# Running Dolphin Locally with Ollama

Wanna chat with Dolphin locally? (no internet connection needed)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701643254622/de524c2f-ee73-4151-ad25-018c55ff9dad.png align="center")

Here is the easy way - Ollama.

1. [install ollama.](https://ollama.ai/download) after you finsh you should be able to run ollama from the command line. Also you will see the ollama icon up top like this:
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701644497179/52991096-2393-408a-a093-ca6ea3321d32.png align="center")

Iff you are curious - anytime you see that icon, that means ollama is running in the background and it also has a port open (11434) that can accept api calls.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701644732571/e75f905b-569f-4d11-ab79-cf80bd16fd8c.png align="center")

Sadly, as of this writing there is no swagger.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701644701436/c18c5573-9223-42f6-b778-160e36b5fcd1.png align="center")

Back to business.

1. download your dolphin. You can choose which size you want from [here](https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF). I am choosing size Q5\_K\_M which is just over 5gb.
    

```bash
mkdir ~/models
cd ~/models
wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf
```

Now you need a Modelfile.

```bash
vim Modelfile.dolphin.mistral
```

Add the following text to Modelfile.dolphin.mistral

```plaintext
FROM ./dolphin-2.2.1-mistral-7b.Q5_K_M.gguf
TEMPLATE """{{ if and .First .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """You are Dolphin, an uncensored, unbiased AI assistant"""
PARAMETER stop <|im_end|>
```

Now you can use Ollama to install this model.

```bash
ollama create dolphin.mistral -f Modelfile.dolphin.mistral
```

Now look, you can run it from the command line.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701645161483/f965260b-15c0-4207-a6d5-b4e03ed29f85.png align="center")

Which is cool enough. But we are just getting started.

If you want, you can install [samantha](https://huggingface.co/TheBloke/samantha-1.2-mistral-7B-GGUF) too so you have two models to play with.

```bash
wget https://huggingface.co/TheBloke/samantha-1.2-mistral-7B-GGUF/resolve/main/sama
ntha-1.2-mistral-7b.Q5_K_M.gguf
vim Modelfile.samantha.mistral
```

And enter the following into Modelfile.samantha.mistral

```plaintext
FROM ./samantha-1.2-mistral-7b.Q5_K_M.gguf
TEMPLATE """{{ if and .First .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """You are Samantha, an AI companion"""
PARAMETER stop <|im_end|>
```

Then install the model

```bash
ollama create samantha -f Modelfile.samantha.mistral
```

And now you can also chat with Samantha from the command line.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701645770021/323aa1bc-74f0-40c0-aed7-78e6d4fd4f99.png align="center")

Cool yeah? We are just getting started.

Let's get Ollama Web UI installed.

```bash
cd ~
git clone https://github.com/ollama-webui/ollama-webui.git
cd ollama-webui
npm i
npm run dev
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701646311466/aa9a944b-49be-4845-b763-74c9d3bdab2e.png align="center")

Now you can open that link http://localhost:5173 in your web browser.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701646385344/f1c808f9-8cac-4ee2-979a-041f08663120.png align="center")

now you can choose dolphin or samantha from the dropdown

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701646459858/985212d3-1a15-4d8d-ae71-11e510b4306e.png align="center")

(I have installed a few others too)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1701646519748/81e4a8dd-753e-48ef-992a-9b328d190b98.png align="center")

Well talking to these models from the command line and the web ui is just the beginning.

Also, frameworks such as [langchain](https://github.com/langchain-ai/langchain), [llamaindex](https://github.com/run-llama/llama_index), [litellm](https://github.com/BerriAI/litellm), [autogen](https://github.com/microsoft/autogen), [memgpt](https://github.com/cpacker/MemGPT) all can integrate with ollama. Now you can really play with these models.

Here is a fun idea that I will leave as an exercise - given some query, ask dolphin to decide whether a question about coding, a request for companionship, or something else. If it is a request for companionship then send it to Samantha. If it is a coding question, send it to [deepseek-coder](https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF). Otherwise, send it to Dolphin.

And just like that, you have your own MoE.