Ollama from zero: running your first local model

Ollama is a tool that simplifies the process of running locally hosted language models (LLMs). Once installed, you can quickly download and interact with various models without any data leaving your machine. This guide will walk you through setting up Ollama and running your first local model.

What is Ollama?

Ollama is a Docker-like environment for running locally hosted LLMs. It handles the complexities of downloading and managing models, as well as providing an API server and GPU detection. With just one command, you can start chatting with a pre-packaged model that runs entirely on your machine.

Installation

Linux/Mac

For Linux and Mac users, installation is straightforward:

curl -s https://ollama.com/install.sh | sh

Always read the script before running it to understand what it does. You can also use the official Docker image or a Windows installer if preferred.

Windows

Windows users have the option of using an installer or the official Docker image with GPU passthrough enabled by installing the NVIDIA Container Toolkit.

Ollama runs as a service listening on localhost:11434.

First Commands

Here are the essential commands to get you started:

• Download and run a model:

  ollama run llama3.2
  

  This command downloads the llama3.2 model and starts an interactive chat session.

• Download a model without running it:

  ollama pull qwen2.5:7b
  

  Use this to download models for later use or reference.

• List downloaded models:

  ollama list
  

• List models in memory (running):

  ollama ps
  

• Remove a model from disk:

  ollama rm name
  

Choosing the Right Model

The choice of model depends on your hardware capabilities:

• 8 GB RAM/VRAM: Suitable for models in the 3B class or smaller.
• 16 GB RAM/VRAM: Ideal for models around 7-8B.
• 24 GB+ RAM/VRAM: Best for larger models like those above 14B.

By default, models are Q4 quantized to optimize performance and memory usage. For more details on this and other options, refer to the quantization documentation.

API Usage

For advanced integration, Ollama provides an API that allows you to interact with models programmatically:

• Interactive chat:

  curl -X POST http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{"prompt": "What is the weather like today?"}'
  

• Text generation:

  curl -X POST http://localhost:11434/api/generate -H "Content-Type: application/json" -d '{"prompt": "Write a short story about a cat."}'
  

These APIs are compatible with various programming languages and tools, making it easy to integrate Ollama into your projects.

Tips for Beginners

• First Response Delay: The first response after an idle period may take longer as the model loads into VRAM. Use the keep_alive option to keep the model active.

• Context Length: Default context length is small, which can be increased using the num_ctx option but will consume more memory.

• Model Storage: Models are stored in ~/.ollama, so monitor disk usage closely as they can take up significant space (around 30 GB).

• Model Size Mismatch: If you download a model that doesn't match your hardware, it may run slowly or not at all. Use the Ollama "can I run it" calculator to check compatibility.

• Security: Exposing localhost:11434 outside of your local network can be risky. Ensure proper binding and firewalling to prevent unauthorized access.

By following these steps, you can set up and use Ollama effectively for running and managing locally hosted AI models.