How to Run Large Language Models Locally with Ollama

Table of Contents

• How to Run Large Language Models Locally with Ollama
  • Introduction & Architecture
  • Prerequisites & Setup
    • System Requirements
    • Dependencies
    • Installation Commands
• Install Go
• Clone the Ollama [8] repository
• Navigate to the project directory
• Build and install Ollama [4]

📺 Watch: Neural Networks Explained

Video by 3Blue1Brown

---

Introduction & Architecture

Running large language models (LLMs) locally can be a significant development for developers and researchers, offering benefits such as reduced latency, enhanced privacy, and better control over computational resources. This tutorial introduces Ollama, an open-source tool that simplifies the process of downloading and running LLMs on your local machine.

As of April 18, 2026, Ollama has garnered significant popularity with 169,300 stars on GitHub (Source: GitHub), reflecting its widespread adoption in the developer community. The latest version, v0.6.1, was released on the same day and includes several improvements over previous iterations.

Ollama's architecture is designed to be lightweight yet powerful, allowing users to run a variety of LLMs such as Kimi-K2.5, GLM-5, MiniMax, DeepSeek, gpt [5]-oss, Qwen, and Gemma (Source: DND:Github Trending). The tool leverages the simplicity of a command-line interface (CLI) to manage these models efficiently.

The underlying technology behind Ollama involves downloading pre-trained model weights from remote servers and serving them locally. This approach ensures that users can leverage the full power of LLMs without relying on cloud-based services, which might be costly or restrictive in terms of data privacy.

Prerequisites & Setup

To set up your environment for running large language models with Ollama, you need to ensure that your system meets certain requirements and has the necessary dependencies installed. Here’s a detailed guide:

System Requirements

• Operating System: Linux (Ubuntu 20.04 or later), macOS (10.15 Catalina or later)
• Hardware:
  • CPU: Multi-core processor with at least 8GB of RAM
  • GPU: Optional but recommended for faster inference times; NVIDIA GPUs are supported

Dependencies

• Go: Ollama is written in Go, so you need to have the Go programming language installed. You can download it from the official website (https://golang.org/dl/).
• Docker or Podman: These tools are used for containerization and will help manage dependencies more efficiently.
• Git: For cloning the Ollama repository.

Installation Commands

# Install Go
sudo apt-get update && sudo apt-get install golang

# Clone the Ollama repository
git clone https://github.com/ollama/ollama.git

# Navigate to the project directory
cd ollama

# Build and install Ollama
make build

Why These Dependencies?

• Go: The choice of Go as the primary language for Ollama is due to its performance, simplicity, and ease of deployment.
• Docker/Podman: Containerization tools are essential for managing dependencies in a consistent manner across different environments.

Core Implementation: Step-by-Step

This section will guide you through the process of running an LLM locally using Ollama. We’ll start by setting up the environment and then proceed to download and run a model.

Step 1: Initialize Ollama

First, initialize Ollama with your preferred configuration settings.

# Initialize Ollama
ollama init

# Start the server
ollama serve

Step 2: Download an LLM Model

Next, download and install a specific model. For this example, we’ll use Qwen.

# List available models
ollama list

# Install Qwen
ollama pull qwen

Step 3: Run the Model

Once the model is installed, you can start interacting with it using the Ollama CLI or API.

# Start a session with Qwen
ollama run qwen --prompt "What is the weather like today?"

Why This Approach?

• Initialization: Initializing Ollama sets up necessary configurations and ensures that all dependencies are correctly installed.
• Model Management: The pull command allows you to easily download and install models from a central repository, making it straightforward to experiment with different LLMs.

Configuration & Production Optimization

To take your local LLM setup into production, several considerations need to be addressed. This includes optimizing configuration options, managing resources efficiently, and ensuring scalability.

Resource Management

Ollama allows you to configure resource limits for each model instance.

# Set memory limit for Qwen
ollama config qwen --memory 8GB

# Adjust CPU settings if necessary
ollama config qwen --cpu 4

Batch Processing & Asynchronous Jobs

For production environments, handling multiple requests efficiently is crucial. Ollama supports batch processing and asynchronous job management.

# Submit a batch of prompts to Qwen
cat prompts.txt | ollama run qwen

# Use async mode for better performance
ollama run qwen --async

Hardware Optimization

Leverage GPU resources if available, as they significantly improve inference speed.

# Run Qwen on a specific GPU device
OLLAMA_DEVICE=0 ollama run qwen

Advanced Tips & Edge Cases (Deep Dive)

Running LLMs locally introduces unique challenges such as prompt injection and memory management issues. Here’s how to handle these scenarios effectively.

Prompt Injection

Prompt injection can be mitigated by carefully sanitizing input data.

# Example of sanitization in Python
def sanitize_input(prompt):
    # Remove potentially harmful characters or patterns
    sanitized = re.sub(r'[^\w\s]', '', prompt)
    return sanitized

sanitized_prompt = sanitize_input("What is the weather like today?")

Memory Management

Monitor and manage memory usage to prevent out-of-memory errors.

# Check current memory usage of Qwen
ollama status qwen --memory

# Adjust model settings dynamically based on resource availability
OLLAMA_MEMORY=4GB ollama run qwen

Results & Next Steps

By following this tutorial, you have successfully set up and configured Ollama to run large language models locally. You can now experiment with different LLMs and tailor the setup according to your specific requirements.

What’s Next?

• Experimentation: Try running other models available through Ollama.
• Customization: Modify configurations to optimize performance for your use case.
• Documentation & Support: Refer to the official documentation (https://ollama.ai/docs) for more advanced features and troubleshooting tips.

---