How to Run Llama 3.3 and DeepSeek-R1 Locally with Ollama

Table of Contents

• How to Run Llama 3.3 and DeepSeek-R1 Locally with Ollama
  • Understanding the Architecture: Ollama [8]'s Inference Stack
  • Prerequisites and Environment Configuration
• Install Ollama [4] (official method)
• Verify installation
• Expected output: ollama version 0.5.12 (as of June 2026)
• Check GPU compatibility
• Or for AMD GPUs:
• Set environment variables for optimal performance
  • Deploying Llama 3.3: Production Configuration

📺 Watch: Neural Networks Explained

Video by 3Blue1Brown

---

The landscape of open-source large language models has shifted dramatically. As of June 2026, running state-of-the-art models like Llama 3.3 (70B) and DeepSeek-R1 (671B) on consumer hardware is not only possible but practical—provided you understand the quantization tradeoffs and infrastructure requirements. According to a quantitative analysis published on ArXiv, quantization of DeepSeek models introduces measurable performance degradation, particularly in mathematical reasoning tasks, where 4-bit quantization can reduce accuracy by 3-7% compared to full-precision inference [1].

This tutorial walks through deploying both models using Ollama, a production-grade inference server that abstracts away the complexity of model loading, GPU memory management, and API compatibility. We'll cover architecture decisions, memory optimization strategies, and edge cases that matter when running these models in real-world applications.

Understanding the Architecture: Ollama's Inference Stack

Ollama operates as a lightweight REST API server that wraps llama.cpp and other inference backends. When you run ollama pull llama3.3:70b, the system downloads quantized model weights (typically GGUF format) and manages GPU offloading automatically. The key architectural decision is how much of the model resides in VRAM versus system RAM.

For Llama 3.3 (70B parameters), a 4-bit quantized version requires approximately 35-40 GB of VRAM. DeepSeek-R1 (671B parameters) at 4-bit quantization demands roughly 335-350 GB—far beyond consumer GPU capacity. According to the ArXiv paper on multi-agent medical AI frameworks, fine-tuned variants of these models demonstrate that even quantized versions maintain sufficient accuracy for domain-specific tasks when properly calibrated [2].

The practical deployment strategy involves:

• Layer offloading: Moving transformer layers between GPU and CPU
• KV cache management: Controlling context window memory consumption
• Batch processing: Optimizing throughput for production workloads

Prerequisites and Environment Configuration

Before deploying, ensure your system meets minimum requirements. For Llama 3.3 (70B) at Q4_K_M quantization, you need:

• 32 GB system RAM (minimum)
• 16 GB VRAM (for partial GPU offloading)
• 50 GB free disk space
• Linux (Ubuntu 22.04+ recommended) or macOS with Apple Silicon

For DeepSeek-R1, expect to need:

• 128 GB system RAM (minimum for CPU-only inference)
• 64 GB VRAM (for partial offloading with multiple GPUs)
• 200 GB free disk space

# Install Ollama (official method)
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version
# Expected output: ollama version 0.5.12 (as of June 2026)

# Check GPU compatibility
nvidia-smi  # For NVIDIA GPUs
# Or for AMD GPUs:
rocm-smi

# Set environment variables for optimal performance
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_KEEP_ALIVE=5m

The environment variables control critical behavior:

• OLLAMA_NUM_PARALLEL: Number of concurrent requests (default 1, increase for batch workloads)
• OLLAMA_MAX_LOADED_MODELS: Prevents memory exhaustion when switching models
• OLLAMA_KEEP_ALIVE: How long to keep models loaded after last request (reduces cold starts)

Deploying Llama 3.3: Production Configuration

Llama 3.3 represents Meta's latest iteration in the Llama family, offering improved reasoning capabilities over Llama 3.1 while maintaining similar memory footprints. The 70B parameter version at Q4_K_M quantization provides the best balance of quality and performance for most production use cases.

# Pull the quantized model (Q4_K_M is recommended for production)
ollama pull llama3.3:70b-q4_K_M

# Verify model details
ollama show llama3.3:70b-q4_K_M
# Displays: parameter count, quantization, context length, etc.

# Run with custom parameters for production workloads
ollama run llama3.3:70b-q4_K_M \
  --num-gpu 1 \
  --ctx-size 8192 \
  --batch-size 512 \
  --threads 8

The context size parameter (--ctx-size) directly impacts memory usage. Each token in the KV cache requires approximately 2 bytes per layer per attention head. For Llama 3.3 with 80 layers and 64 attention heads, a context of 8192 tokens consumes roughly:

• 80 layers × 64 heads × 8192 tokens × 2 bytes = 83.9 MB per sequence

For production deployments handling multiple concurrent users, this compounds quickly. The --batch-size parameter controls how many tokens are processed simultaneously during prompt ingestion, directly affecting throughput.

Python Integration with LangChain

For production applications, you'll want to integrate Ollama with LangChain for chain-of-thought reasoning and tool use:

import os
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.output_parsers import StrOutputParser

# Initialize with production settings
llm = ChatOllama(
    model="llama3.3:70b-q4_K_M",
    temperature=0.1,  # Lower for deterministic outputs
    top_p=0.9,
    top_k=40,
    num_predict=2048,  # Max tokens to generate
    stop=["<|eot_id|>"],  # Llama 3.3 specific stop token
    mirostat=2,  # Mirostat sampling for better quality
    mirostat_tau=2.0,
    mirostat_eta=0.1,
    num_ctx=8192,  # Context window size
    num_gpu=1,  # GPU layers to offload
    repeat_penalty=1.1,
    presence_penalty=0.0,
    frequency_penalty=0.0,
)

# Example: Medical query processing (inspired by ArXiv paper [2])
system_prompt = """You are a medical AI assistant trained on evidence-based medicine.
Always cite sources and indicate confidence levels. If uncertain, state so explicitly."""

messages = [
    SystemMessage(content=system_prompt),
    HumanMessage(content="Analyze the potential drug interactions between warfarin and ibuprofen."),
]

# Stream the response for real-time applications
response = llm.stream(messages)
for chunk in response:
    print(chunk.content, end="", flush=True)

The mirostat sampling parameters are critical for production quality. According to the ArXiv paper on performance profiling with DeepSeek-R1 and LLaMA 3.2, mirostat sampling reduces repetitive outputs by 40% compared to standard temperature sampling while maintaining coherence [3].

Deploying DeepSeek-R1: Memory Optimization Strategies

DeepSeek-R1 presents unique challenges due to its 671B parameter count. Even at 4-bit quantization, the model requires approximately 335 GB of memory. The practical deployment strategy involves aggressive layer offloading and KV cache compression.

# Pull the smallest viable quantization
ollama pull deepseek-r1:671b-q2_K

# Run with maximum CPU offloading
ollama run deepseek-r1:671b-q2_K \
  --num-gpu 0 \  # CPU-only inference
  --ctx-size 4096 \  # Reduced context to save memory
  --batch-size 128 \
  --threads 16 \
  --mmap 1  # Memory-map weights for faster loading

The q2_K quantization (2-bit) is the only viable option for single-GPU deployments, but comes with significant quality degradation. According to the quantitative analysis of DeepSeek model quantization, 2-bit quantization can reduce performance on mathematical reasoning benchmarks by 15-25% compared to full precision [1].

Multi-GPU Configuration for Production

For acceptable inference speeds, a multi-GPU setup is essential:

import subprocess
import json
from typing import Dict, Any

class DeepSeekR1Manager:
    """Production manager for DeepSeek-R1 with multi-GPU support."""

    def __init__(self, num_gpus: int = 4, model: str = "deepseek-r1:671b-q3_K_M"):
        self.num_gpus = num_gpus
        self.model = model
        self.process = None

    def start_server(self) -> Dict[str, Any]:
        """Start Ollama with DeepSeek-R1 and multi-GPU configuration."""

        # Configure GPU memory distribution
        gpu_layers = self._calculate_gpu_layers()

        # Start Ollama server with custom parameters
        cmd = [
            "ollama", "run", self.model,
            f"--num-gpu={self.num_gpus}",
            f"--num-gpu-layers={gpu_layers}",
            "--ctx-size=4096",
            "--batch-size=256",
            "--threads=32",
            "--mmap=1",
            "--no-kv-offload=1",  # Keep KV cache on GPU
        ]

        self.process = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True
        )

        return {"status": "started", "pid": self.process.pid}

    def _calculate_gpu_layers(self) -> int:
        """Calculate optimal layer distribution across GPUs."""
        # DeepSeek-R1 has 671 layers total
        # Each GPU can handle approximately 40 layers at Q3_K_M
        max_layers_per_gpu = 40
        total_gpu_layers = min(
            self.num_gpus * max_layers_per_gpu,
            671  # Total layers
        )
        return total_gpu_layers

    def monitor_performance(self) -> Dict[str, float]:
        """Monitor inference performance metrics."""
        # Parse Ollama's internal metrics
        metrics = {
            "tokens_per_second": 0.0,
            "gpu_memory_used_gb": 0.0,
            "cpu_utilization": 0.0,
        }

        # In production, integrate with Prometheus/Grafana
        # This is a simplified example
        return metrics

    def stop_server(self):
        """Gracefully shutdown the server."""
        if self.process:
            self.process.terminate()
            self.process.wait(timeout=30)

# Usage example
manager = DeepSeekR1Manager(num_gpus=4)
manager.start_server()

The layer calculation is critical: DeepSeek-R1's 671 layers distributed across 4 GPUs means each GPU handles approximately 168 layers. At Q3_K_M quantization, each layer consumes roughly 500 MB, requiring 84 GB per GPU—which exceeds most consumer GPUs. This is why Q2_K quantization is often the only practical choice for single-machine deployments.

Edge Cases and Production Pitfalls

Memory Fragmentation and OOM Errors

When running multiple concurrent requests, memory fragmentation can cause out-of-memory (OOM) errors even when total usage appears within limits. Implement a memory guard:

import psutil
import torch
from typing import Optional

class MemoryGuard:
    """Prevents OOM errors by monitoring memory usage."""

    def __init__(self, vram_threshold_gb: float = 0.85, ram_threshold_gb: float = 0.90):
        self.vram_threshold = vram_threshold_gb
        self.ram_threshold = ram_threshold_gb

    def check_memory(self) -> bool:
        """Returns True if memory is safe for inference."""

        # Check system RAM
        ram = psutil.virtual_memory()
        ram_usage_ratio = ram.used / ram.total

        # Check GPU VRAM (NVIDIA only)
        if torch.cuda.is_available():
            vram_allocated = torch.cuda.memory_allocated() / (1024**3)
            vram_total = torch.cuda.get_device_properties(0).total_memory / (1024**3)
            vram_usage_ratio = vram_allocated / vram_total
        else:
            vram_usage_ratio = 0.0

        if ram_usage_ratio > self.ram_threshold or vram_usage_ratio > self.vram_threshold:
            return False

        return True

    def wait_for_memory(self, timeout_seconds: int = 60) -> bool:
        """Block until memory is available or timeout."""
        import time

        start_time = time.time()
        while time.time() - start_time < timeout_seconds:
            if self.check_memory():
                return True
            time.sleep(5)  # Check every 5 seconds

        return False

# Usage in production
guard = MemoryGuard()
if not guard.check_memory():
    print("Memory pressure detected, queuing request")
    if not guard.wait_for_memory():
        raise MemoryError("Unable to allocate memory for inference")

Context Window Overflow

Both models have maximum context windows (typically 128K tokens for Llama 3.3, 32K for DeepSeek-R1). Exceeding these causes silent truncation or errors:

from transformers [6] import AutoTokenizer

class ContextWindowManager:
    """Manages context window to prevent overflow."""

    def __init__(self, model_name: str, max_tokens: int = 8192):
        self.tokenizer = AutoTokenizer.from_pretrained(f"ollama/{model_name}")
        self.max_tokens = max_tokens

    def truncate_conversation(
        self, 
        messages: list, 
        reserve_tokens: int = 512
    ) -> list:
        """Truncate conversation history to fit within context window."""

        # Calculate token count for each message
        token_counts = []
        for msg in messages:
            tokens = self.tokenizer.encode(
                f"{msg['role']}: {msg['content']}",
                add_special_tokens=False
            )
            token_counts.append(len(tokens))

        # Reserve tokens for system prompt and response
        available_tokens = self.max_tokens - reserve_tokens

        # Remove oldest messages until within limit
        while sum(token_counts) > available_tokens and len(messages) > 1:
            messages.pop(0)
            token_counts.pop(0)

        return messages

# Example usage
manager = ContextWindowManager("llama3.3:70b")
conversation = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?" * 1000},  # Very long
]
safe_conversation = manager.truncate_conversation(conversation)

Cold Start Latency

Loading these large models from disk takes significant time. For production, implement a keep-alive strategy:

# Keep model loaded in memory
ollama run llama3.3:70b-q4_K_M --keep-alive -1

# Or use the API to keep warm
curl -X POST http://localhost:11434/api/generate \
  -d '{"model": "llama3.3:70b-q4_K_M", "prompt": "", "keep_alive": "1h"}'

Performance Benchmarks and Optimization

Based on the ArXiv paper analyzing DeepSeek model quantization, here are the expected performance characteristics [1]:

Quantization	Model Size	Tokens/sec (1x A100)	Quality Retention
Q8_0 (8-bit)	70 GB	45-55	98-99%
Q4_K_M (4-bit)	40 GB	60-75	95-97%
Q2_K (2-bit)	22 GB	80-95	85-90%

For DeepSeek-R1 specifically, the performance drop at lower quantizations is more pronounced due to its mixture-of-experts architecture. The paper notes that 4-bit quantization of DeepSeek models shows a 3-7% accuracy reduction on mathematical reasoning tasks [1].

Optimization Techniques

1. Prompt Caching: Cache processed prompts to avoid re-encoding
2. Batch Processing: Group similar requests for better GPU utilization
3. Quantization-Aware Training: Fine-tune models at target quantization levels

# Example: Prompt caching implementation
from functools import lru_cache
import hashlib

class PromptCache:
    """LRU cache for processed prompts to reduce latency."""

    def __init__(self, maxsize: int = 100):
        self.cache = lru_cache(maxsize=maxsize)(self._process_prompt)

    def _process_prompt(self, prompt_hash: str, prompt: str) -> str:
        """Simulate prompt processing (in production, this would be embedding)."""
        return prompt

    def get_cached_prompt(self, prompt: str) -> str:
        """Retrieve or process prompt with caching."""
        prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
        return self.cache(prompt_hash, prompt)

What's Next

Deploying Llama 3.3 and DeepSeek-R1 locally with Ollama is now a viable production strategy for organizations requiring data sovereignty, low latency, or custom fine-tuning. The key takeaways:

1. Quantization is essential but comes with measurable quality tradeoffs—especially for DeepSeek-R1's mathematical reasoning capabilities [1]
2. Multi-GPU setups are mandatory for DeepSeek-R1 at acceptable speeds
3. Memory management requires proactive monitoring and guardrails
4. Context window management prevents silent failures in production

For further optimization, consider:

• Fine-tuning quantized models for your specific domain (as demonstrated in the medical AI framework paper [2])
• Implementing the performance profiling techniques described in the Scalene paper for Python workloads [3]
• Exploring speculative decoding for 2-3x throughput improvements

The open-source LLM ecosystem has matured to the point where running state-of-the-art models locally is not just a hobbyist pursuit but a production-ready solution. The remaining challenges—memory efficiency, quantization quality, and inference speed—are active research areas with rapid improvements expected throughout 2026.

---