How to Deploy LLMs with vLLM for Production Inference

Table of Contents

• How to Deploy LLMs with vLLM for Production Inference
  • Why vLLM Matters for Production Inference
  • Prerequisites and Environment Setup
• Create a clean Python environment
• Install vLLM with CUDA support
• For production monitoring
• Verify installation
• Should show CUDA 12.1 or higher
  • Setting Up the vLLM Inference Server
• server.py - Production vLLM server with custom configuration

📺 Watch: Neural Networks Explained

Video by 3Blue1Brown

---

Running large language models in production is harder than training them. The gap between a working notebook and a reliable inference endpoint that handles thousands of requests per second while keeping latency under 200ms is where most teams fail. This tutorial walks through deploying LLMs using vLLM, an open-source inference engine that has become the standard for production serving.

As of June 2026, vLLM has 72,929 stars and 14,263 forks on GitHub, making it the most popular open-source LLM serving framework. Originally developed at UC Berkeley's Sky Computing Lab, vLLM centers on PagedAttention, a memory-management method for transformer key-value caches that dramatically improves throughput compared to naive implementations. The project is written in Python and supports continuous batching, distributed inference, and quantization.

Why vLLM Matters for Production Inference

The core problem vLLM solves is memory inefficiency in transformer inference. When generating tokens, the model must store key-value (KV) cache entries for every token in the sequence. With a 70B parameter model serving 32 concurrent requests at 4096 tokens each, the KV cache alone consumes roughly 32 * 4096 * 2 * 80 * 4 bytes ≈ 80 GB of GPU memory. Traditional implementations allocate fixed contiguous memory blocks for each request, leading to massive frag [5]mentation and wasted capacity.

PagedAttention treats the KV cache like virtual memory in an operating system. It divides the cache into fixed-size blocks (pages) and maps them non-contiguously across GPU memory. This eliminates fragmentation and allows memory sharing across requests during parallel sampling or beam search. The result is 2-4x higher throughput than frameworks like Hugging Face's Text Generation Inference or NVIDIA's Triton Inference Server, according to the vLLM paper's benchmarks.

Prerequisites and Environment Setup

You need a machine with NVIDIA GPUs. For this tutorial, I assume a single A100 80GB or H100, but vLLM supports multi-node deployment. The setup works on Ubuntu 22.04 with CUDA 12.1+.

# Create a clean Python environment
python3 -m venv vllm-env
source vllm-env/bin/activate

# Install vLLM with CUDA support
pip install vllm==0.6.0

# For production monitoring
pip install prometheus-client fastapi uvicorn

# Verify installation
python -c "import vllm; print(vllm.__version__)"

The vLLM package includes all dependencies including PyTorch [6], transformers, and flash-attention. If you hit CUDA compatibility issues, verify your driver version:

nvidia-smi | grep "CUDA Version"
# Should show CUDA 12.1 or higher

Setting Up the vLLM Inference Server

vLLM provides a built-in OpenAI [10]-compatible API server. This is the fastest path to production because you can swap in any OpenAI SDK client without code changes.

# server.py - Production vLLM server with custom configuration
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
from vllm.entrypoints.openai.api_server import run_server
import argparse

def create_engine():
    """Configure the vLLM engine for production workloads."""
    engine_args = AsyncEngineArgs(
        model="meta-llama/Meta-Llama-3-70B-Instruct",
        # Critical: match your GPU memory
        max_model_len=8192,
        # Enable continuous batching
        max_num_seqs=256,
        # Tensor parallelism for multi-GPU
        tensor_parallel_size=2,
        # Quantization to fit larger models
        quantization="fp8",
        # Enable prefix caching for shared system prompts
        enable_prefix_caching=True,
        # GPU memory utilization target
        gpu_memory_utilization=0.90,
        # Swap space for KV cache overflow
        swap_space=4,
        # Trust remote code for custom models
        trust_remote_code=True,
    )
    return AsyncLLMEngine.from_engine_args(engine_args)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", type=str, default="0.0.0.0")
    parser.add_argument("--port", type=int, default=8000)
    args = parser.parse_args()

    run_server(
        engine=create_engine(),
        host=args.host,
        port=args.port,
        # Enable Prometheus metrics
        enable_metrics=True,
    )

Start the server:

python server.py --port 8000

The server downloads the model on first run if not cached. For Llama-3-70B with FP8 quantization, expect 140GB of model weights. Download time varies from 10 minutes (100 Gbps) to 2 hours (1 Gbps).

Building a Production Client with Error Handling

The OpenAI-compatible API means you can use the standard Python client. Here's a production-grade client that handles rate limits, retries, and streaming:

# client.py - Production client with retry logic and streaming
from openai import OpenAI
import time
from typing import Generator, Optional
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class VLLMClient:
    """Production client for vLLM with automatic retry and monitoring."""

    def __init__(
        self,
        base_url: str = "http://localhost:8000/v1",
        api_key: str = "token-abc123",  # vLLM requires a dummy key
        max_retries: int = 3,
        timeout: float = 120.0,
    ):
        self.client = OpenAI(
            base_url=base_url,
            api_key=api_key,
            max_retries=max_retries,
            timeout=timeout,
        )
        self.request_count = 0
        self.total_tokens = 0

    def generate(
        self,
        prompt: str,
        system_prompt: Optional[str] = None,
        max_tokens: int = 1024,
        temperature: float = 0.7,
        stream: bool = False,
    ) -> str:
        """Generate text with automatic retry and token tracking."""
        messages = []
        if system_prompt:
            messages.append({"role": "system", "content": system_prompt})
        messages.append({"role": "user", "content": prompt})

        for attempt in range(self.client.max_retries):
            try:
                response = self.client.chat.completions.create(
                    model="meta-llama/Meta-Llama-3-70B-Instruct",
                    messages=messages,
                    max_tokens=max_tokens,
                    temperature=temperature,
                    stream=stream,
                )

                if stream:
                    return self._handle_stream(response)

                content = response.choices[0].message.content
                self.request_count += 1
                self.total_tokens += response.usage.total_tokens

                logger.info(
                    f"Request {self.request_count}: "
                    f"{response.usage.total_tokens} tokens, "
                    f"{response.usage.completion_tokens} completion tokens"
                )

                return content

            except Exception as e:
                logger.warning(f"Attempt {attempt + 1} failed: {e}")
                if attempt == self.client.max_retries - 1:
                    raise
                time.sleep(2 ** attempt)  # Exponential backoff

    def _handle_stream(self, response) -> Generator[str, None, None]:
        """Handle streaming responses with token counting."""
        for chunk in response:
            if chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content

    def get_metrics(self) -> dict:
        """Return client-side metrics for monitoring."""
        return {
            "total_requests": self.request_count,
            "total_tokens": self.total_tokens,
            "avg_tokens_per_request": (
                self.total_tokens / self.request_count 
                if self.request_count > 0 else 0
            ),
        }

# Usage example
if __name__ == "__main__":
    client = VLLMClient()

    # Non-streaming
    response = client.generate(
        "Explain the difference between PagedAttention and traditional KV cache management.",
        system_prompt="You are an expert in LLM inference optimization.",
        max_tokens=500,
    )
    print(response)

    # Streaming
    for chunk in client.generate(
        "Write a haiku about GPU memory.",
        stream=True,
    ):
        print(chunk, end="", flush=True)

Implementing Continuous Batching and Prefix Caching

The two features that separate vLLM from naive implementations are continuous batching and prefix caching. Here's how to configure them for maximum throughput:

# batching_config.py - Advanced batching and caching configuration
from vllm import SamplingParams
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class BatchConfig:
    """Configuration for optimal batching behavior."""
    max_num_seqs: int = 256
    max_num_batched_tokens: int = 8192
    enable_prefix_caching: bool = True
    use_v2_block_manager: bool = True  # Faster block allocation

class BatchManager:
    """Manages request batching with priority queuing."""

    def __init__(self, config: BatchConfig):
        self.config = config
        self.priority_queue: List[dict] = []
        self.batch_size = 0

    def add_request(
        self,
        prompt: str,
        priority: int = 0,
        max_tokens: int = 1024,
        temperature: float = 0.7,
    ) -> None:
        """Add request to priority queue."""
        self.priority_queue.append({
            "prompt": prompt,
            "priority": priority,
            "sampling_params": SamplingParams(
                max_tokens=max_tokens,
                temperature=temperature,
                # Enable prefix caching for this request
                use_prefix_cache=self.config.enable_prefix_caching,
            ),
        })
        # Sort by priority (lower number = higher priority)
        self.priority_queue.sort(key=lambda x: x["priority"])

    def get_batch(self) -> Optional[List[dict]]:
        """Get next batch of requests respecting max batch size."""
        if not self.priority_queue:
            return None

        batch = []
        current_tokens = 0

        for request in self.priority_queue:
            request_tokens = len(request["prompt"].split()) + request["sampling_params"].max_tokens
            if current_tokens + request_tokens <= self.config.max_num_batched_tokens:
                batch.append(request)
                current_tokens += request_tokens
                if len(batch) >= self.config.max_num_seqs:
                    break

        # Remove batched requests from queue
        self.priority_queue = self.priority_queue[len(batch):]
        return batch

# Usage in server
batch_manager = BatchManager(BatchConfig())

# Add requests with different priorities
batch_manager.add_request(
    "Critical system query..",
    priority=0,  # Highest priority
    max_tokens=100,
)
batch_manager.add_request(
    "Batch analytics task..",
    priority=10,  # Lower priority
    max_tokens=2000,
)

Prefix caching is particularly valuable when serving chatbots or RAG systems where every request starts with the same system prompt. vLLM caches the KV cache for the common prefix and reuses it across requests, reducing time-to-first-token by up to 80% for long system prompts.

Pitfalls and Production Tips

After deploying vLLM in production across multiple clusters, here are the issues that actually cause outages:

1. GPU Memory Fragmentation from Model Loading

When you load a model with tensor_parallel_size=2 on 4 GPUs, vLLM distributes weights across the specified GPUs only. The remaining GPUs sit idle. If you later try to load a second model, the memory is fragmented. Always use CUDA_VISIBLE_DEVICES to isolate model instances:

# Run two model instances on separate GPU pairs
CUDA_VISIBLE_DEVICES=0,1 python server.py --port 8000 --model meta-llama/Llama-3-70B
CUDA_VISIBLE_DEVICES=2,3 python server.py --port 8001 --model mistral [9]ai/Mixtral-8x22B

2. The max_num_seqs Trap

Setting max_num_seqs too high causes OOM errors during peak load. The relationship is non-linear because each sequence's KV cache grows as it generates tokens. A safe formula:

max_num_seqs = (gpu_memory_gb * 0.85 - model_size_gb) / (max_model_len * kv_cache_bytes_per_token)

For an A100 80GB with Llama-3-70B (140GB in FP8 across 2 GPUs = 70GB each):

max_num_seqs = (80 * 0.85 - 70) / (8192 * 2 * 80 * 4 * 1e-9) ≈ 256

3. Prefix Caching Memory Bloat

Prefix caching stores KV cache blocks in a hash table. With 10,000 unique system prompts, the cache can consume 20GB+ of GPU memory. Monitor vllm:prefix_cache_hit_rate metric. If it drops below 0.3, disable prefix caching or implement a cache eviction policy:

# In engine configuration
engine_args = AsyncEngineArgs(
    enable_prefix_caching=True,
    # Limit cache to 50% of available KV cache blocks
    max_num_cached_blocks=0.5,
)

4. Tokenizer Latency at Scale

The tokenizer runs on CPU and becomes a bottleneck at high throughput. vLLM uses Hugging Face tokenizers, which are fast, but at 1000+ requests/second, tokenization adds 5-10ms per request. Pre-tokenize common prompts and cache the token IDs:

from transformers import AutoTokenizer
import functools

class CachedTokenizer:
    def __init__(self, model_name: str):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.cache = {}

    @functools.lru_cache(maxsize=10000)
    def encode(self, text: str) -> list:
        return self.tokenizer.encode(text)

5. The Silent OOM on Long Sequences

vLLM's PagedAttention handles variable-length sequences efficiently, but a single 32K token request can exhaust the KV cache block pool. Set per-request limits:

sampling_params = SamplingParams(
    max_tokens=4096,  # Hard limit per request
    min_tokens=1,
)

And configure the engine to reject requests exceeding the limit:

engine_args = AsyncEngineArgs(
    max_model_len=8192,
    # Reject requests longer than this
    max_num_batched_tokens=8192,
)

Monitoring and Observability

vLLM exposes Prometheus metrics on the /metrics endpoint. Here's a minimal monitoring setup:

# monitoring.py - Prometheus metrics collection
from prometheus_client import start_http_server, Gauge, Counter, Histogram
import time

# Define metrics
requests_total = Counter('vllm_requests_total', 'Total requests processed')
tokens_generated = Counter('vllm_tokens_generated', 'Total tokens generated')
request_latency = Histogram(
    'vllm_request_latency_seconds',
    'Request latency in seconds',
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
)
gpu_memory_usage = Gauge('vllm_gpu_memory_usage_bytes', 'GPU memory usage')

def start_monitoring(port: int = 8001):
    """Start Prometheus metrics server."""
    start_http_server(port)

    while True:
        # Update GPU memory metric
        import subprocess
        result = subprocess.run(
            ['nvidia-smi', '--query-gpu=memory.used', '--format=csv,noheader,nounits'],
            capture_output=True, text=True
        )
        memory_mb = sum(int(x) for x in result.stdout.strip().split('\n'))
        gpu_memory_usage.set(memory_mb * 1024 * 1024)

        time.sleep(15)

What's Next

This setup handles production workloads for most teams. The next step is scaling horizontally with a load balancer in front of multiple vLLM instances. For multi-node inference, vLLM supports tensor parallelism across machines using NCCL, but this requires InfiniBand or 100 Gbps Ethernet.

If you need lower latency than vLLM provides, look into speculative decoding or Medusa heads. For higher throughput, consider model quantization to FP4 or INT4 using the vllm add-quantization plugin system.

The vLLM ecosystem continues to evolve rapidly. Check the GitHub repository for the latest features, and always benchmark with your specific workload before deploying to production.

---