How to Build a Production AI Glossary with Vector Search

Table of Contents

• How to Build a Production AI Glossary with Vector Search
  • Real-World Use Case and Architecture
  • Prerequisites and Environment Setup
  • Core Implementation: Building the Semantic Glossary
    • Step 1: Define Your Glossary Schema
• config.py
  • Step 2: Build the Embedding [1] Pipeline
• src/embedder.py
  • Step 3: Implement the Vector Store
• src/vector_store.py

📺 Watch: Neural Networks Explained

Video by 3Blue1Brown

---

If you're building an AI glossary for your team or product, you've probably noticed that most implementations are just static markdown files or JSON dictionaries. They work for beginners but fail in production when you need semantic understanding, versioning, and real-time updates. This tutorial walks through building a production-grade AI glossary using vector embeddings and semantic search, so your glossary actually helps engineers make decisions rather than just collecting dust.

The core problem: a static glossary doesn't adapt to how people actually ask questions. When an engineer searches "how do I handle missing data in my training pipeline", they don't want to see the definition of "imputation" - they want the specific technique, trade-offs, and code example. We'll build a system that understands intent, not just keywords.

Real-World Use Case and Architecture

Your AI glossary needs to serve multiple roles in production:

• Onboarding: New team members searching for terminology
• Decision support: Engineers comparing techniques (e.g., "batch normalization vs layer normalization")
• Documentation: Linking glossary terms to actual code and experiments

The architecture uses three components:

1. Embedding pipeline: Converts glossary entries into vector representations using sentence transformers [4]
2. Vector store: LanceDB for efficient similarity search with metadata filtering
3. Query service: FastAPI endpoint with semantic search and hybrid retrieval

Why LanceDB? It's embedded (no separate server), supports multi-modal data, and handles versioning natively. For a glossary that changes frequently, this matters more than raw query speed.

Prerequisites and Environment Setup

You'll need Python 3.10+ and these packages:

pip install lancedb sentence-transformers fastapi uvicorn pydantic

Create your project structure:

ai-glossary/
├── data/
│   └── glossary_entries.json
├── src/
│   ├── embedder.py
│   ├── vector_store.py
│   └── api.py
└── config.py

Core Implementation: Building the Semantic Glossary

Step 1: Define Your Glossary Schema

First, define what a glossary entry looks like. This matters because production glossaries need structured metadata for filtering and versioning.

# config.py
from pydantic import BaseModel, Field
from typing import Optional, List
from datetime import datetime

class GlossaryEntry(BaseModel):
    term: str
    definition: str
    category: str  # "technique", "architecture", "metric", "concept"
    code_example: Optional[str] = None
    related_terms: List[str] = []
    version: str = "1.0.0"
    created_at: datetime = Field(default_factory=datetime.utcnow)
    updated_at: datetime = Field(default_factory=datetime.utcnow)
    author: str = "system"
    tags: List[str] = []

The category field is critical for filtering. In production, you'll often want to search only within "technique" entries when someone asks about implementation details.

Step 2: Build the Embedding Pipeline

The embedding pipeline converts text into vectors. We use sentence-transformers/all-MiniLM-L6-v2 because it's fast (384-dimensional vectors) and handles technical language reasonably well.

# src/embedder.py
from sentence_transformers import SentenceTransformer
import numpy as np
from typing import List, Dict
import logging

logger = logging.getLogger(__name__)

class GlossaryEmbedder:
    def __init__(self, model_name: str = "all-MiniLM-L6-v2"):
        self.model = SentenceTransformer(model_name)
        self.dimension = self.model.get_sentence_embedding_dimension()
        logger.info(f"Loaded model {model_name} with dimension {self.dimension}")

    def embed_entry(self, entry: Dict) -> np.ndarray:
        """
        Create a composite embedding from term + definition + code example.
        This gives better semantic matching than embedding just the term.
        """
        text_parts = [
            entry.get("term", ""),
            entry.get("definition", ""),
            entry.get("code_example", "")
        ]
        combined = " ".join([p for p in text_parts if p])

        # Handle edge case: empty entry
        if not combined.strip():
            logger.warning(f"Empty entry for term: {entry.get('term', 'unknown')}")
            return np.zeros(self.dimension)

        embedding = self.model.encode(combined, normalize_embeddings=True)
        return embedding

    def embed_query(self, query: str) -> np.ndarray:
        """Embed a search query with the same normalization."""
        if not query.strip():
            raise ValueError("Query cannot be empty")
        return self.model.encode(query, normalize_embeddings=True)

The composite embedding approach is a production trick. If you only embed the term, "batch normalization" and "layer normalization" will have similar vectors because they share "normalization". By including definitions and code, the model learns the actual differences.

Step 3: Implement the Vector Store

LanceDB stores vectors locally and supports metadata filtering. This is perfect for a glossary that needs to be portable and versioned.

# src/vector_store.py
import lancedb
import pyarrow as pa
from typing import List, Dict, Optional
import numpy as np
from datetime import datetime
import json
import logging

logger = logging.getLogger(__name__)

class GlossaryVectorStore:
    def __init__(self, uri: str = "./data/lancedb"):
        self.db = lancedb.connect(uri)
        self.table_name = "glossary"
        self._ensure_table()

    def _ensure_table(self):
        """Create table if it doesn't exist with proper schema."""
        schema = pa.schema([
            pa.field("term", pa.string()),
            pa.field("definition", pa.string()),
            pa.field("category", pa.string()),
            pa.field("code_example", pa.string()),
            pa.field("related_terms", pa.list_(pa.string())),
            pa.field("version", pa.string()),
            pa.field("created_at", pa.timestamp("us")),
            pa.field("updated_at", pa.timestamp("us")),
            pa.field("author", pa.string()),
            pa.field("tags", pa.list_(pa.string())),
            pa.field("vector", pa.list_(pa.float32(), 384)),
        ])

        try:
            self.table = self.db.create_table(self.table_name, schema=schema, mode="overwrite")
            logger.info(f"Created table {self.table_name}")
        except Exception as e:
            self.table = self.db.open_table(self.table_name)
            logger.info(f"Opened existing table {self.table_name}")

    def add_entries(self, entries: List[Dict], embedder) -> int:
        """
        Add multiple glossary entries with their embeddings.
        Returns count of successfully added entries.
        """
        records = []
        for entry in entries:
            try:
                vector = embedder.embed_entry(entry)
                record = {
                    **entry,
                    "vector": vector.tolist(),
                    "created_at": datetime.utcnow(),
                    "updated_at": datetime.utcnow()
                }
                records.append(record)
            except Exception as e:
                logger.error(f"Failed to embed entry {entry.get('term')}: {e}")
                continue

        if records:
            self.table.add(records)
            logger.info(f"Added {len(records)} entries to glossary")
        return len(records)

    def search(self, query: str, embedder, k: int = 5, 
               category: Optional[str] = None) -> List[Dict]:
        """
        Semantic search with optional category filter.
        Returns top-k results with similarity scores.
        """
        query_vector = embedder.embed_query(query)

        # Build search with optional filter
        search_query = self.table.search(query_vector.tolist()).limit(k)
        if category:
            search_query = search_query.where(f"category = '{category}'")

        results = search_query.to_list()

        # Format results with distance scores (lower = more similar)
        formatted = []
        for r in results:
            formatted.append({
                "term": r["term"],
                "definition": r["definition"],
                "category": r["category"],
                "similarity_score": 1 - r["_distance"],  # Convert to similarity
                "version": r["version"],
                "tags": r["tags"]
            })

        return formatted

The _distance field from LanceDB gives cosine distance (0 = identical, 2 = opposite). Converting to similarity score makes it more intuitive for API consumers.

Step 4: Build the FastAPI Service

The API layer handles query routing, error cases, and provides a clean interface for frontends or other services.

# src/api.py
from fastapi import FastAPI, HTTPException, Query
from pydantic import BaseModel
from typing import Optional, List
import uvicorn
from embedder import GlossaryEmbedder
from vector_store import GlossaryVectorStore
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = FastAPI(title="AI Glossary API", version="1.0.0")

# Initialize components
embedder = GlossaryEmbedder()
store = GlossaryVectorStore()

class SearchRequest(BaseModel):
    query: str
    k: int = 5
    category: Optional[str] = None

class SearchResult(BaseModel):
    term: str
    definition: str
    category: str
    similarity_score: float
    version: str
    tags: List[str]

@app.get("/health")
async def health_check():
    """Simple health check endpoint."""
    return {"status": "healthy", "version": "1.0.0"}

@app.post("/search", response_model=List[SearchResult])
async def search_glossary(request: SearchRequest):
    """
    Search glossary entries semantically.

    Edge cases handled:
    - Empty query: returns 400
    - No results: returns empty list (not 404)
    - Invalid category: returns results without filter
    """
    if not request.query.strip():
        raise HTTPException(status_code=400, detail="Query cannot be empty")

    try:
        results = store.search(
            query=request.query,
            embedder=embedder,
            k=min(request.k, 50),  # Cap at 50 results
            category=request.category
        )
        return results
    except Exception as e:
        logger.error(f"Search failed: {e}")
        raise HTTPException(status_code=500, detail="Search service unavailable")

@app.post("/entries/bulk")
async def add_entries(entries: List[dict]):
    """
    Bulk add glossary entries.
    Accepts a list of entry dicts matching the GlossaryEntry schema.
    """
    count = store.add_entries(entries, embedder)
    return {"added": count, "total": len(entries)}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Step 5: Seed Your Glossary with Real Data

Create a sample dataset to test the system:

# seed_data.py
import json
from src.embedder import GlossaryEmbedder
from src.vector_store import GlossaryVectorStore

sample_entries = [
    {
        "term": "Batch Normalization",
        "definition": "A technique to normalize the inputs of each layer to improve training stability and speed. Normalizes across the batch dimension using mean and variance.",
        "category": "technique",
        "code_example": "import torch.nn as nn\nnn.BatchNorm1d(128)",
        "related_terms": ["Layer Normalization", "Instance Normalization"],
        "version": "1.0.0",
        "author": "system",
        "tags": ["normalization", "training", "deep learning"]
    },
    {
        "term": "Layer Normalization",
        "definition": "Normalizes inputs across the feature dimension rather than the batch dimension. Commonly used in transformers and RNNs.",
        "category": "technique",
        "code_example": "import torch.nn as nn\nnn.LayerNorm(768)",
        "related_terms": ["Batch Normalization", "Transformer"],
        "version": "1.0.0",
        "author": "system",
        "tags": ["normalization", "transformer", "deep learning"]
    },
    {
        "term": "Gradient Descent",
        "definition": "An optimization algorithm that iteratively adjusts model parameters to minimize the loss function by moving in the direction of steepest descent.",
        "category": "technique",
        "code_example": "# Pseudocode\nfor epoch in range(num_epochs):\n    gradients = compute_gradients(loss, params)\n    params -= learning_rate * gradients",
        "related_terms": ["Stochastic Gradient Descent", "Adam", "Learning Rate"],
        "version": "1.0.0",
        "author": "system",
        "tags": ["optimization", "training", "fundamental"]
    }
]

if __name__ == "__main__":
    embedder = GlossaryEmbedder()
    store = GlossaryVectorStore()
    count = store.add_entries(sample_entries, embedder)
    print(f"Seeded {count} entries")

Pitfalls and Production Tips

1. Embedding Quality Degradation

The biggest gotcha: your embeddings will drift over time as you add entries. If you add 1000 entries about "transformer architectures", the vector space shifts. Solution: periodically re-embed all entries with the same model version and track embedding model version in metadata.

2. Cold Start Problem

When you first deploy, you have no user queries to tune results. Start with a curated set of 50-100 high-quality entries covering your domain's core concepts. The none concept from set theory (the empty set) is actually relevant here - an empty glossary returns no results, which is useless.

3. Query Ambiguity

Users will search "normalization" and expect both batch and layer normalization. Our composite embedding handles this, but you should also implement query expansion: when someone searches "normalization", automatically boost entries tagged with "normalization".

4. Versioning Nightmares

When you update a glossary entry, the old vector is still in the store. LanceDB supports versioning, but you need to explicitly handle it. Implement a deprecated flag and filter it out in searches:

def search(self, query, embedder, k=5, include_deprecated=False):
    query_vector = embedder.embed_query(query)
    search_query = self.table.search(query_vector.tolist()).limit(k)
    if not include_deprecated:
        search_query = search_query.where("deprecated = false")
    return search_query.to_list()

5. Memory and Performance

Each 384-dimensional vector is ~1.5KB. For 10,000 entries, that's 15MB - fine for memory. But if you're embedding code examples that are 1000+ tokens, your embedding model will truncate. Always check token limits:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("all-MiniLM-L6-v2")
tokens = tokenizer(combined_text, truncation=True, max_length=256)

Testing Your Implementation

Run the API and test with curl:

# Start the server
python src/api.py

# In another terminal, test search
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "how to stabilize training", "k": 3}'

# Expected output includes Batch Normalization and Layer Normalization
# because they're semantically related to training stability

What's Next

This glossary system is production-ready for teams of 10-50 engineers. For larger deployments, consider:

1. Multi-modal embeddings: Add image embeddings for architecture diagrams
2. Feedback loop: Track which results users click and fine-tune embeddings
3. Cross-encoder reranking: Use a cross-encoder model to rerank top-20 results for better precision

The key insight: a glossary isn't just a dictionary - it's a knowledge base that should understand intent. By building it with vector search, you're creating a system that grows with your team's understanding, not one that becomes obsolete the moment someone asks a question you didn't anticipate.

---