Glossary

An LLM with tools and a loop that lets the LLM decide when and how to use them. Distinguished from a chain by non-determinism in tool sequence.

Microsoft multi-agent framework. Conversational agents, strong for research; less production-hardened than LangGraph.

A predetermined sequence of LLM calls and tools. Deterministic; no choice in tool sequence. LangChain LCEL builds these.

Agent framework with role/task abstractions. Opinionated; simpler than LangGraph for some patterns.

Stanford framework for programmatic LM optimization. Compiles prompts via metric-driven search; orthogonal to LangGraph.

Pausing an agent for human approval, input, or correction. LangGraph supports natively via interrupt nodes.

Stateful agent framework: nodes (functions), edges (transitions), shared state, conditional routing, cycles, human-in-the-loop. Production-grade alternative to ad-hoc ReAct.

LangChain Expression Language — declarative composition of chains via | operator. The right tool for linear pipelines.

Open-source memory framework (formerly MemGPT). Hierarchical memory model emulating OS memory paging.

Open-source memory layer for LLM agents. Manages long-term, episodic, and semantic memory abstractions.

Agent pattern: planner LLM decomposes the task into steps; executor LLM (or chain) runs each step. Better for long-horizon tasks than pure ReAct.

Reason + Act loop (Yao et al. 2022): Thought → Action → Observation, repeated. Modern function-calling APIs subsume the prompt format; the loop structure remains.

Pattern where the model critiques and revises its own output. Useful for correctness-critical generation; costs an extra LLM call.

Evaluating an agent's path (tool calls and order), not just its final answer. Catches wasteful or right-for-wrong-reasons behavior.

Zero-shot retrieval benchmark across 18 datasets. Tests generalization beyond training distribution.

Open-source multilingual embedding (BAAI). Supports dense + sparse + multi-vector outputs in one model. Strong on Arabic; common self-hosted choice.

An encoder model that produces independent embeddings for queries and documents, enabling fast nearest-neighbor search. The default for retrieval.

Classical sparse retrieval: term frequency × inverse document frequency, length-normalized. Still complements dense retrieval for exact-term matches.

Splitting source documents into retrievable units. Strategies: fixed-size, recursive, semantic, sentence-window, parent-document, hierarchical, layout-aware.

Cohere's multilingual embedding model. Strong cross-lingual EN↔AR; vendor-managed alternative to BGE-M3.

Angle-based similarity: a·b / (||a|| ||b||). Standard for normalized embeddings; equivalent to dot product on unit-norm vectors.

A learned function from text (or image, audio) to a fixed-size vector, where geometric distance ≈ semantic similarity.

Non-relevant documents that are nearest neighbors of a query — the training pairs that teach the embedder to discriminate.

Combining sparse (BM25, SPLADE) and dense retrievers, fused with RRF. The production-grade default.

Contrastive loss for embedding training: pull positive pairs together, push hard negatives apart. The default fine-tuning objective.

Document parsing that preserves structure (tables, headings, figures) — using libraries like unstructured, Docling, LlamaParse.

Embedding training technique (Kusupati et al. 2022) where the model learns nested representations: truncating dimensions still preserves quality. Used by OpenAI text-embedding-3.

Multilingual retrieval benchmark across 18 languages including Arabic. The right benchmark for non-English retrieval.

Massive Text Embedding Benchmark. The standard English-heavy embedding leaderboard. Useful first filter, not a final verdict.

Few-shot fine-tuning recipe for sentence transformers. Trains a classifier with few examples via contrastive learning on Sentence-Transformers.

Learned sparse retrieval. Predicts term importance per query/doc; output is a sparse vector with contextual term weights, beating BM25 on most benchmarks.

Access Control List. The mechanism enforcing per-user/group permissions on documents and chunks in enterprise RAG.

EU AI Act. Risk-based regulation; high-risk systems require risk management, data governance, transparency, human oversight, robustness/cybersecurity.

IBM AI Fairness 360. Open-source bias-detection and mitigation toolkit with metrics and pre/in/post-processing methods.

Immutable record of every (user, query, retrieved IDs, model version, output, action) tuple. Required for regulated industries; WORM storage typical.

OpenAI models hosted by Microsoft inside Azure. Trades velocity for residency, governance, private endpoints, compliance certs.

AWS Bedrock — managed multi-vendor LLM service (Claude, Llama, Mistral, Cohere, Amazon Titan). AWS-side governance/residency.

IBM's containerized software bundle on OpenShift. Cloud Pak for Data and Cloud Pak for Integration are the AI-relevant ones.

Microsoft fairness toolkit. Sklearn-friendly; metrics and mitigations for distributional bias.

IBM Message Queue. Mainstay enterprise messaging in finance/insurance; XA transactions, units of work. Common boundary between AI services and legacy.

Vendor commitment to defend customers against IP claims arising from model outputs. IBM, Adobe, Microsoft, Google all offer; OpenAI varies by tier.

Mitchell et al. 2019. Document declaring capabilities, limitations, intended use, evaluation, fairness analysis. Required artifact for regulated deployments.

Architecture serving multiple isolated customer organizations from one system. Patterns: namespace-per-tenant, filter-by-tenant, index-per-tenant.

Vector DB partition scoping data per tenant or use case. Pinecone namespace, Weaviate class, Qdrant collection. Production default for SaaS multi-tenancy.

Red Hat / IBM enterprise Kubernetes distribution. The deployment substrate for IBM AI workloads.

Provisioned Throughput Units — Azure OpenAI's reserved capacity pricing. Predictable but expensive at low utilization.

IBM's AI platform: watsonx.ai (model serving + tuning), watsonx.data (lakehouse), watsonx.governance (audit/lineage).

Scaled dot-product attention: `softmax(QK^T / √d_head) V`. The only step where tokens exchange information. Multi-head runs h of these in parallel with smaller per-head dimension.

Activation-aware Weight Quantization. Identifies salient channels (where activations are large) and protects them from quantization error. 4-bit quality often matches BF16. Default 4-bit format for many open-weight deployments.

Byte-Pair Encoding. Subword tokenization that iteratively merges the most frequent adjacent pair into a new token until a target vocab size is reached. Used by GPT, Llama, Mistral, Qwen, DeepSeek.

Use the special `[CLS]` token's final-layer vector as the sentence embedding. Default for BERT-family bi-encoders trained for it.

Quantized model file format used by llama.cpp and downstream tools (Ollama, LM Studio). Supports 2-, 3-, 4-, 5-, 6-, 8-bit variants (Q4_K_M, Q5_K_M, Q8_0, etc.). Designed for CPU + small-GPU inference.

Post-training weight quantization for transformers. Calibration-data driven, reduces weights to 4-bit (sometimes 3-bit) with minimal quality loss. Common for self-hosted open weights.

Grouped-Query Attention. Keep h query heads, but share K/V across groups (e.g. 8 query heads share 1 KV head set). Cuts KV cache memory ~4–8× with no quality loss. Default in modern open-weight LLMs.

During autoregressive generation, cache K and V tensors from past tokens so each new step only computes Q for the new token. Memory cost = `2 × n_layers × n_kv_heads × d_head × seq_len × dtype_bytes` per sequence — what bounds long-context concurrency.

Normalize each token vector to zero mean / unit variance, then apply learned scale γ and shift β. Stabilizes training. Original transformer norm.

Average all (non-padding) token vectors from the final layer. Default for Sentence-Transformers, BGE-M3, E5. More forgiving than CLS — no single token must learn to aggregate.

Run h independent attention heads in parallel with `d_head = d/h`, concatenate outputs, project with W_O. Each head can specialize (induction, copy, syntactic, semantic).

vLLM mechanism that splits KV cache into fixed-size blocks managed like OS virtual memory. Enables continuous batching, prefix caching, preemption. The default for high-throughput LLM serving.

Root Mean Square normalization. `x / sqrt(mean(x²) + ε) · γ`. No mean centering, no shift. Cheaper than LayerNorm with no quality loss; default in Llama, Mistral, Qwen.

Rotary Position Embeddings (Su et al. 2021). Encodes position by rotating Q and K vectors by an angle proportional to position; the dot product becomes a function of relative position. Llama, Mistral, Qwen all use RoPE.

Language-agnostic tokenizer (Kudo & Richardson 2018) that operates on raw strings including spaces. Spaces become `▁`. Used by T5, Llama, Mistral, Gemma. Supports unigram-LM and BPE training modes.

A small draft model proposes k tokens; the target model verifies them in one forward pass and accepts the longest matching prefix. ~2-3× speedup at no quality loss with matched draft/target pairs. Medusa is the multi-head self-distillation variant.

Activation/gating combo used in modern FFN layers (Llama, Qwen). Replaces simple `x · W1 → activation → x · W2` with a gated formulation that improves quality at similar parameter count.

OpenAI's fast Rust-backed BPE tokenizer for GPT models. Use it to count tokens accurately before sending prompts; string length is not a reliable proxy.

BERT-family subword tokenizer. Like BPE but the merge criterion is likelihood of the training data given the vocabulary. Subword pieces inside a word are prefixed with `##`.

Typed graph of entities and relationships. Underpins GraphRAG; enables multi-hop reasoning over connected data.

Graph community detection algorithm. Used by Microsoft GraphRAG to cluster entity graphs into hierarchical communities for summarization.

The model-specific token format wrapping system/user/assistant messages. Use tokenizer.apply_chat_template to avoid silent quality loss.

Anthropic's LLM family (Opus, Sonnet, Haiku). Strong on coding, agent steerability, long-context reasoning, faithful instruction following.

Cohere's RAG-optimized LLM family (Command R, R+). Built-in citations, structured outputs, multilingual including Arabic.

Anthropic's alignment technique: model self-critiques against a 'constitution' of principles. Underpins Claude's behavior.

Maximum tokens the model can attend to in one call. Modern frontier: 128K-200K typical, 1M+ for Gemini Pro. Quality often degrades past 128K — see lost-in-the-middle.

Open-weights LLM lab. DeepSeek-V3 / R1 are very strong on math/code at low cost; MoE architecture.

Direct Preference Optimization. Trains models on preference pairs (chosen vs rejected) without an explicit reward model. Simpler than PPO/RLHF.

Updating model weights on domain data. Bakes behavior, format, persona into weights. Combined with RAG in production.

API mechanism for LLMs to emit structured tool calls. Modern APIs support parallel calls per turn.

Google's LLM family (Pro, Flash, Ultra). Native multimodal, 1M+ context, integrated with Vertex AI.

OpenAI's multimodal flagship; the most-used 'default' frontier model. Strong tool use, native multimodal.

IBM's enterprise-focused LLM family (3B-20B + code/embedding variants). IP indemnity, data lineage, strong fit for regulated industries.

Python library wrapping LLM APIs with Pydantic schemas + retry on validation failure. The de facto Python standard for structured output.

Constrained decoding that forces the LLM to emit valid JSON, optionally matching a schema. OpenAI strict mode is the gold standard.

Meta's open-weights LLM family. Llama 4 family is the open-source frontier as of writing — strong, commercial-permissive license.

Low-Rank Adaptation. Fine-tuning trick that updates small adapter matrices instead of full weights. Cheap, mergeable, default for open-source FT.

European LLM lab (Mistral, Mixtral, Codestral). Strong open + managed mix; EU data residency story.

Mistral's Mixture-of-Experts model (8x22B etc.). Activates a subset of experts per token, giving large-model quality at smaller compute.

Mixture-of-Experts architecture. Multiple expert FFN sub-networks; a router activates a subset per token. Mixtral, DeepSeek, Qwen MoE variants.

OpenAI reasoning-tuned model. Slow, expensive, much better at math/code/planning than GPT-4o.

Python data validation library. Standard for defining input/output schemas; integrates natively with OpenAI strict mode and Instructor.

Quantized LoRA. Loads base model in 4-bit, trains LoRA adapters in higher precision. Lets you fine-tune 70B models on a single GPU.

Alibaba's open-weight LLM family. Strong multilingual, top of many open benchmarks. Increasingly competitive with frontier closed models.

Reinforcement Learning from Human Feedback. The classic alignment technique behind ChatGPT; uses a reward model trained on preference data.

LLM-generated training data. Common for embedding fine-tuning (synthetic query-doc pairs) and instruction tuning.

LLM capability to invoke external functions/APIs. The mechanism agents use to act on the world.

MSA-trained BERT variant. Standard transformer baseline for MSA Arabic NLP tasks.

Linguistic concerns specific to Arabic: morphology, diacritics, dialect variation (MSA vs Egyptian/Gulf/Levantine), mixed-script content.

Vowel/consonant marks in Arabic script (تشكيل/tashkeel). Often omitted; should be normalized at ingest for retrieval.

Regional Arabic varieties (Egyptian, Gulf, Levantine, Maghrebi). Diverge heavily from MSA in vocabulary, syntax, grammar; generic models often fail.

Arabic NLP toolkit (QCRI): segmentation, POS, NER, diacritization. Useful for sparse retrieval pre-processing.

Arabic BERT variant trained on dialect tweets. Strong for downstream classification on dialect Arabic; lighter than as an embedder.

Modern Standard Arabic. Formal/written register, standardized across the Arab world. Generic multilingual models default here.

OpenAI's joint image-text encoder. Embeds both modalities into a shared vector space; default for cross-modal retrieval.

Google's CLIP variant with sigmoid loss. Stronger than CLIP at the same scale; common image-text encoder choice.

LLM with native image input (GPT-4o, Claude 4 with vision, Gemini Pro). Used for multimodal RAG generation and image captioning.

RAGAS metric: how well the answer addresses the question. Computed via reverse generation + similarity.

Provider APIs (OpenAI, Anthropic) for non-urgent jobs at ~50% off. Great for offline eval, classification, summarization.

Routing a small % of traffic to a new model/prompt version before full rollout. Standard safer-deployment pattern for LLM apps.

Pattern that stops sending traffic to a failing dependency until it recovers. Standard for AI services that depend on external models.

RAGAS metric: % of retrieved chunks that are actually relevant. Diagnoses retriever noise.

RAGAS metric: % of needed information that was actually retrieved. The ceiling on RAG quality.

RAGAS metric: % of answer claims that are grounded in retrieved context. Diagnoses generator hallucination.

Runtime toggle that controls which prompt/model/config is active. Enables atomic rollback and A/B testing.

Semantic-cache library: embed query, return cached answer if near-neighbor exists. Reduces cost and latency for repeated/paraphrased queries.

Input/output validation, PII redaction, prompt-injection defense, schema validation around LLM calls. The non-negotiable production layer.

Drop-in observability proxy for LLM APIs. Easy to integrate; useful for cost / latency dashboards.

% of queries where the relevant document appears in top-K. Cheap deterministic retrieval-only metric.

Property of an operation that can be safely retried without duplicate effect. Critical for tool design and async LLM pipelines.

Injection where the malicious payload lives in retrieved/external content, not the user's direct input. The harder variant to defend.

Managed prompt-injection and abuse-detection service. Drop-in input/output guardrail layer.

Open-source LLM observability and eval platform. Self-hostable LangSmith alternative with strong feature parity.

LangChain's hosted observability and eval platform. First-class trace inspection, dataset eval, online evaluation.

Single SDK abstracting 100+ LLM providers behind one API. The default abstraction for portability and routing.

Using an LLM to evaluate other LLM outputs against a rubric. Cheap automation but biased — use a different model from the one judged.

Microsoft prompt-compression library. Trains a small model to drop low-importance tokens; 60-80% input reduction with small quality loss.

Pattern where queries are routed first to cheap models, escalated to frontier on confidence/complexity. Routinely cuts costs 60-90%.

Normalized Discounted Cumulative Gain. Standard ranking quality metric: rewards correct items appearing earlier in the ranked list.

NVIDIA's Colang-based guardrails framework. Declarative dialog rails over LLMs.

CNCF observability standard. Many LLM observability tools (Phoenix, custom) emit traces via OTEL for ingestion into existing observability stacks.

Open-source LLM observability with strong trajectory visualization. OpenTelemetry-native.

Microsoft's open-source PII detection and redaction toolkit. Common pre-LLM input filter.

Provider-side caching of long static prompt prefixes (system prompt + retrieved docs + tool schemas). Anthropic, OpenAI, Gemini all support; cuts input cost up to 90%.

Adversarial input (direct or in retrieved content) that overrides the system prompt. Defended via privilege separation, classifiers, output checks. Greshake et al. 2023.

Open-source prompt testing framework. CLI + CI for eval, regression testing, and provider comparison.

RAG evaluation framework: faithfulness, answer relevancy, context precision, context recall. The standard set of LLM-judge metrics for RAG.

LMSYS-developed learned router that distills GPT-4-quality routing decisions into a small classifier — cuts cost without quality loss.

Running a new system in parallel with the old, logging outputs but not serving them. Highest-safety deploy pattern.

Server-sent token-by-token output. Reduces perceived latency dramatically; supported by all major providers.

Hugging Face Text Generation Inference server. Production-grade serving; competitor to vLLM.

Time-to-first-token. The latency users feel; often more important than total latency in chat UX.

High-throughput LLM serving engine. PagedAttention for efficient KV cache management; the default for self-hosted inference.

RAG with interventions at pre-retrieval (query rewriting, HyDE), retrieval (hybrid + re-rank), and post-retrieval (compression, dedup) stages.

Retrieval inside an agent loop. The LLM decides when to retrieve, what to retrieve, and when to stop, instead of a fixed pipeline.

Shrinking retrieved context via summarization (LLMLingua, manual) to mitigate lost-in-the-middle and reduce tokens.

Corrective RAG. Adds a retrieval grader between retrieve and generate; routes to web search or knowledge refinement on poor retrievals.

A model that takes (query, document) as a single input and outputs a relevance score. More accurate than bi-encoders but quadratic in pair count, so used only at re-rank stage.

RAG over a knowledge graph. Microsoft's approach: entity extraction → graph build → community detection (Leiden) → hierarchical summarization → local/global query modes.

Hypothetical Document Embeddings. Generate a fake answer with the LLM, embed it, and use that embedding for retrieval. Bridges query–document vocabulary mismatch.

Liu et al. 2023 finding that LLM attention over long context is U-shaped — content in the middle of a long context is recalled poorly. Drives compression and re-ranking.

RAG architected as a DAG of pluggable modules (retrievers, routers, predictors) rather than a fixed linear pipeline.

RAG over mixed-modality corpora (text + images + tables). Two patterns: joint embedding space (CLIP-style) or vision-LLM captioning + text indexing.

The baseline RAG pipeline: chunk, embed, store, retrieve top-K, stuff into prompt. The reference implementation against which improvements are measured.

Index small chunks for retrieval precision, but expand to the parent paragraph or document for generation context.

Reformulating the user query (often via LLM) before retrieval to better match corpus vocabulary.

Retrieval-Augmented Generation. A pattern where an LLM is grounded on retrieved evidence at inference time instead of relying on parametric knowledge alone.

Re-scoring retrieved candidates with a stronger model (cross-encoder, Cohere Rerank, BGE Reranker, LLM-as-judge) to improve top-K ordering.

A small classifier or LLM judge that scores retrieved chunks for relevance before passing them to the generator. Core component of CRAG.

Reciprocal Rank Fusion. score(d) = Σ 1/(k + rank_i(d)) across retrievers; uses rank only so retrievers' incomparable scores don't matter. k=60 is standard.

RAG variant where the model emits reflection tokens ([Retrieve], [IsRel], [IsSup], [IsUse]) controlling its own retrieval and grounding behavior.

Asking the LLM to first generalize the query, retrieve for the general form, then answer the specific. Useful for multi-hop.

The number of documents/chunks retrieved per query. Larger K → higher recall but more noise and tokens; tune via retrieval eval.

Approximate Nearest Neighbor search. Trades exact recall for sublinear query time. Implementations: HNSW, IVF-PQ, DiskANN, ScaNN.

Embedded / lightweight vector DB. Best for local development; not a production-scale database past ~5-10M vectors.

Microsoft graph-based ANN designed for SSD: scales to billions on a single node. Pinecone serverless uses this internally.

Meta's ANN library. Fast primitives but not a database — you build the storage, replication, and metadata layer yourself.

Hierarchical Navigable Small World. Multi-layer graph index. Tunables: M (neighbors), ef_construction (build), ef_search (query). Default ANN choice.

Inverted File + Product Quantization. Cluster vectors then quantize. Lower recall than HNSW but compressed; suits billion-scale on commodity hardware.

Columnar vector DB (Lance format). Good fit when you also want analytics over the same data.

Open-source vector database (Zilliz commercial). Designed for billion-scale, strong distributed story.

Postgres extension for vector search. The right answer when you already run Postgres and corpus is below ~10M vectors.

Managed vector database. Serverless tier, strong filtering, low operational friction. Common SaaS choice.

Rust-based vector database, fast self-host, rich payload filtering. Common on-prem choice.

Google's anisotropic-quantization ANN. Used in Vertex AI Matching Engine. Strong recall/speed tradeoff.

Vector database with native hybrid (BM25 + dense) search and a modules ecosystem. Managed or self-host.