Notes - Inference Engineering

Philip Kiely | August 3, 2026

Chapter 0: Inference

Training vs. Inference

Inference is defined as the second phase of a generative AI model's lifecycle, following training. Training is the process of learning model weights from data; inference is the process of serving generative AI models in production. During the prior decade's machine learning boom, large numbers of data scientists and ML engineers gained familiarity with both phases for classic ML models. Inference for classic ML models (e.g., XGBoost) was relatively simple — an early example given is running such models on lightweight CPUs with a basic software stack (referencing early Bas experience). Generative AI inference is categorically more complex: it is not sufficient to take model weights, acquire GPUs, and expect fast, production-reliable results at scale.

The Three Layers of Inference

Doing generative AI inference well requires three layers working together to support mission-critical inference at scale:

• Runtime: optimizing the performance of a single model on a single GPU-backed instance.
• Infrastructure: scaling across clusters, regions, and clouds without creating silos while maintaining uptime.
• Tooling: giving inference engineers the right level of abstraction to balance control with productivity.

Runtime Layer

The runtime layer ensures an individual model running on a GPU (or across multiple GPUs in a single instance) runs as performantly and efficiently as possible. This layer depends on a sophisticated software stack spanning CUDA, PyTorch, and inference engines such as vLLM, SGLang, and TensorRT-LLM. Low-level optimization matters significantly — kernels like FlashAttention are cited as delivering significant performance gains.

The runtime layer relies on model performance techniques that apply new research to the specific challenges of generative AI inference:

• Batching: running incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.
• Caching: reusing the KV cache (the cached results of the attention algorithm) between requests that share prefixes.
• Quantization: lowering the precision of select pieces of the model to access more compute and reduce memory burden.
• Speculation: generating and validating draft tokens to produce more than one token per forward pass during decode (illustrated in Figure 0.2 as a technique that improves inference latency).
• Parallelism: efficiently leveraging more than one GPU to accelerate large models witht introducing new bottlenecks.
• Disaggregation: separating the two phases of LLM inference — prefill and decode — onto independently scaling workers.

A non-obvious point: these model performance techniques are not limited to LLMs. They apply across modalities — vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation — each of which extends AI system capabilities and requires its own inference optimizations.

Infrastructure Layer

Runtime optimizations alone are insufficient. Regardless of how performant a single model server instance is, it will eventually receive more traffic than it can handle. This is explicitly framed as not a CUDA or PyTorch problem, but a systems problem to be solved at the infrastructure layer.

Infrastructure problems evolve with scale, in three stages:

1. Autoscaling stage: knowing when to add and remove replicas and doing so quickly.
2. Capacity stage (past roughly a few hundred GPUs):rs must spread workloads across multiple regions and cloud providers to access enough GPU capacity — this quickly creates silos, where some clusters are starved for resources while others sit with unused capacity.
3. Global unification stage: the final level of infrastructure scale, where all available resources are treated as a single unified pool of compute (illustrated in Figure 0.3, which depicts unifying capacity across multiple cloud service providers).

Practical benefits of thoughtful multi-cloud infrastructure include improved reliability (protection against downtime in any single region or cloud provider) and, for global applications, improved end-to-end latency by running inference near end users.

Tooling Layer

Once runtime and infrastructure capabilities exist, they must be presented at an appropriate level of abstraction. Both inference providers (e.g., Baseten) and internal teams building inference must decide what tooling and developer experience to offer as the third critical layer Developer experience for inference is described as inherently subjective, spanning a spectrum:

• One extreme: a black box — supply model weights, receive back an API.
• Other extreme: exposing only basic constructs for compute, network, disk, and similar primitives.

The practical recommendation is that the right developer experience sits somewhere in the middle — giving inference engineers enough control to run mission-critical inference confidently, while providing enough abstraction to remain productive.

Book Roadmap

Inference Engineering is framed as a map of technologies and techniques powering inference across the three layers (runtime, infrastructure, tooling). Chapter previews given:

• Chapter 1, Prerequisites: product thinking and AI engineering work needed before inference engineering begins — use case definition, latency and cost budgeting, and selecting/evaluating which generative AI models to optimize and deploy.
• Chapter 2, Models: technical architecture of AI modelm LLMs to image and video generation models), establishing where inference bottlenecks exist, with special focus on optimizing attention.
• Chapter 3, Hardware: starts from GPU spec sheets, breaking down compute and memory, disambiguating architectures and SKUs within NVIDIA's datacenter-grade lineup, then briefly surveys other accelerators on the market.
• Chapter 4, Software: builds up abstractions from CUDA through frameworks (PyTorch, Transformers, Diffusers) and inference engines (vLLM, SGLang, TensorRT-LLM); also introduces Dynamo, NVIDIA's latest system for large-scale distributed model serving.
• Chapter 5, Techniques: covers key model performance optimization techniques adapted from cutting-edge research and applied in production — quantization, speculative decoding, KV cache re-use, model parallelism, and disaggregation.
• Chaer 6, Modalities: expands inference engineering beyond LLMs into voice and visuals. Notes that vision-language models, embedding models, ASR models, and speech synthesis models adapt LLM architectures — meaning the same tools and techniques used for LLMs can run them. Image and video generation models, by contrast, have their own distinct architectures and associated optimization techniques.
• Chapter 7, Production: concludes with the important problems to solve in operating infrastructure for, and building performant applications on top of, optimized model inference services.
• Appendices A and B: a glossary of inference engineering terms and a collection of recommended resources for further reading, respectively.

Knowledge Cutoff

An explicit caveat is included: like LLMs, books have knowledge cutoffs, and this book was finished in January 2026.

Chapter 1: Prerequisites

Inference engineering adds speed and scale to AI products by optimizing production serving of generative models. Optimization means identifying the best solution among a range of options, and this requires knowing what "best" means for a given product before optimizing performance or building infrastructure — many performance improvements come from tradeoffs among latency, throughput, and quality rather than maximizing a single factor.

A non-obvious framing: optimization is often about finding the right balance, not maximizing one dimension. NFL players are big, fast, and strong, butot as big as sumo wrestlers, as fast as Olympic sprinters, or as strong as champion powerlifters — their bodies are optimized for the specific demands of their position over a season. Inference systems similarly muste specialized for the specific demands of a model, product, and traffic pattern. The more constraints introduced, the better the achievable outcomes.

Before optimizing, teams should know:

• Model requirements: which model(s) inference needs to run on.
• Application interface: how inputs are delivered and how outputs are expected to be formatted.
• Latency budget: end-to-end, how fast the product needs to respond to a user action.
• Unit economics: what's reasonable to spend per-request, per-user, or per-month.
• Usage patterns: how many concurrent users, and whether usage follows a pattern (e.g., business hours spikes).

Practical warning: early in building an AI product, these answers are often unclear. At that stage it's better to use off-the-shelf APIs rather than invest in dedicated inference. Only once the product scales and requirements become clear does inference engineering become worthwhile.

Scale and Specialization

Two ways to add AI models to a product:

• Shared inference: send traffic to a public API endpoint, paying per million tokens or another consumption-based metric.
• Dedicated deployments: rent (or purchase) GPUs and run an inference service exclusively for the application, paying per hour of GPU time.

Non-obvious point: shared vs. dedicated inference is not the same axis as closed vs. open models — there are shared endpoints for open models, and many labs offer dedicated setups for closed models to large customers. However, a key motivation for adopting open models is th it unlocks unrestricted dedicated inference.

Most AI products start with pay-per-token APIs because the tradeoffs favor that approach while searching for product-market fit:

| Pros of shared inference | Cons of shared inference | |---|---| | Zero overhead, only pay for consumption | Cost scales linearly with usage | | No cold start times, model always available | Provider uptime caps product uptime, noisy neighbors | | Minimal engineering work, just need an API key | No control over latency, model quality, or rate limits |

Three reasons to eventually shift to dedicated deployments:

• Scale: traffic volume makes paying per GPU more economical than paying per million tokens.
• Specialization: running a custom or fine-tuned model, or specific latency/uptime requirements.
• Orchestration: multiple models and multi-step pipelines requiring minimized network latency and deployment complexity.

Warning: switching to dedicated deployments hands you full ownership of inference engineering — more flexibility and control, but also more engineering surface area and a higher floor for monthly inference spend. Only switch once there's a clear and immediate business need.

About Your App

Every inference engineering decision is downstream of the specific use case — analogous to a sports recruiter selecting different physical traits depending on whether they coach basketball (tallest) or gymnastics (shortest).

Two cases require building at the highest level of generality:

• Foundation models: a self-trained model sold via a public shared inference API, needing to suppmany usage patterns.
• Inference platforms: building an inference platform (internal or product) that must support any model and any use case.

Most AI-native applications, however, are vertical apps (code editors, customer service agents) where the goal is to add as many constraints as possible by getting specific about the use case.

AI-Native Applications

Different application categories rely on different models/modalities and need tailored inference considerations:

| Category | Example | Considerations | |---|---|---| | Agents | Prospecting agent for sales teams | One user action triggers many inference calls | | Chat | Front-line customer support chat with RAG | Time to first token makes chat feel fast | | Voice | Real-time translation between languages | End-to-end latency for natural conversation | | Media | Virtual try-on for clothes, shoes, jewelry | Balance output quality vs. speed | | Search | Legal document discovery | Offline corpus filling vs. online user requests | | RecSys | E-commerce product recommendations | Consistent latency with high request volume | | Completion | Tab completion for coding in an IDE | Full completion chunk at user's typing speed | | Moderation | Scan user-generated content for safety | High throughput for cost-effective checks |

This is only a small sample; as models get faster, cheaper, and smarter, entirely new use cases not yet imagined will emerge.

Online versus Offline

A primary tradeoff in inference engineering is latency versus throughput: lower latency makes an application faster, but higher throughput makes it cheaper at scale (fewer GPUs for the same number of users).

Most AI applications (code completion, chat, voice agents) are online — real-time, with an impatient user waiting — and should be optimized for latency.

Some applications have offline batch inference needs, better served by high-throughput deployments where individual requests can be slow but the system processes far more requests per hour in parallel. Example offline workloads:

• Catalog transcription: transcribing a bactalog of podcasts, interviews, or other audio to make it searchable.
• Document processing: embedding, converting, or analyzing documents on a regular cadence.
• Corpus preparation: cleaning, embedding, or preparing massive corpora for model training.

Non-obvious practical point: a single model can serve both needs — Whisper (speech-to-text) could run both a real-time dictation app and a batch transcription job — but if both use cases have enough volume, it's more cost-effective reate two separate deployments of the same model, one tuned for latency and one for throughput.

Consumer versus B2B

Consumer and business applications have different inference needs.

• Consumer apps are more cost-sensitive with less predictable usage; many are designed for virality, where a single launch or marketing campaign can spike usage overnight. Engineers should prioritize marginal cost and flexibility while keeping latency and availability at a decent (not necessarily top) standard.
• B2B products often have better margins and more stable usage but require high availability and consistently low latency, since mission-critical software in the revenue path is held to a high performance/reliability standard. Engineers must favor latency and uptime, with cost and scale as secondary concerns.

Warning: in both consumer and business contexts, compliance can limit infrastructure options, especially in regulated industries. Essential considerations:

• Data sovereignty: are GPUs located in a region where user data is legally allowed to be sent?
• User privacy: are model inputs/outputs kept private and secure?
• Regulatory compliance: are you and underlying providers compliant with all relevant regulations?

Inference engineers must work closely with security and legal experts to ensure infrastructure compliance.

Model Selection

All else being equal (hardware, runtime, optimizations, architecture), a smaller model with fewer parameters is faster and cheaper than a larger one. This makes model choice — not the runtime engine or speculation algorithm â single most important decision in model performance optimization.

Practical guidance: early-stage AI engineers should just use prebuilt pay-per-token APIs for powerful frontier models (Kimi, DeepSeek, or closed models like GPT and Gemini) — before product-market fit, it's not worth spending time or money on custom inference.

When it's time to scale, the advice flips: find or create the smallest, easiest-to-run model that's smart enough for the task. In many cases this will still be a triion-parameter frontier model, but it's always worth checking whether a smaller, cheaper, faster model can do the job.

Non-obvious point: model choice also determines which inference optimizations are available, since inference engines vary in depth of support across model architectures. Sticking with popular architectures ensures robust support across the performance tooling landscape.

Model Evaluation

Model evaluation ("evals") is systematically measuring model intelligence, and is described as a prerequisite for inference engineering. Evals help engineers:

• Spend time wisely: confirm a model is useful before investing in making it fast.
• Establish a baseline: some performance optimization techniques risk reducing model quality, so a baseline is needed for comparison.

Key distinction: evals are tailored to specific products, domains, and tasks, unlike standard intelligence benchmarks (MMLU, SWE-bench) that measure general capability against common tasks.

Warning/non-obvious point: intelligence benchmarks are useful for shortlisting models but have become saturated or even gamed, invoking Goodhart's Law — "when a measure becomes a target, it ceases to be a good measure" — since frontier labs have heavy incentive to show record-breaking benchmark scores with each release. Better overall intelligence gauges exist (e.g., Elo rating via head-to-head win rate), but there's no substitute for directly measuring how a model performs on your specific application.

Tips for useful model evaluation:

• Look at your data: check eval lts against intuition for the product and problem space.
• Be precise: know the hardest problems the model must solve, and focus evaluation there.
• Use tools: don't reinvent the wheel on this fundamental AI engineering problem.

(Further eval tooling recommendations are noted as covered elsewhere, outside core chapter content.)

Fine-Tuning for Domain-Specific Quality

Fine-tuning takes a pre-trained foundation model and adapts it to a specific use case by introducing new data, changing model weight values while keeping the same overall architecture.

Practical benefit: fine-tuning a small model to pass evals makes it easier to hit latency and cost targets.

Concrete example: translating English into SQL. General-purpose coding models handle SQL well but run into the hundreds of billions of parameters. Because SQL is a relatively constrained language, a tiny fine-tuned model of just a few billion parameters can reach equivalent performance on this narrow task.

Caveat: text-to-SQL is an extreme example — many domains won't support such a vast size reductn — but it illustrates what's achievable with a cleanly scoped domain, a strong set of evaluation criteria, and high-quality labeled data for fintuning.

Distillation

Distillation asks: what if you could retain most of the intelligence of a large model at a fraction of the size? It uses a large "teacher" model to train a smaller "student" model to emulate the teacher's behavior.

Key technical distinction: unlike fine-tuning on synthetic input-output pairs, distillation shows the student model the teacher's actual probability distributions, not just final answers. Where fine-tuning teaches better performance in a specific domain, distillation teaches the model to emulate the teacher's behavior — good and bad.

Non-obvious point: distillation sees substantially less real-world use than fine-tuning. When a frontier lab releases a family of models of different sizes, the smaller models are generally not distilled from the larger ones — they're independently trained, to prevent the larger model's biases from artificially limiting theller ones. Distillation becomes valuable mainly when a lab has only trained a large model and wants to make it more accessible in smaller form.

Concrete example: in January 2025, DeepSeek released their flagship reasoning model DeepSeek-R1 (671B parameters) along with distilled versions built on the popular Llama 3 and Qwen 2.5 architectures. These distilled models showed similar reasoning behavior to the main model, albeit with worse intelligence benchmark scores, but were small enough to leverage existing performance tooling built for Llama/Qwen architectures. At time of publication, these DeepSeek-R1 distills remain among the most popular distilled models on Hugging Face, alongside distills of models like Whisper (audio transcription) and some image generation models.

Measuring Latency and Throughput

The two most common LLM performance metrics are TTFT (time to first token) and TPS (tokens per second); modality-specific metrics beyond LLMs are covered elsewhere.

| TTFT | TPS | |---|---| | With streaming output, how long until a user sees the first output token | How many tokens per second the user receives after the first token generates | | Based on compute-bound prefill | Based on bandwidth-bound decode | | Lower TTFT = better latency | Higher TPS = better latency |

Non-obvious precision issue: TPS is an ambiguous term — it can mean a latency metric (tokens/second per user) or a throughput metric (tokens/second for the entire service). Most people default to the per-user latency meaning, but more precise terms exist when needed:

• Perceived TPS: observed tokens per second per user after the first token (latency).
• Total TPS: total tokens generated per second by the entire inference service (throughput).
• Inter-token latency (ITL): time between subsequent tokens; an ITL of 10 milliseconds equates to 100 tokens per second per user.

TTFT and TPS are most useful for user-facing streaming LLM systems like chatbots. For other request types, such as an agent's tool call, individual tokens aren't useful on their own — instead, measuatency as total response time.

Latency Percentiles

Important distinction: which percentile is being measured when discussing/comparing metrics. Naively looking at mean (average) TTFT or TPS doesn't tell the whole story, because LLM total response time is generally a right-skewed distribution — most times cluster around a mode, but outliers can take significantly longer. Mean latency is generally higher than P50 latency due to these outliers.

Warning: outliers can dramatically affect ur experience and trust — it's not good enough for most interactions to feel snappy if one in every ten takes several seconds. Hence engineers measure latency in percentiles:

| Percentile | Meaning | Interpretation | |---|---|---| | P50 | Median latency | 1 in every 2 requests is slower | | P90 | 90th percentile latency | 1 in ery 10 requests is slower | | P95 | 95th percentile latency | 1 in every 20 requests is slower | | P99 | 99th percentile latency | 1 in every 100 requests is slower |

While driving down average latency matters, good performance work also focuses on reducing P90/P99 latencies for a more reliable user experience.

End-to-End Metrics

Another important distinction: whether a metric measures solely inference time (on-GPU time to generate tokens) or end-to-end time (accounting for network latency and queue time).

Both are valuable: inference-only time reveals how effective model performance work is, while end-to-end metrics reveal users' actual perception of speed. Practical diagnostic rule: when inference time is fast but end-to-end time is slow, the problem lies in infrastructure rather than model performance optimization, and attention should shift accordingly.

Chapter 2: Models

Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable without sacrificing quality. Doing this well requires a strong intuition for how models work under the hood. Generative AI models are compositions of large, complex neural networks whose lineage traces back to 1950s perceptrons for binary classification, through multi-layer perceptrons with back-propagation (introducing hidden states and iterative weight adjustment), to deep networks with dozens of layers researched in the 2000s. AlexNet (2012) proved deep neural networks could deliver real-world capability and demonstrated the effectiveness of GPUs for deep learning, spurring architectures like word embedding models and Generative Adversarial Networks (GANs). The real starting point for modern generative AI is 2017's "Attention Is All You Need" (Vaswani et al.), which introduced the transformer — a neural network with an attention mechanism that learns relationships between parts of a sequence. Transformers underpin every modality: text, voice, image, and video. Two major transformer-based generation styles exist across modalities: autoregressive token generation (predict thnext token from a tokenized sequence) and iterative denoising (refine random noise toward a likely output via diffusion). This chapter covers LLM architecture (autoregressive) and image generation architecture (iterative denoising).

Neural Networks

A basic intuition for neural network concepts is essential for productive inference engineering. The fundamental unit is a node (neuron): a short program that multiplies an input by weights, adds a bias, and returns a result. A group of nodes forms a layer; nodes within a layer compute independently, while the "network" connections exist between layers, where each layer's output feeds the next layer's input. LLM neural networks contain dozens to hundreds of layers, of three types: the input layer (accepts/processes input), hidden layers (iteratively transform input toward output), and the output layer (returns the prediction). Outputs of hidden layers are called hidden states — a form of internal representation whose dimensionality (vector size) matters greatly. Text representations increase dimensionality (encoding text chunks into vectors of hundreds/thousands of numbers to capture semantic meaning), while image model representations reduce dimensionality (from millions of pixels down to a manageable se).

Two network roles handle these representations: encoders (create an internal representation of input, adding semantic meaning) and decoders (use the internal representation to generate output). Networks are composable — combinable into a single model or chained into pipelines. Modern LLMs are decoder-only; encoder-only models are rarer today, with BERT-family embedding models as a prominent example. Many non-text models use encoder-decoder architecture — for instance, Whisper (open autranscription model) uses an encoder for audio input and a decoder to generate text tokens.

Linear Layers and Matmul

The most essential neural network operation is matrix multiplication (matmul): an input vector multiplied through a matrix produces an output vector. A linear layer is the simplest matmul form, applying a weight matrix and adding a bias vector to an input vector. A linear layer's weights are a small fraction of a model's total weights; specific weight values are set during training.

Activation Functions

Matrix multiplication is composable — multiplying by two matrices in sequence equals multiplying by the product of those matrices. This is a problem for multi-layer networks: a stack of linear layers would collapse mathematically into a single equivalent laye defeating the purpose of depth (deep networks are useful because more layers use parameters effectively and encode richer meaning in hidden states). Activation functions solve this by breaking linearity between layers. They must be non-linear (to prevent collapse) and differentiable or mostly differentiable (to support back-propagation).

ReLU (Rectified Linear Unit) is one of the most basic activation functions used in inference: return X if X is greater than zero, else return zero. Dozens of activation functions exist — including one nicknamed "Swish" for its resemblance to the Nike logo — but most follow the same pattern of zeroing out negative values while keeping positive values unchanged. ReLU, SiLU, Swish, and SwiGLU are fast to run, easy to train on (mostly differentiable with usable grats), and effectively break linearity for multi-layer networks.

LLM Inference Mechanics

LLMs generate new tokens one at a time based on every previous token (autoregressive generation). Tokens are the atomic units of language models — numbers representing text chunks. Modern LLMs use subword tokenization: each token is a word or fraction of a word (one token per common word/punctuation, but splitting less common words into multiple tokens). Converting between text and tokens requires noeural network computation — a tokenizer is simply a mapping between strings and numerical token IDs. A model's vocabulary is the full token-to-string mapping. Vocabularies and tokenizers differ by model, and newer models tend to use more efficient tokenization schemes — fewer tokens needed to produce an output means faster en-end inference. Most models exceed 100,000 vocabulary tokens; other modalities (e.g., speech synthesis) expand vocabularies to let tokens represent things like audio waveforms.

Inference involves two or three token sequences: the input sequence (prompt, chat, context, functions, etc.), an optional reasoning sequence (intermediate "thinking" output for reasoning models), and the output sequence (the final response). Combined, these are bounded by the model's context window (total tokens processable/generatable per request); a request can further cap output length via max_tokens. Although the input sequence is ultimately a single string, LLMs are trained to accept varied structured inputs — multi-turn chat with roles, function/tool signatures, sometimes multimodal inputs. These get combined into one sequence via the chat template, which differs subtly model to model and must be implemented correctly by the inference engine — a notable non-obvious pitfall, since a subtly wrong chat template silently degrades quality.

Tokenizing the input with the chat template applied is "step zero." Two primary inference phases follow:

• Prefill: processes the input sequence to calculate attention for each input token and stores results in a KV cache.
• Decode: orms forward passes to generate tokens autoregressively.

Each decode forward pass must produce a token, but neural networks natively output vectors, not tokens — requiring extra steps. The output layer produces a vector of logits sized to the vocabulary; after normalization, these represent per-token probabilities. The output token is chosen via weighted random sampling over normalized probabilities, tunable using:

• Temperature: adjusts logits before normalization.
• Top-k: keeps oy the k most likely tokens after normalization, then re-normalizes among them.
• Top-p: keeps the smallest set of tokens whose cumulative probability reaches p.

Lower temperature/top-k/top-p makes output more predictable by constraining selection to high-likelihood tokens; temperature=0 or top-k=1 makes selection fully deterministic (always pick the highest-probability token). For structured output (e.g., JSON schemas), additional tools like logit biasing steer generation after each forward pass; correct implementation of these is essential for tool-use quality and high-quality inference generally — a practical warning that sloppy structured-output handling breaks core LLM capabiliti. Generation continues until a special stop token appears (unless context window or max_tokens is hit first). The two core inference-loop costs are KV cache generation (prefill) and logit-vector/token generation (decode) — these dominate time and resource use because they depend on large neural network computation.

LLM Architecture

Every Hugging Face LLM ships a config.json detailing architecture — decisions made during training about each component's nature and shape. Within one architecture there can be multiple sizes (e.g., Llama 8B and 70B), multiple variants (base vs. instruct sharing the same architecture), and unlimited fine-tunes (methods like LoRA change behavior, not architecture). Architecture matters practically because it determines runtime/engine support: a highly optimized deployment of one architecture variant carries over performance gains to other variants of the samchitecture.

Architecture names are structured — e.g., Qwen3MoeForCausalLM parses as: Qwen (model family/brand), 3 (major architecture version), MoE (Mixture of Experts indicator), CausalLM (causal language model indicator). A causal language model predicts the next token from previous tokens only, unlike a masked language model that fills blanks using both left and right context; all generative LLMs today are causal. Beyond the architecture name, config.json details layer dimensions anthe vectors passing through them during inference.

Transformer Blocks

An LLM's main body is dozens to hundreds of transformer blocks, forming a network with three layer types: the embedding layer (input layer, tokens → embeddings), transformer blocks (hidden layers generating predictions), and the output layer (a.k.a. language modeling head/LMHead, converting hidden states to a logits vector sized to vocabulary). Within transformer blocks, sublayers handle attention, a feed-forward network, and normalization. The feed-forward network (aulti-layer perceptron) comprises the majority of trainable weights; attention sublayers are the second-largest component; normalization and activation functions are comparatively negligible in size. Despite linear sublayers dominating weight count, attention is the more complex operation for inference.

Attention

Attention relates a given token to other tokens in the sequence — mirroring how humans interpret relationships between words. Example: in "I decided to write a book because I thought it would be easy, but it was actually hard," attention reveals that "it" refers to writing a book. The standard form is scaled dot-product attention, taking three inputs: Q (queries — embedded representation of the token being generated/updated), K (keys — representations of all prior tokens), and V (values — computed attention values for prior tokens). Attention sublayers are multi-head, with each head running one attention operation in parallel with others on the same sublayer; different heads may specialize in different relationship ty., subject-verb agreement vs. co-reference resolution).

Two main attention types exist: self-attention (Q, K, V from the same sequence) and cross-attention (Q from a different sequence than K/V, conditioning Q on external information). LLMs use self-attention with a causal mask (preventing looking ahead); image generation/multimodal models also use cross-attention (e.g., between the image being generated and its text prompt).

Because attention checks the current token against every previous token, it is quadratic-time with respect to sequence length — as context grows, attention slows. In practice, the KV cache makes attention effectively linear: it stores key-value pairs for each previous token so they can be looked up instead of recomputed, built during prefill and used/updated during decode. The KV cache liv on GPU memory by default and is a major topic in inference engineering (detailed later in section 5.3).

Mixture of Experts Models

Network density is defined by the number of connections between layers — denser nworks retain more information, sparser networks need less compute/memory. Mixture of Experts (MoE) adds sparsity to linear layers: instead of one giant matrix, an MoE model has hundreds of smaller matrices ("experts") and routes each input to a small subset of them (activation). Example: Qwen3-235B-A22B activates only 22 billion of its 235 billion total parameters per request. This low active-parameter count makes MoE models highly efficient for single-request local inference. Important warning/non-obvious point: in batched production inference, different requests activate different experts, so almost all model parameters end up active at any given time — unless sparsity is deliberately preserved via large-scale Expert Parallelism (covered in section 5.4.2).

Expert routing is granular — every forward pass generating one token works through every model layer, and a tiny internal "router" model picks which experts activate at each layer. In the Qwen example (128 experts), the router selects 8 experts at each of 94 layers per generated t. MoE architectures are especially popular for larger models (100B+ parameters), though some MoE models exist as small as 20-30B parameters. MoE unlocks Expert Parallelism, a new inference parallelism form enabling high-throughput inference for large models across multiple GPUs. Models under 32B parameters — especially under 8B — tend to use dense architectures efficiently instead; domain-specific models (e.g., tab completion) also gain little from MoE since the whole model effectively fuons as one expert.

Image Generation Inference Mechanics

Image generation models predate the public LLM boom slightly — closed Midjourney and open Stable Diffusion models both first released in summer 2022. Unlike monolithic LLMs, image generation models are pipelines of multiple models working together, with three essenti components:

• Text encoder: converts the text prompt into instructions the image model can use.
• Denoising model: the core component, iterating from noise to an image based on the prompt.
• Variational Autoencoder (VAE): converts model output from latent space to pixel space.

Beyond the base pipeline, image inference commonly adds LoRAs (lightweight fine-tunes changing style/enhancing quality) and ControlNets (outlines/edges steering output to match broad shapes and colors). A rich open-source ecosystem, including tools like ComfyUI, enables building complex pipelines by swapping components for unique outputs.

The entire pipeline operates in latent space rather than pixel space, because an ordinary HD image (1024x1024) exceeds a million pixels, and calculating attention over the whole image in parallel at full pixel resolution would be infeasible. Latent space is a much lower-dimensional representation — a latent matrix might be 128x128, roughly 1% of the pixel-space value count. Latent space initializes as random noise; the denoising model refines that noise into an image over a series of steps guided by the text prompt. Unlike LLMs (which process tokens sequentially one at a time), each denoising step updates the entire latent space atnce. Most image models take 30 to 50 steps for high quality.

Within each step, the model actually runs two forward passes — one with conditioning (the text prompt) and one without — then combines them based on a guidance scale. This means a "50-step" generation is really 100 forward passes — a non-obvious cost multiplier worth remembering when estimating compute. Key inference arguments controlling this process:

• Prompt: describes desired image content.
• Negative prompt: separspecifies styles/objects to exclude.
• Number of steps: trades speed for quality (30–50 typical).
• Guidance scale: balances creativity vs. prompt adherence (typically ~4).
• Image size: selects resolution/aspect ratio from a fixed menu.

Image Generation Model Architecture

Image generation models are built on ffusion transformers — architecturally similar to LLM transformers but processing image data rather than discrete token embeddings. Diffusion transformers view images in patches: training feeds images in via overlapping 2x2 or 4x4 pixel patches embedded into latent space; inference reverses this, transforming latent space back into pixels once generation finish.

Stable Diffusion XL (SDXL) offers a clean example pipeline (text encoder, denoising model, VAE) and remains architecturally relevant despite its age. SDXL's pipeline uses two diffusion models for denoising — a base model (pure noise → coherent image) and a refiner model (adds detail, ensures prompt adherence), trained for these separate tasks.

Modern models substantially outperform SDXL in prompt adherence, accurate faces/hands/details, legible text rendering, and image-to-image support. Models like the Qwen Image family follow a broadly similar pipeline shape but with much larger, more capable components at every stage:

| Component | SDXL (2023) | Qwen Image (2025) | |---|---|---| | Text encoder | CLIP-based model | Qwen 2.5 VL (7B) | | Denoiser | <4B parameters | 20B parameters | | VAE | Single encoding | Dual encoding |

This capability jump comes from larger component models and moreplex pipelines — new abilities like legible text and photorealistic faces stem from swapping tiny NLP text encoders for full LLMs and scaling denoiser parameter counts roughly fivefold. Bigger models need more resources, but growing GPU power hasn't been sufficient alone — inference engineers still can't rely purely on hardware gains for efficiency.

The newest research direction blends diffusion transformer architecture with LLM architecture: anything tokenizable can be modeled as an LLMd LLMs bring baked-in text understanding while already serving as pipeline components. LLMs address diffusion's structural limitations — diffusion models can only produce fixed-size output, while LLMs autoregressively produce variable-length output; diffusion needs up to 100 forward passes per image, while LLMs generate tokens a single forward pass each. Models like HunyuanImage-3.0 use this LLM-style approach.

Few-Step Image Generation Models

The most time-consuming part of image generation is the 30-50 denoising steps, so an alternative optimization strategy is simply reducing step count rather than speeding up each step. Few-step image generation models are trained to produce high-resolution images with eight or fewer denoising steps, running 80-90% faster out of the box — though with noticeably lower output quality (a practical speed/quality tradeoff warning). Two creation methods:

• Latent consistency: train a model to predict the target latent image vector directly, repeating the prediction two to four times to enhance quality.
• Distillation: use adversarial and/or progressive distillation to train a small model to emulate a larger one in fewer steps.

Distillation is more common today; when new image models (FLUX, Qwen Image) release, the open community produces distillations alongside quality/style LoRAs. Practical application: for latency-sensitive, quality-tolerant use cases like real-time generative filters, few-step models are worth considering.

Video Generation

Video generation models are architecturally similar to image models but bigger three to five times more parameters and encoding 10-100x more information in latent space. Early video generation used a naive frame-by-frame ("framewise") approach: generate a starting frame, then use it to generate the next, and so on. The critical flaw (a clear warning) is error accumulation — small early errors compound frame over frame and quality degrades rapidly.

Modern video models instead hold the entire video in latent space and modify it on each denoising step, with every frame tending to every other frame and updating on every forward pass. If image latent space represents two dimensions (X, Y), video latent space represents three: X, Y, and T (time). The main limitation is fixed video length, analogous to image models' fixed aspect ratios — modern video models create just a few seconds of footage. T binding constraint is compute: even on the latest GPUs, attention over massive latent space is extremely expensive, taking several seconds of inference per second of video. Video generation is so compute-intensive it typically runs at batch size one — a full 8-GPU node dedicated to a single request. Despite this expense per step, video models use roughly the sa total step count as image models (~50 steps).

Video generation is a newer modality than LLMs and image generation, and its current limitations mirror where LLMs stood roughly two years earlier:

| LLMs (Late 2023) | Video gen models (Late 2025) | |---|---| | High TTFT, Low TPS | Slow generation times | | Frequent hallucinations | Unrealistic physics | | Maxed out Ampere GPUs | Max out Blackwell GPUs | | Limited context windows | Short video outputs |

Most of these LLM limitations have since been largely eliminated; removing analogous limitations from cinematic video generation (and related areas like world models and 3D object generation — XYZ dimensions rather than XYT) remains a highly active research area. One promising direction returns to autoregressive generation ideas — blending LLM architecture into image/video models. Rather than pure framewise generation with its unusable errorumulation, techniques like Self Forcing combine a global quality view with an iterative generation approach. Adding autoregressive components can partially address the attention bottleneck, though attention remains the single most important and expensive inference component for video.

Calculating Inference Bottlenecks

In a perfectly optimized system, every resource stays fully utilized. GPUs have two main resources: compute (floating-point operations per second) and memory bandwidth (bytes moved per second). Ideally compute never idles waiting on memory, and memory bandwidth never goes unused waiting on compute. Real systems have bottlenecks — imbalances where one resource idles while another saturates. Identifying bottlenecks is the essential first step to improving performance: optimizing compute when a syem is memory-bound (or vice versa) accomplishes nothing — an important practical warning against misdirected optimization effort.

Typical inference bottleneck pattern:

• LLM prefill (KV cache construction) is compute und.
• LLM decode (token generation) is memory bound.
• Image and video generation are compute bound.

Optimization goals target making the relevant bottleneck less limiting system-wide. Example: batching multiple requests together makes LLM decode less memory bound, because a batch uses more compute for the same memory traffic.

Ops:Byte Ratio and Arithmetic Intensity

Every GPU has a compute speed (ops/second) and memory bandwidth (GB or TB/second); comparing these gives the ops:byte ratio. Example: an H100 GPU in FP16 delivers 989 teraFLOPS against 3.35 TB/s bandwidth, yielding an ops:byte ratio of about 295 — meaning perfectly balanced FP16 inference on an H100 needs 295 floating-point operations per byte of memory accessed.

To compare against this ratio, calculate the arithmetic (operational) intensity of the algorithm at hand — the ratio between work and memory traffic for a specific calculation, measured across a single function/algorithm's execution (versus ops:byte's per-second hardware measure). Arithmetic intensity is alized via a roofline model, charting performance against a diagonal bandwidth ceiling and a horizontal performance ceiling. Where arithmetic intensity exceeds the hardware's ops:byte ratio (hitting the horizontal ceiling), the algorithm is compute bound; where it's lower (hitting the diagonal ceiling), it's memory bound. Finding bottlenecks means examining arithmetic intensity of the most expensive calculations — for inference, that calculation is attention.

LLM Inference Bottlenecks

refill determines time to first token (TTFT) and is compute-bound; decode determines tokens per second (TPS) and is memory-bound. For each phase, comparing the most important operation's arithmetic intensity to hardware ops:byte ratio proves the bottleneck. In both phases attention is the most expensive operation, with exact arithmetic intensity depending on model architecture (dimensions, heads, etc.), input sequence length, and attention algorithm implementation.

The essential difference: prefill processes the entire input sequence in parallel (loading model weights once, then running large matrix multiplications against attention matrices — many calculations per memory read, hence high arithmetic intensity), while decode generates tokens one at a time (loading model weights per token for a comparatively cheap vector-matrix multiplication — floating-point operations relative to the memory cost of loading full model weights, hence low arithmetic intensity).

Worked example: consider a decode step for a model with a 128-dimensional attention head (d=128) over a 4096-token sequence (N=4096), using standard unoptimized attention. Matrix sizes: Q, K, V are Nxd (4096x128); S, P are NxN (4096x4096); O is Nxd (4096x128). Assuming FP16 (2 bytes per value), a 4096x4096 matrix is about 32 MiB — roughly the size of a high-resolution RAW DSLR photo, a helpful concrete anchor. Each line of the attention algorithm follows the pattern: load from memory, compute, store result back to memory. Summing reads and writes across all steps gives total memory movement; summing operations across sts gives total compute. Dividing total compute by total memory movement gives arithmetic intensity — in this example, 62, far below the H100's ops:byte ratio of 295, confirming decode's memory-bound nature (exact numbers vary by model, sequence length, and hardware, but the general principle holds). Calculating arithmetic intensity by hand like this is described as an academic exercise rather than a routine engineering task, but useful to work through once for intuition-building.

Imageeneration Inference Bottlenecks

Image and video models are relatively small (roughly a tenth the parameters of frontier LLMs) but their attention mechanism is equally computationally demanding, since they use iterative denoising rather than autoregressive token generation. Just as LLM prefill attention processes the whole input sequence at once, media generation attention must consider the entire image/video object as represented in latent space. Like LLM prefill, image and video generation inference is compute bound (specific optimization techniques covered later in sections 6.5 and 6.6).

Optimizing Attention

For LLMs, attention scales quadratically with input sequence length since each attention calculation depends on the K and V values of every previous token. In practice, decode-time attention runs in linear time thanks to the KV cache storing prior key/value computations. Even linear-scaling attention remains one of the most expensive parts of inference across models and architectures, making attention optimization an important, highly active research area. Attention optimization is delicate — because each token depends on every previous token, small errors accumulate quickly, so care is required when modifying it.

The basic attention algorithm is simple to implement, but that basic form is inefficient: intermediate matrices S and P are stored to memory at the end of one step and then immediately reloaded in the next — wasted memory traffic. Two optimization strategies exist:

• Implementation improvements: write higher-performance kernels using memory/compute efficiently — still bounded by quadratic time complexity, but lossless (no quality impact), making long-sequence inference feasible on current hardware.
• New algorithms: create attention variants with better-than-quadratic time/space complexity — these trade off some quality for efficiency, though training techniques can minimize the impact.

FlashAttention is the most famous implementation-level optimization: where basic attention needs only a handful of code lines, FlashAttention tens of thousands of lines of hand-fused, GPU-specific kernels (FlashAttention for H100 differs from FlashAttention for B200 — a non-obvious hardware-specificity point). It works by eliminating excess memory reads/writes and precisely fitting the algorithm's layout to a given GPU's capabilities, and is especially useful for cpute-bound operations like LLM prefill and video generation.

PagedAttention addresses a different problem: KV caches grow large, filling GPU memory and taking time to read. It partitions the KV cache into blocks ("pages") accessed via a lookup table, so the cache can be stored across fragmented GPU memory rather than requiring one contiguous block — a practical memory-management win.

FlashAttention and PagedAttention are valuable but don't change attention underlying quadratic algorithmic nature. New attention variants change the actual time/space complexity:

• Sliding window attention: computes attention only over a sliding window of w previous tokens, changing complexity from O(N²) to O(Nw), with w often in the 8K-32K range.
• Gated attention: training-introduced layer types that approximate attention for certain context chunks in linear time relative to chunk length.
• Linear attention: replaces the quadratic softmax equation with a linear-time approximation algorithm.
• Compressed attention: periodically compresses earlier context, with attention then considering both compressed context and uncompressed recent tokens.
• Multi-latent attention: approximates attention within low-dimensional latent space.

There's an intuitive raionale behind windowed/localized approaches: nearby tokens in a sequence affect each other more than distant ones (analogous to how a sentence relates more strongly to the sentence right before it than one from much earlier in a chapter). This intuition can be reinforced through training — applying techniques like sliding window attention during training produces models that maintain quality when the same technique is later used at inference time.

A further research avenue avoids attentionltogether via different architectures. Mamba is a selective state-space model replacing self-attention with a recurrent state update, achieving linear scaling with sequence length. Hybrid models mix Mamba-style state-space blocks with transformer blocks; while state-space model applications remain limited, hybrid architectures are gaining popularity, with open models like NVIDIA Nemotron 3 Nano adopting hybrid designs.

Chapter 3: Hardware

Inference engineering depends on accelerators capable of loading terabytes of data and performing trillions of operations per second. GPUs are the dominant accelerator type for inference, with NVIDIA as market leader. Three GPU categories exist: datacenter GPUs (e.g., NVIDIA B200, racked in refrigerator-sized chassis with standardized power, networking, cooling), workstation GPUs (e.g., RTX Pro 6000), and personal computing GPUs (e.g., GeForce RTX 5090). Datacenter GPU inference runs in three modes: cloud (rented from hyperscalers like AWS/GCP or neoclouds like Coreweave/Nebius — the most common approach), on-premise (purchased and installed in a self-controlled datacenter), and air-gapped (on-premise but requiring physical access to run inference, used by large enterprises/governments). Navigating the hardware landscape is complex due to cloud provider variation and NVIDIA's naming conventions.

GPU Architecture

GPUs are throughput machines optimized for simple, massively parallel workloads, unlike CPUs which excel at complex sequential execution. Since AI inference is fundamentally vector and matrix multiplication, GPUs are a naturalit. Despite the simplicity of the core parallel-computation principle, GPU hardware itself is extraordinarily complex; inference engineers work above this complexity but still need a solid mental model of what happens inside the chip.

Compute

GPU "cores" differ from CPU cores. GPUs contain Streaming Multiprocessors (SMs), each holding multiple cores of three types:

• CUDA Core: operates on individual scalars.
• Tensor Core: operates on vectors and matrices.
• Special Function Unit (SFU): accelerates specific math operations like sin, cos, log.

For measuring inference-relevant compute, Tensor Core throughput is what matters — SFUs matter for softmax, but Tensor Cores handle Matrix Multiply and Accumulate (MMA) instructions foundational to inference. "Accumulate" means adding the product of two matrices to a base matrix to produce the output (matrix A × matrix B + matrix C = matrix D).

Threads are conceptually similar between CPUs and GPUs, but GPUs have tens to hundreds of thousands of concurrent threads (vs. dos to hundreds on CPUs) that can switch tasks in a single clock cycle.

Compute is measured in FLOPS (floating point operations per second), with datacenter GPUs reaching teraFLOPS or petaFLOPS. Spec sheets list two measurements:

• Dense: raw FLOPS when every tensor element is used.
• Sparse: in tensors with 2:4 structured sparsity (50% zero values), Tensor Cores skip multiplying by zero, often roughly doubling FLOPS versus dense at the same precision — but not always.

Warning: inference is dense by default, so always compare FLOPS without sparsity when evaluating hardware. FLOPS generally double with each halving of numeric precision (e.g., 1 petaFLOP at 16-bit becomes 2 petaFLOPS at 8-bit) — relevant when comparing GPUs, since comparisons must be made at identical precisions. Compute is the bottleneck for LLM prefill and for image/video generation, so for these workloads prioritize accelerators with higher FLOPS.

Memory and Caches

GPUs use high-speed onboard memory called VRAM (the "V" harking back to GPUs' original o/graphics purpose), implemented today as HBM3, HBM3e, or HBM4 (high-bandwidth memory), with dozens to hundreds of gigabytes available. Two memory types exist on any chip:

• DRAM: general-purpose off-chip memory, measured in gigabytes (VRAM is a type of DRAM).
• SRAM: faster, more expensive, on-chip memory, measured in kilobytes or megabytes (used for caches).

GPUs have three cache levels:

• L0: instruction cache for a single Tensor Core.
• L1: shared memory per Streaming Multiprocessor (e.g., 256 KB per SM on an H100).
• L2: global cache shared across SMs (e.g., 50 MB total on an H100).

VRAM bandwidth is the peak transfer rate between GPU cores and VRAM, determining how fast data can be fed into the cache hierarchy.

Practical application: total VRAM caps the size of model that can be loaded. VRAM needs to hold model weights plus at least 50% headroom for KV cache (more for long context, high batch sizes, or video generation). Warning: insufficient VRAM for weights causes an OOM (out of memory) load failure; insufficient headroom causes slow inference or an OOM crash during runtime.

Memory bandwidth is the bottleneck for LLM decode at low-to-medium batch sizes. Practical tip: when selecting a GPU to maximize tokens per second, choose higher memory bandwidth (e.g., H200 over H100).

GPU Architecture Generations

Hardware iteration cycles are slow — years elapse between finalizing an architecture and shipping GPUs, meaning new architectures were designed before today's AI capabilities existed. This requires foresight into how use cases evolve over a GPU's market lifetime. Training was historically the primary GPU use case; now inference dominates, and newer architectures increasingly add inference-focused features.

GPU names (e.g., B200) combine a letter (architecture generation) and number (model within generation). Numbering is somewhat arbitrary across generations, but generally higher numbers mean larger, more powerful, costlier GPUs. Lettering, denoting architecture, is meaningful — NVIDIA releases a new architecture roughly every onetwo years, each improving base compute/memory speeds and adding efficiency features. Since 1998, NVIDIA has named architectures after prominent scientists.

Non-obvious point: inference engineers generally only need to work within the three-to-five most recent generations. Modern architectures' efficiency often makes them more cost-effective even for cost-sensitive workloads at scale, despite higher unit prices. Turing (T4) and Ampere (A10, A100) still appear in low-traffic/legacy systems, but most current deployments use Lovelace (L4, L40), Hopper (H100, H200), or Blackwell (B200, B300). Hopper and Blackwell offer low-precision Tensor Cores, high-bandwidth memory, and inference-focused features; Lovelace is used for low-cost small-model inference. Upcoming Rubin (2026) and Feynman (2028) promise further gains.

Hopper GPUs

| GPU | FP8 compute (dense) | Memory | Bandwidth | |---|---|---|---| | H100 | 1,979 teraFLOPS | 80 GB | 3.35 TB/s | | H200 | 1,979 teraFLOPS | 141 GB | 4.8 TB/s |

Named for Rear Admiral Grace Hopper, first released March 2022 with the H100. Hopper introduces FP8 support, an 8-bit floating-point format. FP8 Tensor Cores run twice as fast as FP16 and require half the memory bandwidth to move values — though this gain is not always a linear 2x perfornce improvement (detailed further in section 5.1). Hopper adds fourth-generation Tensor Cores in more/faster SMs, plus dynamic programming instructions, thread block clusters, and distributed shared memory, giving CUDA engineers more tools for high-performance kernels. One notable kernel enabled by these features is FlashAttention 3, which improves attention performance and memory efficiency by exploiting Hopper's new asynchronous data transfer/execution capabilities. Practical note: H100/H200 are among the most widely used inference accelerators because the architecture is new enough to be performant yet established enough for broad industry support and highly optimized kernels — right-sized for common workloads across modalities.

Ada Lovelace GPUs

| GPU | FP8 compute (dense) | Memory | Bandwth | |---|---|---|---| | L4 | 242 teraFLOPS | 24 GB | 300 GB/s | | L40 | 362 teraFLOPS | 48 GB | 864 GB/s |

Named for the first computer programmer, released just six months after Hopper. Lovelace acts more as a graphics-oriented counterpart than a true successor, though it also supports FP8. Major limitation/warning: Lovelace GPUs lack NVLink interconnect support, unlike Hopper/Blackwell 8-GPU nodes that rely on NVLink for efficient parallelism — Lovelace GPUs must run individually or via efficient methods like Pipeline Parallelism. Practical application: L4 GPUs are a cheap, convenient choice for small models in modalities like text embeddings and computer vision. Warning/non-obvious point: L40 GPUs are generally a poor inference choice — for the same memory footprint, fractional H100s via multi-instance GPU (M, section 3.3.2) deliver much higher compute and memory bandwidth.

Blackwell GPUs

| GPU | FP8 compute (dense) | Memory | Bandwidth | |---|---|---|---| | B200 | ~5 petaFLOPS | 192 GB | Up to 8 TB/s | | B300 | ~5 petaFLOPS | 288 GB | Up to 8 TB/s |

Named for mathematician David Blackwell, first released November 2024 with the B200, followed by the B300 (the B100 also exists but is uncommon for inference). Blackwell extends low-precision computing further with FP4 (4-bit floating point) plus microscaling formats (MXFP8, MXFP4, NVFP4) that better retain quality during inference (detailed in section 5.1). It builds on Hopper's asynchronous programming paradigm with additional features for loading/storing between tensor and global memory. The updated FlashAttention 4 kernel heavily relies on tiling loads, computations, and writes in asynchronous pipelines. B200/B300 are described as the new gold standard for inference, offering top performance for LLMs and demanding workloads like video generation; software support, optimized kernels, and general availability have matured in recent months, marking an important industry transition.

Rubin GPUs

Named for astronomer Vera Rubin, launching 2026 as NVIDIA's next-generation architecture. Warning/practical guidance: when evaluating new architectures, reserve judgment until real-world performance benchmarks are available — new architecture rollouts plus industry-wide software support typically take about a year to fully mature. Known details: Rubin uses HBM4 (upgrading from HBM3/HBM3e), which will benefit memory-bandwidth-bound tasks like LLM decode. Rubin also introduces CPX, a separate chip built for compute-bound tasks like LLM prefill, which will be part of NVIDIA's rack-scale systems for higvolume inference. After Rubin, Feynman is expected in 2028, likely with larger/more powerful chips and faster memory architecture (few concrete details known yet).

Grace and Vera CPUs

NVIDIA also makes ARM-based CPUs integrated with GPUs on superchips like the GH200 and GB200 ("G" stands for Grace, paired with Grace Hopper naming). What matters for inference is Grace's much higher-bandwidth CPU-GPU connection: NVLink Chip to Chip delivers up to 900 GB/s bi-directional bandwidth between CPU and GPU memory — several times faster than PCIe or standard CPU-GPU connections. Practical application: setups that offload information like LoRA fine-tune weights or KV caches from previous inference calls to the much larger CPU memory retrieve that data far faster with Grace CPUs. For the Run architecture, the Vera CPU (also named for Vera Rubin) replaces Grace.

Instances

The atomic unit of cloud GPU allocation is an instance — a virtual machine including GPUs (device), CPUs (host), host memory, storage, networking, and interconnect (physical GPU-to-GPU and node-to-node connections). Instances vary across cloud providers even though NVIDIA offers reference architectures, since each provider builds systems to its own preferences. Non-obvious point: even the GPU itself can differ by instance — for example, A100 GPUs come in PCIe and SXM form factors, and most A100 inference runs on SXM because it has 5% higher memory bandwidth than PCIe. Warning: when provisioning instances, understand exactly what's included, since any component — not just the GPU — can be a bottleneck or faint.

Multi-GPU Instances

Models too large for a single GPU, or workloads needing multiple GPUs for performance, commonly require two, four, eight, or more GPUs (e.g., for large models like DeepSeek or video generation). The standard GPU unit is a node containing eight GPUs (e.g., a B200 node = eight B200s). Nodes connect GPUs via:

• NVLink: one-to-one communication layer between GPUs, up to 1,800 GB/s on Blackwell and 900 GB/s on Hopper.
• NVSwitch: all-to-all communication layer built on NVLink for coordinating all GPUs in a node.

These high-bandwidth interconnects allow spreading inference across up to a full 8-GPU node. For workloads needing more than eight GPUs, node-to-node interconnect is required — the NVIDIA standard is InfiniBand (NVIDIA acquired Mellanox, InfiniBand's manufacturer, in 2019), mpeting with Ethernet. InfiniBand reaches up to 400 Gb/s per Network Interface Controller (NIC) — much slower than NVLink, but still the fastest node-to-node option (Ethernet maxes out at 100 Gb/s per NIC). Warning: noevery cloud provider uses InfiniBand — some offer proprietary interconnects or only provide it on some GPUs, so always verify what interconnect and bandwidth a provisioned instance actually delivers.

NVIDIA's high-end NVL72 GB200 system combines 72 Blackwell GPUs and 36 Grace CPUs in a full-rack system forassive throughput on the largest models with intense traffic. The next generation is the NVIDIA Vera Rubin NVL144 CPX, with updated Vera CPUs, Rubin GPUs, and the new Rubin CPX chip. Practical guidance: when working with multi-GPU/multi-node systems, keep relative interconnect bandwidths in mind — an interconnect an order of magnitude faster (like NVLink vs. InfiniBand) can handle far more data before becoming a bottleneck; parallelism and disaggregation techniques (sections 5.4 and 5.5) are designed to navigate this topology.

Multi-Instance GPUs

The opposite problem also occurs: the GPU is too big for the model. Newer high-performance architectures (Hopper, Blackwell) come with large compute/memory allocations that are hard to fly utilize for models with only a couple billion parameters or fewer, even at large batch sizes — wasting valuable GPU resources. Rather than running lightweight workloads on older, lower-performance GPUs, Multi-Instance GPU (MIG) — a hardware-level capability on larger GPUs including A100, H100, H200, and B200 — allows splitting a GPU into up to seven fractional instances, each receiving a slice of CPU, RAM, storage, and other resources needed to form a full instance.

Example: an H100nstance with three slices gets about 3/7 of available compute, up to half the total VRAM (40 GB), and roughly half of CPU cores, CPU memory, storage, and network bandwidth. Non-obvious detail: while software engineering typically works in multiples of two, MIG uses seven compute slices because GPUs generally don't have a clean multiple-of-two SM count (e.g., an SXM H100 has 132 SMs) — compute slices are built from SMs, so seven evenly-sized slices are created and any leftover SMs sit idle. Practical application: for small models like Orpheus TT(3B parameters), running two MIG instances can be a more efficient use of resources than allocating an entire GPU to a single instance.

Other Datacenter Accelerator Options

While NVIDIA's AI hardware leadership made it the world's most valuable company, it isn't the only hardware provider — competitors from established giants (Amazon, Google) to well-funded startups are investing billions in alternatives:

• AMD (public): MI350 GPU — a datacenter GPU with competitive specs on AMD's own software stack.
• AWS (public): Inferentia and Trainium — purpose-built chips for inference and training respectively.
• Cerebras (startup): WSE-3 — a wafer-scale chip with extremely high memory bandwidth to remove decode bottlenecks.
• Etched (startup): Sohu — an Application-Specific Integrated Circuit (ASIC) built for the transformer architecture.
• Furiosa (startup): RNGD — a power-efficient accelerator for tensor contraction operations.
• Google (public): TPU — an AI-specific ASIe and training.
• Groq (startup): LPU — a composable language processing unit relying on SRAM for high memory bandwidth.
• Qualcomm (public): Cloud AI 100 Ultra — a full-sized GPU composed of multiple power-efficient mobile GPUs.
• Sambanova (startup): RDU — a Reconfigurable Dataflow Unit with large memory allocation for trillion-parameter models.

Since GPUs are fairly general-purpose accelerators, each competitor bets on a specific edge:

• Memory bandwidth: startups like Cerebras and Groq chase high tokr-second via ultra-high-bandwidth memory acceleration of decode.
• Power efficiency: companies like Furiosa and Qualcomm target lower power consumption for cheaper operating costs.
• Platform integration: enterprises like Amazon and Google build deep integration with their own cloud platforms and proprietary closed models.

Shared challenges across all these alternatives:

• Software: without CUDA, providers must rebuild the entire inference stack from scratch for their hardware.
• Manufacturing: assembling what is described as the most complex object humankind has created.
• Distribution: after manufacturing, chips still need to be installed and brought online to reach the market.

Competition accelerates innovation, benefiting every inference engineer through more hardware options in the datacenter — and this competitive pressure extends to non-datacenter options like local inference.

Local Inference

Local inference — also called edge, client-side, or on-device inference — runs AI model inference directly on an end user's device rather than a centralized server. Four major advantages over server-side inference:

• Zero network latency: no communication overhead, saving tens to hundreds of milliseconds.
• Independence: no dependency on internet connectivity, and no impact from server traffic or downtime.
• Improved privacy: user data never leaves the device.
• Cost: no datacenter GPU expense — edge inference is free for the developer, enabling new business models.

Warning: despite sounding ideal, local inference has four practicalses limiting its applications:

• Hardware capabilities: even high-end prosumer desktops offer only a fraction of datacenter GPU speed/power.
• Thermal constraints: worse cooling than datacenters further limits speed and power.
• Fragmented support matrix: countless hardware/software combinations make standardization difficult.
• Battery life: inference is demanding and quickly drains laptop/smartphone batteries.

Practical guidance: when building for local inference, account for audience — an AI enthusiast may have top-tier hardware, but a median user likely has older, less powerful devices. Local inference is maturing from experimentation to production, backed by a strong hardware/software ecosystem and a community closely tied to open models.

Desktop Inference

The classic local device is aorkstation/gaming PC with one or two high-end consumer GPUs (NVIDIA or AMD), though researchers/enthusiasts using these setups represent a small share of the desktop inference market. Apple is increasingly the desktop inference leader: its M-series CPUs/GPUs share a single unified memory pool, giving GPU inference access to far more memory, albeit at slower speeds. Comparison of highest-end current options illustrates the memory-vs-speed tradeoff:

| Hardware | NVIDIA RTX 5090 | Apple M3 Ultra | |---|---|---| | Memory | 32 GB | 512 GB | | Bandwidth | 1,792 GB/s | 819 GB/s | | Cost (full computer) | $5,000 | $10,000 |

This tradeoff persists through midrange hardware at more reasonable prices. Low-end computers like Chromebooks cannot run meaningful local inference. The enthusiast open-source ecosystem focuses on running frontier open models on desktops/laptops; aggressively quantized 100B+ parameter models are now runnable on high-end personal hardware using tools like Ollama and llama.cpp. The rise of Mixture of Experts architectures is a tailwind for desktop inference, since these models have fewer active parameters — an individual request only touches a fraction of total weights. Image generation is also popular on personal computers, especially v ComfyUI, a tool for assembling multiple image model components into a single pipeline. Smaller language models and other modalities like speech are increasingly feasible on midrange computers, with browser inference libraries like WebLLM and other cross-platform standards rapidly bringing these capabilities from early adopters to the mainstream.

Mobile Inference

Mobile devices represent the majority of on-device inference workloads today. Both major mobile operating systems provide developer tooling for edge inference:

• Android: Google's AI Edge SDK and ML Kit GenAI APIs interface with Gemini Nano and OSS Gemma models.
• iOS: Apple's Foundation Models and Core ML frameworks provide APIs across modalities.

Warning: mobile devices have extremely limited hardware capabilities and battery capacity, making inference even more challenging — even high-end phones struggle beyond one or two billion parameter models. Still, some modalities suit edge inference well on phones: transcription and speech synthesis are laten-sensitive and some models run small and fast enough for real-time phone execution; other discrete tasks like translation can be handled by small fine-tuned edge models. Overall framing: the future of inference is not local-versus-cloud but both working together — small models and quick queries running on end-user devices, while more demanding workloads remain on datacenter GPUs in the cloud.

Chapter 4: Software

NVIDIA's market dominance in inference stems significantly from its mature software ecosystem, not just its hardware. Hardware iterates slowly (yearly or two-year cycles from companies like Apple and NVIDIA), while software iterates fast — running a newly released open model on day zero often requires nightly builds or prerelease versions of dependencies. This fast iteration and low barrier to entry dramatically expands the inference engineering landscape beyond hardware players.

Key software players include NVIDIA (CUDA up through Dynamo, often proprietary), Hugging Face (model registry plus transformers and diffu libraries), The Linux Foundation (hardware-agnostic PyTorch and vLLM), and LMSYS Org (SGLang and evaluation tools). Thousands of other companies, universities, and research institutions contribute essential open-source work. This chapter presents foundational, long-term-relevant technologies organized by increasing abstraction: CUDA (direct GPU communication), deep learning frameworks (Python abstractions over CUDA), inference engines (configurable PyTorch-backed inference), and NVIDIA Dynamo (orchestration atop inference engines). Most inference engineering work happens at the higher abstraction levels, but a strong mental model of adjacent layers is essential regardless of where you work in the stack.

CUDA

CUDA is NVIDIA's proprietary computing platform and programming model for running parallel tasks on GPUs. Its component parts:

• CUDA kernel: a user-defined function executing parallelized code on the GPU.
• CUDA graph: a directed acyclic graph (DAG) of kernels and other GPU operations, used to optimize repeated workflows.
• CUDA driver: a low-level interface managing memory and execution between application and GPU hardware.
• CUDA runtime: a developer-facing API for launching kernels and managing memory.

CUDA stands for Compute Unified Device Architecture, though the acronym is rarely expanded today. It is the foundation of the entire generative AI ecosystem on NVIDIA GPUs. Notably, CUDA is not itself a programming language — CUDA programs are written in a language like C++ and compiled into separate CPU and GPU code by a compiler such as nvcc. A "CUDA kernel" can simply be understood as "a piece of code written for NVIDIA GPUs." A simple example kernel doubles every element of an array — on a CPU this runs sequentially, but on a GPU thousands of elements can be processed simultaneously.

A non-obvious point: writing CUDA kernels shifts thinking from algorithms to implementations. The traditional attention algorithm can be expressed in a few dozen lines of code, but FlashAttention — mathematically the same operation — takes teousands of lines to implement in a memory-efficient way for a specific GPU.

CUDA Kernels for Inference

Writing CUDA kernels doesn't mean building from scratch — prior art predates CUDA by decades. BLAS (Basic Linear Algebra Subprograms), first implemented for Fortran in the 1970s, specifies common linear algebra operations like dot products and matrix multiplication. cuBLAS is CUDA's implementation of BLAS, providing pre-built kernels for essential linear algebra operations, while cuD provides primitives for neural networks specifically.

GEMM (General Matrix-Matrix Multiplication) is the most frequently used BLAS operation in inference, since every linear layer relies on matrix multiplication; cuBLAS provides a strong starting point. But engineers aren't limited to cuBLAS — fine-grained control may be neede such as writing different GEMM kernels for different matrix shapes or specific GPU architectures. CUTLASS is a template library providing building blocks for high-performance kernels (FlashAttention 3 uses it), and CuTe is another template library introducing abstractions for tiled tensor operations on recent architectures — both let engineers write kernels at a higher abstraction level while retaining performance. FlashInfer is another library offering high-performance LLM inference kernels, including optimized attention kernels and fused sampling functions.

CUDA Kernel Selection

Most inference engineers will never write their own kernels, but kernel selection — choosing the best kernel from available options — is an important part of optimization. Kernel implementations are highly specialized and hard-coded to specific hardware details like memory bandwidth or Tensor Core layout. A key warning: a kernel written for an H100 will likely fail to exploit a B200's architecture and extra memory, while a kernel written for a B200 could be backwards incompatible with the previous-generation Hopper architecture — porting handwritten kernels across GPU generations takes substantial engineering work.

Most kernel selappens automatically: deep learning frameworks and inference engines have pre-configured kernels for various architectures, and PyTorch and TensorRT-LLM include automatic kernel selection during compilation. However, manual kernel selection for essential algorithms can speed up inference significantly. Example: most production GEMM kernels come from cuBLAS, but when DeepSeek released an updated DeepSeek-V3, they also released DeepGEMM, providing more efficient FP8 GEMM kernels specifically for the Hopper architecture. Manual kernel selection lets you plug in a kernel like DeepGEMM to speed up a specific step (e.g., multiplying matrices of precise dimensions) — but compatibility matters: upgrading to a B200 would require swapping the kernel back, waiting for Blackwell support (now available as of publication), or porting it yourself.

Reducing Memy Accesses with Kernel Fusion

Running two kernels back-to-back on the same data wastes memory reads and writes. Example: running multiply_by_2 then multiply_by_3 on a vector requires reading input from memory, computing, writing output, then reading that same output back in for the second operation, computing, and writing again — six total memory-touching steps,ith an unnecessary round trip between steps three and four. During decode — the bandwidth-bound phase of LLM inference — an inference engine cannot afford these unnecessary memory operations.

Kernel fusion re-implements two or more kernels into a single kernel handling both operations (e.g., fusing multiply_by_2 and multiply_by_3 into multiply_by_6), reducing the sequence to a single read, compute, and write. In practice, fusion is far more complex since functions are more intricate and data overlap isn't clean, but common inference fusion patterns exist, like combining matrix multiplication, bias adding, and activation into one kernel.

Kernel fusion can be automatic or manual. Compilers can identify straightforward fusion opportunities and generate fused kernels automatically, but more sophisticated algorithms like FlashAttention require handwritten fused kernels used as plugins during infee.

Deep Learning Frameworks and Libraries

These frameworks and libraries bridge direct CUDA work and off-the-shelf inference engines like vLLM, and are used in both training and inference. PyTorch has emerged as the clear leader. Two other frameworks are mentioned briefly: TensorFlow, Google's end-to-end ML platform prominent in the 2010s but now fallen out of favor, and JAX, a Google-associated research project with a simpler interface but — per its own documentation — sharp edges toect.

PyTorch

PyTorch is a Python package for describing tensor operations, originally created at Meta and now part of the Linux Foundation; it is the industry standard for both training and inference of generative AI models. A practical/personal note from the author: writing low-level C++ is difficult, but PyTorch allows writing highly performant inference code in Python for CPUs and GPUs while still allowing dips into CUDA via specific kernel plugins when needed.

PyTorch can train any kind of neural network, and its autograd module automatically computes gradients for any differentiable function — defining a computation graph yields a gradient to train against, which is what makes PyTorch so powerful for training. But PyTorch is also powerful for inference, balancing builtn functions and automatic performance optimization with manual control where needed.

Compilation via torch.compile is the step that transforms a model from training to inference readiness — it targets a specific GPU and performs automatic kernel selection and fusion. A significant limitation/warning: torch.compile cannot fuse plugin kernels like DeepGEMM, FlashAttention, or other custom kernels, limiting its usefulness for LLM inference (where most kernels are custom). However, it remains useful for optimizing less common model architectures and compiling long sequences of lightweight kernels. When optimizing a model with a custom or rare architecture, functions may need to be rewritten to be more abstract — especially regarding Python-specific language features — for compilation to succeed.

del File Formats

Safetensors, created by Hugging Face, is the dominant format for serializing model weights, replacing generic formats like .bin that were designed for arbitrary data. Its "safety" comes from holding only tensor data, not executable code — unlike general formats that can execute arbitrary Python code during deserialization. Since generative AI models have hundreds of gigabytes of weights split across dozens of safetensors files, the format uses memory mapping so files can loaded without allocating full memory upfront, making loading faster and safer.

ONNX (Open Neural Network Exchange) is another leading format that stores weights together with an execution graph, unlike safetensors which separates weights from architecture. ONNX bundles them together and is highly portable, with deep PyTorch integration and multi-hardware support — a good alternative when you need to store the full model graph, not just weights.

ONNX Runtime and TensorRT

Both are high-performance inference runtimes. PyTorch models exrt to ONNX format, which ONNX Runtime executes directly or which TensorRT compiles into a highly optimized engine. Comparison:

• ONNX Runtime: open source, Linux Foundation–associated; first-class PyTorch exporter; supports many GPU types.
• TensorRT: mix of proprietary and open components built by NVIDIA; integrated via Torch-TensorRT; NVIDIA GPUs only.

The export process resembles Torch compilation, but neither standard supports every PyTorch data structure, type, and operation — export can surface these issues, though this gets tricky with complex models. Example warning: DeepSeek V3's Multi-Latent Attention (MLA) as implemented in PyTorch is difficult to export, though the transformers architecture overall is simple enough that hand-fusing kernels remains feasible.

A notable industry trend: it's increasingly common to skip the intermediate representation step entirely and go directly from PyTorch to an inference engine like vLLM or TensorRT-LLM for supported models, exporting weights only as safetensors. ONNX Runtime andsorRT remain widely used, especially TensorRT for image/video model runtimes, but the industry is bifurcating between handwritten PyTorch control versus prebuilt inference engine convenience.

Transformers and Diffusers

Hugging Face's transformers and diffusers libraries are built on PyTorch but are not designed for large-scale production inference — they offer reference implementations for learning from and adapting. Despite being tinkering toolboxes, they include essential informatio(like the config.json file shipped with models) and useful utilities, such as those for downloading model weights, which are widely used regardless of production setup. Sample code in model cards on Hugging Face is great for understanding a model's exact input/output spec or for local inference and notebooks, but for production you should either write and compile PyTorch code directly or use a production-ready inference engine.

Inference Engines

Three competitive inference engines exist: vLLM, SGLang, and TensorRT-LLM, all offering good out-of-the-box performance for LLMs and similar-architecture modalities (covered in chapter 6). As of late 2025, vLLM and SGLang began supporting image and video generation via vLLM Omni and SGLang Diffusion respectively; TensorRT-LLM does not support image or video generation (though TensorRT or PyTorch directly can be used for those, per section 6.5).

Inference engines are powerful because they're configurable — engineers work with pre-optimized components at a higher abstraction level, spending time testing technique combinations rather than repeating routine implementations. At a high level, vLLM and SGLang are more general, easier to adopt, with day-zero support for more models, while TensorRT-LLM has a steeper learning curve but usually achieves the best performance.

| Engine | Performance | Ease of use | Model support | Hardware | License | |---|---|---|---|---|---| | vLLM | Good | Easy | Most | GPU, TPU | Apache 2.0 | | SGLang | Good | Easy | Most | NVIDIA, AMD | Apache 2.0 | | TensorRT-LLM | Best | Hard | Some | NVIDIA only | Apae 2.0 |

All three run continuous batching out of the box and support the main performance optimization techniques: post-training quantization, speculative decoding, prefix caching, parallelism, and disaggregation. At Baseten, all three are used, though TensorRT-LLM the most — inference engineers should be familiar with all three and choose on a deployment-by-deployment basis.

vLLM

vLLM has the largest market share among inference engines — GitHub stars are a rough popularity measuand at time of publication vLLM had twice as many stars as SGLang and TensorRT-LLM combined. First released summer 2023, it's the oldest of the three by a few months, originally created at UC Berkeley and now hosted by The PyTorch Project within The Linux Foundation.

vLLM's best selling point is broad support: the most hardware options (NVIDIA, AMD, Intel GPUs, plus Google TPUs) and the most models/architectures — nearly every open LLM integrates with vLLM from day zero. vLLM Omni extends it for multimodal inference (image, audio, video inps and outputs).

A core inference engineering principle noted here: the more constraints you introduce, the better performance you can achieve. vLLM's broad platform achieves impressive results when properly configured, but per the author's experience it falls short of the highest-end performance achievable with narrower frameworks like TensorRT-LLM. Developer experience centers on the vllm serve command with configuration passed as flags. vLLM is pip-installable and provides official Docker images with pre-bundled dependencies and hardware architecture support.

Use vLLM when:

• You want to quickly stand up a model server with solid out-of-the-box performance for almost any open model.
• You want to run an "Omni" model with multiple input/output modalities.
• You're using a smaller GPU or older architecture where TensorRT-LLM offers few benefits.

SGLang

SGLang is the other major community-driven fast inference framework, first released December 2023. It has risen to prominence alongside Chinese open models like DeepSeek and Qwen, and is the engine of choice for inference at xAI.

Its unique angle is a developer experience pairing a fast backend runtime with a flexible frontend language — meaning individual engine components can be chosen for deep customization without rewriting everything else from scratch. SGLang supports both NVIDIA and AMD GPUs with strong day-zero model support, and works closely with labs like DeepSeek, Qwen, Kimi, and Z AI to release optimized implementations of new architectural features li Multi-Latent Attention.

SGLang has invested heavily in supporting large-scale MoE LLM deployments, specifically multi-node deployments on systems like GB200 NVL72 for high throughput — these systems offer extremely cost-efficient inference for large, high-traffic models. Developer experience centers on the sglang.launch_serr command with flag-based configuration.

SGLang Diffusion supports image and video generation inference via a pipeline abstraction orchestrating multiple stages, closely mapping to the architecture of image/video generation models. For performance, it adds diffusion-specific parallelism methods and reuses the scheduler and optimized kernels from the main SGLang package.

Use SGLang when:

• You want excellent out-of-the-box throughput with decent latency on large MoE models like DeepSeek and Kimi.
• You want an inference engine experience for image and video generation models.
• You want control, customization, and community participation.

TensorRT-LLM

TensorRT-LLM is NVIDIA's open-source inference engine, offering the highest performance and most flexibility to expert users among the three options. Important naming note: there are two major versions, and only the older one is actually related to TensorRT — TensorRT-LLM V0 (0.X.Y) is a plugin for NVIDIA TensorRT, while TensorRT-LLM V1 (1.X.Y, released summer 2025) is a standalone package based directly on PyTorch with no TensorRT dependency, bypassing the intermediate representation. Deployments of the older version remain common, so always check which version is in use.

TensorRT-LLM achieves theest performance largely because it has access to kernels written by NVIDIA engineers, including some closed-source kernels — these handwritten, manually fused kernels offer excellent support for the latest architectures (Hopper, Blackwell) and NVIDIA-specific number formats like NVFP4. It offers a robust in-flight batching implementation (token-level continuous batching) for throughput, and supports essentially every model performance optimization setting: quantization, speculation algorithm prefix caching, chunked prefill, flexible parallelism, and disaggregation.

With V1, the developer experience resembles vLLM and SGLang, but in addition to flag arguments on the trtllm-serve command, it expects a config.yaml file for deeper customization. The best installation method is via one of NVIDIA's official Docker containers.

Use TensorRT-LLM when:

• You're running a well-supported model architecture on a Hopper or later GPU.
• You're willing to do extra engineering work for the best possible performance.
• Optionally, you plan to use NVIDIA Dynamo for serving and want the most deeply integrated engine.

NVIDIA Dynamo

NVIDIA Dynamo is a distributed system for model serving, first announced at NVIDIA GTC in March 2025. It works with every inference engine — vLLM, SGLang, and TensorRT-LLM — as backends, providing an orchestration layer for large-scale deployments on top of them.

Dynamo provides support for essential model performance techniques (detailed further in chapter 5):

• KV cache re-use: retaining KV information between requests and routing requests based on prefix match.
• Disaggregation: separating prefill and decode onto individually optimized engines with independent scaling.
• Multi-node parallelism: optionally using two or more GPU nodes in a single model replica, usually with Expert Parallelism.

As with inference engines, there's substantial configuration work required to get maximum performance from Dynamo. Its thoughtful abstractions for distributed KV routing, disaggregation, and multi-node model parallelism enable high-performancntime information aggregation and real-time configuration adjustments as traffic fluctuates — for example, automatically scaling prefill and decode workers up/down using an SLA-based planner operating on user-defined TTFT and TPS constraints.

General principle: the more scale you have, the more optimization tools and techniques are available to you. Dynamo is built for scale — big models, big traffic — excelling at serving foundation models like the trillion-parameter Kimi family to lambers of concurrent users; for smaller models it can still offer moderate performance improvements at large scale. It's a great choice if building an inference API for a built-from-scratch foundation model or serving an open model in a high-usage product.

Warning: many deployments don't need Dynamo's additional complexity — unss operating at volume where disaggregation and KV-aware routing matter, Dynamo is unnecessary overhead, and inference engines should be used directly instead. Dynamo is the newest project covered in the chapter, with features still being built out; it's open source (Apache 2.0) with an active community, public CI, and NVIDIA engineer support.

Performance Benchmarking and Load Testing

Benchmarking is essential to performance optimization — without precise, accurate benchmarks there's no way to know if optimizations aractually working. A high-quality benchmark simulates real life as closely as possible; the best benchmark is shadowing real-world production traffic onto the test system, copying incoming requests to benchmark without affecting the original request.

If shadowing isn't possible, simulated traffic must match expected production workload across multiple dimensions:

• Sequence lengths: time to first token and memory usage depend on input sequence length (ISL) and output sequence length (OSL) — number of tokens in prompt and response.
• Volume and pattern of traffic: batching and server load depend on concurrent request count; jitter traffic to mimic real usage.
• Request contents: actual prompt content affects performance facts like cache hit rate and draft token acceptance.
• Input parameters: settings like temperature and reasoning effort should be set to anticipated production values.

Key warning: optimization is about tradeoffs and constraints — maximizing benchmark performance against unrealistic/bad inputs means production performance won't match expectations.

Performance Benchmarking Tooling

Since benchmarking setups should closely reflect production traffic, everyone's setup looks somewhat difrent, but common tools include:

• SGLang Genai-bench: a CLI and dashboard by the SGLang team for benchmarking models deployed with any inference framework.
• NVIDIA GenAI-Perf: a client-side tool for measuring latency and throughput on varied traffic.
• Locust: an open-source, non-GenAI-specific load-testing tool that can simulate up to millions of simultaneous users.

Open-source eval datasets are also useful for benchmarking — from general evals like MMLU and gsm8k to domain-specific ones like SWE-bench. Though benchmarking measur performance, not output quality, these datasets serve two purposes: providing varied and realistic inputs, and spot-checking that performance optimizations haven't degraded model output quality. Non-obvious tip: choose an eval dataset matching your production use case, like HumanEval when reducing latency for a code completion system.

Performance Benchmarking Tips

Great benchmarks are realistic and consistent — send enough traffic for a good performance read without being swayed by outliers; run benchmarks multiple times and average results when in doubt. Before optimization work, establish a solid baseline benchmark. Keep a consistent benchmarking configuration as you test optimizations, testing each optimization individually as well as collectively, since optimizations can sometimes work against each other (e.g., speculative decoding with large batch sizes). The principle of changing one variable at a time applies to benchmarking configuration itself — when testing various traffic patterns or sequence shapes, change only one vale at a time to get clear results.

Profiling Performance

Profiling goes one level deeper than benchmarking. A benchmark gives a single figure (e.g., P90 TTFT is 350ms), while a profiling tool shows where each of those milliseconds was spent in the inference process — benchmarking tells you how a system performs, profiling tells you why.

Most inference engineers won't need profiling in daily work — when using an already high-performance tool like TensorRT-LLM, the workflow is a cycleconfiguration and benchmarking, making profiling extraneous. However, profiling should be part of your toolkit if you're contributing to a framework like vLLM or SGLang, writing your own inference service in PyTorch, or operating at the cutting edge of a new modality like video generation.

Most popular profiling tools:

• PyTorch Profiler: easy-to-use library capturing step-by-step performance metrics (CPU time, GPU time, memory usage) during inference.
• NVIDIA Nsight Systems (NSys): featureful but complex tool for GPU/CPU sampling and tracing, providing system-wide analysis across multiple GPUs and their interconnects.
• NVIDIA Nsight Compute (NCU): profiling utility and CLI for in-depth analysis of individual CUDA kernels on compute and memory usage.

Frameworks like TensorFlow and TensorRT also ship with their own built-in profilers. Profilers matter because they give granular compute/memory usage information, guiding optimization work toward the most expensive pipeline steps. Practical example: using PyTorch Profiler, you might find activation functions taking unusually long due to excess memory reads, leading you to write a fused kernel running activations alongside attention to prevent those excess reads — then insert that kernel into your PyTorch code and re-run system-level benchmarks to check whether latency targets are met. Together, profiling and benchmarking provide the information needed to improve system performance and the confidence to deploy optimizations to production.

Chapter 5: Techniques

Inference engineering distinguishes itself from mostndustries in that new academic techniques move from research papers into production within months or weeks rather than years. This chapter frames a core principle: introducing more constraints into an inference system tends to improve performance. It also introduces a companion principle specific to model performance techniques: the more traffic a system serves, the more optimizations become viable, since higher model parallelism, KV-aware routing, and dynamic disaggregation only make economic sense at large GPU scale (often multi-node). Real-world traffic resists fixed constraints, but with sufficient volume, systems can be tuned continuously — through iterative deployments or dynamic runtime adjustments — rather than as a one-time task. Finding effective combinations of techniques requires patient experimentat one anecdote describes a Baseten engineer testing 77 different configurations by hand during an internal hackathon before finding a non-obvious setup that doubled TPS for a customer's model. Techniques can be symbiotic or incompatible with each other — for example, quantizing the KV cache eases a disaggregation bottleneck, while increasing batch size reduces the compute headroom available for speculation. The goal is always a balanced set of optimizations at delivers more than the sum of its parts. Five categories of applied research are covered: quantization, speculation, caching, parallelism, and disaggregation, with attention paid throughout to when each technique should be used and what tradeoffs it introduces.

Quantization

Quantization improves both TTFT and TPS, increases throughput, and creates headroom for other optimizations (disaggregation, speculation, prefix caching) to work even better — but poorly executed quantization can materially degrade output quality. Models are natively trained in BF16 or FP16 (with 8-bit and 4-bit native precisions becoming more common in training). Post-training quantization converts weights and other values to lower precision, which helps both inference phases: compute-bound prefill runs faster on lower-pcision Tensor Cores with more FLOPS, and memory-bound decode loads half as much data per value, effectively doubling memory bandwidth. Quantized data introduces its own overhead, so performance gains are not linear — dropping one precision level typically yields 30–50% better performance for LLMs, not a full 2x.

The risk is that quantization introduces precision errors that compound. A numeric illustration: squaring and cubing different precisions of Pi (3.14159 vs. 3.14 vs. 3) shows howll precision losses expand as calculations stack — most quantization engineering effort goes into preventing and minimizing this compounding error.

Number Formats

Key number formats, listed with approximate first supporting GPU architecture: FP64 (Fermi, 2010), FP32 (Kepler, 2012), FP16 (Pascal, 2016), BF16 (Ampere, 2020)FP8 (Hopper, 2022), MXFP8 (Blackwell, 2024), INT8 (Pascal, 2016), FP6 (Blackwell, 2024, experimental), FP4 (Blackwell, 2024), MXFP4 (Blackwell, 2024), NVFP4 (Blackwell, 2024, proprietary), INT4 (Turing, 2018). FP64 is reserved for scientific computing, not AI; FP32 is occasionally used in training but almost never inference; FP6 remains experimental (though AMD is adopting it quickly). In practice, 16-, 8-, and 4-bit formats are the primary inference precisions.

Number formats are characterized by precision (bit count), type (integer vs. floating point), and scale factor (multiplier mapping low-precision values back to higher precision). These combine into two critical qualities: dynamic range (spread between smallest and largest representable value) and granularity (how many values share a single scale factor). Dynamic range matters enormously — 16 bits can represent 65,536 distinct values versus only 256 for 8 bits. Floating-point formats outperform integer formats for inference because of their three-part structure (sign bit, exponent bits, mantissa bits) — the exponent gives floating point much higher dynamic range, which matters because outlier values are significant in inference and are better preserved after quantization in floating-point for. An FP8 number in "E4M3" format has a 4-bit exponent, 3-bit mantissa, and 1 sign bit.

Granularity varies by quantization level applied: tensor level (one scale factor for an entire QKV tensor), channel level (a distinct scale factor per feature vector), and block level (dividing each feature vector into blocks of N values, each with its own scale factor). Finer granularity better preserves outliers and quality but adds overhead for storing/applying more scale factors. MXFP8 and MXFP4 ("microscaling" formats introduced with Blackwell) compute a blockwise scale factor every 32 parameters, mitigating their lower native dynamic range. NVFP4 pushes granularity further with a block size of 16 plus a secondary 32-bit global scale factor to combat 4-bit quality loss. The tradeoff of microscaling is that small-block scale factors themselves consume memory and require extra compute to apply — though Blackwell GPUs offset this by applying scale factors directly in Tensor Cores.

Quantization is also essential for local/edge inference of large models — GGUF is the dominant binary format for distributing heavily quantized models on Hugging Face, letting individual researchers squeeze huge models like DeepSeek onto consumer hardware such as Apple computers. These local strategies o use dynamic quantization, leaving certain layers at original precision while quantizing others to as little as one bit, and averaging out to fractional bit counts — e.g., Unsloth's popular 1.58-bit quantization. Despite being impressive engineering, dynamic/integer quantization schemes are not recommended for quality-sensitive production systems because integer formats lack sufficient dynamic range; production inference engineers should stick to floating-point formats. FP8/MXFP8 are generally the sweet spot for performance without quality loss; FP4 (especially NVFP4, with its added granularity) is promising but FP8/MXFP8 remain most flexible, particularly for KV cache quantization.

Quantization Approaches

Larger models tend to be less sensitive to quantization since any individual parameter mters less, but careful quantization remains essential even at scale. Quantization happens either during training (quantization-aware training, where weights and scale factors are learned together for accuracy at a target precision) or after training (post-training quantization, converting finished weights via calibration). Some labs ship quantization-aware-trained models (e.g., GPT-OSS in MXFP4, Kimi K2 Thinking in INT4), but engineers working with open weights are generally limited to post-training quantization. NVIDIA TensorRT Model Optimizer (ModelOpt) is a leading open-source post-training quantization tool that also supports pruning, distillation, and sparsity, with outputs compatible across vLLM, SGLang, and TensorRT-LLM.

Two decisions follow choice of precision: (1) which model components to quantize (weights, activations, KV cache, attention) and (2) which number format best matches needed dynamic range and granularity. Model components have differing sensitivity to quantization, from least to most sensitive:

1. Weights (specifically linear layers) — least sensitive.
2. Activations — the intermediate outputs of activation functions are only somewhat sensitive (note: the activation functions themselves are rarely quantized since they represent a tiny fractio total weights).
3. KV cache — moderately sensitive.
4. Attention — highly sensitive, especially operations like softmax.

Even within low-sensitivity categories like linear layers/activations, early and late network layers (input/output layers) are often kept at original precision because they are more sensitive than middle layers. KV cache quantization has a special multiplier effect: beyond direct performance gains, it boosts prefix caching and disaggregation by letting more cache fit in memory and be read faster. However, since each token's KV cache entry is reused by every subsequent token, precision errors compound across the sequence — this compounding effect is exactly why attention is riskiest to quantize: attention is highly sensitive to dynamic range, and because each attention step s on all prior attention results, errors accumulate rapidly over long sequences. Consequently, nearly all but the most aggressive quantization schemes keep operations like softmax in original precision.

A moderate, practical approach: use a high-dynamic-range format like FP8 (ideally a microscaling variant like MXFP8) to quantize select linear layers, activations, and often KV cache, while leaving attention-layer components largely unquantized even under aggressive schemes.

Measuring Quality Impact

Production-grade quantization targets zero perceptible quality loss, verified through thorough testing against the unquantized baseline. Three measurement methods:

1. Perplexity — feed the model expected output sequences and measure how likely it was to predict them (rather than generating output). Higher perplexi means the model is more "surprised," which is undesirable. Look for minimal perplexity increase post-quantization.
2. Intelligence benchmarks — standard benchmarks like MMLU or SWE-bench compared before/after.
3. Cust evals — product-specific evaluation suites compared to original weights.

In every method, the target difference should be statistically indistguishable from the normal run-to-run noise inherent to non-deterministic LLMs. Perplexity is the simplest check; benchmarks/custom evals (ideally domain-matched to real usage) give a more comprehensive quality picture — running all three types together against the same original baseline gives the fullest picture. Quantization should be treated as a tunable spectrum, not a binary choice: quantizing to FP8 instead of FP4, or quantizing fewer components (e.g., weights-only), offers a lower-risk way to gain some performance. For quality-sensitive domains unwilling to risk any degradation, every other technique covered in the chapter (speculation, caching, parallelism, disaggregation) is lossless with respect to quality.

Speculative Decoding

Decode is autoregressive (one token generated at a time) and bottlenecked by memory bandwidth, leaving compute mostly idle at low-to-moderate batch sizes aseights stream from memory. Speculative decoding exploits that idle compute to attempt generating multiple tokens per forward pass through the target model, improving TPS/ITL (though not TTFT). The shared mechanism across algorithms:

1. A speculator proposes one or more draft tokens.
2. The target model validates whether these draft tokens match what it would have generated itself.
3. The target model accepts valid draft tokens and generates one additional token of its own, completing the pass.

This yields N+1 tokens per forward pass, where N is the number of accepted draft tokens. Generating drafts isn't free (costs compute and memory), but validating a draft token is much cheaper for the target model than generating an original token from scratch — analogous to how solving a sudoku is hard but checking a finishesudoku is easy.

Speculative decoding's uplift depends on three factors: (1) draft token cost (time to generate a draft), (2) draft sequence length (drafts generated per pass), and (3) token acceptance rate (percentage of drafts accepted). Acceptance rate is highest early in a draft sequence and degrades deeper into it — once a single draft token is rejected, every subsequent token in that draft sequence is also rejected — so short, high-acceptance sequencee generally preferable, since generation/validation still carries meaningful (if smaller) overhead. Acceptance rate is sensitive to many factors: higher temperature makes token distributions harder to predict and reduces speculative effectiveness; subject-matter mismatch matters too — if a draft model/head is stronger in math than history, acceptance will vary by topic. Speculative decoding is most useful at low batch sizes where spare compute exists; at higher batch sizes it should be dynamically disabled since compute becomes too saturated to afford verification overhead.

Draft-Target Speculative Decoding

The original method, using two models: a draft model (smaller, generates speculative tokens) and a target model (the original model, now also verifying drafts). Choosing the draft model is e key decision — a good draft model balances high acceptance rate against minimal resource cost. Draft models are usually smaller members of the same model family as the target (shared tokenizer/behavior), with a rule of thumb that the draft model should be at least 10x smaller by parameter count. Fine-tuning or distillation can raise acceptance rate by teaching the small model to better mimic the target. This method is a good quick-start choice requiring no training, but generally offers wse performance than newer algorithms — it introduces the most overhead of any speculation method because the inference engine must additionally store the draft model's weights, activations, and KV cache, and dedicate compute to draft-model prefill, all while carefully coordinating the two models so they don't compete for resoues (some engines, like TensorRT-LLM, handle this orchestration automatically).

Medusa

One of the first alternatives to draft-target speculation. Rather than running a separate draft model, Medusa fine-tunes the target model itself by grafting on two to four additional decoder heads (beyond the ordinary single head) that generate sequential draft tokens. Draft tokens are still validated on the following forward pass, as with draft-target speculation. Medusa remains limited in draft token count and acceptance rate and is not widely used in production today, but it inspired more effective successor techniques like EAGLE.

EAGLE

Off-the-shelf pretrained models (like Qwen 0.5B) make poor draft models because they're designed as standalone cheap-hardware LLMs, not as speculators tuned for high-end GPUs like the B200 — they run inefficiently and have relatively low acceptance rates. EAGLE instead is a purpose-built draft model trained from scratch to generate sequences of up to eight draft tokens (2x Medusa's capacity) with a very high acceptance rate. Its key insight: LLMs accumulate rich context in hidden states between layers during inference, information ordinary draft models never see. EAGLE is trained to take hidden states as input (specifical three states — from an early, middle, and late layer) and produce speculative tokens as output. EAGLE models are typically under one billion parameters and scale well with more training data. In practice, EAGLE implementations with engines like TensorRT-LLM tend to be straightforward, using post-training EAGLE creation with single-sequence speculation. EAGLE can be attached to the same PyTorch module as the target model, so a single forward pass runs both target-model inference and EAGLE eculation together — this unified pipeline eliminates the multiple CPU round-trips that draft-target decoding needed to coordinate separate models. EAGLE is the general-purpose go-to speculation algorithm for teams with the resources to train EAGLE heads, and is well-supported across inference engines. Like other speculation thniques, using EAGLE for improved latency requires reduced batch sizes, which lowers throughput and raises cost.

N-gram Speculation and Lookahead Decoding

N-gram speculation works completely differently — there'so draft model at all. Instead, while building the KV cache during prefill, the inference engine constructs an n-gram dictionary mapping a starting token to an observed sequence of N subsequent tokens. During decode, each generated token is checked against this dictionary, and any matching suffix is proposed as draft tokens, which the target model then verifies as usual on the next pass. The advantage over EAGLE is sequence length — EAGLE tops out around eight draft tokens with decent acceptance, while n-gram sequences can exceed ten tokens. The catch is that acceptance rate is only high when model output closely resembles model input — n-gram speculation is mainly useful for code completion and code revision, where syntax is predictable and output closely mirrors input, but it strongly outperforms EAGLE within that specific domain.

Lookahead Decoding is a related method that generates n-grams dynamically during inference to populate the dictionary (rather than relying on pre-existing repetitive context), making it more general-purpose tn-gram speculation, at the cost of extra compute needed to generate the n-grams. Every speculative decoding algorithm ultimately aims to reduce total forward passes needed to complete an output sequence, improving latency and specifically tokens-per-second-per-user during decode; n-gram speculation excels at code-completion-style tasks while Lookahead Decoding generalizes better in compute-rich systems.

Caching

During prefill, the engine builds a KV cache (stored keys/values per token); during decode, this cache is updated for each new token, since autoregressive generation means each token's value depends on every prior token in the sequence. All inference engines use KV caching by default within a single request — without it, inference would be unbearably slow since every prior value would need recomputatiofor every new token. But engineers can extract more value by reusing KV cache across requests, not just within a single sequence.

Prefix Caching and KV Cache Re-Use

Consider two four-token prompts sharing the first two tokens ("Weather in"). Normally the engine runs prefill on all four tokens of each prompt, but with prefix caching, the KV cache computed for the shared prefix during the first request can be reused for the second, skipping prefill on those tokens and improving TTFT. This is the mechanism behind pay-per-token APIs charging less for "cache hit" versus "cache miss" input tokens — cached tokens require very little compute or time to reuse, and inference engineers can apply the same principle to their own deployments to cut latency and cost.

While saving two tokens is trivial, prefix caching can skip prefill on thousands of tokens in domains including: complex system prompts (agents, customer chatbots, RAG scaffolds, tool calls with long boilerplate on every call), code completion (sharing thousands of lines of code as context), documents and retrieval (repeated context ahead of user prompts in summarization/QA/retrieval), and multi-turn conversations (each new turn repeats the entire prior chat history, so savings compound with every rn).

A critical, non-obvious point: prefix caching only works from the start of the sequence up to the first non-matching token — even if all subsequent tokens are identical between two sequences, no caching benefit exists if the sequences diverge earlier. In the weather example, even though the fourth token (a question mark) is shared, the cache reuse stops at the third token because that's where divergence occurs before it, and the boundary example given shows that if the very first tokediffers between two sequences, there is zero savings from prefix caching, even if every later token in both sequences is identical. Because prefixes only extend to the first unique token, context engineering directly determines TTFT savings — the practical implication is to structure prompts so that novel/dynamic tokens appear late in the context as possible, keeping shared boilerplate at the front.

Prefix caching dominates because LLMs are autoregressive: each token influences every subsequent one, so even a single novel token shifts the model's internal representation of everything after it, even when a human reader would perceive the surrounding text as identical. Active research exists on non-prefix KV cache reuse (caching arbitrary mid-prompt sequences), which requires correcting positional embeddings and selectively recomputing certain KV entries to preserve quality; tools like CacheBlend and LMCache support this, expanding what's cacheable beyond strict prefixes.

Where to Store the KV Cache

KV cache is extremely valuable but consumes large amounts of memory, and GPU VRAM is limited. Engines let you configure how much memory to allocate to KV cache (e.g., a specific TensorRT-LLM setting) — a worked example: on a B200 with 180GB VRAM, using 100GB for model weights/buffers and allocating 80% of the remaining VRAM would yield 64GB for KV cache. This allocation fills quickly under load, forcing eviction of saved caches and increasing future cache-miss risk.

To gain more room, cache can be offloaded from VRAM to other storage tiers, in descending bandwidth order:

• G Device Memory (GPU VRAM) — terabytes/sec, tens to hundreds of GB capacity.
• G2: Host Memory (CPU RAM) — tens to hundreds of GB/sec, hundreds of GB to terabytes capacity.
• G3: Local SSD — 5–10 GB/sec, terabytes capacity.
• G4: Networked SSD — gigabytes/sec, tens of terabytes capacity.

Certain SKUs like GB200 include CPUs and interconnects delivering much faster G2 storage, making them well-suited for KV cache offloading. NVIDIA Dynamo supports this via KVBM (KV Block Manager), whes APIs for moving KV cache blocks between memory tiers. General rule: keep the most frequently accessed blocks in higher-bandwidth tiers, relegating less-used blocks to slower storage until needed.

Cache-Aware Routing

Production systems run multiple replicas behind a load balancer, typically routed based on replica busyness. But if prefix caching is heavily used, routing logic must account for it — a user in an ongoing conversation or repeatedly querying the same codebase should be routed back to the same replica whenever possible to g cache hits, producing faster and cheaper requests (rather than being routed purely by load balance, which would scatter their requests and cause cache misses).

Another approach is building a global KV cache across replicas using G4 networked storage. Routing still matters in this setup — a replica with ahot" local G1 cache serves faster than one reading from G4 networked storage — but the global cache guarantees all replicas can eventually reach any precomputed sequence, and cached sequences aren't lost when nodes cycle or are spun down during autoscaling.

Long Context Handling

"Long context" is a somewhat circular definition: a sequence becomes "long context" precisely when its KV cache grows large enough to cause inference problems. Depending on model, hardware, engine, and traffic, problems can emerge past common thresholds like 32K, 64K, or 128K tokens — a practical warning is to specifically test very large input sequences during performance benchmarking to validate behavior at these boundaries. Foundation labs use scalinchniques like RoPE to extend usable context windows, but longer context introduces new inference challenges: attention scales linearly with sequence length when accounting for KV cache, and with long sequences attention can become the dominant VRAM consumer — competing directly with decode's own memory needs.

Model-specific approaches like sliding window attention, compressed attention, and sparse attention address this on a per-architecture basis, but there are also general optimization thniques for the standard attention algorithm:

• FlashAttention: optimized attention kernels that reduce memory reads/writes needed to compute attention.
• PagedAttention: a memory management technique storing KV cache in fixed-size pages, reducing fragmentation and duplication.
• Chunked Prefill: splits large input sequences into chunks that can run alongside decode as resources allow, avoiding overwhelming the engine with one huge sequence at once.

When even these optimizations aren't enough VRAM for KV cache on a single GPU, the next step is parallelizing inference across multiple GPUs.

Model Parallelism

Every frontier LLM today is too large to fit on a single GPU for batch inference — model growth has outpaced GPU memory growth and shows no sign of reversing. In FP8, roughly one gigabyte of VRAM is needed per billion parameters of weights. For DeepSeek-V3.1 (671B parameters), weights alone would trigger an out-of-memory error on a single B200. It's not enough to barely fit weights either: on 4x B200 GPUs (720GB total VRAM), DeepSeek's weights could theoretically fit, but with no room left for KV cache (which often consumes 80%+ of remaining VRAM after weights), four B200s couldn't serve any reasonable sequence length or batch size — a full 8-GPU node is actually required for real production traffic on a DeepSeek-sized model. A practical estimation method: multiply precision, parameter count, and expected KV cache allocation together to estimate minimum GPU count, then round up to the next available instance size. Even for midsize modelke GPT-OSS, teams often provision beyond the bare minimum GPU count to enable larger KV caches and better per-user latency.

Scaling parallel inference efficiently is limited by inter-GPU communication overhead. As covered in Chapter 3, interconnects include NVLink/NVSwitch within a node and InfiniBand between nodes — both are high bandwidth relative to networking generally, but a fraction of the speed of VRAM itself. Since decode is memory-bandwidth-bound, multi-GPU inference must be carefully designed to avoid communicati bottlenecks — a discipline called topology-aware parallelism.

Three primary forms of model parallelism:

• Pipeline Parallelism (PP): splits model layers across GPUs, with each GPU handling one stage of forward/backward passes. Not recommended generally, due to poor latency and utilization from its step-by-step pipeline nature. Tensor Parallelism (TP): splits tensors within each layer across GPUs, with compute-heavy operations like matmuls divided up. Requires cross-GPU synchronization, making it unsuitable for multi-node use.
• Expert Parallelism (EP): shards entire experts of MoE models across different GPUs, keeping in-expert inference fast within a single GPU. Requires routing between GPUs to reach different experts but is better suited for throughput.

Tensor Parallelism is generally the best choice for low-latency inference within a single node; Expert Parallelism improves throughput for MoE models; Pipeline Parallelism is reserved for multi-node scenarios only. Data-parallel strategies like Context Parallelism split computation across devices too, but these are rare in LLM inference — they matter more for video generation (covered in section 6.6).

Tensor Parallelism for Lower Latency

TP should be the default multi-GPU strategy — it supports both dense models (e.g., Llama 405B) and the MoE models that dominate today's open model landscape. TP splits apart each layer itself (unlike PP, which keeps layers intact) and distributes fragments across GPUs, so the expense of reading weights and running matrix multiplication for each layer is shared. However, each l's partial results need to be combined via an all-reduce operation into a single output before the next layer can run — a step minimized by high-bandwidth intra-node NVLink/NVSwitch. Increasing TP degree improves per-user TPS, assuming the model is large enough and sequences long enough that communication overhead doesn't outweigh the benefit of a faster forward pass — a condition true for most frontier models.

Expert Parallelism for Higher Throughput

EP cleanly divides experts acroPUs — e.g., a 128-expert model served as EP8 across eight GPUs would place 16 full experts per GPU. EP improves total system throughput and lowers cost: with each expert processing tokens independently, individual token latency stays the same, but the system overall handles more simultaneous tokens. Many production deploymentsombine TP and EP together (e.g., using TP for attention layers and EP for the sparse MoE layer) to capture both benefits. EP requires less inter-GPU communication than TP — the Expert Router (which determines which exrts each token activates) is small enough to be replicated on every GPU, and while inter-GPU communication is still needed to pass tokens between experts, EP does not require collecting/combining results at every layer the way TP does. This lower communication overhead lets EP scale well to multi-node deployments and systems with limited interconnect bandwidth.

Multi-Node Inference

Serving a very large model at high precision, supporting multi-million-token sequences, or simply maximizing speed can require more than eight GPUs. Multi-node training has long been standard, but multi-node inference introduces new challenges: infrastructure (reliably provisioning and abstracting across two or more interconnected nodes across cloud providers, covered in Chapter 7) and parallelism (efficiently communicating over InfiniBand, which is much slower than NVLink).

InfiniBand's lower bandwidth relative to NVLink complicates topology-aware parallelism — TP generally requires too much cross-GPU communication to be a good multi-node fit. Two viae multi-node strategies instead:

1. For dense models: TP within each node, PP between nodes (e.g., "TP8PP2" notation).
2. For MoE models: Expert Parallelism (e.g., "EP16") works well too since it has lower communication overhead than TP.

For MoE models specifically, TP8PP2 tends to deliver lower per-user latency while EP16 tends to deliver higher overall system throughput. A key warning: unless model size and KV cache genuinely require multi-node inference, it's usually not the best use of extra hardware — teams are often better off using additional nodes for horizontal replica scaling or disaggregated serving instead.

Disaggregation

Disaggregation synthesizes three ideas from earlier in the chapter: (1) prefill is compute-bound and determines TTFT, while decode is memory-bound and determines TPS; (2) specialization (in kernel selection, parameter tuning, etc.) improves performance; and (3) parallelizing model serving across multiple GPUs or nodes is viable if interconnect bottlenecks are avoided. When prefill and decode shara node under heavy load, they compete for resources — ideally prefill would consume mostly compute while decode consumes mostly memory and the twcoexist peacefully, but with larger batches and more compute-intensive optimizations, this separation breaks down and the two phases interfere with each other.

How Disaggregation Works

Disaggregation separates prefill and decode into distinct engines running on separate GPUs or nodes, turning inference into a three-step process: (1) the prefill engine processes the input sequence, generates the KV cache, and computes the first token; (2) the prefill engine transmits the KV cache over the hardware interconnect to the decode engine; (3) the decode engine computes all subsequent tokens.

Conditional disaggregation improves on this basic flow for real-world traffic: requests are first sent to the decode engine, which checks whether the input is already cached or short enough to handle locally — if so, the decode engine performs prefill itself, skipping disaggregation entirely; if not, it forwds the request to a prefill engine. This is described as generally better suited to real-world traffic than always disaggregating. A further benefit of disaggregation is that each engine type can be tuned independently as well as as a system — for instance, the compute-bound prefill engine can use a lower Tensor Parallelism degree than the memory-bound decode engine.

When to Use Disaggregation

Disaggregation is powerful but demands multiple GPUs and substantial engineering effort, so should only be adopted when all relevant conditions hold:

1. Serving large traffic volume — starting around one hundred million to one billion tokens per day, depending on model size.
2. Serving a larger model — at least a hundred billion parameters.
3. Traffic is prefill-heavy with long input sequences.

If either conditionr 2 doesn't hold, disaggregation likely wastes money on extra hardware for minimal gain. If condition 3 doesn't hold, it's likely better to use the extra GPUs for horizontal replica scaling instead, since standard decode engines are already efficient for short sequences or prefix cache hits. A strong real-world use case: serving a frontier LLM inside a code editor, where many developers simultaneously submit large, varied chunks of code as context — heavy prefill volume against a trillion-parameter model is described ashe textbook disaggregation workload.

Dynamic Disaggregation with NVIDIA Dynamo

Dynamo provides production-ready disaggregation support built for heterogeneous real-world traffic, offering: a prefill queue to hold requests when prefill engines are saturated; robust conditional disaggregation with configurable thresholds for input sequence length (after prefix cache) and prefill queue size; and efficient NIXL-based KV transfer from prefill to decode engines, including a kernel that transposes KV blocks between layouts when the two engine types use different TP configurations.

Together these features enable dynamic disaggregation, where the number of prefill and decode engines can be adjusted at runtime to match changing traffic patterns. Disaggregation need not use a one-to-one ratio of prefill to decode engines — real systems typically run multiple of each, denoted as "xPyD" (for example, 5P3D means five prefill engines and three decode engines working together to serve one model deployment).

As these systems scale, new bottlenecks emerge. With disaggregation, the key new bottleneck is prefill queue size — it must not be allowed to grow too large, managed both by setting a reasonable threshold for handling prefiocally on the decode engine and by reconfiguring xPyD ratios at runtime to allocate more resources to prefill when needed. The other major bottleneck is running out of KV cache capacity on decode engines under high load, which can be mitigated by increasing KV cache availability through quantization and KV cache offloading — teniques covered earlier in the chapter.

Chapter 6: Modalities

Modality describes what types of input a model accepts and what types of output it creates. Prior chapters focus on inference engineering for LLMs (text in, text out); this chapter extends that discussion to other modalities. Generative AI covers a rich array of input/output/category combinations, including text+image/video → text (vision language), text/image → vector (embedding), audio → text (transcription), text → audio-voice (speech synthesis), texo-music (music generation), audio → audio (speech-to-speech), text/image → 3D model (generative CAD), text/image → image/video (image/video generation), image/video → text (captioning), image/video → mask (segmentation), and text+image → image (image editing).

Despite this variety, there are only two broad model archetypes:

• Autoregressive token generation: starts from a tokenized sequence and predicts the most likely next token. LLMs are the most famous example, but vision language models, embedding models, ASR models, and TTS models all rely on similar architectures, and many of the same inference engines and techniques used for LLMs apply to them.
• Iterative denoising: starts from random noise and refines towarikely output. Image and video generation models rely on this approach, though hybrid diffusion transformer models are increasingly setting the frontier. Some of the same optimization philosophies (kernel selection, parameter tuning) apply here too, but the implementation details end up quite different.

A non-obvious point: for each new modality, the way latency, throughput, and quality are measured must be rethought. For example, a single token of TTS audio output isn't meaningful on its own — instead of time-to-first-token (TTFT), the useful metric is time to first word or time to first sentence.

6.1 Vision Language Models

Vision language models (VLMs) take one or more images or videos plus a text prompt and generate a text response. A VLM usually has two modules: a standard LLM, and a small vision encoder tt converts raw images/videos into image tokens. The LLM is far larger than the encoder — in Mistral Large 3, the vision encoder is 2B parameters versus a 673B-parameter LLM. Despite being small, the vision encoder is ctical for inference, and because VLMs use varied encoder architectures, runtime support is more fragmented, which increases the importance of vLLM and SGLang for serving VLMs.

Rule of thumb: a high-resolution input image adds about 1,000 visual tokens to the input sequence. Image tokens behave similarly to regular tokens at a high level but add up fast. The primary optimization challenge across VLMs is handling the longer input sequence and larger KV cache, which affects both phases:

• Prefill: images are patched, embedded, tokenized, and fed in as part of the input sequence.
• Decode: same mechanics as LLMs, but with longer context, and some models add attention variants for image tokens.

Every LLM optimization technique from the prior chapter applies: KV cache quantization reduces memory bandwidth/storage overhead for longer sequences; decode can be accelerated with speculation (especially EAGLE); prefix caching reuses KV cache for images across multi-turn chats and repeated queries; Tensor Parallelism speeds inference while accessing more VRAM for large models/long contexts; and disaggregation moves prefill to independently scaling specialized workers for long sequences.

VLMs also introduce a new quality-speed tradeoff: downsampling. Images/videos can be converted to visual tokens at different resolutions — high resolution takes about 4x more tokens than low resolution but carries more detail. Downsampling generally isn't needed for single-image inputs but may be necessary for multiple images or video clip

6.1.1 Video Processing for Vision Language Models

A video is more than the sum of its frames — it may contain audio (though many VLMs can't process audio directly and require separate transcription added into the prompt), and frames express motion through time that static images lose. VLMs are trained on video clips spefically to understand this time dimension, and high-quality inference requires processing an entire video clip in a single call.

A striking example of scale: one second of cinematic video contains 24 frames; each frame costs roughly 1,000 tokens as a high-definition image, so a four-second clip would produce nearly 100,000 input tokens. In practice, video inputs don't generate quite this long a sequence because downsampling — reducing resolution and frame rate — is practically obligatory to fit an entire clip into a single inference request. Even so, video understanding models are still only capable of handling very short clips. After tokenization and encoding, inference proceeds like image inference but with much longer context, making prefix caching, KV cache offloading, and optimized attention implementations even more important for these tens-of-thousands-of-token sequences.

6.1.2 Omni-Modal Models

VLMs are part of a broader trend toward "omni" models that accept and produce multiple modalities. There's a tradeoff: omni models offer unique blended capabilities, but smaller specialized models are often faster and more accurate within specific domains. Example: many VLMs have built-in text recognition trained their image processing, but this lags behind dedicated OCR models that are a fraction of the size.

Practical implication: production VLM inference often involves coordinating a pipeline of multiple models and preprocessing steps — individual preprocessors for extracting data from PDFs, OCR for reading text from images, or transcription for audio from video. Each pipeline component must be individually optimized for speed and should scale independently to avoid bottlenecks.

6.2 Embedding Models

An embedding model traforms a variable-length chunk of text (or another modality like an image) into a fixed-length vector representation capturing semantic meaning. Encoding content into a shared semantic vector space allows comparing distance between items with simple math. Embedding models, together with vector databases, underpin agent memory, RAG, search, and recommendation systems.

Embedding workloads have two distinct traffic profiles:

1. High-throughput backfills: bulk operations like indexing millions of documents, updating product catalogs, or preparing data for LLM pre-training.
2. Low-latency lookups: individual user-facing queries for search, retrieval, or recommendation, where every millisecond matters.

Practical guidance: inference engineering for embedding models starts by clarifying which profile is needed. If both are required and traffic justifies the cost, it's better to build separate systems for each usage type.

6.2.1 Embedding Model Architecture

Despite tens of thousands of embedding models on Hugging Face, all use one of two transformer architectures:

• BERT-style models: encoder-only, usually under 1B parameters, originally built for masked token prediction.
• LLM-based models: modern language models, generally up to 8B parameters, repurposed to generate embeddings.

LLM-based embedding models today offer substantially greater capability, though BERT-style models remain useful for simple, latency-sensitive tasks like classification.

Embedding models introduce their own speed/quality tradeoff via embedding dimensionality — the size of the output vecr (a few hundred to a few thousand values), where longer vectors encode more information. Modern embedding models use Matryoshka representations to unlock dynamic tradeoffs between dimensionality and quality while retaining more information even on shorter vectors. Non-obvious point: dimensionality doesn't materially affect inference time, but it does affect storage, retrieval, and similarity computation time within a system. Warning: vectors from different embedding models generally cannot be meaningfully compared, even at the same length, since they encode inputs into different semantic spaces.

6.2.2 Embedding Model Inference

For LLM-backbone embedding models (e.g., Qwen 3 Embed 8B), optimization shares tools and techniques with high-volume, low-latency deployment of small LLMs. Runtimes include vLLM, SGLang, Infinity, and TEI (Text Embedding Inference by Hugging Face), but the best performance comes from adapting TensorRT-LLM, which brings an optimized XQA kernel for fast attention and kernel fusion to reduce memory access overhead — making it the most performant engine for both latency and throughput on supported models.

Quantization offers further gains: while smaller models generally lose more quality from quantization, FP8 quantization of embedding modeleights offers improved performance with minimal quality loss. Practical verification method: run identical inputs through the original and quantized model and check cosine similarity of output vectors — 100% similarity means identical vectors, and at least 99% similarity should be the target for confidence in the quantization.

Since embedding models process tokens in parallel, prefix caching and disaggregation aren't relevant optimizations, and given their small size, multi-GPU parallelism is not effective either. Instead, high-traffic deployments should scale horizontally, with each GPU as its own replica. Batching and queueing matter greatly: embedding models support much higher batch sizes than other models — a single request may batch dozens or hundreds of text inputs, and many requests can rn parallel on one GPU since even demanding embedding models are relatively small and fast. Warning: traffic (large backfills or usage surges) can still exceed even these large batch sizes, so a robust queuing system is essential infrastructure.

6.3 ASR Models

Automatic speech recognition (ASR) models take audio input and produce text output, powering transcription and dictation apps. The most popular open ASR model is Whisper (OpenAI), supporting dozens of languages with accurate transcription. Even Whisper's largest, highest-quality variant is just 1.55B parameters, and it runs extremely fast on fractions of large GPUs like H100 via Multi-Instance GPUs (MIGs). Despite many sizes, variants, distillations, and quantizations existing, in practice most latency budgets can be satisfied with the highest-quality models: Whisper 3 Large and Whisper 3 Turbo.

Whisper is an encoder-decoder model: the encoder takes a processed audio waveform (log-Mel spectrogram) and encodes it into audio features; the decoder converts these features into text tokens. The overwhelming majority of inference time is spent on the decoder, which is an autoregressive transformer very similar in architecture to an LLM — meaning excellent existing LLM optimization tools apply to the main btleneck. The primary tool is TensorRT-LLM, providing in-flight batching for the decoder and an optimized C++ runtime with efficient CUDA kernels; it works especially well with recent architectures like Hopper and Blackwell, making MIGs an even better option for ASR inference.

6.3.1 Single-Chunk Latency Optimization

For real-time transcription (dictation apps, voice agents), the relevant metric is round-trip time for a single audio chunk. Target: 200 milliseconds, the average human reaction time. With Whisper already running on an optimized TensorRT-LLM engine, there's little runtime-level work left to do — most gains for real-time Whisper instead come from orchestration and infrastructure.

The biggest product-experience upgrade for ASR is streaming, implemented at the API server layer (not thmodel runtime layer) via a WebSocket connection, streaming audio in and text out continuously. At the runtime layer, nothing changes; instead, a Voice Activity Detection (VAD) model monitors the incoming stream and segments it into discrete chunks for the ASR model. Inference runs normally on chunks, streaming text results back via the WebSocket. This setup handles several concurrent streams and has the advantage of keeping transcription sequential — when each chunk runs on the same GPU, therevious chunk's output sequence can be used as a prefix for the next chunk, improving transcription quality.

6.3.2 Long File Latency Optimization

Whisper only supports 30-second chunks, so transcribing long files (e.g., hour-long podcasts) requires different optimizations. Performance for long files is measured with Real-Time Factor (RTF): if the world's fastest typist could manually transcribe an hour of audio in 30 minutes, that's RTF 2X. An optimized Whisper deployment can transcribe an hour of audio in under four seconds — RTF 1000

Fast long-file transcription requires a multi-step pipeline: first, a VAD model (running on dedicated hardware) removes silence and chunks meaningful audio segments rather than splitting by fixed time intervals — avoiding the risk of cutting words in half. Chunks are then processed in parallel, ideally across multiple GPUs or MIGs; RTF improves roughly linearly with GPU count, and each GPU processes multiple chunks at once via in-flight batching for high utilization. Finally, chunked transcripts are stitched back together by timestamp.

Non-obvious tradeoff: parallelizing chunk transcription removes the ability to use the previous sequence as a prefix for the next chunk (unlike the sequential single-chunk case), but other quality-improvement techniques more than compensate. ASR output can be automatically checked for hallucinations like repeated words/phrases by measuring compression ratio and words-per-minute. When a chunk shows an issue:

1. Re-run at higher temperature — counterintuitive, sinceher temperatures generally cause more hallucinations, but the goal is to break cycles of repeated words and produce a different output.
2. Re-chunk the entire audio (or a segment) into smaller chunks and re-run transcription.

In practice these techniques make the previous-sequence-as-prefix trick unnecessary, unlocking efficient, accurate parallel transcription for long files.

6.3.3 Diarization

Diarization — annotating a transcript with who is speaking when — is an adjacent problem to transcription. Diarization ms categorize audio by voice feature, then segment and cluster across the file to timestamp speaker changes. This is a completely different class of model: where Whisper is an encoder-decoder transformer, diarization systems like pyannote audio are pipelines of classic ML models (segmentation, embedding, and clustering models). Optimizing diarization means running each model fast and orchestrating the whole pipeline efficiently, using tools like PyTorch and pyannote along with optimizations like Torch compilation. Warning: even highly optimized diarization implementations take at least twice as long to process an audio file as transcription does.

6.4 TTS Models

Text-to-speech (TTS) models, or speech synthesis models, take text input and produce speech audio. In 2025, open models like Orpheus TTS introduced extremely lifelike speech synthesis to the open ecosystem, and many companies fine-tuned Orpheus for improved vocal quality and product-specific voices, driving high adoption of open models in voice AI.

Modern TTS models are fine-tuned LLMs — Orpheus TTS derives from Llama 3.2 3B — so many LLM runtime and performance optimizations apply directly. TTS models have small parameter counts (Orpheus at 3B is on the larger end), so MIGs on H100s are highly efficient. Unlike ASR models (which generally run FP16), TTS model weights and KV cache can be quantized to FP8 for better performance, in addition to optimized kernels and in-flight batching from TensorRT-LLM.

Architecture detail: LLM-backbone TTS models are trained by expanding the LLM's vocabulary with tens of thousands ocoded audio tokens, then trained on text-input/tokenized-audio-output pairs. This means using TTS models in practice requires an audio decoder that converts output audio tokens into a waveform — a potential bottleneck. The audio decoder should be implemented in PyTorch, compiled for the target GPU, and use dynamic batching with a short timeout (e.g., 15 milliseconds); in-flight batching is not possible for the audio decoder.

TTS performance uses different metrics than LLMs:

• TTFB ime to first byte): the speech-synthesis equivalent of TTFT.
• Time to first sentence: a more user-oriented metric — time to generate the first meaningful phrase or sentence.
• TPS: decode speed in tokens per second, as with an LLM.

As with LLM TTFT, the goal is to minimize TTFB — for Orpheus, as low as 150 millisecois achievable on a single H100. However, TPS goals differ from LLMs: generated tokens convert to audio waveforms, and depending on the model it may only take 80–100 tokens per second to generate audio in real time — nd that threshold, additional tokens per second provide no benefit. Instead, performance work focuses on scaling throughput in terms of concurrent real-time outputs a GPU can support; more concurrent users per GPU dramatically lowers per-user cost.

6.4.1 Streaming Real-Time Text to Speech

As with ASR, once the runtime layer is optimized (TensorRT-LLM, quantization, compiled SNAC decoder), further real-time performance gains come from infrastructure rather than the runtime. Streaming over WebSockets is the biggest unlock versus sending text and receiving audio in discrete chunks. Practical step: test the inference engine to determine how many concurrent real-time streams it can generate, then set the batch size and active WebSocket count to match, keeping usage high but stable.

Warning: TTS models are rarely used outside real-time applications, and if used for batch cases (e.g., backfilling a large document corpus to audio for accessibility), note that TTS models handle long inputs poorly — speech quality starts degrading after about0 seconds.

6.4.2 Speech-to-Speech Models

An exciting research area is speech-to-speech models — models taking audio input and producing audio output directly. Today, most voice systems use a cascading approach: an ASR model, LLM, and TTS model work in a pipeline to listen, think, and respond, with auxiliary components like VAD and embedding models facilitating natural conversation and context.

Speech-to-speech models, like OpenAI's gpt-realtime, augment a core LLM with audio consption and production capabilities, unifying the pipeline into a single model — possible because ASR, LLM, and TTS share similar architectures, especially on the decoder side. Warning: at time of publication, there are no commercially viable open speech-to-speech models, and closed options like gpt-realtime are significantly lescapable and more expensive than cascading multi-model setups. Research in this space remains robust, and this emerging modality will require its own inference engineering approaches soon.

6.5 Image Generation Models

Working with image and video generation models differs fundamentally from LLMs on several axes. First, architecture: while some recent models like HunyuanImage-3.0 resemble LLMs more closely, most image/video generation models are iterative denoisers, not autoregressive token generators — pipelines of multiple small models working together in latent space rather than the uniform decoder architecture of an LLM. Tooling reflects this difference: at time of publication, SGLang Diffusion and vLLM Omni are brand new, and most image/video generation inference is implemented lower in the stack, working directly with PyTorch or TensorRT.

Constraints differ too: image generation models are 10–20x smaller than frontier language models, and inference is compute-constrained, not bandwidth-constrained. Perhaps the most significant difference is that image and video generation models offer more direct quality-to-speed tradeoffs.

Non-obvious challenge: evaluating image model output quality programmatically is difficult. Automatic pipelines using vn language models give only directional signal at best and may diverge from human preferences — the human eye is "mysterious," and most image quality evaluations work by having humans pick among thousands of images to aggregate vibes and preferences into quality benchmarks.

6.5.1 Image Generation Kernel Optimization

Model card inference examples for image generation models generally use the diffusers library with very few optimizations. Non-obvious point: while image generation is toretically compute-bound, memory-efficient kernel selection and kernel fusion are often necessary just to reach that compute bottleneck.

High-performance image model inference uses one of three libraries:

• SGLang Diffusion: performant inference engines built for popular image/video generation architectures.
• TensorRT: high-quality black-box implementations of popular models using NVIDIA's in-house kernels.
• PyTorch: careful kernel selection and fusion yields control, flexibility, and improved high-end performance.

Practical guidance: for something that works well right away, use SGLang Diffusion or TensorRT implementations. PyTorch offers advanced inference engineers an opportunity for deep customization.

The most essential kernel is the attention kernel — many image generation models use FlashAttention 2 out of the box, but FlashAttention 3 and 4 yield better performance on Hopper and Blackwell GPUs respectively. Smaller kernels, especially normalization functions like RMSNorm, are good fusion candidates for efficient memory usage. GEMM kernels matter for compute-bound inference — they apply to linear layers and are generally safe to quantize to 8-bit floating point formats, accessing 2x higher FLOPS on Tensor Cores; kernels from CuTe, CUTLASS, or DeepGEMM may prove most efficient on a model-by-model basis.

Torch compilation includes automatic kernel fusion with a plugin system for manually selected kernels, and the resulting engine can be cached for faster load times at node startup — important because compilation itself takes several minutes. Warning:most high-performance engines, Torch compilation targets the specific GPU model/architecture performing the compilation — if the model will run on a B200, compilation should happen on a B200.

6.5.2 One Weird Trick for Faster Image Generation

Beyond legitimate kernel selection and Torch compilation, there are informal optimization hacks. Image generation time tracks linearly with step count, which is why few-step and latent consistency models are much faster than full 50-step models but reducing step count risks dropping image quality below an acceptable threshold.

Key mechanism: each denoising pass runs at batch size two, since each step includes a pass with and without prompt guidance. The guidance parameter controls how much the prompt-guided image is weighted when combining the two iterations per step; if guidance is zero, the prompt-guided image doesn't need to be generated at all.

Non-obvious insight: after the first few steps, the basic outline of the image is established, and remaining steps fill in details — pmpt adherence matters more in early steps that set broad strokes, since the model won't change its mind on later steps and switch a dog into a cat mid-generation. Practical trick: turning off guidance partway through generation saves denoiser passes without reducing step count. Example: skipping guidance for the last 20 steps of a 50-step run yields only 80 total passes through the model instead of 100, while quality generally remains high.

6.6 Video Generation Models

Video generation is the most demanding modality. These models should run on Blackwell GPUs (or Rubin, once available) whenever possible, since these GPUs offer high memory capacity for Context Parallelism, fast Tensor Cores for attention computation, and microscaling data formats for more precise quantization.

Architecturally, video generation resembles image generation, just rendering a full video rather than a single frame from latent space — following the principle that greater scale unlocks more techniques, video generation uses all the same techniques as image geration plus additional optimizations. Video generation is compute-bound and uses iterative denoising over latent space, generally taking about the same number of denoising steps as image generation (~50), but each step processes far more data.

Non-obvious point: because video generation models are compute-bound, batching isn't useful the way it is for text generation. Video generation models usually run on full nodes of eight GPUs with batch size one — all eight GPUs work together to crea one video at a time. Unlike batched workloads where latency-throughput tradeoffs come from adjusting batch size, the only way to improve video generation throughput and cost is to make the model itself faster.

Historical context: early video generation models were framewise, generating frames one at a time, which reduced quality and coherence. Today's models run denoising steps on the video as a whole in latent space, where latent space represents three dimensions (width, height, time) instead of image generation's two (width, height). This means passing huge amounts of data through each attention calculation — attention accounts for 70–80% of compute time for video models, making it the most important optimization target.

6.6.1 Attention Optimization and Quantization

Atton optimization starts with kernel selection: test FlashAttention, DeepGemm, CuTe, and CUTLASS kernels to find the best performer for a given model. Where language models use the KV cache to accelerate attention, video generation models use other caching patterns to reuse model outputs — reusing parts of the attention computation can make video generation 30–40% faster in practice. Two fundamental caching approaches (with implementations continuously evolving):

• Timestep-based caching: caching and reusing outputs of certain timesteps to skip entire steps.
• Transformer-based caching: caching and reusing hidden states to skip layers within the transformer.

Warning: caching algorithms and implementations range from negligible quality degradation to unusable output — test carefully beforuction use.

Beyond kernels and caching, quantization is the main tool for speeding up attention. Non-obvious distinction from LLMs: for bandwidth-constrained language model inference, quantization's benefit is reduced data loaded through memory; for video models, quantization instead means accessing double the FLOPS by switching to lower-precision Tensor Cores. Language model quantization focuses on weights (large linear layers where quantization impact is negligible), but for video models, while quantizing weights still helps and these layers dominate memory bandwidth, they represent only a small fraction of compute time. Instead, video model quantization focuses on attention — the riskiest part of any model to quantize since errors accumulate over inference. With only ~50 steps versus thousands of autoregresve iterations in token generation, the risk is somewhat lower for video models but still significant.

The first method for reducing quality impact: use blockwise quantization and a microscaling data format (MXFP8), available on Hopper and Blackwell — microscaling formats better preserve outlier values, which strongly affect attention accuracy. The most sophisticated approach is selectively quantizing within the model:

• By step: keep early steps in FP and quantize later steps — following the same insight as the classifier-free guidance trick in image generation, since early steps establish the image outline and matter more for prompt adherence/accuracy.
• By layer: keep first and last layers unquantized and quantize hidden layers, since first/last layers handle input and final output while hidden layers perform intermediate calculations that tolerate approximation better.

By quantizing only less important parts of the process, quality is preserved. These tactics appear in kernels like SageAttention, an 8-bit attention kernel usable for quality low-precision attention on video generation models.

6.6.2 Context Parallelism

While video generation models generally run on a full node of eight GPUs, they use Context Parallelism rather th Tensor Parallelism. Context Parallelism copies model weights onto every GPU; video models are small enough that replicating weights eight times uses a meaningful amount of memory but remains feasible on B200. Instead of splitting the model across GPUs, Context Parallelism splits the attention calculation across GPUs, coordinated via a mechanism like ring attention, where each GPU holds a piece of the context and passes intermediate results to the next GPU in the ring.

Non-obvious detail: attention for transformer models is multi-head (usually eight or more heads), and attention heads are independent, so they can be run separately with results combined afterward. Attention isn't the only parallelizable component — the latent decoding step using the variational autoencoder takes 3–5% of total inference time and clso be run across GPUs.

These parallelism techniques are what make AI video generation feasible at all, and as video sequences get longer and models get larger, parallelism will remain the most critical technique for video generation model inference.

Chapter 7: Production

Inference engineering aims to make generative AI models faster, less expensive, and more reliable, but this promise only materializes if optimization work survives contact with production traffic, including hypergrowth and viral spikes. Scaling production traffic rigorously tests assumptions — sequence shapes, traffic patterns, and even what topics users chat about all affect observed performance. Maintaining secure, robust infrastructure is a distinct skillset from optimizing model inference on a GPU: no matter how efficient a single instance is, enough traffic will overwhelm it, and this is an infrastructure problem requiring different tools and mindset, not a PyTorch or CUDA problem. Scaling introduces new complexities: where and how to acquire GPUs, balancing traffic across them, preventing downtime, and reconciling cost accounting when moving from per-token API pricing to paying directly for infrastructure. Latency in production comes from more than prefill and decode — systeust be evaluated end-to-end, eliminating inefficiencies in server, network, and even client code where influence over the client is possible.

Containerization

Containerization packages an application with its dependencies to standardize deployment, turning a program into an artifact that runs anywhere ("no more it works on my machine"). Containers are lightweight because they share the host's Linux kernel (distinct from a CUDA kernel), making them well-suited to packaging inference services. Key terminology: a container is an actively running isolated environment; an image is the executable package containing everything needed to run software; a Dockerfile is a human-readable, machine-interpretable set of instructions for building an image; a registry is a central repository for storing and distributing images (Docker Hub is a popular AI registry, analogous to Hugging Face for weights or PyPI for Python packages).

Docker containers are built from layers: a base image (an OS distribution like Ubuntu, or a more complex pre-built image, itself composed of multiple layers), additional layers (filesystem changes from dependencies, application code, and config as specified in the Dockerfile), and a container layer — a thin, ephemeral layer created at runtime holding any changes made while running, which is lost when the container terminates. Inference engines like vLLM and SGLang offer official base images for active releases, and it's generally better to start from one of these proven images than to build from scratch.

Dependency Management

Dependency chains for inference are long and fragile, and reaching a working build is hard — this makes containerization essential for preserving a known-good build in an ecosystem with frequent breaking changes. Images are built for a specific GPU architecture and model, and a container bundles: the CUDA toolkit version (CUDA, cuDNN, and driver versions compatible with the rest of the stack), Python packages (torch, transformers, diffusers, etc.), the specific inference engine version (vLLGLang, TensorRT-LLM), and system packages (e.g., ffmpeg, common for audio/image/video models).

Like a backpacker packing light, images should include only strictly necessary dependencies for fast deployment and efficient operation — inference images are often many gigabytes. Another best practice is pinning versions: specifying exact dependency versions keeps runtime behavior consistent across environments and makes builds reproducible. Tools like uv, poetry, or pip flag version incomtibilities at build time; once an image with pinned versions builds successfully, it will always resolve to the same versions, protecting against future breaking changes.

Breaking changes are especially common with newly released models — when models like DeepSeek drop, the whole inference ecosystem races for "day zero" support, and engineers often must rely on overnight builds or developer pre-releases of dependencies rather than stable releases. These early versions are more bug-prone and typically need to be rebuilt on stable releases in e following days or weeks.

NIMs

NVIDIA Inference Microservices (NIMs) are pre-built Docker containers for popular open models, making inference implementations portable. There are two types: Multi-LLM NIM, a flexible container for running a family of models on a supported GPU architecture, and LLM-specific NIM, an engine optimized for one specific model on one specific GPU configuration for maximum performance. A NIM can be used as a starting point to build on, a reference architecture to learn from, or an out-of-the-box inference service. However, if maximum control is the goal rather than a done-for-you configuration, it's generally better to build a custom container from a less opinionated base image than to adapt a NIM.

Autoscaling

Autoscaling aims to ensure enough resources exist to serve incoming requests while maintaining latency SLAs, without wasting money on idle GPUs. Autoscaling systems typically run on Kubernetes, an open-source container orchestration system, paired with cluster-level provisioning/deallocation. Kubernetes clusters have two component types: a control plane (makes routing and scaling decisions) and a worker plane (runs the containerized applications, including GPUs and other hardware each container needs).

Since traffic is rarely perfectly consistent, there's no single ideal replica count. Autoscaling dynamically adjusts replica counts using two signal types: utilization (GPU memory/compute usage) and traffic (request volume). These don't always align — e.g., in L prefill, a few requests with hundreds of thousands of uncached input tokens can cause much higher utilization than many small, high-cache-hit-rate requests. Traffic-based scaling can be proactive, while utilization is a lagging indicator; combining both keeps resources matched to demand.

Designing a traffic-based autoscaler requires configuring five factors: min replicas (floor kept running regardless of traffic), max replicas (ceiling for high traffic), autoscaling window (sliding timeframe for measuring traffic), scale down delay (wait time after a scale-down signal, guarding against spikey traffic returning), and concurrency target (requests each replica can handle at once). For example, increasing scale-down delay avoids premature scale-downs during spiky traffic but risks unnecessary spend once traffic has truly cooled.

Concurrency and Batch Sizing

Effective autoscaling requires understanding how much concurrent traffic each instance can handle via batching. Three batching approaches: static batching (server waits until batch is full before starting), dynamic batching (waits until batch is full or a timeout passes), and continuous batching (continuously runs inference, swapping in new requests as slots open — also called in-flight batching by TensorRT-LLM). Continuous batching operates at the token level and is implemented robustly by vLLM, SGLang, and TensorRT-LLM, minimizing latency relative to static batching.

Batch sizing trades latency for throughput: larger batches increase overall throughput but worsen each user's individual lency. Performance should be tested across multiple batch sizes to find the right fit for model, instance, latency target, and budget. This tradeoff is controlled at the autoscaling level via concurrency target and at the replica level via batch size — these should match. Once every active replica hits max concurrency, the autoscaler spins up more replicas; if many replicas are running half-full batches, it's time to scale down.

Cold Starts

A cold start is the time needed to spin up new model replica. Overall autoscaler performance depends on cold start speed — slow spin-up makes confident scale-down difficult, leading to over-provisioning. Four factors affect cold start time: GPU procurement (speed of adding/allocating GPUs to the cluster), image loading (speed of loading the container image ontohe instance), model loading (speed of loading weights into the container), and engine startup (speed of starting the inference engine, including any compilation).

GPU procurement speed is mostly a function of the cloud provider (unless a pool of warm nodes is flexed between models), and node start time is a negotiable contract factor. Engineers have more control over image/weight loading and startup. Loading images and weights is a function of writing bandwidth for gigabytes (often hundreds of GB) of data — the t levers are making data smaller or getting more bandwidth. Minimizing images to necessary dependencies speeds builds, and quantizing weights speeds loading during cold starts.

For small models, weights used to be baked into the image for simpler caching, but now that most models have dozens or hundreds of billions of parameters, weights dwarf the image size and are better loaded separately. Where weights are loaded from massively impacts bandwidth: loading from a third party like Hugging Face is limited by their egress speed, and storing weights in an S3 bucket introduces network latency and transfer costs. For multi-hundred-billion-parameter models, gigabytes-per-second bandwidth is needed — best achieved by loading over the network thin a node from a source physically cached near the GPU instance in the same datacenter.

Engines like vLLM and SGLang start up fast, but engines like TensorRT-LLM and PyTorch-optimized models have a hardware-specific compilation step that can take several minutes. Caching built engines massively improves cold start times in these cases — both TensorRT-LLM and PyTorch support engine caching — but a cached engine must be loaded onto an instance with the exact same GPU type, CUDA version, software dependencies as the environment it was built in to function correctly.

Routing, Load Balancing, and Queueing

Once multiple replicas are online, the system must decide which requests go where. Routers work at the request level, answering "where should this request go?" Load balancers work at the system level, answering "where could this request go?" In complex systems there isn't just one router and one load balancer — routing occurs throughout the stack with load balancers injected at key points.

Simple even-split routinge.g., "3 replicas, 12 requests, 4 each") doesn't work because requests vary in input token count — a 10,000-token request will unbalance a systemuilt around a 100-token average — and some replicas are better suited to certain requests. Examples: KV cache-aware routing (send a request to a replica with matching prefix already in KV cache) and LoRA-aware routing (send to a replica that already has the needed LoRA weights in memory). Intelligent routing uses engine and orchestrator information (e.g., NVIDIA Dynamo) to route on sequence length, prefix, and LoRA needs.

Load balancing and routing alone aren't sufficient — when traffic exceeds capacity, a queue holds requests while resources scale up or free up. A standard queue is FIFO, but more complex implementations like priority queues can, for example, give paid users priority over free users during high traffic. As new replicas come online, the queue must recognize them immediately and assign each new replica requests up to its concurrency limit right away, rather than lettg requests continue waiting on existing replicas.

Scale to Zero

Advanced autoscalers support scale to zero: scaling down to zero active replicas when there's no traffic, then automatically scaling up when traffic returns. This relies on two prerequisites: fast cold starts (since users may be waiting live) and robust queueing (to hold incoming requests until a replica becomes available).

Scale to zero isn't a fit for all workloads. It's great for development (bursty testing, first-request latency unimportant) and for production applications with periodic traffic (e.g., an agent used only during business hours in one country, or an offline daily batch job). However, relying on scale to zero to control costs in a latency-sensitive application with light, unscheduled traffic is a warning sign that the application isn't yet ready for dedicated infrastructure — pay-per-token APIs should be used until greater scale is reached.

Independent Component Scaling

AI applications increasingly run as multi-model, multi-stage compou AI workloads requiring coordination of multiple steps per request. These steps can have very different hardware needs — a voice activity detector needs far less power than the transcription model it feeds, while the LLM processing the resulting transcript may need a full multi-GPU node — and each step'sling parameters differ too. Autoscaling decisions should be decomposed and each step scaled individually to right-size resources and avoid both bottlenecks and overprovisioning.

However, every model in the pipeline should run in the same cluster: if intra-cluster messaging takes 10ms and inter-cluster messaging takes 50ms, that 40ms difference across a 5-step pipeline eats 20 percent of a one-second latency SLA.

Multi-Cloud Capacity Management

Single-cluster autoscaling has limits — high-volume, globally distributed deployments need thousands of GPUs spread worldwide. Building multi-cloud inference as siloed compute pools per provider is easy but prevents fluid use of inter-cloud compute, making workload migration tedious and err-prone. True multi-cloud inference requires a multi-region, multi-provider bin-packing tool that treats distinct compute pools as fungible, taking a global view like Kubernetes does within a cluster, enabling self-healing and global scheduling.

True multi-cloud inference unlocks: capacity (pooling multiple providers for more flexible GPU access), redundancy (splitting inference across providers for outage resilience), latency (running inference close to end users), and compliance (running inference to meet data sovereignty/regulatory requirements). This requires a coordination layer with a control plane (handles model deployment and global scaling decisions, receives real-time event streams) and workload planes (handle direct inference traffic and in-cluster scaling, report utilization/demand). This separation ensures workload planes serve traffic independently — a control plane or workload plane failure shouldn't affect other workloads.

GPU Procurement

Three major GPU provider types: hyperscalers (large clouds like AWS/GCP), neoclouds (GPU-focused clouds like Coreweave or Nebius), and resellers (secondary markets like SF Compute Coany). Providers vary in capacity, availability, and reliability, with hyperscalers commanding a premium and tradeoffs everywhere between cost and uptime SLAs, support, regional availability, instance configuration, and cluster size.

Securing capacity is the first challenge — getting the latest hardware, especially in large clusters, is difficult, as relatively few providers offer blocks of hundreds of nodes, and many providers allocate the majority of in-demand GPUs to their largest long-term-reservation customers. Working across multiple providers is often necessary to get needed GPUs in the right regions. GPUs are procured via three mechanisms: reserved (blocks reserved for months/years at discounted rates), on-demand (individual instances available up to quota at high per-hour cost), and spot (discounted on-demand instances pre-emptible with agreed notice, often minutes). Largecale inference generally blends sources: a baseline of low-cost reserved instances plus on-demand and spot for handling peaks, distributed across multiple clusters worldwide for user proximity.

Geo-Aware Load Balancing

Global user bases need a global load balancer, not just a per-cluster one. The goal is balancing not letting requests sit queued when spare capacity exists elsewhere against not habitually routing, e.g., a request from Singapore to a San Francisco server. Rule of thumb: it takes about five milliseconds for a request to cross a time zone, so New York to San Francisco is about fifteen milliseconds one-way — given tight latency budgets, running workloads as close to end users as possible matters.

Building for Reliability

GPUs have a notably high failure rate in production, and engineers rning large-scale training runs must account for hardware failure. Meta's Llama 3 paper reported that running 16,000 GPUs for 54 days produced 419 unexpected interruptions, primarily from hardware failure — roughly one ilure per 50,000 GPU-hours. That sounds like a long interval, but a single 8-GPU node running inference for a full year already exceeds 70,000 GPU-hours, so inference engineers should expect hardware failure as a normal occurrence.

GPU health is a node-level concern: when one GPU fails, others on the same node often fail next or need maintenance. Proactively noting failures, cordoning nodes, and cycling pods keeps clusters healthy. Beyond GPU failures, cloud providers have scheduled maintenance and unscheduled downtime, so every infrastructure layer must be reinforced for reliability. Multi-cloud inference enables two high-reliability postures: active-active (multiple regions/clusters actively serve live traffic simultaneously; if one plane fails, traffic continues seamlessly on others) and active-passive (a "hot standby" cluster/region stays ready but idle, with traffic cut over if the active plane fails). Seamless failover across clusters, regions, or providers keeps reliability high and latency low.

Security and Compliance

Cloud security and compliance have been scrutinized for over twenty years, and mission-critical AI inference must meet both bars. Conversations center on three areas: user data (protecting inputs and outputs), model weights (an invaluable trade secret for fine-tuned/proprietary models), and infrastructure (GPUs and intelligence access as abuse targets).

One of the easiest security improvements: simply don't store user inputs or model outputs, if logging requirements or user agreements don't force retention — this reduces attack surface. Securing AI inference workloads is similar to securing any containerized workload: data encryption, container security, network and access controls, and workload isolation, validated by extensive third-party penetration testing, remain the gold standard.

Inference engineers increasingly must support regulated industries and compliance-heavy regions. Multi-cloud infrastructure helps here — certifications like SOC 2 Type II or regulations like HIPAA require providers to also be compt, so being able to shift workloads to compliant providers is valuable. Multi-cluster infrastructure also supports running one model across multiple regions to satisfy data residency requirements, where user data from one country cannot be processed on servers elsewhere — for example, a cluster near Torontand another near New York keeps Canadian data in Canada and American data in the US, with minimal latency overhead.

Testing and Deployment

Beyond replica-level testing and benchmarking during inference engine configuration, end-to-end system testing before deployment is essential. Three testing strategies: manual testing (scripts or manual clicks sending synthetic traffic), load testing (automatically sending large traffic volumes to test scaling and performance), and shadow traffic (copying live traffic to test deployments under real-world conditions).

Testing inference is expensive — it costs engineering time to configure and measure, and GPU time to run test traffic. While this is a cost of doing business, minimizi testing expense matters — for example, shadow traffic testing could start with a random sample of production traffic, followed by a shorter load test. Testing should account for AI product usage fluctuating on daily and weekly cycles.

Zero-Downtime Deployment

High-availability deployment strategies avoid downtime. The traditional approach is blue-green deployment: two identical environments (original "blue," new "green") where full traffic cuts over from blue to green once green ready, with blue kept available for rollback. However, blue-green doesn't suit large-scale inference well due to GPU capacity/cost — if blue uses 100 GPUs, green needs another 100 GPUs before cutover.

Instead, canary deployments (named after canaries used to detect gas in coal mines) achieve similar benefits with lower GPoverhead by catching errors before they affect large numbers of users. A canary deployment is a 4-step process: (1) build the new deployment and ready it to handle requests, (2) direct a small percentage of live traffic to it, (3) monitor and revert if there are issues, (4) gradually increase traffic while monitoring until the new deployment handles 100 percent. Canary rollouts can be done in minutes or ramped slowly for stability. With autoscaling, canary deployments don't increase cost much at scale, since reduced traffic to the production system lets it scale down replicas — but the new deployment must maintain enough active replicas throughout, or users see latency spikes as requests queue while autoscaling catches up.

Cost Estimation

Moving from consuming public API tokens to running dedicated GPU inference changes how cost must be thought about. Public API cost is simple — price per million tokens times tokens used, with some variables like cache hit/miss rates and volume discounts, but fundamentally a linear function of usage. One motivation for inference engineering investment is escaping per-token pricing to control unit economics, but the mental transition is difficult: dedicated inference cost becomes a function of many variables, which s control but makes estimation harder. Factors include: batch sizing (optimized for latency with low batch sizes, or throughput with high batch sizes), traffic patterns (consistently saturating GPUs, or capacity going spare), and sequence lengths (average and outlier input/output token counts).

Given this complexity and the input/output token price difference, it's more productive to convert token price into a total cost and compare it to dedicated cost, rather than trying to reverse-engineer a per-token price from GPU spend. Cost estimates should use a long time horizon, ideally at least a week, to smooth out usage variation. Beyond GPU costs, the engineering time spent building and maintaining inference systems should be factored in — while justified by increased reliability, security, and control, ishould be added to GPU costs to form a complete total cost of ownership (TCO) picture.

Observability

Inference is mission-critical and must be monitored accordingly, with alerting, logs, and observability built at the right level of abstraction. Inference observability should measure: total volume (requests received), request and response sizes (input/output sequence lengths), response codes (counts of 2XX/4XX/5XX), latency (time to first token, tokens per second, end-to-end latency at P50/P90/P99), replica count (active and starting instances), utilization (CPU, host memory, GPU, GPU memory), and queue depth (enqueued requests for asynchronous systems).

These metrics are interdependent — a latency spike could stem from request volume or from long input sequences, and viewing metrics together lets engineers understand not just what happened but why. When issues occur, logs (both server logs and audit logs of service changes) provide real-time diagnostic information. Observability should never be siloed — it should be deeply integrated with existing observability/alerting tooling (Grafana, Datadog, PagerDuty, Sentry) to put inference information in context with the rest of the application.

Client Code

Client code is itical but often overlooked area when optimizing for latency and scale. A call to an inference service has two sides: the client (browser, agent, or application making the request) and the server (the inference service handling it and returning results). The industry-standard client is the OpenAI SDK, supporting many compatible providers beyond OpenAI; popular frameworks like LangChain, Vercel AI SDK, LiteLLM, and LlamaIndex can also act as clients. Whether using an existing library or custom code, latency overhead or throughput bottlenecks can arise on the client side, and real-time applications may need a non-HTTP protocol like WebSockets for continuous connections. On-server inference time is only a fraction of total end-to-end latency for a request.

Client Latency Overhead

Establishing a session between client and server takes a few dozen milliseconds, depending on connection and protocol. In a high-performance system with a 300ms P95 end-to-end latency SLA, a TLS handshake alone can cost at least ten percent of the latency budget before inference even starts. Future requests from the same client should reuse existing sessions to save time — this isn't a n idea, and tools like the OpenAI SDK handle it silently, but when building custom clients for non-standard modalities, session reuse should be followed as a best practice.

Asynchronous Inference

Some systems are built for throughput rather than latency — bulk document processing and corpus embedding aren't latency-sensitive, making asynchronous jobs a good fit. Asynchronous requests use a "fire and forget" approach. Ordinary synchronous requests have a timeout (generally a few minutes) after which they fail; asynchronous jobs fix this by immediately acknowledging the request and later returning results to a webhook supplied in the original request. Asynchronous jobs still have time limits, but these are measured in hours rather than minutes. Combined with strong server-side queuing, asynchronous requests make high-throughput, latency-insensitive systems more robust and efficient.

Strming and Protocol Support

Streaming makes applications feel instant. For language models, streaming text output over HTTP suffices, but other modalities — especially live voice and video — need both input and output streams capable of carrying more data; one-time HTTP request/response cycles fit text chat but not continuous streaming. Two common bi-directional streaming protocols: WebSockets (for streaming use cases without strong schema enforcement requirements) and gRPC (for weefined service-to-service communication).

WebSockets suit unstructured, real-time data like audio, where the receiving server parses and processes it downstream; a server supports up to a fixed, developer-configurable number of concurrent WebSocket clients, and once that concurrency is reached, new connections must wait for a free slot or another replica to scale up. gRPC also enables bi-directional streaming but for structured data — requests must follow a predefined schema, removing the burden of parsing input, though this additional validaon layer makes gRPC slightly slower than WebSockets.