Introduction

LLM inference at production scale demands a layered optimization strategy. Naive serving frameworks leave 70-80% of GPU capacity idle. By combining continuous batching, PagedAttention KV cache management, quantization, and speculative decoding, teams can reduce cost-per-token by 5-10x while simultaneously improving latency.

A typical naive implementation — one request per GPU at a time — achieves 15-25% GPU utilization. The GPU spends most of its time waiting for memory transfers and sequential token generation rather than performing parallel matrix multiplications. At $2-4/hour for an H100, this translates directly to wasted capital.

Modern inference optimization addresses this through four complementary techniques: continuous batching to maximize GPU occupancy, PagedAttention to eliminate KV cache fragmentation, quantization to reduce memory footprint, and speculative decoding to reduce per-token latency. Together, these techniques can reduce cost-per-token by 5-10x compared to naive implementations.

Continuous batching

Traditional static batching waits for a batch of requests to arrive, processes them together, and returns all results before accepting new requests. This approach fails for LLMs because requests have variable output lengths — a short response finishes while others are still generating, leaving GPU capacity idle.

Continuous batching (also called iteration-level scheduling) solves this by inserting new requests into the batch at each token generation step. When a sequence completes, its slot is immediately filled with a waiting request. This keeps the GPU fully occupied regardless of output length variance.

The key implementation detail is the iteration-level scheduler. At each forward pass, the scheduler decides which sequences to include in the batch, respecting memory constraints and priority policies. Sequences that have finished generation are evicted and replaced with queued requests.

PagedAttention

The KV cache stores key and value tensors for all previously generated tokens, enabling efficient attention computation without recomputing past context. In naive implementations, each sequence is allocated a contiguous block of GPU memory for its maximum possible KV cache size. This causes severe memory fragmentation — memory is reserved but unused.

PagedAttention, introduced in the vLLM paper, applies virtual memory concepts from operating systems to KV cache management. Memory is divided into fixed-size pages (typically 16 tokens per page), and sequences are allocated pages on demand. Non-contiguous physical pages are mapped to contiguous logical addresses via a block table.

The result is near-zero memory waste. Traditional systems waste 60-80% of KV cache memory to fragmentation and over-reservation. PagedAttention reduces this to under 4%, enabling significantly larger batch sizes on the same hardware.

Quantization strategies

Quantization reduces the numerical precision of model weights and activations, decreasing memory footprint and increasing throughput. The tradeoff is potential quality degradation, though modern quantization techniques minimize this impact.

For production serving, INT8 quantization (specifically W8A8 — 8-bit weights and activations) is the recommended starting point. It halves memory requirements with minimal quality impact, enabling larger batch sizes and higher throughput. FP8 on H100 hardware offers similar memory savings with even lower quality impact.

Speculative decoding

Autoregressive generation is inherently sequential — each token requires a full forward pass through the model. For large models, this creates a latency floor that cannot be reduced by adding more GPUs. Speculative decoding breaks this constraint by using a small draft model to propose multiple tokens simultaneously, then verifying them in parallel with the target model.

The draft model (typically 1-7B parameters) generates a sequence of candidate tokens. The target model then verifies all candidates in a single forward pass using parallel attention. Accepted tokens are kept; rejected tokens cause the sequence to revert to the last accepted position.

In practice, speculative decoding achieves 2-3x latency reduction for tasks where the draft model has high acceptance rates (typically 70-85%). Tasks with predictable outputs (code completion, structured generation) benefit most.

Inference stack architecture

A production inference stack integrates all optimization techniques into a coherent serving pipeline. Each layer addresses a specific bottleneck in the serving process.

Framework comparison

Choosing the right serving framework is one of the highest-impact infrastructure decisions. Each framework makes different tradeoffs between throughput, latency, ease of deployment, and operational complexity.

For most production deployments, vLLM is the recommended starting point. It implements all major optimizations (continuous batching, PagedAttention, speculative decoding), has an active open-source community, and provides an OpenAI-compatible API. TensorRT-LLM offers higher peak throughput on NVIDIA hardware but requires more operational expertise.

Optimization ROI calculator

Estimate the financial impact of inference optimization for your workload. Adjust the sliders to match your current usage and cost profile.

Production configuration guide

Translating optimization techniques into production configuration requires understanding the interaction between batch size, KV cache allocation, and GPU memory. The following guidelines apply to vLLM deployments on H100 hardware.

Tensor parallelism should match the number of GPUs per node. For a 70B model, 4-way tensor parallelism across 4 H100s is optimal. Pipeline parallelism adds latency and should only be used when the model does not fit within a single node's GPU memory.

Enable prefix caching for applications with shared system prompts. For a 2,000-token system prompt, prefix caching eliminates 2,000 tokens of computation per request — a 50-80% reduction for short user messages.

Frequently asked questions