In this article, you will learn how to match an LLM inference optimization technique to the actual bottleneck in your workload by working through a six-question decision tree.
Topics we will cover include:
- What each major inference optimization technique, from quantization to prefill-decode disaggregation, assumes about your workload.
- How to work through a decision tree that routes you to the right technique based on your model and traffic.
- How the techniques compare on trade-offs, and which symptoms show that a chosen technique isn’t working.
Introduction
LLM inference gets slow and expensive as traffic grows. Requests queue behind long generations, GPU memory fills with cached attention state, and the hardware spends much of its time waiting on memory reads.
Inference optimization is the set of techniques that help here. Some techniques target memory, some target compute, and some target how requests are scheduled against each other. Each technique assumes something specific about your workload, so the right choice depends on which assumptions hold for yours. This article covers:
- What each major inference optimization technique assumes about your workload
- A decision tree that routes you to a technique
- How the techniques compare on latency, throughput, and implementation cost
- The symptoms that show a technique isn’t working
We start with what each technique assumes.
What Each Inference Optimization Technique Assumes
LLM inference has many optimization techniques, but each one addresses a different type of bottleneck. Before using one, it helps to understand the workload it is designed for.
| Technique | What It Assumes |
|---|---|
| Quantization / sparsity / distillation | The model itself is too large for the hardware or the latency budget, and shrinking it is worth some accuracy risk |
| Prefix caching | Many requests share an identical starting sequence of tokens, so recomputing that prefix is wasted work |
| Continuous (in-flight) batching | Requests arrive concurrently with unpredictable output lengths, and waiting for the slowest one in a batch wastes the GPU |
| PagedAttention | KV cache memory is fragmented or over-provisioned, capping batch size below what the hardware could otherwise hold |
| Speculative decoding | Decode is the bottleneck, tokens are generated one at a time, and a cheaper draft model can guess several of them correctly |
| Prefill-decode disaggregation | Prefill and decode are contending for the same GPU, and that contention is hurting both time-to-first-token and inter-token latency at once |
| Tensor / pipeline parallelism | The model doesn’t fit in a single device’s memory, or per-device compute is the ceiling on throughput |
| KV cache offloading | Useful cache entries are being evicted too early because GPU memory can’t hold everything you’d like to keep |
| Offline batch inference | Requests don’t need an interactive response, and total throughput outweighs per-request latency |
Choose the technique based on the bottleneck in your workload to avoid unnecessary complexity.
Working Through a Decision Tree to Choose a Technique
The tree below has six branching questions. Each one narrows the field based on a specific property of your model or your traffic. Every question ends the same way: a direct answer if the condition holds, and a pointer to the next question if it doesn’t.
Question 1: Does the Model Fit Comfortably on Your Target Hardware?
Before moving on to the serving stack, check whether the model itself is the problem. A model that barely fits in GPU memory leaves almost no room for KV cache, which caps your batch size no matter how good your scheduler is.
If the model is too large, or inference is compute-bound even at batch size one, start with model-level compression:
- Quantization reduces the precision of weights and activations.
- Structured sparsity removes less important weights to reduce computation.
- Distillation trains a smaller model to mimic a larger model.
These techniques are mostly independent of the serving stack, and you may want to consider them first because they determine how much hardware capacity remains.
If the model already fits with headroom to spare, move to Question 2.
Question 2: Do Your Requests Share Repeated Prompt Structure?
Chat applications resend the full conversation history on every turn. Agent frameworks reuse the same system prompt across thousands of calls. Retrieval-augmented generation (RAG) pipelines often prepend the same instructions to different retrieved contexts.
If a meaningful share of your traffic has an identical prefix, whether that’s a system prompt or a growing chat history, prefix caching is the highest-leverage technique available. It skips recomputing the KV cache for tokens the model has already processed, and the gains compound as conversations get longer.
⚠️ Prefix caching only works when the shared prefix matches exactly, character for character. Put stable, reusable content first and variable content at the end of the prompt.
If your prompts are mostly unique with little shared structure, move to Question 3.
Question 3: Are Requests Arriving Concurrently With Variable Output Lengths?
A support chatbot that returns a one-sentence answer and a code assistant that returns four hundred lines rarely finish at the same time, even if they started together. Under naive batching, the whole batch waits on the slowest request.
If your traffic mixes short and long generations and arrives in overlapping bursts, continuous batching fixes the idle time this creates. Finished sequences leave the batch immediately, allowing new requests to start without waiting for a batch boundary. Most serving engines enable this by default.
If your traffic is already well-batched and GPU idle time is not a problem, move to Question 4.
Question 4: Is KV Cache Fragmentation Limiting Your Batch Size?
Some engines reserve a fixed, contiguous block of memory per request sized for the maximum possible sequence length, even when most requests are much shorter. That reserved space sits unused for the life of the request and blocks other requests from using it.
If KV cache is wasting memory or limiting batch size, PagedAttention can help. It splits the cache into fixed-size pages, allowing memory to be allocated in smaller chunks instead of large contiguous blocks. Most modern serving engines already support it, so check whether it is enabled before adding new configuration.
If memory is not the constraint, move to Question 5.
Question 5: Is Per-Token Decode Latency the Metric You’re Focusing On?
If your goal is faster token streaming for each user rather than higher overall throughput, consider speculative decoding.
A smaller draft model predicts several tokens ahead, and the main model verifies them in a single pass. When the draft model predicts correctly, multiple sequential decode steps can be completed together, reducing inter-token latency.
The trade-off is the extra model and verification overhead. If you don’t have a suitable draft model or its predictions are accepted too rarely, the added complexity may not pay off. In that case, move to Question 6.
Question 6: Are Prefill and Decode Contending for the Same GPU at Scale?
This is mainly a production-scale concern because it changes the serving architecture.
If long prefills are delaying decode requests and hurting both time-to-first-token and inter-token latency, prefill-decode disaggregation can separate the two phases onto dedicated hardware. Prefill nodes process prompts, then pass the KV cache to decode nodes optimized for token generation.
This adds KV-cache transfer and routing overhead, so it makes sense only when the traffic justifies that complexity. For smaller workloads or non-interactive requests, offline batch inference is often simpler. If none of these conditions apply, start with continuous batching and prefix caching.
Mapping the Decision Tree to an Optimization Technique
Working through the tree lands you on one of the techniques in the following table:
| Technique | Best For | Trade-off |
|---|---|---|
| Quantization / sparsity / distillation | Models too large for the target hardware or latency budget | Some accuracy loss, and quantized activations are harder to get right than quantized weights |
| Prefix caching | Chat, agents, and RAG workloads with repeated prompt structure | Breaks on any character-level mismatch in the shared prefix |
| Continuous batching | Concurrent traffic with variable output lengths | Minimal downside; mostly a matter of whether it’s enabled |
| PagedAttention | Workloads bottlenecked by KV cache fragmentation | Adds a page table layer to attention computation |
| Speculative decoding | Latency-sensitive single-user generation | Needs a draft model with a reasonably high acceptance rate to pay off |
| Prefill-decode disaggregation | Large-scale traffic where prefill and decode contend for the same GPU | Network cost of moving the KV cache between nodes, plus routing complexity |
| Tensor / pipeline parallelism | Models that don’t fit on a single device | Pipeline bubbles and inter-device communication overhead |
| KV cache offloading | Long-context or high-concurrency workloads running out of GPU memory for cache | Fetching offloaded cache back adds latency |
| Offline batch inference | Non-interactive workloads prioritizing total throughput | Not usable for anything with a strict response-time requirement |
Recognizing Common Inference Optimization Pitfalls
A technique that you think will work may behave quite differently in practice. The signals below tell you when that is happening and what to check first.
| Symptom | Likely Cause | Fix |
|---|---|---|
| Prefix cache hit rate is near zero despite repeated prompts | Non-deterministic serialization, or dynamic content placed early in the prompt | Front-load static content and fix key ordering in serialized context |
| Batch size won’t grow even with memory to spare | PagedAttention isn’t enabled, or page size is misconfigured | Check the serving engine’s batching and cache flags directly rather than assuming defaults |
| Speculative decoding is slower than plain decoding | Draft model’s acceptance rate is too low for the target model | Pick a draft model closer in behavior to the target, or drop speculative decoding for this workload |
| Disaggregation made latency worse, not better | Workload is too small, or most requests are short and cache-heavy | Fall back to local prefill and revisit disaggregation only past a defined traffic threshold |
| Throughput gains from quantization didn’t show up | Only weights were quantized while activations stayed at full precision, and the GPU still converts back before compute | Profile where time is spent before assuming quantization alone will fix throughput |
If a technique consistently underperforms, check its underlying assumption before adding more complexity.
Wrapping Up
Choosing an inference optimization technique starts with identifying your bottleneck. Model size, prompt overlap, traffic concurrency, memory fragmentation, and the balance between throughput and per-user latency each point toward a different answer, and most systems will need a combination of a few of these rather than relying on just one.
If you’re interested in learning more, you may find The Roadmap to Mastering LLM Inference Optimization useful.
Happy learning!
No comments yet.