Measuring Performance of Transformer Inference

When you optimize the inference performance of an LLM, you need to know how to measure it. Without measurement, it is easy to make a model more complicated without making it faster, or to improve throughput while making user-visible latency worse.

An LLM service has several kinds of performance. A user cares about how long it takes to see the first token and how quickly the rest of the answer streams. An operator cares about how many requests the hardware can serve, how much memory is used, and how much each generated token costs. A researcher may care about whether an optimization changes the model’s output quality.

In this chapter, you will learn about:

• Latency and throughput metrics
• Time to first token and time per output token
• Measuring CPU and GPU inference
• Using CUDA events
• Benchmarking multiple requests
• Thinking about multiple GPUs and multiple machines

Let’s get started.

Measuring Performance of Transformer Inference
Photo by Tomas Anton Escobar. Some rights reserved.

Overview

This chapter is divided into eight parts; they are:

• Metrics for LLM Inference
• Measuring a Single Request
• Warmup and Synchronization
• Measuring GPU Work with CUDA Events
• Measuring Memory Usage
• Measuring Concurrent Requests
• Multiple GPUs and Multiple Machines
• Cost per Token

Metrics for LLM Inference

The most common inference metrics are:

• Latency: How long a request takes from start to finish.
• Time to first token (TTFT): How long the user waits before the first output token appears.
• Time per output token (TPOT): The average time between generated tokens after the first token.
• Throughput: How many tokens or requests are processed per second.
• Memory usage: How much CPU memory or GPU memory is used.
• Utilization: How busy the accelerator is during the benchmark.
• Cost per token: The hardware or service cost divided by the number of tokens processed.

For LLMs, a single latency number is usually not enough. Consider two requests:

• Request A: 2,000 prompt tokens and 20 output tokens
• Request B: 20 prompt tokens and 2,000 output tokens

Request A stresses prefill. Request B stresses decode. They may have the same total number of tokens, but they have different performance profiles. This is why you should record prompt tokens and output tokens separately.

Tail latency also matters. If most requests complete in one second but a few take ten seconds, users will notice. Report high-percentile latencies such as p90, p95, and p99 in addition to the mean or median. The high percentiles describe the worst cases better. You can easily find these percentiles from a list of values using NumPy:

These numbers are simple, but they prevent a common mistake: optimizing the average while making the worst cases slower.

Measuring a Single Request

The simplest measurement uses time.perf_counter(). It is a built-in high-resolution wall-clock timer suitable for measuring elapsed time in Python. It is more accurate than time.time().

The following example measures prefill and decode separately for a Hugging Face causal language model:

This function does not use the model’s generate() method. That is intentional. The goal is to expose prefill and decode so they can be measured separately. The number of output_tokens includes the tokens generated by both prefill and decode. The seconds_per_output_token is the average time per output token in the decode phase.

There are two details to notice:

• use_cache=True asks the model to return the KV cache.
• During decode, the model receives only next_token, not the whole sequence.

This is the same idea as Chapter 1, but using a library model.

Warmup and Synchronization

When you measure performance, note that some one-time costs should not dominate the result. In Python, import of a module can be slow but subsequent import of the same module is instant. Similarly, the first execution of some code may be slower than subsequent executions due to initialization of data structures or warmup of caches. You want to measure steady-state work, not that setup overhead.

Therefore, benchmarks should include warmup. The first few iterations may be slower for various reasons. Instead of measuring the total time and dividing by the number of iterations, you should measure the time for each iteration and analyze the steady-state ones. For example, if you use the model to generate multiple tokens, you will likely put the generation in a loop. Measure each iteration as follows, then ignore the first few results:

If you use GPU to run your LLM inference, you also need to initialize the kernels when you first run them. Unfortunately, many GPU operations are asynchronous. That is, while you launched an operation on GPU, Python may continue with your code immediately while the GPU is still working. Therefore, a naive approach to measure the time would be incorrect. Instead, you should use torch.cuda.synchronize() to wait for the GPU to finish the operation before you stop the timer:

This gives a wall-clock measurement that includes the actual GPU work. For accurate prefill and per-token decode timings on GPU, call sync_if_needed(device) after each timed model(...) call in measure_one_request(), not only once at the end of the request.

Measuring GPU Work with CUDA Events

CUDA events measure elapsed time the GPU spent executing kernels, not the end-to-end user latency. This time does not include any Python overhead. Below is an example of how to use CUDA events to measure the time:

You can use it to measure one forward pass:

CUDA event timing and wall-clock timing answer different questions:

• Wall-clock timing measures what the application experiences.
• CUDA event timing measures how long the GPU work took.

For an inference service, wall-clock timing is usually the primary metric because users experience queues, tokenization, scheduling, network overhead, and streaming. CUDA events are useful when you are optimizing kernels or comparing model execution paths.

For deeper GPU profiling, use tools such as PyTorch Profiler, Nsight Systems, Nsight Compute, or CUPTI-based monitoring. These tools can report kernel timelines, memory copies, GPU utilization, and operator-level breakdowns. They are more complex than a timer, but they are necessary when a simple benchmark says the model is slow and you need to know why.

Measuring Memory Usage

Memory is a different dimension to measure because it limits not speed for one user so much as how many users your system can serve. Usually the GPU memory is the bottleneck. In PyTorch, you can report allocated and reserved memory like the following:

The allocated value is memory used by tensors. The reserved value is memory held by PyTorch’s caching allocator. The maximum allocated value is often the most useful number for capacity planning.

The allocated and reserved memory are real-time snapshots but the max allocated value is a peak over time. For accurate measurement, you should reset the peak statistic before a benchmark:

Memory should be measured together with tokens. A run with a longer prompt or more generated tokens will naturally use more KV cache memory.

Measuring Concurrent Requests

To create a server that runs a language model, evaluate the system by how many requests you can serve per second. Throughput depends on both how fast you fulfill one request and how many requests you can run concurrently, though concurrency does not scale linearly under contention.

Production systems should handle multiple users, and the scheduler may batch their work together. The following simple benchmark runs several requests concurrently using Python threads. This does not implement continuous batching. It only measures how a model wrapper behaves when several callers use it at the same time.

This benchmark is for illustration only. It is not a replacement for a real serving benchmark. It does not model HTTP overhead, streaming, request queues, batching, cancellation, or cache eviction. But it is a useful next step after a single-request benchmark where you can run the model in parallel and observe the per-request latency. The Python GIL (Global Interpreter Lock) is usually not the main concern here because heavyweight model execution is often offloaded to compiled code. Concurrent use of the same model or tensors from multiple threads is unsafe without synchronization, so share one model on CUDA only with a lock or a single worker thread.

When benchmarking a real server, record at least:

• Number of concurrent users
• Prompt token distribution
• Output token distribution
• Request rate
• TTFT (Time to first token) percentiles
• Inter-token latency percentiles
• Total tokens per second
• Error rate and timeout rate

The distributions matter. A benchmark with all prompts at exactly 128 tokens and all outputs at exactly 128 tokens is easy to compare, but it may not represent your application.

Multiple GPUs and Multiple Machines

Multiple GPUs can be used for inference in several different ways. Different approaches can drastically change the performance of your system.

The simplest approach is replication. You load one copy of the model on each GPU and route different requests to different replicas. This increases throughput and is easy to reason about, but each GPU must have enough memory for the full model and its KV cache.

Another approach is to split one model across multiple GPUs. Tensor parallelism splits weight matrices across devices. Pipeline parallelism places different layers on different devices. Context parallelism partitions sequence work. Expert parallelism is used for mixture-of-experts models. These techniques allow larger models to run, but they introduce communication overhead and can increase latency.

Multiple machines add another layer. A system may use many replicas across machines for high request volume. It may also split a single large model across machines, but this is more difficult because network communication is slower than communication within one machine. For low-latency serving, crossing machine boundaries inside one forward pass should be treated as expensive.

Measuring performance of a system with multiple GPUs or multiple machines adds a new dimension of communication and synchronization overhead. Before choosing a multi-GPU or multi-machine design, answer these questions:

• Are you serving one large model or many smaller models?
• Are you limited by model weight memory or KV cache memory?
• Do you need lower latency, higher throughput, or both?
• Are requests independent, or do they share long prompt prefixes?
• Can one GPU hold the model, or must the model be partitioned?

These questions matter because the best design depends on the bottleneck. Adding GPUs does not automatically make a single request faster. It may help throughput through replication, or it may make a larger model possible through partitioning. The benchmark should show which effect you are getting.

Cost per Token

Cost is a performance metric. A faster system that uses much more expensive hardware may not be better for an application.

A simple cost estimate is:

If a GPU instance costs 3 dollars per hour and the service generates 1,000 output tokens per second:

This is less than one millionth of a dollar per output token for hardware alone. A real calculation may also include idle capacity, storage, networking, engineering time, orchestration overhead, and failed requests.

Cost should be compared with quality. Quantization, smaller models, and routing can reduce cost, but they may change model behavior. An efficient inference system is not merely the fastest one. It is the one that meets quality and reliability requirements at the lowest practical cost.

Further Reading

Below are some resources you may find useful:

• Little’s law, on Wikipedia.
  This is a useful queueing-theory result for relating average concurrency, arrival rate, and response time. It is a helpful mental model when reasoning about request rate, latency, and the number of in-flight inference requests.
• Metrics, in NVIDIA NIM LLMs Benchmarking.
  This page defines common LLM inference metrics such as time to first token, end-to-end latency, inter-token latency, tokens per second, and requests per second.
• MLPerf Inference, by MLCommons.
  This is a widely used benchmark suite for measuring inference performance across deployment scenarios. It is not limited to LLMs, but it provides useful discipline around repeatable benchmarking and reporting.
• torch.profiler, in the PyTorch documentation.
  This is the main PyTorch profiling interface for collecting CPU and accelerator activity, operator timings, memory information, tensor shapes, and traces that can be inspected later.
• NVIDIA Nsight Systems User Guide, by NVIDIA.
  Nsight Systems is useful when wall-clock timers are not enough and you need a timeline of CUDA API calls, GPU kernels, memory copies, CPU work, and synchronization.
• Taming the Titans: A Survey of Efficient LLM Inference Serving, by Zhen et al.
  This survey gives a broader view of LLM inference serving, including request scheduling, model placement, storage management, disaggregation, load balancing, and cluster-level serving issues.

Summary

In this chapter, you learned how to measure LLM inference performance. You saw why prefill and decode should be measured separately, how to use wall-clock timers and CUDA events, how to record memory usage, and how to report latency percentiles. You also learned that multiple GPUs can mean replication for more throughput or partitioning for larger models, and that the benchmark should make this distinction clear.

In the next part of the book, you will begin studying techniques for making one model faster, starting with floating-point precision.