Measuring Performance of Transformer Inference

When you optimize the inference performance of an LLM, you need to know how to measure it. Without measurement, it is easy to make a model more complicated without making it faster, or to improve throughput while making user-visible latency worse.

An LLM service has several kinds of performance. A user cares about how long it takes to see the first token and how quickly the rest of the answer streams. An operator cares about how many requests the hardware can serve, how much memory is used, and how much each generated token costs. A researcher may care about whether an optimization changes the model’s output quality.

In this post, you will learn about:

  • How to measure latency, throughput, memory use, and cost
  • How warmup and GPU synchronization affect benchmark results
  • How to compare single requests with concurrent serving workloads

Let’s get started.

Measuring Performance of Transformer Inference
Photo by Tomas Anton Escobar. Some rights reserved.

Overview

This post is divided into eight parts; they are:

  • Metrics for LLM Inference
  • Measuring a Single Request
  • Warmup and Synchronization
  • Measuring GPU Work with CUDA Events
  • Measuring Memory Usage
  • Measuring Concurrent Requests
  • Multiple GPUs and Multiple Machines
  • Cost per Token

Metrics for LLM Inference

The most common inference metrics are:

  • Latency: How long a request takes from start to finish.
  • Time to first token (TTFT): How long the user waits before the first output token appears.
  • Time per output token (TPOT): The average time between generated tokens after the first token.
  • Throughput: How many tokens or requests are processed per second.
  • Memory usage: How much CPU memory or GPU memory is used.
  • Utilization: How busy the accelerator is during the benchmark.
  • Cost per token: The hardware or service cost divided by the number of tokens processed.

For LLMs, a single latency number is usually not enough. Consider two requests:

  • Request A: 2,000 prompt tokens and 20 output tokens
  • Request B: 20 prompt tokens and 2,000 output tokens

Request A stresses prefill. Request B stresses decode. They may have the same total number of tokens, but they have different performance profiles. This is why you should record prompt tokens and output tokens separately.

Tail latency also matters. If most requests complete in one second but a few take ten seconds, users will notice. Report high-percentile latencies such as p90, p95, and p99 in addition to the mean or median. The high percentiles describe the worst cases better. You can easily find these percentiles from a list of values using NumPy:

These numbers are simple, but they prevent a common mistake: optimizing the average while making the worst cases slower.

Measuring a Single Request

The simplest measurement uses time.perf_counter(). It is a built-in high-resolution wall-clock timer suitable for measuring elapsed time in Python. It is more accurate than time.time().

The following example measures prefill and decode separately for a Hugging Face causal language model:

This function exposes prefill and decode instead of calling generate(). The field output_tokens includes the token selected after prefill and later decode tokens. The metric decode_seconds_per_token excludes the first token and is None when there is no decode step. The metric first_token_compute_seconds starts after tokenization and excludes queueing, transport, and streaming. It is not user-visible TTFT. Timestamp request arrival and the first token sent to the client to measure TTFT. There are two details to notice:

  • The setting use_cache=True asks the model to return the KV cache.
  • During decode, the model receives only next_token, not the whole sequence.

This is the same idea as a previous post, but using a library model.

Warmup and Synchronization

When you measure performance, one-time costs should not dominate the result. Imports, kernel initialization, and graph compilation can make the first call slower than later calls. Warm up the path you intend to measure, then report steady-state work separately from startup time.

The following block reuses summarize(), measure_one_request(), model, and tokenizer from the earlier blocks. It runs complete requests and excludes the warmup requests from the summary:

GPU operations can be asynchronous: Python may continue while the GPU still works. Synchronize before stopping a wall-clock timer. This standalone example times one warmed-up matrix multiplication and also runs on CPU:

This gives a wall-clock measurement that includes the actual GPU work. The complete measure_one_request() example already synchronizes around each timed phase. Synchronizing every token adds host-side overhead, so these phase timings describe this educational loop rather than an optimized serving engine’s throughput.

Measuring GPU Work with CUDA Events

CUDA events measure elapsed time between two markers on a CUDA stream. They exclude tokenization and most application overhead, but a gap between queued GPU operations can still contribute to the interval. They are not the same as the sum of kernel durations or end-to-end user latency. This standalone example measures the same kind of matrix multiplication as the previous block:

CUDA event timing and wall-clock timing answer different questions:

  • Wall-clock timing measures what the application experiences.
  • CUDA event timing measures the interval between the recorded stream events.

For an inference service, wall-clock timing is usually the primary metric because users experience queues, tokenization, scheduling, network overhead, and streaming. CUDA events are useful when you are optimizing kernels or comparing model execution paths.

For deeper GPU profiling, use tools such as PyTorch Profiler, Nsight Systems, Nsight Compute, or CUPTI-based monitoring. These tools can report kernel timelines, memory copies, GPU utilization, and operator-level breakdowns. They are more complex than a timer, but they are necessary when a simple benchmark says the model is slow and you need to know why.

For example, collect an operator trace for a CUDA matrix multiplication:

Open the trace in a timeline viewer to inspect kernels and copies. CUDA events and this trace describe execution time; neither alone is a reliable measure of sustained GPU utilization. For utilization, sample device-level metrics such as SM activity and memory bandwidth with Nsight or the deployment monitoring stack while a representative workload runs.

Measuring Memory Usage

Memory also limits how many users a system can serve. GPU memory is often the constraint. This example allocates a tensor and reports PyTorch’s allocated and reserved CUDA memory:

The allocated value is memory used by tensors. The reserved value is memory held by PyTorch’s caching allocator. The maximum allocated value shows the peak PyTorch tensor allocation during the measured interval. Capacity planning also needs reserved memory, weights, external allocations, and headroom; a serving engine may reserve a KV pool before requests arrive.

Allocated and reserved memory are current snapshots, while maximum allocated memory is a peak. The following block reuses gpu_memory_summary() from above and resets the peak before an example allocation:

Memory should be measured together with tokens. A run with a longer prompt or more generated tokens will naturally use more KV cache memory.

Measuring Concurrent Requests

Evaluate a language-model server by how many requests it can serve per second. Throughput depends on both how fast you fulfill one request and how many requests you can run concurrently, though concurrency does not scale linearly under contention.

Production systems should handle multiple users, and the scheduler may batch their work together. The following simple benchmark runs several requests concurrently using Python threads. This does not implement continuous batching. It only measures how a model wrapper behaves when several callers use it at the same time.

This benchmark is only a threaded workload probe. It does not model HTTP, streaming, request queues, continuous batching, cancellation, or cache eviction. Concurrent read-only inference on one model can work, but separate calls compete for CPU and accelerator resources, and not every model or cache implementation is thread-safe. If thread safety is uncertain, run requests through one worker or a serving engine. Do not interpret this probe as a continuous-batching benchmark.

When benchmarking a real server, record at least:

  • Number of concurrent users
  • Prompt token distribution
  • Output token distribution
  • Request rate
  • TTFT (Time to first token) percentiles
  • Inter-token latency percentiles
  • Total tokens per second
  • Error rate and timeout rate

The distributions matter. A benchmark with all prompts at exactly 128 tokens and all outputs at exactly 128 tokens is easy to compare, but it may not represent your application.

Multiple GPUs and Multiple Machines

Multiple GPUs can be used for inference in several different ways. Different approaches can drastically change the performance of your system.

The simplest approach is replication. You load one copy of the model on each GPU and route different requests to different replicas. This increases throughput and is easy to reason about, but each GPU must have enough memory for the full model and its KV cache.

Another approach is to split one model across multiple GPUs. Tensor parallelism splits weight matrices across devices. Pipeline parallelism places different layers on different devices. Context parallelism partitions sequence work. Expert parallelism is used for mixture-of-experts models. These techniques allow larger models to run, but they introduce communication overhead and can increase latency.

Multiple machines add another layer. A system may use many replicas across machines for high request volume. It may also split a single large model across machines, but this is more difficult because network communication is slower than communication within one machine. For low-latency serving, crossing machine boundaries inside one forward pass should be treated as expensive.

Measuring performance of a system with multiple GPUs or multiple machines adds a new dimension of communication and synchronization overhead. Before choosing a multi-GPU or multi-machine design, answer these questions:

  • Are you serving one large model or many smaller models?
  • Are you limited by model weight memory or KV cache memory?
  • Do you need lower latency, higher throughput, or both?
  • Are requests independent, or do they share long prompt prefixes?
  • Can one GPU hold the model, or must the model be partitioned?

These questions matter because the best design depends on the bottleneck. Adding GPUs does not automatically make a single request faster. It may help throughput through replication, or it may make a larger model possible through partitioning. The benchmark should show which effect you are getting.

Cost per Token

Cost is a performance metric. A faster system that uses much more expensive hardware may not be better for an application.

A simple cost estimate is:
$$
\text{cost per output token} = \frac{\text{hardware cost per second}}{\text{output tokens per second}}
$$

If a GPU instance costs $3.00 per hour and the service generates 1,000 output tokens per second:

$$
\begin{aligned}
\text{hardware cost per second} &= \frac{\$3.00}{3600 \text{ seconds}} \\
&= \$0.000833 \\
\text{cost per output token} &= \frac{\$0.000833}{1000 \text{ output tokens}} \\
&= \$0.000000833
\end{aligned}
$$

This is less than one millionth of a dollar per output token for hardware alone. A real calculation may also include idle capacity, storage, networking, engineering time, orchestration overhead, and failed requests.

Cost should be compared with quality. Quantization, smaller models, and routing can reduce cost, but they may change model behavior. An efficient inference system is not merely the fastest one. It is the one that meets quality and reliability requirements at the lowest practical cost.

Further Reading

Below are some resources you may find useful:

  • Little’s law, on Wikipedia.
    This is a useful queueing-theory result for relating average concurrency, arrival rate, and response time. It is a helpful mental model when reasoning about request rate, latency, and the number of in-flight inference requests.
  • Metrics, in NVIDIA NIM LLMs Benchmarking.
    This page defines common LLM inference metrics such as time to first token, end-to-end latency, inter-token latency, tokens per second, and requests per second.
  • MLPerf Inference, by MLCommons.
    This is a widely used benchmark suite for measuring inference performance across deployment scenarios. It is not limited to LLMs, but it provides useful discipline around repeatable benchmarking and reporting.
  • torch.profiler, in the PyTorch documentation.
    This is the main PyTorch profiling interface for collecting CPU and accelerator activity, operator timings, memory information, tensor shapes, and traces that can be inspected later.
  • NVIDIA Nsight Systems User Guide, by NVIDIA.
    Nsight Systems is useful when wall-clock timers are not enough and you need a timeline of CUDA API calls, GPU kernels, memory copies, CPU work, and synchronization.
  • Taming the Titans: A Survey of Efficient LLM Inference Serving, by Zhen et al.
    This survey gives a broader view of LLM inference serving, including request scheduling, model placement, storage management, disaggregation, load balancing, and cluster-level serving issues.

Summary

In this post, you learned how to measure LLM inference performance. You saw why prefill and decode should be measured separately, how to use wall-clock timers and CUDA events, how to record memory usage, and how to report latency percentiles. You also learned that multiple GPUs can mean replication for more throughput or partitioning for larger models, and that the benchmark should make this distinction clear.

No comments yet.

Leave a Reply

Machine Learning Mastery is part of Guiding Tech Media, a leading digital media publisher focused on helping people figure out technology. Visit our corporate website to learn more about our mission and team.