Archive | Inference from Transformer Models

connor-wang-r89Sdde8W3o-unsplash

Weight-Only Quantization

During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight. Quantization can also make […]

Continue Reading
chris-boland-1_TNeW5SOhk-unsplash

Fast Attention Kernels and Operator Fusion

A transformer layer is built from familiar operations: linear projections, attention, normalization, activation functions, and residual additions. A direct implementation may launch a separate accelerator kernel for every operation and write every intermediate tensor to device memory. The mathematics is correct, but the execution can spend more time moving values and launching kernels than performing […]

Continue Reading
jean-luc-benazet-VJ4N18Lq2JQ-unsplash

Floating-Point Precision and Its Effects to Model Inference

A model is a collection of numbers. The format used to store and compute with those numbers affects how much memory the model occupies, how quickly the hardware can move the numbers, and which arithmetic units can process them. Changing precision can therefore make inference faster without changing the model architecture. Lower precision is not […]

Continue Reading
petra-reid-WYvxaZGBebg-unsplash

Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient

In the previous article, we saw how a language model processes a prompt during prefill, then generates tokens one at a time during decode, and uses KV cache to avoid repeated computation. In the real world, inference servers handle hundreds or thousands of requests at the same time. How a server schedules those requests determines […]

Continue Reading
neda-astani-KWTkd7mHqKE-unsplash

From Prompt to Prediction: Understanding Prefill, Decode, and the KV Cache in LLMs

In the previous article, we saw how a language model converts logits into probabilities and samples the next token. But where do these logits come from? In this tutorial, we take a hands-on approach to understand the generation pipeline: How the prefill phase processes your entire prompt in a single parallel pass How the decode […]

Continue Reading
colton-duke-UExx0KnnkjY-unsplash

How LLMs Choose Their Words: A Practical Walk-Through of Logits, Softmax and Sampling

Large Language Models (LLMs) can produce varied, creative, and sometimes surprising outputs even when given the same prompt. This randomness is not a bug but a core feature of how the model samples its next token from a probability distribution. In this article, we break down the key sampling strategies and demonstrate how parameters such […]

Continue Reading

Machine Learning Mastery is part of Guiding Tech Media, a leading digital media publisher focused on helping people figure out technology. Visit our corporate website to learn more about our mission and team.