Weight-Only Quantization

During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight.

Quantization can also make computation faster. Many AI GPUs can perform low-precision matrix operations faster than FP16 operations. The actual speedup depends on the hardware, kernels, and model shapes.

Weight-only quantization stores selected weights as integers while keeping activations in FP16 or BF16. This post begins with the asymmetric affine mapping, then shows how to quantize and run models with PyTorch and other inference tools.

In this post, you will learn about:

• How weight-only quantization stores and reconstructs model weights
• How post-training methods and GGUF conversion produce deployable artifacts
• How to check the memory, speed, and quality effects of quantization

Let’s get started.

Weight-Only Quantization
Photo by Connor Wang. Some rights reserved.

Overview

This post is divided into five parts; they are:

• Asymmetric Affine Quantization
• Post-Training Quantization with PyTorch
• GPTQ and AWQ
• Quantizing a Model to GGUF
• Evaluating Weight-Only Quantization

Note: The code examples in this post use TorchAO and require a CUDA-enabled PyTorch installation. You can install the dependencies with:

python -m pip install torchao transformers accelerate

The GPTQ and AWQ sections have their own installation steps. GGUF conversion requires a built llama.cpp checkout.

Asymmetric Affine Quantization

Integer quantization maps a range of real values onto a finite set of integer codes. An asymmetric affine quantizer uses a scale $s$ and a zero point $z$:
$$
q = \operatorname{clamp}\Big(\operatorname{round}\big(\frac{x}{s}\big) + z, q_{\min}, q_{\max}\Big)
$$
Dequantization reconstructs an approximate floating-point value:
$$
\hat{x} = s(q-z)
$$
For an unsigned $b$-bit range, $q_{\min}=0$ and $q_{\max}=2^b-1$. Given the minimum and maximum values to represent, one common choice is:
$$
\begin{aligned}
s &= \frac{x_{\max}-x_{\min}}{q_{\max}-q_{\min}} \\
z &= \operatorname{clamp}\Big(\operatorname{round}\big(q_{\min}-\frac{x_{\min}}{s}\big),q_{\min},q_{\max}\Big)
\end{aligned}
$$
The scale determines the distance between adjacent representable values. The zero point is the integer code for real zero. Including zero in the real range ensures that it can be represented exactly.

The example demonstrates the numerical mapping, not efficient storage. A uint8 tensor always uses one byte per element, even if bits is less than eight. A deployment format must pack sub-byte values to obtain the expected storage reduction. This example uses one scale and zero point for the whole tensor. Per-channel and per-group quantization often preserve LLM weight quality better because each channel or group gets its own range.

For a linear layer $\mathbf{y}=\mathbf{x}\mathbf{W}^\top$, weight-only quantization stores an integer matrix $\mathbf{Q}$ plus scales and zero points instead of $\mathbf{W}$:
$$
\mathbf{y} \approx \mathbf{x}\big(s(\mathbf{Q}-z)\big)^\top
$$
The input remains floating point. An efficient kernel would use the quantized weights directly instead of reconstructing the floating-point copy of $\mathbf{W}$ like the equation shows.

The asymmetric affine mapping is the explanatory model used in this post. Production kernels and checkpoint formats may use a fixed zero point or another encoding. The runtime must support the exact quantization scheme and packed layout stored in the checkpoint.

Post-Training Quantization with PyTorch

PyTorch’s current quantization development is centered in TorchAO. Its quantize_() function replaces the weights of matching torch.nn.Linear modules in place. Applying it to an evaluated, trained model is post-training quantization: no training step is performed.

The following example requests asymmetric, per-channel INT8 weight-only quantization. The activations remain BF16:

PerAxis(0) assigns separate quantization parameters to the rows of each linear weight matrix. IntxWeightOnlyConfig also supports per-group granularity. Its available packing formats and optimized kernels depend on the TorchAO version and target device.

TorchAO also provides hardware-oriented presets such as Int8WeightOnlyConfig and Int4WeightOnlyConfig. These presets may use a different mapping from the explicit asymmetric example. You should use the configuration documented for the target hardware, then benchmark the complete model. A mathematically valid configuration is not necessarily the fastest one.

The transformers library can apply a TorchAO configuration while loading a model. For example:

This is calibration-free weight-only PTQ. The explicit HQQ parameter choice and tiled packing avoid an optional MSLK kernel dependency in TorchAO 0.17 on the tested RTX 4090. Other TorchAO versions and devices may support different layouts. Static activation quantization is a different workflow: it prepares an exported model, runs representative inputs to collect activation ranges, and then converts the model.

GPTQ and AWQ

Round-to-nearest quantization treats each weight locally. GPTQ and AWQ use calibration activations to reduce the effect of quantization error on model outputs.

Rounding each weight independently ignores how often and how strongly its input channel contributes to outputs. Calibration examples let a quantizer estimate this effect and reduce output error, though the result depends on how representative those examples are.

GPTQ quantizes weights layer by layer using approximate second-order information. As weights are quantized, it adjusts the remaining weights to compensate for earlier error. The transformers library now uses GPT-QModel as its GPTQ backend. Install the required packages to use GPTQ for quantization:

Then you can quantize and save a small model. The GPTQConfig setting dataset="c4" provides input samples for calibration. Setting sym=False requests the asymmetric form. Below is an example of quantizing a small model to INT4:

To load the saved GPTQ checkpoint in a fresh process and run one inference:

Asymmetric GPTQ checkpoints have fewer compatible kernels than the default mapping. For example, the Marlin backend does not support asymmetric GPTQ. You should choose the quantization scheme and inference backend together.

Activation-aware Weight Quantization (AWQ) is another quantization method. It identifies weights that are important under representative activations and searches for scaling factors that protect them before low-bit quantization. The --model_path argument points to the original, unquantized model in Hugging Face format. From a checkout of the reference llm-awq implementation with its dependencies installed, run the search first and then create the real INT4 checkpoint:

Install the reference implementation in a separate environment before running its commands:

The reference project documents Python 3.10 for this environment. Use its installation instructions for the matching CUDA extensions. Download the model with

then run the following commands:

The --run_awq command obtains its calibration input internally. It uses 128 samples from the validation split of mit-han-lab/pile-val-backup and forms sequences of 512 tokens. This reference CLI does not expose an argument for selecting another text dataset. Omitting --no_zero_point keeps the implementation’s zero-point quantization.

The raw .pt file produced by --dump_quant is not a complete Hugging Face checkpoint. It contains the quantized weights but not the model configuration or tokenizer. The following script packages them together in a local directory:

The transformers library can then identify the AWQ format from the checkpoint’s quantization_config, load the quantized weights, and run inference:

The model architecture, group size, zero-point convention, packed layout, and installed AWQ backend must be compatible.

Quantizing a large model can take substantial time and memory. Before starting, check whether a compatible quantized checkpoint already exists. If you quantize it yourself, save the tokenizer and complete quantization configuration with the packed weights.

Quantizing a Model to GGUF

GGUF is a container format used by the llama.cpp ecosystem. It stores model metadata, tokenizer information, tensors, and their types in one or more files. GGUF supports many quantization types; it is not itself one quantization algorithm.

The llama.cpp workflow has two stages. First, convert the original Hugging Face model to a high-precision GGUF file (such as bfloat16), then quantize that file. The following example uses the original, unquantized TinyLlama/TinyLlama-1.1B-Chat-v1.0 checkpoint. From a built llama.cpp checkout:

Install the conversion script’s Python dependencies in a separate environment and build the binaries first. The conversion requirements may select a different PyTorch build from the one used for the CUDA examples earlier in this post. This CUDA build needs CMake, a C++ compiler, and the CUDA toolkit; omit -DGGML_CUDA=ON for a CPU build.

Q4_K_M is a mixed GGUF quantization type that uses different encodings for different tensors. Its file size includes scales, metadata, and tensors stored at other precisions, so it is not exactly four bits per model parameter. Its name does not mean every value follows the affine formula shown earlier.

You can use llama-cli to load the quantized GGUF file and generate text directly from its packed weights:

This confirms that llama.cpp can generate from the packed file. TinyLlama may give an inaccurate explanation, so use separate quality checks for your task.

Some GGUF models are available on the Hugging Face Hub and can be run directly:

If you prefer to use Python code, transformers also accepts a GGUF filename for supported architectures:

Use a recent transformers release for the Qwen3.5 example below. Loading may dequantize tensors and require much more memory than the GGUF file size suggests.

But you should not assume that transformers preserves compressed inference. It keeps GGUF blocks packed only for a narrow set of model architectures, devices, and quantization types; other combinations are dequantized while loading. You can use llama.cpp when broad GGUF compatibility and direct execution of packed weights are required. You should quantize from the highest-quality source available. Requantizing an already quantized model compounds approximation error, and llama-quantize warns that it can severely reduce quality.

Evaluating Weight-Only Quantization

Evaluate the quantized model on representative inputs. If quality degrades beyond your application’s tolerance, try a different calibration set, quantizer, or bit width.

Some quantizers use calibration examples to reduce output error between the original and quantized model. That proxy does not replace application quality metrics. Useful checks include:

• Perplexity on representative text
• Accuracy or task-specific scores
• Compare greedy generations side by side
• Compare sampled generations across several seeds, prompt lengths, and temperatures

After creating opt-125m-gptq with the earlier command, run this script on one CUDA GPU. It loads and measures the FP16 baseline and the GPTQ checkpoint one at a time so their peak allocated memory measurements are comparable. The timing includes generate() but excludes model loading and tokenization:

This is a small diagnostic benchmark. For deployment decisions, use the same prompts, output limits, warmup, and concurrency as the real workload. You should measure perplexity or task accuracy as well as sample generations; one matching greedy answer does not establish quality parity. Peak PyTorch allocation may omit memory held by other libraries, so check device-level memory too.

A quantized model may require specialized kernels or a compatible inference engine. You may need specialized hardware for the best performance. You can compare the following measurements on the target platform to decide if quantization is worth the effort:

• Checkpoint size and model-load memory
• Peak memory during prefill and decode
• Time to first token
• Throughput at realistic batch sizes and concurrency

Further Reading

Below are some resources you may find useful:

• PyTorch TorchAO documentation. Quantized Inference. This guide lists current weight-only and dynamic inference configurations.
• PyTorch TorchAO documentation. quantize_. This page documents the in-place post-training quantization API.
• Elias Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. This paper introduces the GPTQ algorithm.
• Hugging Face Transformers documentation. GPTQ. This guide covers quantization and checkpoint loading with GPT-QModel.
• Ji Lin et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. This paper introduces AWQ.
• AWQ reference implementation. By MIT Han Lab. This repository includes commands for searching AWQ parameters and producing real quantized weights.
• AWQ conversion example. In the reference repository. This shows how to package a raw quantized state dictionary with a Transformers configuration and tokenizer.
• LLM Compressor. By the vLLM project. This toolkit provides GPTQ, AWQ, and other post-training compression recipes.
• llama.cpp quantization. In the llama.cpp repository. This guide documents conversion, quantization types, importance matrices, and execution.
• Hugging Face Transformers documentation. GGUF. This page explains the supported GGUF loading paths and fallback dequantization.
• NVIDIA A100 Tensor Core GPU. In the NVIDIA documentation. Its specifications compare peak throughput across FP16, INT8, and other numerical formats.
• AMD Instinct MI300X Platform. In the AMD documentation. Its specifications provide another example of higher peak INT8 matrix throughput than FP16 throughput.

Summary

In this post, you learned how asymmetric affine quantization maps floating-point weights to integer codes using a scale and zero point. Per-tensor, per-channel, and per-group parameters trade metadata and kernel complexity for accuracy. Weight-only post-training quantization derives these parameters after training while keeping activations at higher precision.

TorchAO provides PyTorch-native PTQ, while GPTQ and AWQ use calibration data to preserve model behavior at low bit widths. GPT-QModel, LLM Compressor, and the AWQ reference implementation provide workflows outside TorchAO. GGUF packages quantized tensors for runtimes such as llama.cpp.

Quantization is useful only when the checkpoint format, numerical scheme, packed layout, and inference kernel work together. Evaluate quality, memory, latency, throughput, and compatibility on the actual deployment target.