Decode throughput is bandwidth divided by model size

Here we will explore how far you can get predicting local LLM throughput knowing only the model’s size on disk.

The claim: for single-token decode, tokens per second is bounded by memory bandwidth divided by weight bytes. You need no architecture details, no layer count and no attention variant, just the file size and the spec sheet.

I measured it against six Qwen 2.5 models on an M3 Max. That is one machine, one framework, dense models only, single stream, short prompts.

Why decode is bandwidth bound

Generating one token requires reading every weight in the model exactly once. The arithmetic done with those weights is a matrix-vector product: each weight is loaded, multiplied by one activation, accumulated, discarded.

That is roughly 2 FLOP per byte read. An M3 Max does about 400 GB/s of memory bandwidth and several TFLOP/s of compute, which puts its roofline ridge point far above 2 FLOP/byte. Decode sits on the memory-bound side of the roofline by a wide margin, and kernel optimisation cannot move it across.

So the ceiling is:

tokens/sec <= bandwidth / weight_bytes

For a 4-bit 7B model at 4.28 GB and 400 GB/s, that predicts 93.4 tokens per second.

What the models did

Six 4-bit Qwen models under MLX, greedy decode, best of three runs after a discarded warm-up. Prediction is 400e9 divided by the weight bytes on disk.

Run-to-run spread was 2.0% to 7.9%, widening with model size.

At 7B the spec-sheet division lands within 9% of measured. From 3B upward it holds between 75% and 91%. Below that it falls apart: at 0.5B the model reads only 0.26 GiB per token, so per-token overhead that has nothing to do with weight reads starts to dominate, and the prediction overshoots by nearly 3x.

Read the same numbers as effective bandwidth and the shape is clearer:

The 7B model reaches 365.6 GB/s against a 400 GB/s spec figure. That spec is a theoretical peak no workload reaches, so the real achievable ceiling is lower and every roofline percentage in the table above is understated. The 7B model is running the memory system about as hard as it goes, which is what makes the division worth doing.

The dip at 14B and 32B is smaller than it looks. Both still sustain over 300 GB/s. Larger weight sets mean more of the model is being pulled from further away with less benefit from cache reuse across layers, and the wider run-to-run spread at those sizes suggests thermal and residency effects too.

Where the bound stops applying

Decode is bandwidth bound because the matrix-vector product reads each weight once and uses it once. Batch the requests and the same weight read serves every sequence in the batch. Matrix-vector becomes matrix-matrix, arithmetic intensity rises with batch size, and the workload moves up the roofline toward the compute-bound side.

I measured this directly with a single 8192x8192 fp16 weight matrix, 128 MiB, multiplied by batches of increasing width.

Rows per second climbs by a factor of 62 from batch 1 to batch 512, while bandwidth falls from 187.9 GB/s to 25.7 GB/s. The same weight read is amortised across more and more work.

Batches 1 to 8 cost almost nothing: wall time barely moves while throughput multiplies, because the kernel waits on the same 128 MiB read regardless. Past batch 32 each doubling costs real time. Between 256 and 512 throughput gains 5% for double the batch, which is where compute has taken over as the binding constraint.

I had expected throughput to flatten around batch 16 to 32. Instead it keeps climbing to 512, with returns thinning from about 256.

The synthetic benchmark I dropped

I also wrote a synthetic bandwidth benchmark, allocating buffers of 1 to 3 GiB and timing reads across them, intending to quote a measured bandwidth figure rather than the spec sheet number.

Three runs of the same kernel gave 380, 115 and 167 GB/s.

Two of those runs were contended. A next-build process from an unrelated project was running at 653% CPU during one, and a next-server at 392% during another. Waiting for the machine to go quiet fixed part of it.

The rest was a flaw in the benchmark. A diagnostic printing every repetition instead of the best showed a single 1 GiB read series producing 355.0, 360.9, 358.9, 355.6, 113.2, 95.6, 220.2, 182.0 GB/s without anything changing between calls. A kernel allocating a fresh multi-gigabyte output buffer each iteration is measuring the allocator and page residency, not DRAM.

So there is no measured bandwidth figure in this post. The 400 GB/s above is Apple’s published number for the part, and the effective bandwidth column is derived from decode timings, which allocate nothing per token beyond the KV cache.

The decode and batch numbers were taken on a quiet machine, best of several runs. Best-of-N is the right statistic when you are looking for a hardware ceiling: a slower pass only ever means something else was running.

What this does not cover

The division works on the case I measured. Five things move it, and the first two move it a long way.

Context length. Decoding a token reads the KV cache as well as the weights, and the cache grows with every token in the context. My prompts were short, so the weight read dominated by construction. The honest form of the bound is bandwidth divided by weights plus KV, and the KV term is the one that grows while you watch. I have not measured where it starts to matter, which makes this the first follow-up.

Multiple devices. Split a model across GPUs with tensor parallelism and every layer needs an all-reduce. During decode those payloads carry one token of activations, so they are small and latency bound rather than bandwidth bound. Over NVLink that is tolerable. Over PCIe or Ethernet the all-reduce can dominate, and decode is no longer memory-bandwidth bound at all. Expert parallelism for mixture-of-experts models does an all-to-all per layer, which is worse. The bound describes a single device.

Model architecture. Every model here is dense, so weight bytes and file size are the same number. For mixture-of-experts the denominator is active parameter bytes, not file size: DeepSeek V3 is 671B parameters total and 37B active. Grouped-query and multi-query attention cut KV traffic several-fold against multi-head. Speculative decoding breaks the premise outright, since verifying several draft tokens against one weight read is the entire point of it. Mamba-style layers have no KV cache to grow.

Quantisation details. I used file size as the byte count. A 4-bit model also reads scales and zero points, and pays compute to dequantise before the multiply. Both push the real figure away from the simple one.

Serving stacks. The batch sweep above is one matrix multiplication, not an inference server. Real serving uses continuous batching and paged attention, and mixes prefill and decode within a single step. It shows arithmetic intensity climbing with batch size, which is the roofline point. It does not show what a server would do.

One number in the table is a framework result rather than a hardware one. The 0.5B model hitting 36% of roofline is per-token overhead in MLX, not anything about memory. Small models are where the framework shows up.

What this is good for

Capacity planning without benchmarking: given a model size and a memory bandwidth figure, you can bound single-stream decode throughput on one device to within about 25% for a dense model of 3B or more, before downloading anything.

It also tells you which lever matters. Halving the weights through quantisation roughly doubles decode throughput, because the bound is bytes read per token. Adding compute does nothing for single-stream decode.

Batching is where the serving economics change. Single-stream decode leaves most of the machine idle, and batching is the mechanism that fills it: 87216 rows per second at batch 512 against 1400 at batch 1 on the same weight matrix.