Llama 3.1 8B (Q4_K_M) tokens/sec, RTX 5070 Ti, avg of 3 runs
An article claimed enabling a small draft model alongside a larger one gave a clean 26% token-per-second speedup for local inference: 23.35 to 29.46 tok/s with Llama 3.1 8B and a Llama 3.2 1B draft. I had the exact same model pair, an RTX 5070 Ti sitting idle, and llama.cpp's own speculative decoding flags. So I ran it myself instead of taking the number on faith.
The result surprised me: no meaningful difference at all. 126.7 tok/s without the draft model, 126.9 tok/s with it, averaged over three runs each at temperature 0, identical prompt, identical settings. That's noise, not a speedup.
The setup
The source article used LM Studio's speculative decoding toggle with this exact pairing: Llama 3.1 8B Instruct as the main model, Llama 3.2 1B Instruct as the draft. Same family, compatible tokenizers, draft model an order of magnitude smaller. Textbook setup. Rather than reproduce it through LM Studio, I went one layer down and used llama.cpp's llama-server directly, since that's the engine LM Studio itself wraps, with the -md (model-draft) flag and default drafting parameters (max 3 draft tokens, min 0).
Both models were fully offloaded to the GPU (-ngl 999 on both the main and draft model). I ran the same 300-token completion three times with the draft model disabled, then three times with it enabled, reading predicted_per_second straight out of llama.cpp's own timing output rather than a stopwatch.
Results
| Configuration | Run 1 | Run 2 | Run 3 | Average |
|---|---|---|---|---|
| No draft model | 126.4 tok/s | 125.7 tok/s | 128.1 tok/s | 126.7 tok/s |
| + Llama 3.2 1B draft | 125.2 tok/s | 127.1 tok/s | 128.5 tok/s | 126.9 tok/s |
Key finding
0.2% is not a speedup, it's run-to-run noise. The draft model loaded correctly (confirmed in the server log: loading draft model 'llama32-1b-q8.gguf') and both runs used identical GPU offload, so this isn't a case of the draft model silently failing to engage. The most likely explanation: my main model, at 126 tok/s for an 8B model, is already fast enough on this GPU to be compute-bound rather than memory-bandwidth-bound, which is the specific bottleneck speculative decoding is designed to relieve. The source article's numbers (23 to 29 tok/s) suggest a setup that was memory-bandwidth-limited to begin with, likely a MacBook or a card with less raw throughput than a 5070 Ti running CUDA. Speculative decoding's payoff isn't universal; it's a function of where your specific bottleneck already sits.
Why this doesn't discredit the technique
The mechanism itself is real and well-documented: it's implemented in vLLM, llama.cpp, and LM Studio's own engine for a reason, and published acceptance-rate tables for Llama-family draft pairs run 60 to 80%. A 26% speedup measured on one machine, with one engine, at one point on the compute/memory-bandwidth curve just doesn't transfer as a universal number. On hardware that's already fast enough to be compute-bound rather than bandwidth-bound, the same technique can land at zero.
Caveats
I used llama.cpp's server directly rather than LM Studio itself, so if LM Studio's wrapper uses different default speculative-decoding parameters (acceptance threshold, draft token count) than llama.cpp's defaults (max 3 draft tokens, min 0), that could account for some of the gap. I also tested one prompt type (a factual explanation), not the broader mix of summarization, code, and everyday Q&A the original article describes; speculative decoding's acceptance rate is prompt-dependent, and a different prompt style could shift the result. This was three runs per configuration, not a long benchmark suite.
Sources
- Article being tested: XDA Developers
- llama.cpp: github.com/ggml-org/llama.cpp
- Llama 3.1 8B Instruct GGUF: Hugging Face
- Llama 3.2 1B Instruct GGUF: Hugging Face