Do LLM Reasoning Models Need All Their Memory in HBM?
Reasoning models got better by thinking longer. But every token they think adds an entry to the KV cache, and the KV cache lives in HBM — the fast, tiny memory bolted to the GPU. A long chain-of-thought can run to thousands of tokens, and the cache grows with it until HBM runs out. The usual reflex is to evict tokens. That reflex is dangerous.
Why you can't just delete tokens
The tempting fix is to drop low-importance KV entries to reclaim space. The problem: a token that looks unimportant now can be exactly the one attention needs later. Permanently evicting reasoning tokens can collapse accuracy toward zero on hard problems — the model literally forgets a step it needed. So the question isn't "which tokens can we throw away?" It's "must every token sit in the most expensive memory?"
The shape of the opportunity
Importance is extremely skewed. If you measure how much each cached token actually contributes:
- The distribution is heavy-tailed — a small number of tokens dominate.
- Roughly 80% of tokens fall below a low importance threshold.
- The top ~40% of tokens capture ~80% of total importance.
That skew is the whole opening: if most tokens matter little most of the time, most of the cache doesn't need to sit in HBM.
The idea: tier the cache, don't shrink it
Keep every token — just not all in HBM.
A semantics-aware memory hierarchy places KV entries across tiers by importance: HBM for the hot, high-importance tokens; DDR and compressed storage for the cold ones. Crucially, low-importance tokens are prefetched back before each attention step that will read them, so the model computes on exactly the same terms as if nothing ever left HBM. The result approximates full-cache reasoning at a fraction of the HBM footprint — and the design can be reasoned about via an approximation-error bound on how many tokens are safely offloaded versus how many must remain resident.
Why this is a hierarchy, not a heuristic
The framing matters: this is the classic memory-hierarchy idea (fast/small vs slow/large) applied to the KV cache, with token semantics — not just recency — deciding placement. It's the same instinct as caching in CPUs and databases, adapted to what an LLM actually needs to keep in front of it while it reasons.
Where this fits
Cheaper long-context reasoning pairs naturally with cheaper long-term agent memory: one tiers the KV cache within a single generation, the other tiers retrieval across a whole conversation. Both come from the same principle — spend your scarcest resource only on what the model truly needs now.
Frequently asked questions
Why does long reasoning run out of GPU memory?
Each generated token adds to the KV cache in scarce HBM; long chains produce thousands of tokens, exhausting HBM even though most contribute little.
What is a semantics-aware memory hierarchy?
A KV-cache system that ranks tokens by importance and tiers them — HBM for the important few, cheaper tiers for the rest — prefetching cold tokens back just before they're needed.
Isn't that just KV-cache eviction?
No — eviction deletes tokens and can wreck accuracy; tiering keeps every token but stores it more cheaply and fetches it on demand.
Written by Aojie (Justin) Yuan. "Not All Thoughts Need HBM" is described in arXiv:2605.09490.