Beyond the GPU: Solving the Memory Wall in LLM Inference

Part I / The Problem

Buying the Wrong Hardware

Every AI company is in a race to buy GPUs. Data centers are filling up with chips that cost around $30,000 each, built for one thing above all: doing enormous amounts of math very fast.

The paper “Reimagining LLM Inference Infrastructure with Memory-Centric KV Cache Servers” (Kiyawat & Skadron, University of Virginia, HotInfra ’26) makes a provocative claim. For most of what AI chatbots actually do today, the math isn’t the bottleneck. Memory is. And if that’s true, we’re spending a fortune on the wrong kind of hardware.

To see why, we need to understand one concept first.

What Is a “KV Cache”?

When a large language model answers, it actually works in two phases.

Prefill. The model reads your whole prompt in one go. This is heavy math on lots of tokens at once, and GPUs are brilliant at it.
Decode. The model writes its answer one token (roughly one word piece) at a time. To pick each new token it has to look back at everything that came before.

Recomputing all of that history for every new word would be absurdly wasteful. So the model keeps notes: for every token it has seen, it stores two small vectors called a Key and a Value. That notebook is called the KV cache.

Think of a court stenographer. Instead of replaying the whole trial from memory before writing each new line, they glance back through the transcript. The transcript is the KV cache. The catch is that it grows with every word, and the model has to flip through all of it at every single step.

Two phases

How an LLM answers: prefill once, then decode token by token

Each decode step hauls tens of GB through memory to do very little math. The GPU spends over 70% of decode time waiting on memory.

The Memory Bottleneck

That notebook gets big very fast. The paper’s headline example: for LLaMA-3-70B with a 128,000-token context, the KV cache for a single request is about 34 GB. Add the model’s own weights (about 140 GB) and you need roughly 174 GB of memory.

An NVIDIA H100, the flagship AI GPU, has 80 GB. So before the model generates even one word, the job has to be split across multiple GPUs. Not because it needs their math, but because it needs their memory.

Things are getting worse, not better. In 2023 a typical chatbot answer was 100 to 200 tokens long. Today’s “reasoning” models (DeepSeek-R1, OpenAI o1) routinely think out loud for 8,000 to 32,000 tokens. That’s 40 to 160 times more KV cache pressure in under two years.

The memory wall

LLaMA-3-70B, one request, 128K context: 174 GB against an 80 GB GPU

The GPUs are added for their memory, not their math.

Why Scaling Memory Isn’t That Simple

The obvious fix is to give the GPU more memory. But there are physical limits to doing so:

GPUs are out of beachfront. High-bandwidth memory (HBM) sits in stacks right next to the chip, and there’s only so much edge to attach them to. The H100 already uses six stacks. A seventh means a bigger chip, which runs into manufacturing size limits, or taller stacks, which overheat.
CPUs are out of pins. Each regular DDR5 memory channel takes around 100 connection pads on the processor. There’s no room left on the socket to add more.

Why Modern GPUs Are Overkill for Inference

Here’s the uncomfortable consequence. During decode, GPUs typically use under 30% of their compute, while their memory is completely full. Take serving DeepSeek-R1 (671B parameters) with 32K reasoning tokens: it needs 19 H100s running at about 15% compute utilization. You would be paying for 18,791 TFLOPS of number-crunching power to run a job whose real limit is how fast data can be moved.

Meanwhile, the most valuable thing in many workloads isn’t computing new notes, it’s reusing old ones.

Reuse

Production systems report that 50–90% of KV cache is reused across requests (think of the same system prompt shared by thousands of users, or multi-turn chats). In multi-agent setups it’s 75–95%. Sharing KV across engines alone has delivered 15× throughput gains.

So the question becomes: if the job is mostly storing and fetching notes, why are we building it around a math engine?

Part II / The Proposal

Introducing the KV Cache Server

The paper proposes a new kind of machine for the data center: the KV cache server. It sits between the fast GPUs and slow storage, much as cloud storage split memory from compute long ago.

It’s built on CXL (Compute Express Link), a standard that lets you attach large pools of memory over a single x16 link. Need more? Add more memory modules. No extra chip area, no extra pins. Capacity scales to terabytes.

The authors describe a spectrum of three device types (they call them archetypes):

CXL DDRx expander. Plain, cheap memory. Stores KV cache and lets many machines share it. No compute.
CXL + PNM (Processing-Near-Memory). Adds a small helper next to the memory. It can pick which tokens matter, manage the cache or do light memory-bound work.
CXL + PNM + PIM (Processing-In-Memory). Puts compute inside the memory chips themselves, enough to run decoding or even a whole inference job without a GPU.

A CPU coordinates everything, but hands routine scheduling off to the CXL switch or the memory devices, stepping in only when tasks depend on each other. The authors also imagine future memory-to-memory links, like NVLink does for GPUs, so memory devices can pass KV cache directly to each other.

The proposal

The KV cache server: memory devices on a shared CXL fabric

Left to right, more compute moves inside the memory. The dotted links are the memory-to-memory connections the authors propose for passing KV directly. Capacity grows by adding modules: no extra chip area, no extra pins.

The Three Memory Design Patterns

The authors don’t say “throw out the GPUs.” They describe three operating modes, and real systems can mix them:

Share the prompt (GPU still decodes). One GPU does the prefill and writes the KV cache once into the shared CXL memory pool. Several decode GPUs then read it directly. No copying the same notes into every GPU’s memory and no network transfers. Great for RAG, multi-turn chat and apps where thousands of users share one system prompt.

Decode in memory. The GPU does the prefill and then hands off. Smart memory devices (type II or III) do all the token-by-token decoding themselves, and the GPU is freed for other work. Best for long reasoning outputs with small batches, which is exactly where GPUs sit idle.

Everything in memory. The GPU computes a prefix once and is fully released. A type III device runs the entire inference from then on, including any small additional prefill. Earlier research prototypes (Sangam and CENT) showed order-of-magnitude energy savings versus GPUs in this setup.

Agentic AI is a blend of all three. A shared system prompt lives once in the pool, while each agent keeps its private notes on its own memory node. The agents work in parallel and only sync up, through the CPU or directly between memory devices, when one needs another’s result.

Three operating modes

Each mode needs less GPU time than the one before

Best for: (1) RAG, chat and big batches; (2) long reasoning with small batches, GPU freed; (3) heavy prefix reuse, near-zero prefill.

Part III / The Evidence

What the Benchmarks Show

To ground the idea, the authors do a back-of-the-envelope comparison with DeepSeek-R1 (671B) generating 32,000 reasoning tokens.

Because DeepSeek-R1 is a Mixture-of-Experts model, only 37B parameters are active per token, and its attention design compresses the KV cache about 64 times compared to standard attention. Even so, each new token requires moving about 75 GB of data (74 GB of weights plus about 1 GB of KV cache). Speed is therefore set almost entirely by memory bandwidth.

They compare:

	GPU cluster	KV cache server
Hardware	19 × H100	1 CPU, 2 CXL switches, 23 × 64 GB PIM-DIMMs
Memory	1,520 GB	1,472 GB
Usable bandwidth	63.7 TB/s	150.7 TB/s (6.55 TB/s per stick)

A PIM-DIMM is a memory stick with compute built into each memory bank. Because the compute sits inside the banks, it sees their internal bandwidth rather than the narrow channel out of the stick. The result is roughly the same memory but 2.4× the usable bandwidth at a fraction of the price.

Same job, two machines

DeepSeek-R1-671B generating 32K tokens

Back-of-the-envelope estimates from the paper. Steady-state decode only; GPU prefill cost assumed spread over long outputs.

The Devil in the Details

These numbers are striking, and the authors are upfront that they come from a simplified model, not from a built system. A few things to keep in mind:

Caveats

It’s a projection. Throughput is calculated from memory bandwidth at an assumed 80% efficiency. No full KV cache server has been built and benchmarked at this scale.
PIM-DIMMs aren’t off the shelf. The study assumes they cost $12/GB, 2 to 4× regular DDR5. Real pricing for a new product category could differ a lot.
The comparison isn’t perfectly symmetric. The GPU cluster’s cost and power leave out its CPU and NVSwitches, which favors the GPUs, while the KV cache server’s rental price is simply scaled down from its purchase price.
Prefill is set aside. The study models the long decode phase only. Workloads with heavy prefill still need GPUs, which is why the paper frames this as a new tier alongside GPUs and not a replacement.
The software doesn’t exist yet. Today’s KV cache systems, such as vLLM, SGLang, LMCache and Mooncake, manage the cache in software and could extend to these tiers, but the authors say a full hardware-plus-software study is still needed.

The Main Point

Generating the next word in a modern AI conversation is mostly a job of remembering, not calculating. We’ve built AI data centers around calculating.

The authors make two calls to action:

Treat the KV cache as shared, first-class data with its own dedicated hardware.
Extend GPU-style high-speed links to connect memory devices directly to each other.

Their closing question sums it up: “What is the minimum hardware needed to generate the next token?” For most of today’s inference traffic, they argue, the answer is a memory device with a little compute built in, not a $30,000 GPU.