LLM Inference under the hood - Part 2: Prefill, Decode, and Prompt Caching

In Part 1: From Prompt to KV Cache, we learned how a model reads your prompt, turns it into meaning, and stores that context in a short-term memory (the KV Cache).

Now, we need to understand how the model actually generates a response, why it can sometimes feel slow, and how we can use the cache to massively speed things up.

💡 By splitting the workload into “reading” and “writing”, we can reuse pre-calculated memory for identical prompt prefixes. This can drop API bills by up to 90% and make the model respond instantly.

1. The Two Phases: Prefill vs. Decode

LLM inference behaves like two completely different tasks glued together. This is where we hit performance bottlenecks.

• The Prefill Phase (Reading): When you submit a prompt, the computer processes the entire text all at once in parallel. Since it can calculate the context for all words simultaneously, this phase is very fast and efficient. This is when the KV Cache (short-term memory) is built.
• The Decode Phase (Writing): Once the model has read the prompt, it generates the response one single word at a time. To generate word number two, it needs to look at the prompt plus word number one. To generate word number three, it needs to look at the prompt plus words one and two. Because it has to load its entire “brain” into memory just to guess a single word, this phase is much slower.

To read more about these mechanics, check out this guide on the Prefill and Decode Stages.

2. Supercharging Inference with Prompt Caching

Because the “Prefill Phase” (reading) requires building up the short-term memory from scratch, it can still take a few seconds if you paste in a massive document.

This brings us to Prompt Caching (also known as Automatic Prefix Caching).

If multiple separate user requests start with the exact same text—such as a long set of instructions or a static context document—it is incredibly wasteful to rebuild the memory from scratch each time.

With Prompt Caching, modern systems save this memory directly. When a new request arrives, the system checks the beginning of the text: if it matches something it has seen before, it skips the reading phase entirely and pulls the pre-computed memory straight from the cache. The result? Near-instantaneous responses.

Quick Summary

To take advantage of prompt caching, you can use a modern model serving framework like vLLM or cloud services like Amazon Bedrock’s prompt caching.

Visual & Technical Learning Resources

• Sai Lakshmi Allada: Understanding Prefill and Decode Stages
• Amazon Bedrock prompt caching