Context Is a Cache, Not Memory
Why context disappears, why memory must be designed separately, and how better AI systems distinguish temporary working state from durable knowledge.
You have a mental model of what the context window is. You use it every day. You’ve built systems around it.
The mental model is wrong, and it’s costing you.
Here is what you believe, probably without having said it out loud: the context window is where the model remembers things. You put information in, the model holds it, the model uses it.
The bigger the window, the more it can hold, the better it remembers.
Memory scales with size.
That model is intuitive, deeply embedded, and incorrect in almost every way that matters.
The context window is not memory. It is a cache.
And the difference between those two things is not semantic. It changes how you build, what you trust, what you test, and why your agents fail in the specific ways they fail.
What Is Memory
Before “cache,” you need to know what you’re contrasting it with.
Memory, in the sense that matters, is persistent, reliable, and positional.
When something enters memory, it stays until explicitly removed. You can recall it in any order. You can query it by content, not by position.
Nothing about the act of accumulating more information degrades your access to the old information.
Memory is a commitment. What went in is what comes back.
This is also your implicit model of the context window.
You put the system prompt in. You put the user’s history in. You put the retrieved documents in. You expect all of it to be there, faithfully, when the model needs it.
And sometimes it is.
When the context is short, the model recent, and the information recent in the window — it works.
The failure is silent enough that you call it memory and move on.
What a Cache Actually Is
A cache is different from memory in ways that are not small.
A cache has a capacity limit.
When it fills, things leave — not arbitrarily, but not necessarily the things you’d choose.
A cache has no retrieval guarantee — the fact that something entered the cache does not guarantee it comes back in the same form, at the same fidelity, or at all.
A cache operates by locality — recently written and frequently accessed entries are cheap; older, deeper entries are slower and less reliable.
A cache has an eviction policy, whether or not you declared one.
These are the exact properties of the context window, stated precisely.
The window has a capacity limit and behavior degrades as you approach it. Recent information is processed more reliably than deep, early information — models attend more to what’s nearby.
The “lost in the middle” effect is documented and consistent: information placed in the center of a very long context is retrieved less reliably than information at the front or the end.
Presence in the window does not guarantee retrievability. You cannot assume that because you loaded something into context, the model will faithfully access it at the moment it’s needed.
That last sentence is the one worth sitting with.
A recent survey of always-on agents stated it as a finding, not a warning: presence in the window does not guarantee retrievability.
The model cannot be treated as reliably reading everything it was given.
The context window is an unreliable substrate for state.
A cache. Not memory.
The Decay That Nobody Talks About
Here is the specific, quantified version of the failure, and it arrived in a paper four months ago that more people should have read.
An April 2026 study — “Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents” — ran 4,416 trials across six conversation depths to measure a simple thing: how does a model’s adherence to instructions change as the conversation grows?
The finding is uncomfortable.
Commission constraints persist. Omission constraints decay.
What that means in plain language: things you added to the context — information you instructed the model to include, produce, or remember — it holds.
But things you told it not to do, not to say, not to include — those instructions erode. The prohibition weakens as the session deepens.
The deeper the conversation, the more often the model violates your negative constraints, not your positive ones.
Now think about what you put in your system prompt.
Half of it is prohibitions.
Don’t hallucinate. Don’t go off-topic.
Don’t share information across tenants.
Don’t call this tool without confirmation. All of it sitting at the start of the context, getting further away with every turn.
You are relying on a cache to hold a constraint, and the cache is evicting the constraint while keeping the content.
The most important instructions in your system are the ones most vulnerable to this decay.
This is not a model bug you can patch. It is a property of how position and depth interact with attention. You cannot fix it by repeating yourself. You can only design around it — by treating context as the cache it is and not depending on it for things that require memory.
The Stateless Truth
Here is the brutal foundation that the cache metaphor is built on, and the one the industry has been circling for two years without saying plainly.
The model has no memory.
Not a small memory, not a compressed memory, not a selective memory. Zero.
Every inference call resets.
The agent who made an architectural decision on Monday is a different instantiation of the model on Wednesday, and the Wednesday agent has no knowledge of Monday’s session unless you injected it — freshly, deliberately, as input.
What you experience as “the model remembers” is always one of two things: either the information is in the current context (the cache hit), or it was retrieved from an external store and injected into the context before the call (a cache miss that you handled by loading from disk).
The model didn’t remember. You loaded the cache.
The Letta engineering team put the distinction as clearly as it’s been put: RAG is retrieval, memory is persistence. These are not synonyms and they are not interchangeable.
In-context information is available with no latency and no retrieval error — it’s in the cache. Information in an external store must be retrieved, with latency and relevance error and token cost — that’s a cache miss. Memory is the library.
Context engineering is the librarian deciding which books to put on the desk for this session.
The agent who chose RS256 tokens on Monday has no recollection of that decision on Wednesday.
The developer who corrected an import pattern last week will correct it again this week. Unless you built the library. Unless someone is doing the librarian’s job.
Where the Cache Metaphor Gets Precise
The metaphor isn’t poetic. It maps onto computer science vocabulary with enough precision to be useful.
A CPU cache holds a subset of RAM, optimized for the instructions currently executing.
When the CPU needs data, it checks the cache first (fast) and falls back to RAM (slow).
Cache eviction is automatic, policy-driven, and invisible to the program unless you instrument it.
The context window holds a subset of what the agent knows.
When the model attends to a token, it “checks” the context (fast) and has no fallback if the information isn’t there.
Eviction happens when the window fills — and unlike a CPU cache, the LLM’s “eviction” is not explicit removal but degraded attention to distant positions, which is invisible to you unless you measure it.
What this implies for how you build:
The cache is warm only for the current session: Cross-session continuity requires an external store. The agent’s memory between sessions is exactly zero unless you loaded it in. This is not a limitation to engineer around — it is the architecture. Accept it and build accordingly.
Cache capacity is scarce and priced: Every token in the context costs money to process and competes with every other token for the model’s attention. Stuffing the cache with everything available is not a memory strategy. It is a context engineering failure. You are paying to process noise and degrading the signal in proportion.
Cache hits are positional: Information at the front (the system prompt region) and information at the back (the most recent turns) are more reliably attended to than information in the middle. If you need the model to use something reliably, don’t bury it in the middle of a long retrieval dump. This is not a preference. It is an attention property.
Cache misses are silent: A CPU cache miss produces measurable latency. A context cache miss produces a wrong answer, at normal latency, with a confident tone. You will not catch it without measuring retrieval fidelity. It will look like a model problem. It is not. It is a cache miss you didn’t detect.
The Three Things Context Cannot Do
The cache metaphor gives you three negative capabilities — things the context window cannot be trusted to do, no matter how big it gets:
Cross-session persistence: The cache resets on every call. Anything that needs to survive a session boundary needs an external store — persistent memory, a database, a vector index, a structured file the agent reads on startup. The 1M-token window on Fable 5 does not change this. A 1M-token window is a very large cache. It is still a cache. It still resets.
Reliable constraint enforcement over depth: Omission constraints decay. Negative instructions degrade with conversation depth. If your safety property, your access rule, or your behavior constraint is a sentence in the system prompt at position zero of a forty-turn conversation, you should not trust it. Hard constraints go in external, re-injected memory, loaded fresh at the top of the prompt on every call, not accumulated in a growing transcript.
Faithful retrieval from arbitrary positions: Middle-buried information is less reliable. Long retrieval dumps where the relevant chunk is sandwiched between irrelevant ones are less reliable. The model’s attention is not uniform across the window. Anything you need reliably retrieved at inference time should either be in the prompt cache region (front, stable, re-injected), or not in the context at all — pulled on demand from an external store.
What Memory Actually Looks Like
Since we’ve been precise about what context can’t do, be precise about what fills the gap.
Episodic memory — what happened in past sessions, what decisions were made, what the user prefers — lives in an external store, retrieved and injected fresh at session start. Not accumulated in the transcript. Not trusted to a growing window.
Semantic memory — facts, knowledge, domain information — lives in a retrieval index. RAG is the implementation. The retrieval is a cache-miss handler: the model needs something that isn’t in the context, so you fetch it from storage and inject it.
Hard constraints — security rules, access policies, behavioral guardrails — do not live in the conversation history at position zero and drift backward. They are retrieved from persistent memory on every call and injected at the top of the system prompt, where attention is reliable. Position matters. Re-injection matters.
Working state — current task progress, intermediate results, decisions made this session — belongs in the context for now, and belongs written to external state before the session ends, because the cache will reset.
The architecture that falls out of the cache metaphor is a system that knows the difference between what should be in the cache and what should be on disk, and routes accordingly.
Not “put everything in the window and let the model figure it out.”
A curated, intentional cache.
The information the model needs right now, loaded freshly, positioned deliberately, and sized for the attention budget you’ve allocated.
Context engineering is cache management.
The discipline that built L1 and L2 and TLBs and prefetch buffers — all of it was trying to solve the same problem: get the right data in front of the right processor at the right time, without wasting memory on data that won’t be used and without cache-missing on data that will.
The intelligence of the system lives in that discipline. Not in the size of the cache.
Learn more about Caching.
The Bottom Line
Stop thinking of the context window as the place where the model knows things. It is the place where the model currently has access to things — and access is not knowledge, retrievability is not memory, and presence in the window is not a guarantee of use.
The mental model that says “bigger window, better memory” will keep producing systems that fail in strange, session-specific, position-dependent, depth-decaying ways that look like model problems and aren’t.
The mental model that says “context is a cache I actively manage” produces a different question: what deserves to be in the cache right now, what needs to be on disk, and what have I trusted to a volatile substrate that should be persisted somewhere durable?
Ask the second question and most of the mysterious failures resolve.
The constraint that eroded — it should have been re-injected, not trusted to history.
The preference the agent forgot — it should have been written to the memory store, not left to the transcript.
The decision from Monday the agent didn’t know about Wednesday — it should have been on disk, not in a cache that reset.
Memory is a commitment.
A cache is a convenience.
Build the commitment for the things that require it. Cache the rest.