Infinite Context: Active and Passive Memory for LLMs

Every context window is a cliff. You fill it, and the model forgets the beginning. Doubling the window buys you time, not a solution: the attention cost grows with the square of the sequence, so the cliff just moves somewhere more expensive.

This project started from a different question. Not "how do we make the window bigger" but "why is everything the model knows sitting in one flat buffer at all?"

The split

Human recall is not one store. You hold a handful of things in mind right now, and you have a much larger body of things you can retrieve when something cues them. The retrieval is not free, but it is far cheaper than keeping everything in mind at once.

So the architecture has two tiers:

• Active memory is the working set. It lives in the context window, it is small on purpose, and it is what the model attends over on every token.
• Passive memory is everything else the run has seen. It lives outside the window in a vector store, it grows without bound, and it costs nothing to hold.

The interesting part is not either tier. It is the traffic between them.

Promotion and demotion

A passive entry is promoted into active memory when the current turn cues it. A cue is a similarity hit, but similarity alone is a bad trigger: a long conversation is full of passages that look alike and matter differently.

What I ended up scoring on was three signals together:

1. Semantic similarity to the current turn, the obvious one.
2. Recency of last promotion, so something pulled in three turns ago is cheaper to pull again.
3. Write frequency, because a fact the run keeps returning to is load bearing even when the current turn does not name it directly.

Demotion is the harder half. When the active set is full, something has to leave, and the naive choice is to evict the oldest. That is wrong often enough to matter. A task instruction given at turn one stays relevant at turn ninety. What I evict instead is the entry with the lowest combined score, which usually means something that was pulled in once on a weak similarity hit and never touched again.

What broke

The first version promoted too eagerly. Any similarity above a fixed threshold pulled an entry in, the active set churned constantly, and the model got worse rather than better because the working set never stabilised long enough to be useful. Raising the threshold helped less than making promotion hysteretic: an entry has to clear a higher bar to enter than it does to stay. That one change did more for coherence than any amount of threshold tuning.

The second problem was subtler. Passive memory stores text, but text is not what the model needs on recall. It needs the text plus why it was stored. Adding a short provenance note to each entry, what turn it came from and what the model was doing at the time, made the retrieved entries usable instead of merely relevant.

Where it stands

This is still open research, not a shipped system. The parts that work reliably are the two tier split and the promotion scoring. The parts I am still unhappy with are eviction under long multi topic runs, where the active set gets pulled between subjects and holds a poor compromise of both.

What I take from it so far is that the context window is the wrong thing to optimise. The window is a cache. The interesting engineering is in the policy that decides what belongs in the cache, and almost nobody treats it as a policy problem.