Why your AI keeps forgetting.

Twenty minutes into a long AI coding session, it stops following an instruction you gave at the start. That isn't a bug or a mood swing — it's a predictable mechanical consequence of how context windows work. Here's the actual mechanism, and the handful of habits that fix it.

A context window is a token budget, not a memory

Every request you send to an AI model — your messages, its replies, any files or tool output pulled in along the way — gets encoded into tokens, roughly three-quarters of a word each, and packed into a single block the model processes at once. That block has a hard ceiling: the context window. Once a conversation's token count would exceed it, something has to give, and it's never the newest content.

There's no separate "memory" sitting off to the side that the model consults. Everything it knows about the current conversation lives inside that one window. Nothing outside it exists to the model at all — not because it chose to forget, but because it was never shown.

The sliding window: oldest instructions go first

Most chat interfaces handle an overflowing window the same way: truncate from the front. Your opening message — the one with the actual spec, the constraint you cared about, the "always do X, never do Y" — is the first thing to get silently dropped once the conversation grows past the limit. The model isn't being careless. It's answering your latest question with total fidelity to everything currently in its window; your first message just isn't in it anymore.

This is why the failure mode is so specific and so consistent: early ground rules erode first, recent exchanges stay sharp, and the drift feels sudden even though it was gradual the whole time.

Lost in the middle: forgetting inside the window

Even before anything gets truncated, position inside the window matters. Across published long-context evaluations, models are consistently most reliable at the very start and the very end of a prompt, and measurably weaker at pulling out a fact buried in the middle of a long document or chat history — a pattern researchers call "lost in the middle." A detail dropped in message six of a forty-message thread can get less attention than the exact same detail would get in message one or message thirty-nine, independent of whether it technically still fits in the window.

Why "just use a bigger context window" doesn't fully fix it

Frontier models now advertise enormous maximum windows compared to a couple of years ago, and it's tempting to treat that as the fix. It helps, but it doesn't solve the underlying problem for three reasons. First, cost and latency both scale with tokens processed — a 90%-full million-token window is slow and expensive on every single turn, not just the long ones. Second, advertised maximum and effective maximum aren't the same number — retrieval quality tends to degrade well before the window is technically full, so a lot of that headline capacity is theoretical. Third, a bigger window doesn't fix lost-in-the-middle — it just gives you more middle to lose things in.

What actually works

Re-state constraints, don't assume they persist. If a rule matters, put it back near the top of your most recent message once a session runs long, instead of trusting it survived from turn one. It's redundant-looking and it works.

Externalize memory instead of relying on chat history. A short standing file — project conventions, decisions made, constraints that don't change — that gets read fresh each session beats hoping a fact survives forty turns of scrollback. This is the same reason config files and READMEs exist: important state belongs somewhere more durable than a conversation log.

Summarize and compact instead of scrolling forever. Periodically collapsing the last stretch of conversation into a short summary — then continuing from the summary — keeps the signal-to-noise ratio inside the window high, instead of letting old back-and-forth crowd out the parts that still matter.

Retrieve, don't stuff. For anything that resembles a knowledge base rather than a live conversation, pull in only the specific passages relevant to the current question instead of pasting an entire document every turn. A well-targeted 500 tokens beats an unfiltered 50,000 almost every time, and it's cheaper to boot.

Start a fresh thread on purpose. When a session has drifted — the model is contradicting earlier decisions, or visibly straining — a clean restart with a tight, current summary of state usually outperforms continuing to push against a degraded window.

Where this shows up outside the chat box

This isn't just a chatbot quirk — it's the same constraint every AI feature in a real product has to design around. When we built ShotCanvas's AI screenshot picker and headline writer, the prompts stay narrow and single-purpose on purpose: one focused request per call instead of one sprawling conversation carrying accumulated context across a whole design session. It's less impressive-looking than a chatty AI assistant, and it's also a lot harder to derail.

If you're building anything on top of an LLM — or just trying to get a long pairing session with one to stay useful — the fix is rarely a bigger model. It's usually a smaller, better-kept window.