--- title: "Context rot: a bigger context window doesn't mean a smarter model - a practical guide" date: 2026-06-07 description: "A model will accept a million tokens, but it won't understand them the way you think. Context rot is real reasoning degradation under long context. Here's a practical guide: how to arrange, trim and maintain context so the model doesn't get dumber - for chat users and for devs." tags: ["context rot", "context", "ai", "prompting", "agents", "rag"] --- ## TL;DR - **A large context window is input capacity, not a guarantee of reasoning quality.** The advertised window is not the same as the effective window. - **Degradation starts long before the window fills up** and hits multi-step tasks hardest - this phenomenon is called *context rot*. - **Models favor the start and the end of the context**, while information buried in the middle gets lost ("lost in the middle"). - **The overriding rule:** give the model the *minimum sufficient* context, not the *maximum available*. - **For chat users:** fresh chat, key facts at the end, extraction first - reasoning second. - **For devs:** RAG as a filter + reranking, compaction, structured notes, sub-agents, and a hard cap on the effective window measured on your own tasks. ## πŸš€ Intro: "it fits" is not the same as "it understands" You get a model with a million-token context window and the first thought is obvious: finally I can dump in *everything*. The whole repo, the entire email thread, the full set of contracts, six PDFs at once. The model will take it - it fits without a problem. The catch is that "it fits" and "it understands" are two completely different things. It's a bit like emptying an entire filing cabinet onto your desk. Technically everything is within reach, but finding the right sheet and drawing the correct conclusion from it gets harder and harder. The more paper, the worse - even though, in theory, you can "see" every page. That's exactly what happens with language models. The weakening of their reasoning as the context window fills up has earned a name: **context rot**. And, crucially, this is not a hard cliff where the model suddenly stops working at 999,999 tokens. It's a **gradient**: quality drops gradually, often long before the window is full. The Chroma report from July 2025 already showed this: 18 of the era's frontier models (GPT-4.1, Claude 4, Gemini 2.5) degraded as input length grew - *every single one*, no exceptions [1]. And no, the new generation hasn't fixed it. The LOCA-bench benchmark from February 2026 tested today's flagships - Claude Opus 4.5, GPT-5.2, Gemini 3 - and the drops can be even steeper: Opus 4.5's accuracy falls from 96% at 8K tokens to 14.7% at 256K, and GPT-5.2 from 72% to 21.3% [2]. Windows are now measured in millions of tokens, but that doesn't eliminate the problem - at most it shifts the threshold at which it starts to hurt. Good news: the phenomenon is well understood and largely avoidable. This post is a handy guide - first a quick look at *what* breaks and *why*, then concrete practices: separately for someone working in a chat and for a dev building agentic systems. ![U-shaped curve: model accuracy by position of information in the context - highest at the start and end, lowest in the middle](/media/context-rot-krzywa-u-en.webp) _The further from the edge of the context, the harder it is for the model to reach the information. The classic "lost in the middle" effect._ ## 🧠 What actually breaks (and why) Before we get to the practices, it's worth understanding the mechanism - because good practices follow directly from how a transformer works. This isn't a memory-capacity problem; it's about how the model distributes attention. > **Definition:** *Context rot* is the gradual decline in a model's retrieval and reasoning quality as input length grows - occurring even when the model has technical, error-free access to all the information it needs. The loudest symptom is **"lost in the middle"**. The classic study by Liu et al. showed that model performance follows a U-shaped function of information position: it's highest when the relevant fragment sits at the start or the end of the context, and drops noticeably when the model has to reach for something in the middle [3]. Interestingly, this holds even for models marketed as "long-context". Underneath sit several overlapping causes. You don't need to memorize them, but it helps to know this is neither an accident nor the model's "ill will": | Mechanism | What happens | Practical consequence | |---|---|---| | **Lost in the middle** | The model favors the edges of the input | A fact buried in the middle simply disappears | | **Softmax dilution** | A finite "attention pool" spreads over more and more tokens | The signal-to-noise ratio drops | | **RoPE decay** | Weaker "connectivity" between distant tokens | The model loses cause-and-effect relations over distance | | **Effective window < advertised** | Quality sags before the limit is reached | Don't trust the marketing number | | **Distractors and noise** | "Almost relevant" fragments mislead the model | Adding context *worsens* the result, doesn't improve it | | **Reasoning shift** | Under long context the model shortens its own thinking | Less self-checking exactly where it's needed most | The last two rows are especially insidious. The study by Du et al. showed something stronger than "lost in the middle" alone: **input length by itself degrades results even with perfect retrieval** and within the declared window - with reported drops from 13.9% to as much as 85% on math, QA and coding tasks [4]. And a recent preprint on a phenomenon called *reasoning shift* describes how, under long or distracting context, reasoning models can generate up to 65% shorter reasoning traces, checking themselves less often [5]. In other words: long context not only makes it harder to *find* a fact, it also nudges the model toward *shallower* thinking about it. ## 🎯 The overriding rule: minimum sufficient, not maximum available If you remember one sentence from this post, make it this: > Give the model the **minimum necessary information** to do the task - not the maximum you happen to have on hand. It's counterintuitive, because instinct says "I'll throw in this one more document, just in case". But every added token is a smaller attention budget for the rest, and a higher chance that critical information lands in a low-attention zone. The Anthropic team puts it plainly: a transformer forms "every token with every token" relations, so as context length grows the model's ability to capture those relations "stretches too thin" - it's a limited *attention budget* [6]. From this rule follows a second, equally important one: **the advertised window is not the effective window**. NVIDIA's RULER benchmark showed that although the tested models declare a context of 32K or more, only about half actually maintain satisfactory quality at a real 32K - even though nearly all of them ace simple "needle in a haystack" tests [7]. A simple test inflates the model's actual capability; degradation only surfaces on complex tasks. That's why some teams apply a hard ceiling - treating a model that advertises a million tokens as if it really had, say, 200 thousand - and they pick that threshold **by testing on their own tasks**. The exact number is arbitrary and depends on the model and the type of work. What matters is the principle: measure where *your* model stops being reliable, instead of trusting the number from a press release. ![Comparison of the advertised context window with the actual effective window in which reasoning quality stays high](/media/context-rot-efektywne-okno-en.webp) _The advertised capacity is the upper bound of the input. The effective window - the range in which the model still reasons reliably - is often much shorter._ > **πŸ”§ Exercise (effective-window test):** Take a real task you do regularly and run it at increasing context lengths: 20K, 50K, 100K, 300K tokens. Note the threshold at which quality starts to sag. That point - not the marketing number - is your real working window for this type of work. ## πŸ’¬ Practices for chat users (ChatGPT, Claude.ai, Gemini) Working in a chat window, with no API access? You have less leverage, but still plenty. Here are the most important habits. **1. Start a fresh chat more often than you think.** Without an API, a restart is your main tool of context hygiene. Open a new thread when you switch topics, when the model starts to wander or loop, or when the conversation gets long (roughly past a dozen-plus messages). Every additional turn adds noise that degrades the following answers. **2. Key instructions and the question - at the end (and optionally at the start).** Since the edges get the highest attention, don't bury your instruction in the middle of a wall of text. If you're pasting a long document, add a "Task and answer criteria" section *below* it, right before you send. With very long input, repeat the most important constraint at the end - it measurably reduces drift. **3. Extraction first, reasoning second.** This is one of the most effective tricks and it works even in a plain chat. Instead of one giant prompt, split it into two steps: > *Step 1:* "List only the facts from the text below that are needed to answer question X. Ignore the rest." > > *Step 2:* "Using only that list of facts, solve the problem." Du et al. actually tested a simple version of this approach - having the model first recite the relevant evidence and only then solve the task - and got a consistent improvement on the RULER benchmark [4]. You turn a long task into a short one, and a short context means stronger reasoning. **4. Don't paste the whole filing cabinet.** Before you drop in a document, weed out the obvious junk: footers, changelogs, repeating headers, duplicates, low-value attachments. The less noise on input, the fewer distractors that can mislead the model. **5. For important tasks, force verification.** In law, finance, code or math, long context raises the risk that the model finds the right fragment but connects it wrong with the rest. A multi-step prompt helps: "first list the premises, then solve, finally check the result for contradictions". This counters *reasoning shift* - the tendency to shorten thinking under long input [5]. ## πŸ› οΈ Practices for devs and agents (API, RAG, Claude Code) Here you have the full arsenal. Anthropic formalized these techniques as complementary strategies for long-horizon agents [6] - I'm adding a few from RAG and compression practice. **1. RAG as a filter, not a dumping ground.** The point of retrieval is to give the model *only* the most relevant fragments - but retrieval quality has to be high, otherwise you pull in "almost relevant" distractors that make things worse. In *Contextual Retrieval*, Anthropic reported concrete numbers (metric: failed-retrieval rate): contextual embeddings alone cut errors by 35%, combined with contextual BM25 by 49%, and with reranking added by 67% [8]. Reranking - a second stage in which a specialized model picks the 3-5 absolute best fragments from a wider pool - is now standard, not a luxury. **2. Set a hard effective-window cap and measure it.** Don't cap by gut feel. Build a small eval set from your real tasks and find where quality sags (see the exercise above, in the spirit of RULER [7]). Then stick to that threshold in production. **3. Compaction.** When a conversation nears the window limit, summarize its older part and start a new one from that summary. In the Anthropic API, compaction is built in and triggers by default at ~150K input tokens (the threshold is configurable). In Claude Code, auto-compact kicks in automatically at high window fill [9] - and since today's default Opus 4.8 runs on a window on the order of a million tokens, a large window all the more doesn't excuse you from context hygiene. Practical note: compression loses details (file paths, exact names, error messages), so it's usually better to run `/compact` proactively at 50-60% fill than to wait for the automatic one. **4. Structured note-taking.** Have the agent regularly write notes to persistent memory *outside* the context window and fetch them when needed. It's cheap long-term memory that doesn't burden the attention budget - state lives on disk, not in every subsequent call. **5. Sub-agent architectures.** Delegate narrow subtasks to specialized sub-agents, each working in a *clean* window and returning only a condensed distillate of its work. The main agent coordinates but doesn't carry the entire exploration junkyard in its context. This directly addresses the main agent failure mode: noise accumulation during searching, trial and backtracking. ![Comparison of a clean agent context with a context poisoned by noise accumulation over many turns](/media/context-rot-czysty-vs-zatruty-en.webp) _An agent solves the problem as long as its context stays clean. Compaction and sub-agents restore that cleanliness before noise can spoil it._ **6. Extraction before reasoning - as a pipeline stage.** The same principle as in chat, but automated: split the task into extraction β†’ consolidation β†’ reasoning β†’ verification, instead of one prompt for 300K tokens. For hard tasks, separate the roles into distinct calls: extractor β†’ analyst β†’ critic. **7. Prompt compression (optional).** When you really must fit a lot, tools like LLMLingua strip "unimportant" tokens before sending to the model. Reported compression reaches 20x with minimal quality loss - on reasoning tasks about 1.5 points lost at 20x compression [10]. It's a solution for specific cases, not a default step. **8. Positioning and repetition.** Just like in chat: place critical instructions and constraints at the edges, and with a very long prompt repeat the key rules near the end. It's a cheap intervention with a real effect. ![Processing pipeline: from a long noisy context, through evidence extraction, to reasoning on a short context and verification](/media/context-rot-ekstrakcja-rozumowanie-en.webp) _The best workflow for long context: first find the materials, make a short evidence packet, only then reason._ ## πŸ—ΊοΈ Decision cheat sheet Not every situation calls for heavy artillery. A quick map of what to use when: | Situation | Best approach | |---|---| | Simple fact lookup in a single document | Long context is usually enough | | Many documents, emails, contracts, a whole repo | RAG + reranking + a short global summary | | Task requiring multi-step reasoning | Fact extraction β†’ short context β†’ reasoning | | Important info might be in the middle of the prompt | Move it to the end or to a "key facts" section | | The model advertises 1M, but the task is hard | Don't trust the window alone - measure the effective threshold | | The result carries business or legal weight | Add a verification and source-citation step | | A long conversation starts to drift | Fresh chat (UI) or compaction (API/agent) | ## ⚠️ Three traps that are easy to fall into **"More context = better".** The most common mistake. Piling on documents pushes critical information into low-attention zones. Less, but more on-target, almost always wins. **Blind faith in the advertised window.** "If it holds a million tokens, it'll read a million tokens with comprehension" - no. Input capacity and reasoning quality are two different axes. **Forcing chain-of-thought on extremely long input.** The classic "think step by step" generates *more* tokens, which - on an already overloaded window - can deepen the dilution. Better to shorten the context first (extraction), and only then ask for reasoning. ## 🏁 Wrap-up With today's models, a long context window is great as a **warehouse** but poor as the sole **thinking mechanism**. Context rot isn't a defect of a particular model - it's a gradient baked into the transformer architecture, one that affects everyone and starts long before the window fills up. The best systems - and the best users - therefore act like a well-organized analyst: first they find the right materials, then they make a short evidence packet out of them, and only at the end do they reason. Everything else in this guide is a variation on that single idea, tuned to whether you're clicking in a chat or building an agent. One sentence for the road: **don't ask how much will fit - ask how much the model can actually carry.** ![Visual summary of good practices for working with large context and countering context rot](/media/context-rot-podsumowanie-en.webp) _Context rot in a nutshell: the minimum-sufficient-context principle plus a set of practices for the user and the dev._ ## πŸ“š Sources [1] Hong, K., Troynikov, A., Huber, J. (2025). *"Context Rot: How Increasing Input Tokens Impacts LLM Performance"*. Chroma. [https://www.trychroma.com/research/context-rot](https://www.trychroma.com/research/context-rot) [2] Zeng, W., Huang, Y., He, J. (2026). *"LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth"*. arXiv:2602.07962. [https://arxiv.org/abs/2602.07962](https://arxiv.org/abs/2602.07962) [3] Liu, N. F. et al. (2024). *"Lost in the Middle: How Language Models Use Long Contexts"*. Transactions of the ACL, 12, 157-173. [https://aclanthology.org/2024.tacl-1.9/](https://aclanthology.org/2024.tacl-1.9/) [4] Du, Y. et al. (2025). *"Context Length Alone Hurts LLM Performance Despite Perfect Retrieval"*. Findings of EMNLP 2025. [https://arxiv.org/abs/2510.05381](https://arxiv.org/abs/2510.05381) [5] Rodionov, G., Garipov, R., Yakushev, G. (2026). *"Reasoning Shift: How Context Silently Shortens LLM Reasoning"*. arXiv:2604.01161. [https://arxiv.org/abs/2604.01161](https://arxiv.org/abs/2604.01161) [6] Anthropic Applied AI (2025). *"Effective context engineering for AI agents"*. [https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) [7] Hsieh, C.-P. et al. (2024). *"RULER: What's the Real Context Size of Your Long-Context Language Models?"*. COLM 2024, arXiv:2404.06654. [https://arxiv.org/abs/2404.06654](https://arxiv.org/abs/2404.06654) [8] Ford, D. (2024). *"Introducing Contextual Retrieval"*. Anthropic. [https://www.anthropic.com/news/contextual-retrieval](https://www.anthropic.com/news/contextual-retrieval) [9] Anthropic. *Compaction* (API docs) and *Context window* (Claude Code docs). [https://platform.claude.com/docs/en/build-with-claude/compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) [10] Jiang, H. et al. (2023). *"LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models"*. EMNLP 2023, arXiv:2310.05736. [https://arxiv.org/abs/2310.05736](https://arxiv.org/abs/2310.05736) [11] Hsieh, C.-Y. et al. (2024). *"Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization"*. Findings of ACL 2024, arXiv:2406.16008. [https://arxiv.org/abs/2406.16008](https://arxiv.org/abs/2406.16008)