The short version
Context engineering in 2026 means deciding what the model sees on every single call: what stays in the window, what gets dropped or compacted, and what lives outside in files or a retrieval index. Our evals on a production AI tutor showed that with prompt caching, keeping everything is often cheaper, faster, and better at remembering than summarizing, so compaction should be a deliberate response to a named constraint rather than a default.
- Under modern prompt caching, keeping the full history beat every summarization strategy we tested on cost, latency, and memory recall at once. Summarizing rewrites the cached prefix, so you pay full price to recompute everything you just tried to save.
- The cheap tier of compaction is the part that pays. Capping every tool output at a stable size cut our cost per turn by 38% with no measurable loss in memory, because it shrinks the context without rewriting the prefix the cache depends on.
- Compaction is not dead, it is conditional. Name your constraint first: a window that does not fit, a cached input price above roughly $0.55 per million tokens, or measured quality rot. Each one points to a different fix.
- Measure on your own system. Our production defaults looked reasonable, scored 38% on memory probes, cost twice as much as doing nothing, and still produced answers a blind judge rated as good. That combination is why you cannot eyeball this.
You told the agent one rule at turn 1. It agreed. Forty-five turns of greps, diffs, and logs later, it does exactly the thing you told it not to do, and announces it proudly. If you have run any long agent session, you have lived this. I opened our workshop with this exact chat log because everyone in the room immediately recognizes their own terminal.
The model didn’t get dumber. Its context did.
At the AI Engineer World’s Fair, Omar Solano, Samridhi Vaid, and I ran a workshop on what to do about it: context engineering in 2026, shown on our open-source AI tutor instead of on toy examples. We ran the experiments, spent a few hundred dollars on evals, and got results that contradicted our own production setup. This article covers everything from the talk, plus the findings we had to rush past on stage and a few that landed after it. If you prefer to watch, the full workshop is right here, and the article continues below.
Everything here is open source: the tutor’s full codebase, the live demo with all the experiment results, and the slides. You can poke at every number I mention.
Why a tutor is a context engineering problem
At Towards AI, we build courses for AI engineers, and we provide an AI tutor that answers student questions grounded in our lessons. That tutor is the case study for everything that follows, and it had five hard requirements: answers grounded in our content rather than the model’s general knowledge, scoped to the student’s current course, able to hold long debugging sessions with follow-ups, able to read and write code, and fast enough that latency doesn’t ruin the experience. Streaming speed is UX for a tutor.
If we fail at this, the student experience turns into that git-push story above, except the person on the other end paid for the course. During the workshop I illustrated where that leads with a slide about students asking for refunds. It got a laugh, but it’s genuinely the business stake: agents that forget mid-session cost real money.
Two properties of LLMs create the whole problem. First, the context window is finite, and everything competes for one attention budget: instructions, retrieved lessons, tool outputs, code. Second, the model is stateless. Every call starts from zero, and between sessions nothing persists on its own. Within a session, that gives you context management. Across sessions, memory. The workshop, and this article, focus on the first, because you can’t have good multi-session memory if a single session is already falling apart.
What the model actually sees, every call
Here is the part I find people underestimate the most. On every single call, the model re-reads the system prompt, the tool definitions, the entire chat history, every old tool output, the retrieved course chunks, and finally the student’s question. The question is usually the smallest piece, though students paste error logs, so not always.
In our tutor, the segment that balloons is old tool outputs: stale retrieved chunks, tool-call and tool-result pairs, files the agent opened along the way. Our very first finding was that those retrieval payloads, not the chat history, dominate the input. We let each call fill up to 100k tokens of retrieved material, and turns were landing around 200k input tokens. The conversation was never the expensive part. That matters, because most advice about long agent sessions is aimed at trimming chat history, which in our system was rounding error.
And a growing window hurts three times over. Quality degrades as facts get buried, the classic lost-in-the-middle effect that people now file under context rot. The chart I showed on stage for it is worse than most people expect: on a twenty-document window, accuracy starts around 75%, sags into the mid-forties for facts sitting in the middle, and that dip falls below what the same model scores with no documents at all. Handing it the right material in the wrong position was worse than handing it nothing.
Cost climbs too, because the whole history is re-sent and re-billed every turn. And time to first token climbs with input size, because the model reprocesses the window on each call. Our own measurements put median time to first token at roughly 22 seconds below 100k input tokens and around 76 seconds above 800k. Users feel that.
Notice the order I put those in, though. We went in expecting to manage context for quality. The experiments will show that, for our workload, spend and speed were the real reasons. Hold that thought.
The compaction toolbox, from free to fancy
Compaction is a simple idea: keep the smallest context that still contains what you need to answer, and drop or relocate the rest. The techniques form a ladder, and I like presenting it in the order of how much they cost you: trivial tools first, then spending tokens to save tokens, then offloading.
The trivial tier needs no LLM at all. Observation truncation cuts one absurdly large tool output down to its head and tail, like a student’s 300-line stack trace where only the command and the final error matter, with a note telling the model it can re-fetch if needed. Trimming, or a sliding window, keeps the last N turns and drops the oldest. Tool-result clearing swaps consumed outputs for a re-fetchable placeholder while keeping the call record. Cheap, predictable, and honestly where most people should start. Remember this tier, because it wins later.
Then you can spend tokens to save tokens, using a model, and it doesn’t even need to be a big one. Selective retention keeps constraints, decisions, and open tasks while dropping dead ends and duplicates. Summarization continuously replaces old history with a running gist, lossy by design. And full compaction, the Claude Code move, collapses the entire history into one fresh summary and restarts from it when you hit the limit. There is also delta summarization, where you only summarize new turns and append, which matters mostly when you spawn subagents. Our tutor runs fine as one agent, so we skipped that complexity, which is the same “start simple” stance I push in most of my videos.
The third tier is my favorite: don’t delete, relocate. Offload details to files or a memory tool and keep a pointer in context. It’s fully reversible, nothing is lost, and retrieval pulls things back just in time. This is Karpathy’s “LLM wiki” idea, a file tree the agent maintains and re-reads, and it’s the same shape Claude Code and friends converged on with CLAUDE.md and skills: durable files plus a small live window.
We use a concrete recipe from production for this: write chunks to files, cross-linked with pointers, keep one index file that maps them all, then let the agent read only that index. It searches its way back to details per task, so a complex question pulls more and a simple one pulls less. Context scales with task complexity instead of with session length.
A quick parenthesis on skills, because we see this with every client we work with: the field is converging on many small, focused skills rather than one giant instruction file. Names and short descriptions stay visible, bodies load only on use. Progressive disclosure, applied to instructions. Same principle as everything above.
One honest aside on GraphRAG, since people ask constantly. We built it and compared it against plain hybrid retrieval on 41 real cases. Both cited the correct source 100% of the time, they tied on lesson recall at 76%, and classical retrieval actually had the better mean reciprocal rank, 0.70 against 0.65. GraphRAG sent 61% more input tokens and cost 44% more per turn. Building the index over 90 documents cost about $45, and doing the full corpus would have run near $2,000. So we don’t use it. If your dataset is a web of interconnected entities, test it. Ours is lessons and docs, and plain hybrid retrieval was enough.
The twist: prompt caching changes the math
All of that, the GraphRAG index, the rerank calls, the retrieval budget, assumes tokens cost what the price page says they cost. They don’t, and that’s where the 2026 part actually kicks in: the field’s standard advice quietly inverted.
Almost every serious provider now offers prompt caching. When you resend a context the provider has already processed, those tokens are billed at a fraction of the price: roughly 90% off on Gemini, and on DeepSeek the cached rate drops from $0.14 to $0.0028 per million tokens, about 50 times cheaper. Cached tokens are also already computed, so the first token comes back much faster. Some of it is implicit and automatic, which is what our tutor rides, and some of it you pin explicitly, like Anthropic’s cache_control blocks, where the guarantee comes with a TTL and storage rent.
That diagram comes from a thread by @its_ao that I genuinely recommend reading in full.
Now connect this to compaction. The cache only hits when the previous context is resent unchanged. The moment you summarize, compress, or otherwise rewrite the history, the provider sees a brand new prefix and bills every token at full price. Your clever token-saving transformation just threw away a 50x discount. Against DeepSeek’s cache pricing, a summary has to shrink the context by more than 50x to even break even on cost, and good luck doing that without losing the one detail the student asks about next.
Which is why summarization is potentially a trap. Manus AI put it about as strongly as you can in their write-up on building the agent: “If I had to choose just one metric, I’d argue that the KV-cache hit rate is the single most important metric for a production-stage AI agent.” When I first read that line it sounded like an exaggeration. After our eval bill, it doesn’t.
The serious harnesses all reflect this now, and we know how they behave thanks to the leak on one side and Codex being open source on the other. Claude Code compacts rarely and structurally, trimming tool outputs without a model call and only replacing history with a summary near the window limit, with CLAUDE.md and skills surviving the reset. Codex has a local model write a handoff summary. Anthropic and OpenAI both ship context management in the API itself, where old turns get dropped while the system prompt stays cached. Notice the pattern: they all compact rarely, deliberately, and cache-consciously, not continuously.
The practical hygiene that falls out of this, and it’s what we now do ourselves: start fresh sessions when you switch tasks, scope file context narrowly, disconnect unused tools, compact only when needed, optimize for cache hits, route simple tasks to cheaper models, and log everything. That last one is the most skipped and the easiest to fix. I just asked Claude to wire Opik into our stack and it did it. Track cache hit rate, track abnormally long outputs, track user frustration.
All of it together is what context engineering means now: deciding what the model sees, every single call. Prompt engineering grew up, and compaction, memory, and retrieval moved in under one roof.
The tutor, built and measured
Omar took over the second part of the workshop with the part I wish more conference talks had: the actual system and the harness that grades it.
The agent itself is deliberately boring. One agent built with LangChain’s create_agent and an in-memory checkpointer, a middleware stack that can cap, clear, summarize, and scope, a FastAPI backend, and a Next.js chat UI streaming to the student. That’s it. No agent swarm.
The agent is small on purpose. The grounding is what makes it a tutor: a corpus built from 14 sources, our five courses plus nine documentation sets like LangChain, LlamaIndex, OpenAI, and the Claude Code docs. No window holds that, so we embed once and retrieve per question.
Grounding tool one, retrieve_tutor_context, is a classic hybrid pipeline, and every number in it was tuned by earlier experiments rather than vibes. It scopes to the student’s selected source, then runs dense embedding search and BM25 keyword search in parallel, top 15 and top 30. Reciprocal rank fusion merges the two lists, Cohere reranks down to the top 5, anything scoring under 0.10 gets dropped, and whatever survives fills a budget of up to 100k tokens.
Grounding tool two is the fun one. run_kb_command gives the agent a read-only shell over the corpus, the same way coding agents browse a codebase: rg, grep, find, ls, sed, head, cat, wc. It’s jailed to the knowledge base folder, with an 8-second timeout, a 40k-character output cap, and a ceiling of 20 commands per turn, so it can look around but it can’t run away. The knowledge base has three layers: raw markdown mirrors of everything, machine-generated indexes of headings and code symbols, and a wiki of topics and frameworks that an agent wrote offline by reading the raw corpus. When we add a course, an agent updates the wiki following a maintainer file. The tutor only ever reads.
There is a paper making this case properly, and it’s a good read. It calls the idea direct corpus interaction: let the agent search the raw corpus with terminal tools, no embeddings or vector index at all, and it beats strong sparse, dense, and reranking baselines on several benchmarks. The argument is that a single top-k call is a lossy interface, and evidence filtered out early can’t be recovered later no matter how good your reasoning is.
We wanted that to be true for us. And here’s the result I insist on sharing, because negative results almost never get shared: the agent loved the tool, reaching for it on roughly 89% of turns, at about 7.7 knowledge base calls per turn against 0.9 retrieval calls. Then we turned it off and grounding barely moved. Real student questions just aren’t multi-hop enough to need it. It was fun to build. It measured out to almost nothing. We’re keeping the code and shipping with it off.
That one also taught us something about measuring. Our first read of that experiment looked like a big win for turning the tool off, recall jumping from 50% to 96%, and it was an artifact: our recall metric only counted the retrieval tool, so an agent that found the right lesson by grepping got scored as a miss. Once we counted both paths fairly, the browse-enabled setup was at 100% and the browse-off one at 96%. Same runs, opposite conclusion. Most of our measurement bugs turned out to be like that, blind spots in labels and telemetry rather than wrong math.
For context management, the tutor’s production preset stacked three middlewares: clear stale tool outputs past 5k tokens while keeping retrieval results and the latest five, summarize old turns past 30k tokens while keeping the last 20 messages, and source preference for scoping the corpus.
Omar was upfront about the embarrassing part: we picked those defaults because they looked sensible. Production, running on unproven defaults. So, which configuration is actually best?
The harness: run, grade, gate, report
You have to measure, because as we found out, under modern caching the obvious move can invert, and the configuration space is far too big to eyeball. Quick vocabulary, since I’ll lean on these words for the rest of the piece: a preset is one tutor setup with everything else held fixed, a task type is one test set, a run is one preset tried on one model on one task type, and a bundle is what gets saved per turn, the answer plus every tool call, source, token count, and timing behind it.
We tested on two task types built from real students. Single-turn: 60 questions asked once, kept from 151 real academy posts after cleaning out duplicates and premises that our corpus had outgrown, graded on retrieval, key facts, and whether the response was the right kind, teach versus redirect. Omar had Codex scrape those from our academy forum, which felt appropriately meta. One rule there that I’d push on anyone building evals: we never write our own reference answers. Ground truth is the real staff reply, distilled into a handful of atomic, binary-checkable key points.
Sessions are the interesting half. We plant a fact at turn 0, grow the conversation with real course questions until the summarization trigger fires, then probe for the planted fact.
One session, start to finish: turn 0 plants “I’m a Unity dev. My weak topic is RAG evaluation. Dialogue must stream within 300 ms.” Ten turns of real questions push the chat past the trigger. Turn 11 asks, “One free evening, which topic do I drill, and name two metrics?” The expected answer is RAG evaluation, hit rate, MRR.
The harness has four stages: run_battery drives the real production agent code, one bundle per turn, and is the only step that costs money. Grade runs free code checks plus an LLM judge, and a practical note from Omar here: running the judge through a Claude Code or Codex subscription is currently cheaper than paying API rates for it. Check_triggers is the gate: if compaction never fired before the probe, the run is rejected, because the probe wouldn’t be testing memory. Report generates side-by-side tables and token-by-turn curves.
The judge sees only the question, the answer, and the grading criterion, never which preset produced it, so it can’t favor a strategy. It agreed with our human labels 98% of the time, and Omar hand-graded all 96 session probes with zero overrides. Because bundles are saved, we evaluate once and can re-grade whenever we want, which is how several of these findings got corrected later without spending another dollar. We ran 11 configurations in two rounds: a broad screen of eight presets on one trial each, then a tighter follow-up putting the three that mattered most through two trials. full_history and production were the reference points, and the variants included sliding-window, prompt-compression, selective-retention, and context-reset. Same model, same prompt, same retrieval, same dataset. 660 turns, zero API errors, about $88 for the broad screen and $62 for the follow-up, and close to $590 across the whole Gemini program. More than Omar expected, and that bill becomes a plot point in part three.
The surprise: compaction didn’t pay
Here’s session memory recall for every method we tested, on Gemini 3.5 Flash:
Not touching the context at all was the best strategy, in both the broad screen and the deeper follow-up. Our production preset, the one real students were using, sat at 58% in the broad screen and 38% in the tighter two-trial follow-up. The defaults we thought were good enough were worse than doing nothing. With one and two trials the exact percentages are noisy, and the team was clear about that on stage, but the ordering is the story.
Then the head-to-head that named this article. On sessions of 11 to 13 turns, full_history against production:
Keep-everything won all three at once. Cheaper, faster, and better at remembering. Our compacting preset sent about 41% fewer tokens and still paid roughly twice as much, because full_history billed about 87% of its input at the cache discount while every summarization rewrote the prefix and paid full price. And the cleared tool outputs made things worse in a second way: the agent re-retrieved information it used to have. Pay, drop, pay again.
Two details from that collapse are worth more than the headline number. First, the damage tracked the number of lossy rewrites, not the size of the live context. Runs with no compaction events answered every probe. Runs with two or three still answered them. Runs with five or six answered one in three. Each event was squeezing roughly 150k tokens into about 1.7k, so call it ninety to one, and the loss compounds every time you do it again on top of the last summary.
Second, and this is the part that made me uncomfortable: at probe time, the compacting presets were answering from contexts of 95k to 195k tokens and failing, while full_history answered from 363k to 879k tokens and never missed. A bigger context was not the problem. Rewriting it was.
And here is the finding I’d tattoo on anyone shipping an agent. When we scored those same answers holistically, asking a blind judge “is this a good tutor reply,” the production preset scored 97% to 99% while its memory recall in that same screen sat at 58%. On the specific turns where the tutor had demonstrably lost the planted fact, the judge rated the answer good 100% of the time. Every single one. The failure mode is not a visible crash or an obvious hallucination. It’s a confident, well-written, on-topic reply that silently ignores what the student told you ten turns ago. If your eval is an overall quality score, you will never see this. We only saw it because we planted facts and graded only those.
I want to be fair to compaction here, the way we tried to be on stage: these are results for 11-to-13-turn sessions, on a cached cloud model, on our workload. That is precisely what sets up the rest.
The part of compaction that did pay
Killing summarization is not the same as keeping everything forever, and the distinction is where our real production win came from.
Remember the trivial tier, the techniques that need no model call. We ran a clean two-by-two on DeepSeek, 414 turns per arm across three trials: cap tool outputs or don’t, summarize or don’t. Capping alone took cost per turn from $0.189 down to $0.117, a 38% cut, and it won in 14 of the 15 paired trajectories. Cache hit ratio barely moved, from 96.0% to 95.9%, and memory probe accuracy was identical to the uncapped run. That’s a free 38%.
Summarization, meanwhile, added cost on top of either baseline. Roughly 50% more than raw full history, and still 32% more than capped full history. Adding a model call to save tokens made the bill worse in every configuration we tried.
The reason the cap works and the summary doesn’t is the same reason as before. A stable, deterministic cap makes the prefix shorter but leaves it byte-identical from one turn to the next, so the cache still hits. A summary makes the prefix shorter by making it different, so the cache misses. Shrink the context, don’t rewrite it.
We even tried to fix summarization on its own terms. Codex does something clever: instead of replacing the history, it sends the unchanged history plus one instruction asking for a checkpoint summary, which keeps the cache intact for the summarizer’s own call. We built that, and it worked exactly as advertised. The summarizer call’s cache hit rate went from essentially zero to 94%, and the cost of summarizing dropped 87%. Total cost still came in 14% below the naive summarizer and 14% above simply capping and keeping everything. We made compaction as cheap as it can be, and it still lost to the boring option.
So when does compaction actually matter?
Samridhi’s part answered the question everyone should ask next: keep-everything won, but full history on a frontier-priced model is expensive, and a few hundred dollars of eval bills sting. So she pushed the same tests across three regimes: cheaper cached models, documents and tool outputs, and local models at scale. One framing from her section that I think is the most reusable idea in the whole workshop: context is not one thing. A chat history grows every turn. A pasted document or a huge tool output can blow the window in one shot. Each shape breaks differently and needs a different fix.
First lever, a cheaper model. Swap Gemini 3.5 Flash for DeepSeek V4 Flash, rerun the same sessions, and the cost per turn drops about 18 times, from roughly $0.11 to roughly $0.006, helped by that 50x cache discount. Keep-all still won on cost. The more important question was whether it still remembered, and it did: 95% of memory probes passed with the full history, against 32% when we summarized first.
My favorite chart of the entire study is this next one, because it sounds impossible until caching clicks: the cheapest run is the one sending the most tokens. Full history billed the most tokens of any arm, close to 296k per turn with 97% of them cache hits, and still came out cheapest at $0.0063 per turn. Over a 36-turn conversation that’s 1.78 million tokens sent. Every alternative cost more for sending less: the aggressive preset $0.0133, the summarizing production preset $0.0196, profile memory $0.0235. Two to four times the price.
What about context rot, then? For our job, it mostly didn’t show, and the story of how we learned that is a good warning. Our first context-rot run showed middle-of-context recall collapsing past 200k tokens, which is exactly the result everyone expects, so we almost shipped it as a finding. It was an artifact of our own answer cap: we’d limited responses to 300 tokens, and the model was getting cut off before it reached the fact.
With that fixed, recall of distinctive facts, code names and dates, held out to 800k tokens of context with retrieval switched off. Ambiguous facts did drop, roughly by half, so it’s not a free lunch. And the reassuring detail: the misses were the model declining to answer rather than confidently inventing something. The hard wall is real though. Past a million tokens, DeepSeek rejects the request outright instead of quietly compressing to fit.
So finding one buried detail is exactly what a tutor needs, and that task is easy for modern models even in huge contexts. Long-history reasoning is a coding-agent problem. Know which job yours is.
Then the economics turn. Cheap per turn is not cheap at scale: at 100k to 1M turns a day, even DeepSeek lands somewhere between $18k and $180k a month. Closer to home, at a few thousand students and around 10k turns a day, the same math is roughly $34k a month on Gemini against $1.9k on DeepSeek. The model swap is by far the biggest lever available. Past that, the per-token bill is what pushes you toward your own hardware.
Which is where everything flips. Local hardware means a small model, and on the MacBook we tested, a 32k context window. Our lessons alone are bigger than that, and students paste logs past it. If the context doesn’t fit, there is nothing to cache, and the entire keep-everything strategy evaporates. Try to keep the full context and it gets truncated, every single turn. Forced to compact, every technique landed in a sad 27-to-40% memory band, and jumping from a 7B to a 32B model on a much bigger machine didn’t escape it. The window is the limit, not the model.
For documents, though, local has a clean answer: retrieve, don’t stuff. Stuffing one long lesson into the window overflowed at every model size and returned a one-token stub after roughly 340 seconds of prefill. RAG answered at 100% in 25 to 65 seconds, about five times the throughput, off a context of roughly 3k tokens instead of 38k.
And if you retrieve, make it hybrid. This result is why I’ll keep repeating “keep a keyword path” until people are sick of it: dense-only retrieval held about 80% recall on buried facts up to 200k of corpus, then collapsed to 0% at 400k, because the reranker never surfaced the needle among hundreds of chunks. A made-up codename means almost nothing to an embedding. BM25 keyword search held 100% everywhere we tested. Our production hybrid setup exists because of charts like this one.
Side by side, local doesn’t match the cloud on chat memory, roughly a third against the low nineties, and a first token can take anywhere from 20 to 350 seconds. Those memory numbers are directional rather than a matched head-to-head, and the slide said so in a footnote, which I appreciate more than a clean-looking chart. What local does deliver is a dead per-token bill and private document Q&A through RAG at full accuracy. Local is about privacy and throughput. At our scale, it isn’t about cost.
The one number that decides it for you
After all these runs, we got to a rule I can actually hand you, and it came from repricing existing traces instead of running anything new.
Take the same recorded runs, keep-everything against compacting, and bill them at different providers’ rates. At DeepSeek prices, where cached input costs $0.0028 per million tokens, keep-everything wins by 13%. Reprice the identical traces at frontier rates where cached input runs $0.50 per million, and the two arms essentially tie. Cache reads go from about a quarter of the bill to 60% of it. Solve for the crossover and you get a threshold: keeping the full history wins as long as cached input costs less than roughly $0.55 per million tokens.
That single number does a lot of work. DeepSeek sits two hundred times under it, which is why keep-all is not close there. Frontier models sit right at it, which is why the answer flips with conversation length: in that repricing, full history still won a 22-turn session and lost a 36-turn one. No pricing cliff involved, just a longer history crossing the line.
So look up your provider’s cached input price before you write a single line of summarization code. That one lookup will tell you more than any blog post, including this one.
What we actually run now, and what you should take from it
After all of that, our tutor’s setup is almost anticlimactic, which I consider a compliment to the process. Model: DeepSeek V4 Flash, a cached cloud model, because caching makes keep-all the cheapest option at our volume. Retrieval: hybrid, dense plus BM25 plus rerank. Memory: keep everything, with a stable cap on every tool output so the context grows slowly without ever being rewritten. Tool-output clearing is off. The summarization trigger, which used to fire at 30k tokens, now sits at 800k, which in practice means it almost never fires. Our long sessions peak around 287k, so I’ll be honest that the 800k trigger itself is still unvalidated in production. It’s a guard rail we haven’t hit, not a tuned parameter.
The transferable lesson: don’t compact by default. Name the constraint first. If the context doesn’t fit the window, you’re in local-model territory, so retrieve. If the bill doesn’t scale, check your cached input price against that $0.55 threshold and swap to a cheaper cached model before you reach for summaries. If quality actually rots, measure where, because ours held to 800k on the task that mattered. Three different constraints, three different fixes, and only one of them is compaction. And when you do need to shrink the context, start with the tier that doesn’t need a model: cap, truncate, trim. Shrink the prefix, don’t rewrite it.
The repo has the full agent, the knowledge base tooling, and the eval harness, including the findings log where every one of these results is written down in order, corrections and all. The Hugging Face space has the live tutor and every experiment, including ones that didn’t fit the talk. The slides are public too. And if you want to build this exact tutor end to end, evals included, that’s what our Full Stack AI Engineering course walks through across about 60 hands-on hours. This piece pairs well with my earlier one on long context versus RAG, which asked the question these experiments ended up answering with receipts.
So measure on your own system, with your own users’ questions. Every confident default we shipped, the 5k clearing threshold, the 30k summarizer, the fancy browse tool, looked smart and lost to either doing nothing or doing less. Worse, the losses were invisible from the outside, because the tutor kept writing good-looking answers while quietly forgetting things. I’d rather learn that from an eval bill than from a student asking for a refund.
What’s the constraint in your system: the window, the bill, or actual rot? Genuinely curious how many of you are paying the summarization tax without having measured it. Thanks for spending your time here, and I’ll see you in the next one.
FAQ
What is context engineering?
Context engineering is deciding what the model sees on every single call: the system prompt, tool definitions, chat history, old tool outputs, retrieved documents, and the user's question all compete for one finite attention budget. It covers managing context within a session and persisting memory across sessions. It is prompt engineering grown up, with compaction, memory, and retrieval under one discipline.
Should I compact or summarize my agent's chat history?
Not by default. In our evals, summarization dropped in-session memory recall from 92% to roughly a third while costing about twice as much as keeping everything, because rewriting the history breaks the provider's prompt cache. Compact only when you can name the constraint: the context does not fit the window, cached input is expensive enough that resending stops being cheap, or quality measurably degrades.
Why does prompt caching make summarization expensive?
Providers charge much less for tokens they have already processed, up to about 50 times less on DeepSeek and roughly 90% off on Gemini. That discount only applies when the previous context is resent unchanged. Summarizing rewrites that prefix, so the cache misses and every token is billed at full price again. For summarization to pay for itself against a 50x cache discount, it has to shrink the context enormously without losing the details you need.
Is there any compaction technique that still pays off?
Yes, the cheap kind. Capping each tool output at a fixed size before it enters the history cut our cost per turn by 38% in a paired comparison, won in 14 of 15 trajectories, and left memory probe accuracy identical, because a stable cap shrinks the context without rewriting the prefix that the cache depends on. Truncation, trimming, and clearing sit in that same tier and need no model call at all.
Is context rot a real problem?
It depends on the job. Models do lose facts buried in long contexts on some tasks, and the classic chart is brutal: accuracy in the middle of a long window can fall below what the same model scores with no documents at all. But on our own task, recall of distinctive facts held out to 800k tokens of context, and the misses were the model declining to answer rather than inventing something. Ambiguous facts dropped by roughly half, so the driver was ambiguity more than length.
When is a local model worth it for an agent product?
When privacy or per-token economics force it, not before. At 100k to 1M turns a day, even a cheap API like DeepSeek lands between $18k and $180k a month, which is when owning the hardware starts to make sense. But a local model on consumer hardware means a small context window, 32k in our tests, so keep-everything stops working and you must retrieve. Local RAG answered document questions at 100% in our runs while stuffing the window overflowed at every model size.
How do I know if compaction is quietly hurting my agent?
Do not trust an overall quality score, because ours stayed at 97% to 99% while memory recall sat at 58%. Plant a specific fact early in a session, grow the conversation past your compaction trigger, then probe for that exact fact and grade only that. On the turns where our tutor had lost the fact, a blind judge still rated the answer good every time. The failure mode is a well-formed reply that silently ignores what the user told you.