Context engineering: what to put in the window and what to leave out

For a single question, the prompt is the whole story. For an agent that runs for twenty steps, the prompt is a small slice of what the model reads. The rest is tool definitions, earlier replies, tool results, retrieved documents, and notes it wrote to itself. Deciding what fills that space, and what doesn't, is what people now call context engineering.

The term stuck because it names something real. Anthropic's engineering team published a widely read post on effective context engineering for agents in 2025, framing context as a finite resource with diminishing returns. The rest of this article turns that idea into working habits.

Why the window is scarce even when it's huge#

Models now accept very long inputs, so it's tempting to stop caring about size. Three things say otherwise.

Attention isn't uniform. As the window fills, models can lose track of details, mix up similar items, and skip instructions buried in the middle. Bigger windows reduce the hard limit without removing the softer one. People sometimes call this context rot: the more you pile in, the less reliably each piece gets used.

Cost scales with size. Every call re-reads the whole context. A bloated window means paying to reprocess the same clutter on every step, as how the agent loop works shows.

Latency scales too. More input means slower first tokens, which adds up over a long run.

The working principle is to give the model the smallest set of high-signal tokens that makes the desired outcome likely. Everything else is a liability.

What's in the window#

It helps to list what actually occupies the space on a given call.

1. The system prompt. Role, rules, standing guidance.
2. Tool definitions. Names, descriptions, and schemas, every time.
3. Examples. Sample inputs and outputs that show the expected behavior.
4. Retrieved material. Documents, code, search hits pulled in for this task.
5. Conversation history. Earlier user messages, model replies, and tool results.
6. Notes and memory. Anything the agent saved and reloaded.

Most teams over-invest in item one and neglect the others. A perfect system prompt can't compensate for 80,000 tokens of stale tool output crowding the history.

Write the system prompt at the right altitude#

Prompts fail in two opposite ways. Some are brittle: long lists of if-then rules that try to script every case and break the moment reality drifts. Others are vague: "be helpful and accurate," which gives the model nothing to act on.

The sweet spot is specific enough to guide behavior and flexible enough to let the model use judgment. State the goal, the constraints that really matter, the tone, and how to handle the common edge cases. Skip the exhaustive rulebook. If you find yourself adding a rule for every failure, consider whether a better example or a clearer goal would prevent the whole class at once. There's more on wording in prompt engineering that still works.

Just-in-time retrieval beats pre-loading#

The old instinct is to load everything that might be relevant up front. The better pattern for agents is to hold references and fetch details when needed.

Instead of pasting a whole codebase into the prompt, give the agent tools to list files, search, and read one file. Instead of loading every customer record, let it look up the one it needs. The context holds lightweight identifiers such as file paths, ticket numbers, and URLs, and detail arrives on demand.

This mirrors how a person works: you don't memorize the library, you know where to look. It keeps the window small, keeps each piece of information fresh, and lets the agent discover structure by exploring. The trade-off is extra tool calls and some latency, and for many tasks a hybrid works best: preload a small amount of essential context, then retrieve the rest as needed.

Retrieval quality then becomes the bottleneck, which is why RAG that works matters so much for this pattern.

Tame tool results#

Tool output is the fastest way to fill a window. One unfiltered web page or log file can dwarf everything else. A few tactics:

• Return less by default. Give tools a limit and a detail option, and choose brief defaults.
• Filter on your side. Strip HTML boilerplate, drop irrelevant fields, and return the useful lines of a log rather than the whole thing.
• Truncate loudly. If you cut output, say so, so the model can ask for more.
• Clear old results. Once a large result has served its purpose, replace it in the history with a short summary or a pointer. Tool results deep in the past rarely need to stay verbatim.

That last one is easy to overlook. Many long runs are dominated by outputs from step three that nobody needs at step thirty.

Compaction: summarize before you overflow#

When a run gets long, you can compress the history: have the model summarize what happened so far, then continue with the summary in place of the raw transcript. This is usually called compaction.

Good compaction keeps what matters: the goal, decisions made and why, open problems, key facts discovered, and the current state of the work. It drops what doesn't: raw tool output, dead ends, repeated pleasantries. The risk is losing a detail that turns out to matter, so tune it by testing. Start by preserving more than you think you need, then trim what you can prove is safe.

A practical setup triggers compaction when the history crosses a threshold, say 60 to 70 percent of the window, instead of waiting for a hard limit. Some agent tools now do this automatically, and it's worth checking how yours behaves.

Notes outside the window#

For work that spans many steps or many sessions, don't rely on the window at all. Let the agent write notes to persistent storage, such as a progress file, a to-do list, or a scratchpad, and reload the relevant parts when needed.

A coding agent might keep a NOTES.md with the plan, decisions, and what's left. A research agent might save findings to a file as it goes. When the context resets or gets compacted, the notes survive, and the agent picks up where it left off by reading them.

This is the practical core of agent memory. It's cheap, inspectable by humans, and surprisingly effective. It also gives you a debugging artifact: you can open the file and see what the agent believed.

Sub-agents with clean windows#

Another way to manage context is to hand focused sub-tasks to separate agents that start with a fresh window. The main agent delegates ("find every place we call the payment API"), the sub-agent does the exploring in its own context, and returns a short summary. The parent's window only grows by the summary, not by the thousands of tokens of exploration.

This works well for research and code search, where the useful output is small but the process is noisy. It's also where multi-agent designs earn their keep. The cost is more total tokens and extra coordination, so it isn't free, and multi-agent systems looks at when it pays off.

Order, structure, and caching#

How you arrange the window matters too.

Put stable content first. System prompt and tool definitions rarely change, so place them at the start. That lets providers cache the prefix and charge less to re-read it, which is the whole idea behind prompt caching.

Put the task and key instructions where they'll be noticed. For long inputs, restating the question or the important constraints near the end can help.

Use clear structure. Delimiters such as XML-style tags or Markdown headers help the model separate instructions from data. Labeling a chunk as <document> and another as <instructions> reduces the chance of mixing them up, and it also makes hostile text inside a document easier to treat as data.

Avoid contradictions. Two conflicting instructions, one in the system prompt and one three turns back, force the model to guess. Audit long-running prompts for drift.

A worked example: a support agent's window#

Picture a support agent that answers billing questions. Its first version was built the obvious way, and every call carried:

• a 1,500-token system prompt with forty rules accumulated from past incidents,
• twelve tool definitions from three connected servers,
• the full text of the refund policy, the pricing page, and the terms, pasted in "just in case,"
• the entire conversation, including three long tool results from earlier lookups.

Before the customer's question even arrived, the window held a large amount of material that had nothing to do with it. Answers were slow, costs were high, and the agent sometimes quoted the wrong policy because two documents disagreed.

A cleaner redesign:

1. The system prompt shrank to the goal, tone, and the handful of rules that actually matter, with two examples replacing twenty rules.
2. Tools got trimmed to the five the agent really used, and results were capped.
3. Policies moved out of the prompt. A search_policies tool now returns the relevant paragraph with its source, so the agent quotes current text and cites it.
4. Old tool results are summarized after use: "Looked up order A-104: shipped 3 March, delivered 6 March."
5. The stable prefix comes first, so caching applies to the system prompt and tools.

Nothing about the model changed. The window went from crowded to curated, and both accuracy and cost improved. That's the kind of win context engineering usually delivers: unglamorous, measurable, and repeatable.

Anti-patterns to watch for#

A few habits that reliably make context worse:

The kitchen sink. Adding everything that might conceivably help. Each addition feels harmless, and together they bury the signal.

The rule pile. Every incident adds a rule to the system prompt, none ever leave, and eventually the prompt contradicts itself. Schedule a periodic prune, and delete rules that no longer earn their place.

Raw dumps. Passing full API responses, logs, or web pages straight through. Somebody has to read them, and that somebody is a language model billing you per token.

Stale state. Old facts left in the history after they've changed: a file that was since edited, a ticket that was since closed. The model can't tell which version is current unless you make it obvious.

Duplicate context. The same policy pasted into the system prompt, again in a retrieved document, and again in a summary. It wastes space and invites conflicts.

No visibility. Not knowing what's in the window on a given call. If you can't print it, you can't fix it.

How to measure it#

Context engineering can feel like taste, but you can test it.

• Log the token count of each component per call: system, tools, retrieved, history, results.
• Watch which component grows fastest. That's where to cut first.
• Run your test tasks with and without a change, and compare success rate, steps, and cost. The method in evaluating an AI agent applies directly.
• Read transcripts asking a specific question: at the step where it went wrong, did the model have the information it needed, and was it buried?

If a change to context size doesn't move any number, you've learned it wasn't the problem. That's useful too.

A short checklist#

• Is every item in the window there for a reason?
• Are tool outputs capped, filtered, and clearly labeled?
• Do old results get summarized or dropped?
• Is there a plan for long runs: compaction, notes, or sub-agents?
• Are stable parts first, so caching works?
• Are instructions and data clearly separated?
• Have you measured, rather than guessed, what the window contains?

Prompts still matter, but on an agent's fifteenth step they're a rounding error. The teams that build reliable agents are usually the ones treating the whole window as something they design, one call at a time.