Alpesh Kumar

An agent can look impressive in a demo and still fail in a real workflow. Context engineering helps AI agents receive the right information for the next decision. After a few turns, an agent may repeat a question, follow an outdated instruction, or spend most of its prompt on tool logs that do not affect its next action.

Instead, an agent needs a deliberate context pipeline. The pipeline preserves durable facts, retrieves evidence for the task at hand, and leaves enough token capacity to reason, use tools, and respond.

This guide helps developers decide what belongs in memory, what to retrieve on demand, and where to set token boundaries before adding another model, vector database, or memory feature.

Key takeaways

• Treat context as a limited working set for the next action, not an archive.
• Keep durable facts separate from transient history and task-specific evidence.
• Retrieve a small, explainable set, then filter or compress the rest.
• Reserve tokens for instructions, tool results, and the final response.
• Measure whether each source improves task success enough to justify its cost.

Context engineering decides what the agent sees next

A language model can only work with information included in a particular request. However, an agent works across turns, tools, documents, users, and changing goals. Context engineering is the policy that decides which inputs it receives for the next action.

This differs from choosing a model with a larger context window. A larger window gives the system more capacity, but it does not decide what is accurate, current, authorised, or useful. Consequently, old messages and loosely related documents can make the important instruction harder to find.

Teams often bundle three distinct jobs under “agent memory.” In practice, each needs separate rules:

• Memory keeps facts or decisions that may be useful after the current turn.
• Retrieval finds external or historical material relevant to the current task.
• Token budgeting limits how much space each source may use in the prompt.

A useful model is a pipeline:

Task + system rules
        ↓
Select durable memory
        ↓
Retrieve task-specific evidence
        ↓
Rank, filter, and compress
        ↓
Apply token budget
        ↓
Model decision / tool call / response

Thus, every stage should make the next decision easier without introducing unreliable information.

Context engineering and durable memory

Classify information by how long it should live. Not every message deserves a permanent record.

Working memory holds material needed for the current task, such as the objective, a short plan, calculations, and recent tool results. It can disappear when the task is complete.

Session memory captures useful decisions or preferences within an ongoing conversation. For example, a summary is usually more useful than replaying an entire transcript in every prompt.

Long-term memory holds stable, approved facts such as a customer’s preferred language, an account constraint, or a recurring workflow rule. Because those records affect future decisions, they need provenance and an explicit write policy.

Consider a support agent. A language preference is durable, while the last troubleshooting step belongs to the session. By contrast, a raw browser log belongs only to working memory. Storing all three permanently creates noise and may retain data with no future value.

Use a strict retention rule. Save an item only when it is likely to help in a future task, can be stated clearly, and has a trustworthy source or owner. Otherwise, do not treat it as durable context.

Context engineering for retrieval

Memory is not a substitute for retrieval. A well-designed agent keeps stable facts close at hand, then retrieves changing or task-specific evidence when needed.

Shape the retrieval query around the next decision. “Which cancellation policy applies to this order?” is a useful intent. In contrast, “Find everything about cancellations” is likely to produce a broad and distracting result.

1. Form a narrow query from the task, the relevant entity, and the decision being made.
2. Retrieve a small candidate set from approved sources.
3. Then filter candidates for freshness, permission, source quality, and relevance.
4. Re-rank or summarise only the candidates that survive.
5. Finally, send the model an evidence packet with source labels and enough context to interpret each item.

For example, an operations agent deciding whether an order can be cancelled may need the current policy and the order’s present status. It does not need every historic note, every policy revision, or the full customer conversation.

Semantic similarity can help find candidates. Nevertheless, it cannot establish authority, freshness, or permission. Retrieval policy and access controls must perform those checks before the model sees the material.

Context engineering needs token budgets

A context window is capacity, not a target. More prompt material can raise cost and latency, obscure decisive evidence, and leave too little room for tools or a useful final answer.

Total usable prompt budget
├── System instructions and tool schemas: fixed reserve
├── Current task and constraints: protected reserve
├── Durable and session memory: capped reserve
├── Retrieved evidence: capped, ranked reserve
└── Safety buffer for tool results and response: protected reserve

The right numbers depend on the model, tool loop, and workflow. First, cap memory and retrieved evidence. Next, protect the current task and response space. Then test the allocation on representative tasks.

If evidence exceeds its allowance, do not cut the prompt at an arbitrary point. Instead, remove low-value candidates, deduplicate overlapping material, or create a concise summary that preserves source references.

Build a minimum viable pipeline

Start small. Put the current objective and non-negotiable rules in the prompt. Next, load only durable facts approved for that user or account. Do not copy an entire profile just because it exists.

Then retrieve a modest candidate set for the next action. Exclude stale, unauthorised, duplicate, or low-confidence material before it reaches the model. Finally, enforce the token budget and record what the agent used. That record helps diagnose whether a poor answer lacked evidence or lost it during filtering.

context = [system_rules, current_task]
context += select_durable_memory(user, cap=memory_cap)
context += retrieve_and_rank(task, cap=retrieval_cap)
context = trim_low_value_context(context, total_cap)
response = agent.run(context, response_reserve=output_cap)

Keep these policies visible in code and observability. They are product decisions, not incidental prompt-template details.

Make each source earn its place

More memory also creates more ways to be wrong. Stale memory can override current reality, untrusted content can poison memory, and lossy summaries can omit a decisive qualifier. Persistent records also require clear consent, access control, retention periods, and deletion behaviour.

Therefore, evaluate changes with labelled representative tasks. Track task success, grounded answers, tool errors, latency, token use, and the context items that influenced the result.

Choose one workflow and list every source it receives. Define its owner, freshness rule, permission boundary, token cap, and expected task metric. Context engineering makes an agent more dependable when it receives the right evidence for the decision in front of it.