AI agent evaluation starts with a simple reality: an agent can produce a polished final response and still fail the task.
Once an AI system can retrieve information, call tools, request approval and change an external system, the final text is only a report of what happened. It is not proof that the work was completed correctly.
That changes the engineering question. Instead of asking only whether an answer sounds right, teams need to ask whether the agent achieved the intended outcome, respected constraints, used tools correctly and left the system in a valid state.
AI agent evaluation: why final answers are not enough
Imagine a support agent receiving this request: “Cancel my annual subscription and make sure I am not charged again.” It replies: “Your subscription has been cancelled and you will not be charged again.” A response-only grader could reasonably score the message well.
Yet the execution record may reveal that the agent chose the wrong subscription, called a preview endpoint rather than the cancellation endpoint, misread an error response, left a renewal invoice in place or retried a non-idempotent refund request. The answer is persuasive; the outcome is wrong.
This is why agents need system-level evaluation. OpenAI calls end-to-end assessment of an agentic workflow trace grading. Anthropic describes a transcript or trajectory as the record of a trial, including outputs, tool calls, intermediate results and interactions. The practical lesson is simple: evaluate behaviour as well as language.
AI agent evaluation should inspect observable behaviour
Trajectory evaluation does not require a production system to depend on private reasoning traces. The useful trajectory is the observable operational record:
- the user request and relevant conversation state;
- available tools, permissions and policy boundaries;
- retrieval queries and returned evidence;
- tool names, arguments, responses, retries and errors;
- approvals, fallbacks and external state changes;
- the final response, latency, cost and number of tool calls.
This makes failures diagnosable. An agent that receives an empty search result, makes no mutation call and then claims that an account was updated has an observable tool-use and truthfulness failure. An agent that repeatedly submits malformed arguments may need a clearer schema, stronger validation or a better recovery path.
Three surfaces of robust AI agent evaluation
Outcome: did the user get the intended result?
Outcome evaluation checks whether the requested result occurred. For a document-access agent, that could mean Priya received viewer access to the correct Q3 planning document. For a coding agent, it may mean the requested feature works and the relevant tests pass.
AI agent evaluation of trajectory: did the agent act appropriately?
Trajectory checks assess the path to the result. Did the agent use an authorised tool? Did it retrieve the right record? Did it ask for approval before a high-impact action? Did it interpret a failed API response honestly rather than fabricate success?
State and safety: did the environment end correctly?
For agents that affect real systems, state matters more than rhetoric. A correct access-control update gives the intended person the intended permission without changing anyone else’s access. A payment action changes the correct ledger entry exactly once. A file operation creates the expected artifact without exposing confidential data.
Reliable agent behaviour
= correct outcome
+ appropriate tool use
+ verified state transition
+ constraint compliance
+ truthful final communicationAI agent evaluation catches failures that output grading misses
Claiming actions that never happened
An agent may say it cannot access invoices even when an invoice-search tool is available. The more serious reverse case is: “I have sent the invoice,” when no email tool call occurred. Both failures are visible when the final response is compared with the trace.
AI agent evaluation of negative constraints
Many requests contain a boundary as well as a goal: “Draft the reply, but do not send it”; “Do not delete anything”; “Only search this workspace”; “Ask before changing billing.” These conditions should become direct assertions, not vague expectations that the model will remember them.
assert no_tool_call("send_email")
assert no_destructive_action_without_approval()
assert all_actions_within_allowed_tenant()Misreading tool output
Tool output is evidence, not an automatic success signal. This response must block a claim of completion:
{
"success": false,
"error": {
"code": "ACCOUNT_LOCKED",
"message": "Changes require account-owner approval."
}
}An agent that sees this response and says “Your account has been updated” has failed at tool-output handling. Fluent prose does not repair an incorrect state transition.
AI agent evaluation of tool selection and arguments
An agent can understand the user’s request yet call create_customer instead of update_customer, creating a duplicate. It may produce valid JSON with semantically incorrect values: pounds where the API expects pence, a document ID from the wrong tenant or editor access where the user asked for viewer access.
Good schemas reduce these mistakes, but evaluation should still test tool selection, argument validity and the meaning of the resulting state.
AI agent evaluation should not require one exact tool sequence
There is a trap in trajectory evaluation: demanding one predefined sequence for every task. Agents can legitimately find different routes to the same result, especially when retrieval order does not change correctness.
Use strict ordering only for genuine invariants:
- Verify the requester’s identity.
- Confirm the record is eligible for change.
- Obtain approval where required.
- Perform the mutation.
- Verify the persisted result.
- Report the result accurately.
In this workflow, changing the order can create a safety failure. Elsewhere, grade the invariant rather than the exact route. Anthropic warns that overly rigid step matching can punish valid approaches, while LangChain documents strict, unordered, subset and superset trajectory matching for different needs.
A practical access-control example
Consider an internal agent with three tools:
search_documents(query, team_id)
get_document(document_id)
share_document(document_id, user_id, permission)The user asks: “Give Priya viewer access to the Q3 planning document.” A strong evaluation checks more than the final message:
- Outcome: the correct document was found and Priya has viewer access.
- Scope: no other user, document or permission changed.
- Authorisation: the requester was allowed to share the document.
- Tool quality: the sharing call had the right document, user and permission.
- Verification: the persisted access-control list confirms the result.
- Communication: the agent reports the verified state without exposing unnecessary content.
AI agent evaluation needs the right grader for each question
Not every evaluator should be another LLM. The strongest suites layer deterministic checks, rubric-based model grading and targeted human review.
Deterministic checks
Use code where the answer is mechanically verifiable: did a database record change, did an API return success, was a prohibited tool avoided, did the action stay within the correct tenant, did the expected artifact pass validation, and did the run stay within a reasonable latency or cost budget?
AI agent evaluation with rubric-based model graders
Model graders are useful for semantic questions: was the research answer supported, did the agent explain a failure accurately, and was its communication proportionate and helpful? These graders need calibration against human-labelled examples. Otherwise, teams can get tidy scores that do not measure the behaviour they truly care about.
Human review
Human review remains essential for high-impact actions, policy-sensitive cases, new failure categories and calibration. The aim is not to read every trace. It is to route uncertain or risky cases to experts and turn confirmed failures into regression tests.
Build an improvement loop from real traces
The most valuable evaluation cases often come from real behaviour: support tickets, negative feedback, validation errors, timeouts, abandoned tasks, manual corrections and unexplained cost spikes.
- Capture structured traces.
- Identify recurrent failures and near misses.
- Turn representative examples into reproducible evaluation cases.
- Add outcome, trajectory and state graders.
- Improve the prompt, tool definition, policy or orchestration code.
- Run regression tests before release.
- Monitor production for new failure modes.
This extends the idea of an AI agent evaluation flywheel: every real failure becomes a future guardrail.
Start with the workflows that matter most
You do not need hundreds of synthetic tasks. Start with five to ten workflows that make meaningful changes outside the chat window. Define the user goal, prohibited actions, evidence required before completion, expected end state and escalation conditions.
Then monitor a scorecard rather than one broad quality number: task-success rate, constraint-compliance rate, correct-tool-selection rate, valid-argument rate, verified-state-change rate, unsupported-action-claim rate, tool-error recovery, latency and cost per successful task.
A strong overall average can hide a serious failure rate on refunds, access changes or deletions. Segment by workflow and risk level.
From convincing demos to dependable products
Final-answer evaluation asks whether an agent’s words look correct. AI agent evaluation asks whether the system behaved correctly.
That means tracing actions, testing invariants, verifying state and treating the final response as a report of evidence rather than evidence itself. When an agent can act beyond the chat window, that is the difference between a convincing demo and a dependable product.