Agent Evaluation Frameworks Compared for Production Use

You ship an agent, run it through a batch of test prompts, see the outputs look reasonable, and call it done. Then real users show up, and the agent starts looping on edge cases, calling the wrong tool, or quietly failing three steps before the final answer, and nobody notices until a support ticket lands. That gap exists because agent evaluation is a different discipline from LLM evaluation: the thing being tested is behavior across a multi-step sequence, not the quality of one generated output.

LLM evaluation asks a narrow question: given this input, is this output good? It's a bounded, single-turn problem, and benchmarks like MMLU or BLEU scores were built to answer exactly that. Agents break that frame. An agent has to decide what to do, pick a tool, read the result, decide again, and eventually produce an answer, and any one of those decisions can go wrong without the final output ever showing it.

Take a customer support agent that looks up an order, checks a refund policy, calculates a partial credit, and drafts a response. That's four steps, and a failure at any one of them cascades into the next. If you grade only the final response, three intermediate decisions, the ones that actually determined whether the agent did its job, go unchecked. Checking only the final output is like grading a math exam by looking at the answer and never checking the work.

Agent behavior has three distinct layers, and each one needs its own score, because each one answers a different question. Final-answer evaluation checks whether the last message matched what you expected, but that is necessary and far from sufficient. Trajectory evaluation asks whether the agent used the right tools, with the right arguments, in the right order, without looping, and a correct answer reached through a path with two policy-violating calls in the middle still counts as a failing trajectory. Per-turn evaluation asks whether any single turn represented a jailbreak attempt, a policy violation, or a moment where the user got frustrated, and logs, traces, latency numbers, and token counts record none of that.

Most evaluation setups never get past the first layer. They stop at final-answer pass/fail on a held-out benchmark, missing trajectory quality, tool-call correctness, looping behavior, and recovery from errors, and the signal that actually makes an agent better lives in the second and third layers, measured against real traffic.

The three-tier architecture that gives production coverage

Closing that gap takes three layers of evaluation, each built for a different purpose and run on a different schedule, and building them in order is the only reliable path from no coverage at all to real production confidence.

Level 1 is assertion-based unit testing. It's fast and deterministic, and it runs on every commit, so it catches the obvious regressions before anything ships. It's plain code: schema checks, format validation, required fields. If a tool call doesn't match its declared shape, the test fails in milliseconds and the build stops.

Level 2 is trace-based evaluation using an LLM as judge. It's slower and probabilistic, and it runs against curated datasets rather than every commit. It catches subtler quality problems, the kind a careful human reviewer would flag but a schema check never would. This layer looks at the full trajectory: tool selection, argument extraction, how the agent used a tool's result, whether it recovered from an error, whether its plan made sense, and whether it finished the task. These six dimensions are what separate real agent evaluation from a prompt-output check that has a trace view bolted on.

The math at this layer deserves attention. A per-step score that looks almost perfect can hide a failure rate that would sink the product in production. Score trajectories instead of trusting a per-step rubric, because that compounding effect is what a per-step rubric can't tell you.

Level 3 is online evaluation and A/B testing, running continuously against live production traffic, and it catches what no dataset can. Offline evals only show you a snapshot of whatever traffic existed when someone wrote the test set. Production is a moving target that shifts daily, and only live evaluation catches drift, genuinely new failure modes, users who are visibly frustrated, and jailbreak attempts that occur once and never again in the same form.

The right tool here is a per-turn classifier designed for sub-ninety-millisecond latency.

Two failure patterns occur constantly. Some teams skip straight to Level 3, tell themselves they'll just monitor production, and then they ship degraded behavior for weeks before anyone catches it. Others stop at Level 1 and treat a green CI badge as proof that the agent works. Neither is evaluation. Both are partial coverage mistaken for complete coverage.

The gap between offline and online scoring appears across every major framework on the market, which makes this architecture more than a tidy taxonomy. All seven of the dominant frameworks in the 2026 landscape score in CI or in batch, not per-turn inside the live request path. No single tool closes that gap on its own, so the three-tier structure isn't optional scaffolding: it's the only way to get full coverage by combining tools that were each built for one tier.

The six dimensions a framework must score to count as agent eval

Before comparing any tools, it helps to have a fixed scorecard, because without one, every framework comparison turns into a feature list instead of a measurement. Six dimensions separate genuine agent evaluation from LLM evaluation with a trajectory view bolted on, and a framework that can't score these independently is better understood as LLM eval wearing an agent costume.

Tool selection asks if the agent chose the right tool, or if it correctly chose none. It's scored as F1 with an explicit bucket for irrelevant calls, so an agent that called search when it had no reason to gets marked as a graded failure rather than quietly ignored.

Argument extraction asks whether the agent's inputs to a tool were both schema-valid and semantically correct. A date argument can pass a format check and still resolve to the wrong day, and a user doesn't care that the field was technically valid if the agent booked the wrong appointment.

Result utilization asks if the agent actually used what the tool returned, or if it substituted its own guess instead. An agent that reports a number different from the one the tool handed back has hallucinated, even if the final sentence reads smoothly and sounds confident.

Did the agent retry, fall back to another path, or escalate to a human? Per-call rubrics never see this kind of behavior at all; only a trajectory-level metric catches it.

Plan coherence asks whether the agent's path was free of loops, free of dead ends, and reasonably efficient. If you don't attribute cost step by step, a loop can hide inside a correct final answer and stay invisible in a simple trace total.

Task completion asks whether the agent actually finished the user's goal, measured against the state of the system afterward rather than just the words in the final message. tau-bench, for example, checks the database state directly rather than trusting the message, because an agent can claim it booked a flight without the booking ever existing.

Scoring these six separately changes what debugging looks like. Instead of "the agent failed," a team gets something like "the argument extractor regressed on date strings on the flight-booking path," which turns three days of guessing into one targeted bisect. That's the practical payoff of the scorecard: specificity that points straight at the broken component.

Level 1 frameworks: deterministic unit testing in CI

The right tool at Level 1 treats agent evaluation as a software testing problem, not a modeling problem: pytest-style assertions, schema checks, and tool-call verification that run in milliseconds and never require a model call.

UFO approaches this from the infrastructure layer rather than the test-suite layer. Its runtime enforces deterministic preconditions, things like tool schemas, action constraints, and safe-mode controls, at the point of execution, before any evaluation framework ever sees the output. So if a tool call violates a declared schema, it simply cannot execute; it doesn't fail a test after the fact, it never runs. That framing makes deterministic checks first-class infrastructure built into the agent OS itself, not testing afterthoughts layered on top of a general-purpose runtime.

DeepEval is the strongest pure-framework pick for teams that already live in CI. It's Apache 2.0 open source, with roughly 18,400 GitHub stars as of June 2026. Its interface is pytest-style, so evaluation runs as a standard test suite and drops into any CI pipeline without new infrastructure to stand up. It checks tool-call accuracy at the span level alongside agentic trajectory metrics, so it covers most of the six dimensions right at the assertion layer and you don't need a separate tool. Its hosted Confident AI platform extends that into online evaluation on production traces: you get a free open-source tier and a paid Starter plan priced per user or seat, but unlimited seats come only on the higher flat-rate plan. For teams where CI is already the system of record and the pytest workflow is already in place, DeepEval is close to a drop-in fit.

OpenAI Evals is MIT open source and worth knowing, but it needs careful use. It's MIT open source with a substantial GitHub following, though the hosted product is retiring in late 2026. Its eval surface is YAML-defined, which keeps the barrier low for single-vendor stacks where Git already serves as the source of truth. Its trajectory support is thinner than DeepEval's: the completion protocol handles final-answer checks well but doesn't score the full step sequence. The hosted product is retiring and the trajectory coverage is narrower, so it fits single-vendor stacks with modest trajectory needs, not production agent fleets where tool-call correctness carries real weight.

Level 2 frameworks: trace-based evaluation with LLM-as-judge

At Level 2, you can see the real separation between tools built for agents from the start and LLM evaluation tools that added a trace view later. The right pick depends on which part of the trajectory a team needs to score most and what runtime the agent already runs on.

UFO's contribution at this layer comes from what its runtime already captures. Because its agent operating system records the full execution trace, every tool call, argument, intermediate state, and planning decision, any Level 2 framework can consume that trace directly, so teams aren't stuck rebuilding the trajectory layer from scratch before they can even start scoring. Agents running on a purpose-built OS tend to produce richer, more structured traces than agents bolted onto retrofitted cloud tooling, and that matters because an LLM judge working from a detailed trace has more to work with and fewer gaps it has to guess at. Context and memory persist across sessions by design, so you can build evaluation datasets with multi-session trajectories that a single-request runtime simply can't produce.

MLflow is the broadest option on pure metric coverage. It's the most widely deployed open-source AI engineering platform, and its scorer framework evaluates full execution traces, tool calls, reasoning chains, planning decisions, through mlflow.genai.evaluate(). It offers Agent GPA (Goal-Plan-Action) scorers for common evaluation patterns through its TruLens integration, plus custom Python scorers for anything domain-specific. Its judge alignment is built on research-backed algorithms, GEPA and MemAlign, which optimize judge prompts against human labels so the automated score tracks what a reviewer would actually flag rather than whatever a generic rubric assumes. Tracing, prompt optimization, and governance live on one platform, and DeepEval, Ragas, and Arize Phoenix all integrate natively as pluggable scorers. For teams that want the widest metric coverage along with a feedback loop that improves the judges over time, MLflow is hard to beat.

LangSmith is the strongest fit for teams already standardized on LangGraph. Its AgentEvals helpers SDK is MIT-licensed, though the platform itself is closed; the free Developer plan comes with a limited monthly trace allowance, one seat, and 14-day retention, with a Plus tier priced per seat for teams that outgrow that. Its trajectory tracing for LangGraph executions is native: the framework understands what a Runnable is and what a planner's state diff means without anyone wiring up manual spans, and its per-node evaluator setup scores the planner, the tool node, and the post-tool LLM call as separate things. Prompts, datasets, deployment, fleet workflows, and a studio interface all sit on one surface, which makes it the lowest-friction choice specifically when LangGraph is already the runtime. The honest limitations matter too: custom agents, LiteLLM, and direct provider SDKs all sit awkwardly inside it, per-seat pricing gets expensive once more than a couple of people need access, and its metric library is shallower than MLflow's or DeepEval's once the trajectory isn't a LangGraph trajectory. If every agent run on your team already goes through LangGraph, pick anything else and you rebuild a trajectory layer LangSmith already hands over for free.

Arize Phoenix earns its place as a debugging tool first and a scoring tool second. Arize Phoenix handles trajectory and path-convergence evaluation, offers phoenix.evals for LLM-as-judge scoring, and plugs natively into MLflow as a metric. It fits best for teams extending existing ML observability work into LLM evaluation, or anywhere trace debugging, not scoring at scale, is the daily workflow.

RAGAS rounds out the Level 2 field as the fully free option: Apache 2.0 open source, with roughly 14,500 GitHub stars as of June 2026, and no paid tier at all. If a team weighs cost above everything else, that tells you something on its own, even before you compare its metric depth against the rest of the field.