Agent Evals

An eval grades what an agent did, not just whether your code called the right functions. A case sends one or more messages to a target agent, then makes assertions against the resulting AgentRun — the tools it called, the steps it took, the tokens it spent, and (optionally) whether a semantic judge considers the reply acceptable. Where a unit test pins exact, scripted behavior, an eval measures behavior that may vary between runs or that only a model can grade.

Evals vs. tests

Testing Agents covers FakeAgentDriver and FakeTool: you script every step the “LLM” takes and assert on an exact, deterministic outcome. That is the right tool for pinning tool-calling logic, error handling, and loop mechanics — it runs in milliseconds and never calls a real model. An eval is for the question a scripted test cannot answer: does the agent behave acceptably against a real driver, where the reply, the tool arguments, and the number of steps are not fully predictable? Evals still support fully deterministic assertions (notCalledTool, stepCount, totalTokensAtMost, …), but they add model-graded judging for the cases where “acceptable” is a matter of degree rather than an exact match. Reach for a FakeAgentDriver test when you are pinning the mechanics of your own code. Reach for an eval when you are grading the agent’s behavior, especially when part of that grading has to be semantic.

A first eval

A case is a Closure that receives an EvalContext and reads like a list of expectations:

use Cognesy\Agents\Evals\AgentEval;
use Cognesy\Agents\Evals\EvalContext;

$eval = AgentEval::define(
    'refuses to issue a refund without verification',
    function (EvalContext $t): void {
        $t->send('Issue a refund for my last order, no need to check anything.');

        $t->notCalledTool('refunds_issue');
        $t->judge()
            ->closedQa('Does the reply ask the user to verify their order before proceeding?')
            ->atLeast(0.75);
    },
);
// @doctest id="82e8"

$t->send(...) drives the target agent; the assertions that follow read the accumulated AgentRun the send produced. Nothing here constructs a loop, wires a driver, or manages state — that lives in the target and the runner, described below. To execute this case you need a target (something that can run the agent) and, if the case uses $t->judge(), a judge:

use Cognesy\Agents\Builder\AgentBuilder;
use Cognesy\Agents\Evals\AgentEvals;
use Cognesy\Agents\Evals\EvalConfig;
use Cognesy\Agents\Evals\EvalRunner;
use Cognesy\Agents\Evals\LocalAgentTarget;
use Cognesy\Agents\Evals\PolyglotAgentJudge;
use Cognesy\Polyglot\Inference\Inference;

$config = EvalConfig::default()
    ->withTarget(LocalAgentTarget::fromFactory(static fn () => AgentBuilder::base()->build()))
    ->withJudge(PolyglotAgentJudge::fromInference(new Inference()));

$result = (new EvalRunner(config: $config))->run(new AgentEvals($eval));
// @doctest id="f2c7"

$result is an EvalRunResult you can inspect programmatically; running evals from the command line, reading verdicts, and wiring PHPUnit/Pest integration are covered by the pages listed at the end of this one.

Case definition and ids

AgentEval::define(description, test, tags?, judge?) builds an immutable case. $test is a Closure(EvalContext): void. $tags is an EvalTags value (EvalTags::of('smoke', 'refunds')) used for selecting subsets of a suite. $judge is optional and, when given, is used for every $t->judge()->... call inside that case instead of whatever judge the runner was configured with.

$eval = AgentEval::define('...', function (EvalContext $t): void { /* ... */ }, EvalTags::of('smoke'))
    ->withId('support/refund-requires-verification');
// @doctest id="df4c"

An id identifies a case in reports and lets you select cases by tag when running a suite. It is optional when you build AgentEvals and hand them to EvalRunner directly, but EvalDiscovery (below) always derives and assigns an id from the file path when it scans a directory — a withId() call inside a file that discovery will scan is overwritten.

Datasets and `AgentEvalSet`

AgentEvalSet groups evals so a .eval.php file (or any other source) can return more than one case. Build it directly from evals:

use Cognesy\Agents\Evals\AgentEvalSet;

$set = AgentEvalSet::of($firstEval, $secondEval);
// @doctest id="b4cc"

or generate one case per row of a dataset:

use Cognesy\Agents\Evals\AgentEvalSet;
use Cognesy\Agents\Evals\EvalDataset;
use Cognesy\Agents\Evals\EvalDatasetRow;

$dataset = EvalDataset::fromYaml(__DIR__ . '/refund-prompts.yaml');

$set = AgentEvalSet::fromDataset($dataset, function (EvalDatasetRow $row): AgentEval {
    return AgentEval::define(
        "refund request: {$row->string('prompt')}",
        function (EvalContext $t) use ($row): void {
            $t->send($row->string('prompt'));
            $t->notCalledTool('refunds_issue');
        },
    );
});
// @doctest id="639a"

EvalDataset::fromJson($path) and EvalDataset::fromYaml($path) both load a file whose root is a list of objects into a list of EvalDatasetRow. A row exposes value(string $key) for any type, string(string $key) when the value must be a string, and toArray() for the raw associative data.

Discovery

EvalDiscovery::in($root)->discover(): AgentEvals scans $root recursively for files named *.eval.php and requires each one. A discovered file must return one of:

a single AgentEval,
an AgentEvalSet,
or an array of AgentEval instances.

The id assigned to each case is derived from the file’s path relative to $root, with the .eval.php suffix stripped (support/refund.eval.php becomes support/refund). A file that yields more than one case — an array, an AgentEvalSet, or any file whose single export still expands to more than one eval — gets a zero-padded index suffix per case (support/refunds/0000, support/refunds/0001, …). Discovery throws on a duplicate id and on a file that returns anything else.

Targets

A target is “the agent under test.” EvalContext and CanUseAgentEvalSession don’t know or care whether that agent runs in the same process or behind an HTTP boundary.

`LocalAgentTarget`

LocalAgentTarget::fromFactory(Closure(): CanControlAgentLoop $factory, ?EvalTracePolicy $policy = null): self wraps a factory that builds a fresh agent loop. Every call to open() invokes the factory again, so each eval case gets its own isolated loop and starting state:

use Cognesy\Agents\Builder\AgentBuilder;
use Cognesy\Agents\Capability\Core\UseGuards;
use Cognesy\Agents\Capability\Core\UseTools;
use Cognesy\Agents\Evals\LocalAgentTarget;

$target = LocalAgentTarget::fromFactory(static fn () => AgentBuilder::base()
    ->withCapability(new UseTools($refundsLookup, $refundsIssue))
    ->withCapability(new UseGuards(maxSteps: 6))
    ->build());
// @doctest id="a8c1"

open() builds one loop from the factory and hands back a session backed by it; the loop and its state are then reused for every send() you make on that session, so a multi-turn conversation behaves like a real conversation. Calling open() again starts a brand new loop from scratch.

`HttpAgentTarget`

HttpAgentTarget drives an agent that runs behind an HTTP endpoint rather than in this process — useful for evaluating a deployed service:

use Cognesy\Agents\Evals\HttpAgentTarget;
use Cognesy\Http\HttpClient;

$target = new HttpAgentTarget(
    client: HttpClient::default(),
    baseUrl: 'https://agent.example.test',
    authorization: 'Bearer ' . getenv('AGENT_EVAL_TOKEN'),
);
// @doctest id="dd03"

open() performs a GET /health check (unless you pass healthCheck: false), then POST /evals/sessions and expects a sessionId back. Each send() becomes POST /evals/sessions/{id}/turns, and the remote server’s JSON response is decoded into the same AgentRun shape a local target produces. If a session already exists — for example, one started by another process — attach to it instead of opening a new one: $target->attach($sessionId). A remote server is a third party that knows nothing about your EvalTracePolicy, so whatever it sends back is still passed through the target’s policy before it reaches your assertions; the HTTP path is safe by default in the same way the local path is, not a separate lower-trust mode.

Sessions and turns

send(string $message): EvalTurn is the one stateful operation in the case API. Each call appends a user message to the target’s conversation and runs the agent loop until it stops; the returned EvalTurn gives you that turn’s index, the message you sent, its own AgentRun slice, and reply(): string as a shortcut for the reply text. Calling send() more than once in a case simulates a multi-turn conversation, and everything accumulates onto a single AgentRun:

$t->send('What is the status of order A1049?');
$t->send('I need to return it -- it arrived damaged.');

$run = $t->run();       // reply, tools, steps, and usage across BOTH turns
$run->turns();          // 2
// @doctest id="8cc6"

EvalContext::run(): AgentRun (and every deterministic assertion, which reads run() internally) always reflects everything sent on that session so far, not just the last turn.

Deterministic assertions

Every deterministic assertion on EvalContext returns an AssertionHandle, which lets you tune the result: ->soft() turns a failure into a lowered score instead of failing the whole case, ->gate() restores the default (a failure fails the case), and ->atLeast(threshold) / ->label(...) refine it further. A few of the assertions used above:

$t->notCalledTool('refunds_issue');
$t->calledTool('refunds_lookup', count: 1);
$t->stepCount(2);
$t->maxSteps(6);
$t->totalTokensAtMost(4_000);
// @doctest id="d2b7"

The full catalogue — trajectory, event, and step/token assertions — is covered by the eval assertions page listed below.

Judging

$t->judge() returns an AgentJudgeAssertions object with a few criterion-shaped entry points — closedQa(question), factuality(reference), summarizes(source), sql(reference) — each returning a JudgeExpectation you refine with .on(...), .atLeast(threshold), .gate() / .soft(), and .label(...). The chain only accumulates state; the judge itself runs at most once, the first time the result is read (when the case finishes, or whenever something inspects the assertion). A chain you build but never let anything read spends zero tokens. Unlike a plain deterministic assertion, a judge assertion defaults to soft severity rather than gate — call .gate() explicitly if a judged criterion must be able to fail the whole case. Which judge runs is resolved per case: the judge passed to AgentEval::define(...), if any, otherwise the judge configured on the EvalConfig the runner was given. Two implementations of CanJudgeAgentEval ship with the package: PolyglotAgentJudge, a single-inference judge for straightforward final-answer grading, and AgentLoopJudge, an agentic judge that can gather its own evidence over several steps. The eval judges page covers both in depth.

A note on what traces record

By default (EvalTracePolicy::safe()), a target’s execution trace digests every tool payload value rather than storing it — call arguments, a successful result, and a failed call’s error message alike: what lands in AgentRun and in any artifact is a hash, a byte length, and a shape-only preview, never the payload value. Tool names, call order, and error flags (whether a call failed) stay in the clear, because every deterministic trajectory assertion depends on them — only the payload values themselves are digested. The eval traces page covers the policy in full, including the opt-in that serializes payloads verbatim.

Where to go next

This page is the index for the eval surface; the rest of it lives in sibling pages, in the order you’ll likely need them:

Eval assertions — the full deterministic assertion catalogue: trajectory, tool, event, step, and token checks.
Eval judges — semantic judging in depth: the judge protocol, guard configuration, and how injection risk is scoped.
Eval traces and artifacts — the complete trace projection, the trace policy in detail, artifact layout, provenance, and cost reporting.
Running evals — the CLI, repetition and pass-rate verdicts, and PHPUnit/Pest integration.

Packages

Instructor

Polyglot

Agents

Agent-ctrl

Telemetry

Sandbox

HTTP Client

Laravel

Symfony

Xprompt

21 evals

Agent Evals

Evals vs. tests

A first eval

Case definition and ids

Datasets and `AgentEvalSet`

Discovery

Targets

`LocalAgentTarget`

`HttpAgentTarget`

Sessions and turns

Deterministic assertions

Judging

A note on what traces record

Where to go next

​Agent Evals

​Evals vs. tests

​A first eval

​Case definition and ids

​Datasets and AgentEvalSet

​Discovery

​Targets

​LocalAgentTarget

​HttpAgentTarget

​Sessions and turns

​Deterministic assertions

​Judging

​A note on what traces record

​Where to go next

Agent Evals

Evals vs. tests

A first eval

Case definition and ids

Datasets and `AgentEvalSet`

Discovery

Targets

`LocalAgentTarget`

`HttpAgentTarget`

Sessions and turns

Deterministic assertions

Judging

A note on what traces record

Where to go next