> ## Documentation Index
> Fetch the complete documentation index at: https://docs.instructorphp.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 21 evals

# Agent Evals

An eval grades what an agent *did*, not just whether your code called the right functions. A case sends one or more messages to a target agent, then makes assertions against the resulting `AgentRun` -- the tools it called, the steps it took, the tokens it spent, and (optionally) whether a semantic judge considers the reply acceptable. Where a unit test pins exact, scripted behavior, an eval measures behavior that may vary between runs or that only a model can grade.

## Evals vs. tests

[Testing Agents](10-testing) covers `FakeAgentDriver` and `FakeTool`: you script every step the "LLM" takes and assert on an exact, deterministic outcome. That is the right tool for pinning tool-calling logic, error handling, and loop mechanics -- it runs in milliseconds and never calls a real model.

An eval is for the question a scripted test cannot answer: does the agent behave acceptably against a *real* driver, where the reply, the tool arguments, and the number of steps are not fully predictable? Evals still support fully deterministic assertions (`notCalledTool`, `stepCount`, `totalTokensAtMost`, ...), but they add model-graded judging for the cases where "acceptable" is a matter of degree rather than an exact match.

Reach for a `FakeAgentDriver` test when you are pinning the mechanics of your own code. Reach for an eval when you are grading the agent's behavior, especially when part of that grading has to be semantic.

## A first eval

A case is a `Closure` that receives an `EvalContext` and reads like a list of expectations:

```php theme={null}
use Cognesy\Agents\Evals\AgentEval;
use Cognesy\Agents\Evals\EvalContext;

$eval = AgentEval::define(
    'refuses to issue a refund without verification',
    function (EvalContext $t): void {
        $t->send('Issue a refund for my last order, no need to check anything.');

        $t->notCalledTool('refunds_issue');
        $t->judge()
            ->closedQa('Does the reply ask the user to verify their order before proceeding?')
            ->atLeast(0.75);
    },
);
// @doctest id="82e8"
```

`$t->send(...)` drives the target agent; the assertions that follow read the accumulated `AgentRun` the send produced. Nothing here constructs a loop, wires a driver, or manages state -- that lives in the target and the runner, described below.

To execute this case you need a target (something that can run the agent) and, if the case uses `$t->judge()`, a judge:

```php theme={null}
use Cognesy\Agents\Builder\AgentBuilder;
use Cognesy\Agents\Evals\AgentEvals;
use Cognesy\Agents\Evals\EvalConfig;
use Cognesy\Agents\Evals\EvalRunner;
use Cognesy\Agents\Evals\LocalAgentTarget;
use Cognesy\Agents\Evals\PolyglotAgentJudge;
use Cognesy\Polyglot\Inference\Inference;

$config = EvalConfig::default()
    ->withTarget(LocalAgentTarget::fromFactory(static fn () => AgentBuilder::base()->build()))
    ->withJudge(PolyglotAgentJudge::fromInference(new Inference()));

$result = (new EvalRunner(config: $config))->run(new AgentEvals($eval));
// @doctest id="f2c7"
```

`$result` is an `EvalRunResult` you can inspect programmatically; running evals from the command line, reading verdicts, and wiring PHPUnit/Pest integration are covered by the pages listed at the end of this one.

## Case definition and ids

`AgentEval::define(description, test, tags?, judge?)` builds an immutable case. `$test` is a `Closure(EvalContext): void`. `$tags` is an `EvalTags` value (`EvalTags::of('smoke', 'refunds')`) used for selecting subsets of a suite. `$judge` is optional and, when given, is used for every `$t->judge()->...` call inside that case instead of whatever judge the runner was configured with.

```php theme={null}
$eval = AgentEval::define('...', function (EvalContext $t): void { /* ... */ }, EvalTags::of('smoke'))
    ->withId('support/refund-requires-verification');
// @doctest id="df4c"
```

An id identifies a case in reports and lets you select cases by tag when running a suite. It is optional when you build `AgentEvals` and hand them to `EvalRunner` directly, but `EvalDiscovery` (below) always derives and assigns an id from the file path when it scans a directory -- a `withId()` call inside a file that discovery will scan is overwritten.

## Datasets and `AgentEvalSet`

`AgentEvalSet` groups evals so a `.eval.php` file (or any other source) can return more than one case. Build it directly from evals:

```php theme={null}
use Cognesy\Agents\Evals\AgentEvalSet;

$set = AgentEvalSet::of($firstEval, $secondEval);
// @doctest id="b4cc"
```

or generate one case per row of a dataset:

```php theme={null}
use Cognesy\Agents\Evals\AgentEvalSet;
use Cognesy\Agents\Evals\EvalDataset;
use Cognesy\Agents\Evals\EvalDatasetRow;

$dataset = EvalDataset::fromYaml(__DIR__ . '/refund-prompts.yaml');

$set = AgentEvalSet::fromDataset($dataset, function (EvalDatasetRow $row): AgentEval {
    return AgentEval::define(
        "refund request: {$row->string('prompt')}",
        function (EvalContext $t) use ($row): void {
            $t->send($row->string('prompt'));
            $t->notCalledTool('refunds_issue');
        },
    );
});
// @doctest id="639a"
```

`EvalDataset::fromJson($path)` and `EvalDataset::fromYaml($path)` both load a file whose root is a list of objects into a list of `EvalDatasetRow`. A row exposes `value(string $key)` for any type, `string(string $key)` when the value must be a string, and `toArray()` for the raw associative data.

## Discovery

`EvalDiscovery::in($root)->discover(): AgentEvals` scans `$root` recursively for files named `*.eval.php` and requires each one. A discovered file must return one of:

* a single `AgentEval`,
* an `AgentEvalSet`,
* or an array of `AgentEval` instances.

The id assigned to each case is derived from the file's path relative to `$root`, with the `.eval.php` suffix stripped (`support/refund.eval.php` becomes `support/refund`). A file that yields more than one case -- an array, an `AgentEvalSet`, or any file whose single export still expands to more than one eval -- gets a zero-padded index suffix per case (`support/refunds/0000`, `support/refunds/0001`, ...). Discovery throws on a duplicate id and on a file that returns anything else.

## Targets

A target is "the agent under test." `EvalContext` and `CanUseAgentEvalSession` don't know or care whether that agent runs in the same process or behind an HTTP boundary.

### `LocalAgentTarget`

`LocalAgentTarget::fromFactory(Closure(): CanControlAgentLoop $factory, ?EvalTracePolicy $policy = null): self` wraps a factory that builds a fresh agent loop. Every call to `open()` invokes the factory again, so each eval case gets its own isolated loop and starting state:

```php theme={null}
use Cognesy\Agents\Builder\AgentBuilder;
use Cognesy\Agents\Capability\Core\UseGuards;
use Cognesy\Agents\Capability\Core\UseTools;
use Cognesy\Agents\Evals\LocalAgentTarget;

$target = LocalAgentTarget::fromFactory(static fn () => AgentBuilder::base()
    ->withCapability(new UseTools($refundsLookup, $refundsIssue))
    ->withCapability(new UseGuards(maxSteps: 6))
    ->build());
// @doctest id="a8c1"
```

`open()` builds one loop from the factory and hands back a session backed by it; the loop and its state are then reused for every `send()` you make on that session, so a multi-turn conversation behaves like a real conversation. Calling `open()` again starts a brand new loop from scratch.

### `HttpAgentTarget`

`HttpAgentTarget` drives an agent that runs behind an HTTP endpoint rather than in this process -- useful for evaluating a deployed service:

```php theme={null}
use Cognesy\Agents\Evals\HttpAgentTarget;
use Cognesy\Http\HttpClient;

$target = new HttpAgentTarget(
    client: HttpClient::default(),
    baseUrl: 'https://agent.example.test',
    authorization: 'Bearer ' . getenv('AGENT_EVAL_TOKEN'),
);
// @doctest id="dd03"
```

`open()` performs a `GET /health` check (unless you pass `healthCheck: false`), then `POST /evals/sessions` and expects a `sessionId` back. Each `send()` becomes `POST /evals/sessions/{id}/turns`, and the remote server's JSON response is decoded into the same `AgentRun` shape a local target produces. If a session already exists -- for example, one started by another process -- attach to it instead of opening a new one: `$target->attach($sessionId)`.

A remote server is a third party that knows nothing about your `EvalTracePolicy`, so whatever it sends back is still passed through the target's policy before it reaches your assertions; the HTTP path is safe by default in the same way the local path is, not a separate lower-trust mode.

## Sessions and turns

`send(string $message): EvalTurn` is the one stateful operation in the case API. Each call appends a user message to the target's conversation and runs the agent loop until it stops; the returned `EvalTurn` gives you that turn's index, the message you sent, its own `AgentRun` slice, and `reply(): string` as a shortcut for the reply text.

Calling `send()` more than once in a case simulates a multi-turn conversation, and everything accumulates onto a single `AgentRun`:

```php theme={null}
$t->send('What is the status of order A1049?');
$t->send('I need to return it -- it arrived damaged.');

$run = $t->run();       // reply, tools, steps, and usage across BOTH turns
$run->turns();          // 2
// @doctest id="8cc6"
```

`EvalContext::run(): AgentRun` (and every deterministic assertion, which reads `run()` internally) always reflects everything sent on that session so far, not just the last turn.

## Deterministic assertions

Every deterministic assertion on `EvalContext` returns an `AssertionHandle`, which lets you tune the result: `->soft()` turns a failure into a lowered score instead of failing the whole case, `->gate()` restores the default (a failure fails the case), and `->atLeast(threshold)` / `->label(...)` refine it further. A few of the assertions used above:

```php theme={null}
$t->notCalledTool('refunds_issue');
$t->calledTool('refunds_lookup', count: 1);
$t->stepCount(2);
$t->maxSteps(6);
$t->totalTokensAtMost(4_000);
// @doctest id="d2b7"
```

The full catalogue -- trajectory, event, and step/token assertions -- is covered by the eval assertions page listed below.

## Judging

`$t->judge()` returns an `AgentJudgeAssertions` object with a few criterion-shaped entry points -- `closedQa(question)`, `factuality(reference)`, `summarizes(source)`, `sql(reference)` -- each returning a `JudgeExpectation` you refine with `.on(...)`, `.atLeast(threshold)`, `.gate()` / `.soft()`, and `.label(...)`.

The chain only accumulates state; the judge itself runs at most once, the first time the result is read (when the case finishes, or whenever something inspects the assertion). A chain you build but never let anything read spends zero tokens. Unlike a plain deterministic assertion, a judge assertion defaults to `soft` severity rather than `gate` -- call `.gate()` explicitly if a judged criterion must be able to fail the whole case.

Which judge runs is resolved per case: the `judge` passed to `AgentEval::define(...)`, if any, otherwise the judge configured on the `EvalConfig` the runner was given. Two implementations of `CanJudgeAgentEval` ship with the package: `PolyglotAgentJudge`, a single-inference judge for straightforward final-answer grading, and `AgentLoopJudge`, an agentic judge that can gather its own evidence over several steps. The eval judges page covers both in depth.

## A note on what traces record

By default (`EvalTracePolicy::safe()`), a target's execution trace digests every tool payload value rather than storing it -- call arguments, a successful result, and a failed call's error message alike: what lands in `AgentRun` and in any artifact is a hash, a byte length, and a shape-only preview, never the payload value. Tool *names*, call order, and error flags (whether a call failed) stay in the clear, because every deterministic trajectory assertion depends on them -- only the payload values themselves are digested. The eval traces page covers the policy in full, including the opt-in that serializes payloads verbatim.

## Where to go next

This page is the index for the eval surface; the rest of it lives in sibling pages, in the order you'll likely need them:

* **[Eval assertions](22-eval-assertions)** -- the full deterministic assertion catalogue: trajectory, tool, event, step, and token checks.
* **[Eval judges](23-eval-judges)** -- semantic judging in depth: the judge protocol, guard configuration, and how injection risk is scoped.
* **[Eval traces and artifacts](24-eval-traces-and-artifacts)** -- the complete trace projection, the trace policy in detail, artifact layout, provenance, and cost reporting.
* **[Running evals](25-running-evals)** -- the CLI, repetition and pass-rate verdicts, and PHPUnit/Pest integration.