Skip to main content
Wire compatibility is only the first gate. A provider can return valid typed answers and still differ materially in accuracy, calibration, latency, failure rate, and the downstream review workload. Evaluate the exact deployed model and configuration on independently labeled application data before using its output in policy.

​
Keep primitive semantics visible

The bundled model records declare how each primitive is produced: Do not combine native and projected results into one headline number. An uncatalogued route has unknown capabilities; that means Polyglot lacks a fact, not that the primitive is supported or unsupported.

​
Run the fixed example corpus

The repository includes a versioned 12-case support-triage corpus with one Noul, Choice, and Score label per case. It is deliberately small and exists to make the evaluation mechanics reproducible. Replace it with representative, independently labeled application cases before drawing production conclusions.
Each command uses the named bundled preset and therefore the same environment variables as an ordinary Decision call. Self-hosted Jeff and Laya must already be running. Jeff Score evaluations are valid only after /stats reports temperature: 1.0; record the other service facts alongside the JSON output. The report includes:
  • Noul accuracy, Brier score, log loss, expected calibration error, and buckets;
  • Choice accuracy and multiclass Brier score, log loss, and calibration;
  • Score rounded-level accuracy, mean absolute error, multiclass probability metrics, and calibration;
  • attempts, retries, typed failure categories, end-to-end latency, and available provider service, queue, or batcher timing;
  • returned model identities rather than only requested aliases; and
  • the result of an illustrative caller-owned low-confidence review policy.
Calibration confidence is derived consistently from the returned probability distribution: max(p, 1-p) for Noul and the maximum class probability for Choice and Score. Five equal-width confidence buckets feed expected calibration error. Score accuracy rounds the expected level only for the task-correctness view; Score MAE retains the fractional value.

​
Interpret operational facts correctly

End-to-end latency includes Polyglot and transport overhead. A service or batcher header is not interchangeable with it, and Jeff’s batcher time is not a pure queue measurement. Compare cold and warm runs separately and record the hardware, backend, concurrency, and exact model revision. Missing token usage is unknown, not zero. Likewise, a zero software price for a self-hosted model is not zero operating cost: include compute, memory, storage, deployment, observability, and operator time in the application decision. Laya’s modelActionProbability is a model signal, not permission. It may inform an application-owned policy, but authorization and side effects remain outside the model and outside DecisionRuntime.

​
Treat reports as evidence, not guarantees

A small corpus has high sampling error and can be overfit by prompt wording. Use versioned train/development/test boundaries, multiple runs when inference is stochastic, enough examples per segment, and confidence intervals for release decisions. Preserve malformed responses and provider failures as failed cases; do not silently remove them or fabricate fallback distributions.