THE DIRECT ANSWER

AI evaluation compares observed system behaviour with defined expectations. A useful evaluation specifies the task, test cases, criteria and system version. It examines failures as well as averages and distinguishes factual correctness from subjective preferences such as writing style.

Start with real work

Collect representative tasks before comparing models. Include common cases, difficult exceptions and examples where the correct response is to ask for more information. Define what counts as success for each: factual accuracy, correct formatting, a completed action or an appropriate refusal.

Keep the test honest

Hold some examples back while improving prompts or settings. Otherwise it is easy to optimise for the test cases you already know. Record the model version, instructions and relevant tool configuration. A score without that context may be impossible to reproduce after a product update.

Use results to decide

Group failures by cause rather than hiding them inside one average. A system that performs well overall may still fail consistently for one document type or language. In an illustrative support workflow, compare incorrect answers separately from unnecessary escalations because the consequences differ. Evaluation is useful when it changes a decision: what to deploy, what to review manually and what to improve next. A leaderboard position cannot replace a test built around your own work.

A worked example

For a support assistant, prepare answerable questions, conflicting-policy cases and requests requiring escalation. Have a domain owner define acceptable outcomes. Compare versions on the same held-out cases and preserve examples of failure. If a change improves average fluency while inventing a policy exception, the scorecard should make that trade-off visible rather than hiding it inside one overall rating.

COMMON QUESTIONS

Questions readers ask

How many examples are enough for an AI evaluation?

There is no universal number. Coverage should reflect the task's variation and consequences. Start with representative and failure cases, then expand where uncertainty remains.

SOURCES & NOTES

Follow the evidence.

AI-assisted explanation with linked technical references. Examples are illustrative learning exercises, not reported customer deployments or experimental results.

  1. OpenAI: Working with evals

Public sources support the factual background. Analysis and illustrative exercises are identified in the text. How we assess AI · Editorial standards

Spot a factual error or an outdated detail?