THE DIRECT ANSWER

Accuracy tests compare outputs with an established answer or acceptance rule. Preference tests ask which output people favour. A response can be polished and preferred while factually wrong; a correct response can still be hard to use. Neither measure should silently stand in for the other.

Design separate checks

For factual tasks, evaluate support, completeness and appropriate abstention. For experience quality, use representative raters and clear criteria such as readability or usefulness. Randomise presentation where practical and record disagreements rather than hiding them inside one score.

Report the limits

Combine measures when the product needs both correctness and usability, but show what each result means. Keep examples of important failures and test changes against the same baseline. Avoid declaring a universal winner from a small task set or a preference vote alone.

Use separate scorecards for truth and preference

For a factual task, prepare an answer key and score supported correctness. For a writing task, define audience and style criteria and gather preference judgements separately. Hide product names where practical. This proposed evaluation prevents a polished but inaccurate answer from winning a factual test and avoids pretending that every stylistic preference has one objective answer. Publish the task, rubric and limitations with any eventual results.

COMMON QUESTIONS

Questions readers ask

Can user preference establish factual accuracy?

No. A preferred answer may still be wrong. Measure correctness against evidence and record style or usefulness preferences as a separate dimension.

SOURCES & NOTES

Follow the evidence.

AI-assisted editorial decision guide. No hands-on benchmark, product score or winner is claimed. The proposed comparison is a method readers can run; current features, terms and results must be checked for the selected versions.

  1. developers.openai.com — official reference

Public sources support the factual background. Analysis and illustrative exercises are identified in the text. How we assess AI · Editorial standards

Spot a factual error or an outdated detail?