How to Build an Evaluation Set Before an AI Launch

A demo shows what an AI system can do once. An evaluation set shows whether it can keep doing the right thing as prompts, data, tools, and models change.

011. Start With Real Tasks

The first evaluation examples should come from the workflow itself: questions users ask, documents they upload, records they search, fields they edit, and decisions they need to make.

Avoid filling the set with ideal prompts. Real users are incomplete, impatient, and specific in ways demos rarely capture.

Evaluation quality improves when test cases reflect real workflow pressure.

022. Include Refusal Cases

A good AI system should know when not to answer, not to call a tool, and not to invent missing information. Refusal cases test that behavior directly.

Examples include restricted records, unsupported legal or medical advice, missing evidence, ambiguous approvals, and requests outside the system's purpose.

033. Define the Grade Before Running the Test

Decide what good means. Accuracy, completeness, tone, citation quality, tool choice, latency, permission handling, and escalation behavior can all matter, but not equally for every workflow.

If the grade is vague, the eval becomes a debate. If the grade is clear, the team can compare changes without relying on memory.

044. Keep the Set Alive

Evaluation sets should grow after launch. Add production failures, unusual success cases, new business rules, and changed data structures.

The point is not to freeze the AI product. The point is to make change safer.