How to evaluate an AI feature before you ship it, and after

An AI feature that shines in a demo can still let you down in daily use. Here is how to test one on real cases, agree what good looks like and keep checking after launch.

Why a good demo proves very little

Demos mislead for two reasons. The inputs are chosen, often unconsciously, to show the feature at its best, while real ones include typos, blurred scans and surprises. And the outputs vary: a feature built on a language model (an AI system that reads and writes text) can answer the same input differently each time, so one good answer proves little.

Before launch you need to know how often it works on your cases, and what happens when it does not. That takes a repeatable loop, not a single test.

(opens the full-size image in a new tab)

Build a test set from real cases

A test set (sometimes called an evaluation set) is a fixed collection of real inputs, such as past emails, forms and support requests, each paired with what a good result looks like. Include normal work and, deliberately, the awkward cases:

• Common requests, in roughly the proportions you see them.
• Incomplete, ambiguous or badly formatted inputs, and poor scans.
• Inputs from all kinds of users, across languages, dialects and writing styles.
• Cases it should hand to a person, or admit it cannot answer.
• Requests it should refuse, and inputs that try to take control of it, whether typed by a user or hidden in an email or document.

For each case, record the expected outcome: the right answer, such as an invoice total, or, where answers vary or experts might disagree, a short rubric of what a good answer must include and never do. Have an expert check each one rather than copying it from the records, which contain past mistakes.

Start modest, for example with fifty to a hundred cases. That shows common failure patterns, but a gap of a few cases between versions may be chance, and a problem seen once in a few hundred requests may not appear at all. Run important cases several times, treat close results as a tie, and add safety cases before trusting a clean result.

Never reuse these cases as examples in the prompt (the instructions the feature sends to the model). Also hold part of the set back, unread, while you improve the feature, because cases you tune against will flatter it. Base the launch decision on the held-back cases too, refreshing them with new real cases.

Mask personal data where you can, though masked data can often still identify someone, and protect the set like the original records. Check that your data-protection obligations and the provider's terms allow this use, including sending cases to an outside grading service; take legal advice if unsure, and see our guide to keeping data private with AI.

Decide what good looks like

Agree a short list of criteria with the people who own the process:

• Correctness: is the answer right, including figures, facts and decisions?
• Grounded in sources: for features that answer from your documents, such as a retrieval-augmented generation (RAG) assistant, is every claim supported by a passage, not invented or pinned to a source that says otherwise?
• Format: does the output fit what the next step expects?
• Safety: does it refuse what it should, protect personal data and hand over to a person when your rules say so, such as for complaints? Test those rules directly, as a model's sense of its own certainty can be unreliable.
• Tone: does it suit the reader, whether a parent, a patient or a supplier?
• Latency: how long do people wait?
• Cost: what does each request cost at the volume you expect?

A single serious safety or privacy failure should block launch. Latency and cost can be traded for better answers, within any agreed budget ceiling.

Score each criterion with the right method

Most evaluations combine several checks:

• Exact checks: compare the output with the expected answer, such as a date or total, after putting both in one format so "1,000.00" matches "1000". Cheap and dependable.
• Automated checks: rules in code that test format, required fields and length, or flag likely personal details. Patterns miss many names and addresses, so they support human review rather than replace it.
• Human review: people who know the work score answers against the rubric. Slower, but best for tone, usefulness and subtle errors. Reviewers disagree too, so sometimes have two score one sample and tighten the rubric where they differ.
• Model-graded scoring: a language model, often a different one from the feature's own, scores against the rubric. Useful at volume, but it can be inconsistent, favour longer or confident-sounding answers and miss errors an expert would catch. Check it against human scores, and never rely on it alone for safety or decisions about people.

Then read the failures, and break results down by type of case and group of users. A feature may handle routine requests well yet fail on rotated scans, or for people writing in a second language. Each pattern points to a fix; our comparison of prompting, RAG and fine-tuning helps match the fix to the gap. Because the set over-represents awkward cases, its overall score is not the success rate users will see.

Compare with today's process and agree the bar

An AI feature need not be perfect: it must beat today's process, or match it at lower cost or effort, and fail safely. So measure the current process on the same cases against the checked answers, for example by having staff redo a sample, not against what was recorded at the time. This baseline is the first step in our method for choosing a first automation.

Compare the kinds of mistakes too. A person's errors tend to be scattered, while an AI feature can repeat one mistake on every similar case, sounding as confident as when it is right.

Agree launch thresholds before seeing results, so they cannot be bent to fit a decision. For example: accuracy at least as high as the current process, no safety-case failures, and 95 of every 100 responses within a few seconds, since averages hide slow ones. Record who makes the final call.

Passing the safety cases shows the feature resisted the attacks you tested, not every attack. Limit the harm one that gets through could do: read-only access, for example, and a person approving anything it sends, pays or deletes. Our guide to AI agents covers these guardrails.

The result is a scorecard. A feature that falls short on a criterion that matters is not ready.

(opens the full-size image in a new tab)

Keep testing as things change

Re-test after every change

AI features change often: a prompt is edited to fix one complaint, a provider updates or retires a model, source documents are revised. Any of these can fix one problem and quietly create another.

Regression testing means re-running the test set after every change and comparing results with the previous run before anything goes live. Add each failure found in real use to the set.

Where possible, call a specific model version rather than a name that updates automatically, so the model changes only when you choose. Note announced retirement dates to test replacements in time. A retired model cannot be rolled back to, so keep the manual process ready as the fallback.

Watch real use after launch

Real use will surface cases your set missed, and inputs drift. Before launch, set up:

• Sampled human review: an expert regularly scores a random sample of real outputs, plus rare or high-risk requests that random samples seldom include.
• User feedback: an easy way to flag a poor answer, and a route to a person.
• Monitoring: rates of errors, timeouts, corrections and handovers to staff, plus response times and cost per request.
• Rollback: the previous version kept ready, and a switch back to the manual process.
• A change log: what changed, when, why and who approved it, with results before and after.

Where to start

1. Pick one AI feature; write down what it must do, and for whom.
2. Collect real cases with checked expected outcomes or rubrics; hold some back.
3. Agree the criteria and how each will be scored.
4. Measure the current process, then set launch thresholds.
5. Run, score, read every failure, improve, repeat.
6. Before launch, set up sampled review, feedback, monitoring, rollback and a change log.

A spreadsheet of cases and a few scripted checks are enough to begin. When we build AI features, we start by agreeing the cases and criteria with your team, before any prompts are written. Our AI solutions and automation team can help you design both.