Build the evaluation set before you write the prompt
Most AI features stall because nobody can tell whether a change made things better. A small, honest test set fixes that faster than any prompt technique.
- Written by
- Xalicon Engineering · AI practice
- Reviewed by
- Xalicon Editorial · Technical review
- Published
- Updated
- Reading time
- 7 minutes
Draft article
This article is written by the Xalicon team and is in editorial review. The reasoning reflects how we work today; the draft label means it has not yet had a final technical and editorial pass.
The problem with shipping on impressions
A common pattern: a team builds an AI feature, someone tries a few inputs, it looks good, and it ships. Two weeks later a prompt is adjusted to fix a reported issue. Nobody knows whether that adjustment helped overall or quietly broke three other cases, because there is nothing to measure against.
This is not a modelling problem. It is the same problem as shipping code without tests, and it has the same solution — you just have to accept that the assertion is statistical rather than exact.
What a useful evaluation set looks like
It does not need to be large. Thirty to fifty representative cases will tell you more than a vague sense that outputs seem reasonable. What matters is that the cases are real, cover the hard parts, and have an agreed expected outcome.
Collect them from actual usage where you can: support tickets, historical records, the documents people genuinely submit. Synthetic examples tend to be cleaner than reality in exactly the ways that matter.
- Straightforward cases that must never regress
- Edge cases that broke in testing — every one you find belongs in the set
- Cases where the correct answer is to refuse or ask for clarification
- Adversarial inputs, including attempts to override instructions
- Cases where two reasonable people would disagree, documented as such
Scoring without over-engineering
Not everything needs a sophisticated metric. Structured extraction can be scored with exact field matching. Classification uses standard accuracy measures. For generation, a rubric applied by a model — with a sample verified by a human — is usually enough to detect movement between releases.
The goal is not a perfect score. It is a number that moves in the right direction when you improve something, and that catches a regression before a customer does.
Wiring it into delivery
Run the evaluation in CI on any change to prompts, models, retrieval configuration or the surrounding code. Store the results alongside the commit. When a score drops, the pull request should say so.
Then close the loop: when something goes wrong in production, add that case to the set. Over a few months the evaluation set becomes the most accurate description of what your feature is actually expected to do.
What this changes
Teams with an evaluation set ship AI changes at roughly the pace they ship other changes, because the decision to release stops being a judgement call. Teams without one either move slowly out of caution or move quickly and discover the cost later.
It is unglamorous work that takes a day or two. It is also the single highest-leverage thing most teams could do to their AI feature this quarter.
- AI
- Evaluation
- Quality