What belongs in an AI evaluation set?

A demonstration shows what an AI-assisted process can do with the material placed in front of it. It does not tell you whether that material resembles your work, which attempts were discarded, or how much correction the result needed. Before expanding a process, we want a record you can question.

An evaluation set is a small collection of workflow examples paired with an agreed description of acceptable handling. It gives the people considering a system and the people building it a shared basis for judgment. The examples should expose where the proposed process helps, where it adds work, and where it should stop.

This is different from a model benchmark. Benchmarks compare capabilities under defined test conditions. A company evaluation asks whether a particular process, with its sources, permissions, software, and human review, is useful for a particular job. A model's benchmark result cannot answer that operating question for you.

Start with the work the examples must represent

We begin with a valuable problem and a narrow decision. Consider a hypothetical process that turns incoming service requests into draft records for staff. The question might be whether those drafts help staff prepare complete records without having to read and enter everything again. Approving the service request remains outside the test.

The set needs to represent that preparation step. We would gather approved examples from the channels, document types, and working conditions the proposed process will encounter. Ask the people doing the work what a convenient sample would miss: handwritten additions, attachments, seasonal requests, or messages that correct earlier messages.

We would record the period sampled and why each type was included. A folder of memorable problems can reveal weaknesses, but it does not represent ordinary workload. A folder of tidy requests has the opposite limitation. Keep the everyday sample identifiable and add difficult cases deliberately, with a note that their frequency in the test may differ from their frequency at work.

There is no useful universal set size. Start with a collection reviewers can inspect closely, covering the meaningful variations in the bounded task. Record what is absent. If an important request type has no approved examples, the test cannot support a decision about that type yet.

Include cases with different expected endings

For the draft-record example, we would include these groups:

• Normal cases: complete requests in familiar formats. Expected handling is a usable draft whose fields match the source, ready for the agreed review.
• Awkward cases: abbreviations, conflicting dates, a correction in a later message, or a required detail buried in an attachment. Expected handling might be to identify the conflict and ask a specific question rather than choose silently.
• Failure cases: unreadable material, an unavailable source, or a failed connection. Expected handling is a visible incomplete state and a handoff that lets staff continue without assuming the draft is finished.
• Sensitive cases: restricted information, a requester without access, or text attempting to redirect the system beyond its task. Expected handling must respect the access boundary and withhold excluded information or actions.

Not every successful case ends with a completed draft. A request with a missing required detail may be handled correctly by leaving that field unresolved. Conversely, a fluent, complete-looking record can fail if it invents the missing detail. The set should make that distinction visible.

Use real workflow material only within approved data boundaries. Remove unnecessary identifying details and control who can inspect inputs, outputs, and review notes. Keep the ambiguity that matters to the test. If removing a detail changes the task, record that limitation. Constructed cases can help examine rare failures without copying sensitive records, but label them clearly; they do not establish how often those failures occur.

Write the expected handling before seeing the answer

Each example needs enough context for a reviewer to judge it: the input, relevant source material, the role and permissions of the intended user, and the expected next step. A saved message without its attachment or applicable procedure may be an incomplete test case.

We would describe required facts, acceptable alternatives, prohibited behavior, and the destination for unresolved work. Avoid requiring a single exact sentence when several answers would do the job. For a draft record, the important question is whether the fields and unresolved issues are correct, not whether the system uses our preferred phrasing.

Expected handling should come from the people responsible for the process and its rules. An old completed record is evidence to inspect, not automatically a correct answer. Staff may have repaired it using information unavailable at the point being tested. Include only the context the proposed process would actually have.

Score usefulness and consequences separately

We would agree on a short scoring guide before running the set. For each case, reviewers could mark each criterion as met, partly met, or unmet, with a reason and a reference to the evidence. The criteria might cover factual correctness, required information, treatment of uncertainty, access boundaries, and whether the handoff is usable.

Review effort deserves its own judgment. Can staff check the draft against its sources, correct a field, and move on? Or must they reconstruct the request to discover what went wrong? Compare that work with the current process on comparable examples. A readable answer alone does not establish that the task became easier.

Agree which failures prevent expansion regardless of other scores. Disclosing restricted material should not disappear inside an average of otherwise acceptable drafts. Likewise, distinguish a minor wording correction from an unsupported field that could misdirect work. Keep results by case type so routine successes do not conceal repeated failures on awkward inputs.

Retain every attempt in the evaluation record, including failures. For AI-assisted steps, repeat relevant cases to inspect variation. Reporting only the best attempt recreates the limitation of a selected demonstration.

Treat reviewer disagreement as evidence

Have people who know the workflow judge an overlapping portion of the set independently before discussing their ratings. When they disagree, compare the reason, the source, and the rule each person applied.

The disagreement may reveal unclear scoring language, missing context, or a business rule that staff interpret differently. A named workflow owner should resolve policy questions. Technical changes cannot settle a disputed policy.

Preserve the original judgments and the resolution. If the expected handling changes, version the criterion and reconsider affected results. Do not quietly revise the answer key to make the latest output pass. Where disagreement remains unresolved, mark the case as unresolved and limit the conclusion accordingly.

Keep the set stable enough to compare

Give each case a stable identifier and record its source, collection date, approved use, and any changes made to the material. Keep expected handling and scoring criteria under version control. Sensitive source records can remain in an approved restricted store, referenced by identifier rather than copied into a broadly accessible code repository.

Each evaluation run should identify the set version, source versions, model, instructions, application revision, and relevant permission settings. Preserve outputs, ratings, corrections, and reviewer notes together under the same access and retention rules. That record lets you distinguish a changed system from a changed test.

Reserve some representative examples for evaluation rather than using every case to guide revisions. Once a reserved case has shaped a change, stop treating it as unseen evidence. Retain it for comparison and add fresh approved examples when needed. New workflow patterns should enter a new set version, with the reason recorded.

Let the evidence limit the next step

As an AI-native strategy and engineering studio, we connect the business question to a small test that can be inspected. We assess the data and constraints, examine the results, and expand only where justified. The evaluation set supports that decision; it does not certify the whole operation.

If drafts remain difficult to review, narrow the task or stop. If missing information is the recurring problem, a clearer intake process may deserve attention first. If ordinary rules or an existing integration handle the work adequately, AI may add little.

Before expansion, we want you to be able to point to the examples that support it, the failures that still limit it, and the work the test has not examined. That is a more useful basis for a decision than remembering which answer looked convincing during a demo.