Build a task-grounded evaluation harness
Turn real tasks, expected behaviors, known failures, and production feedback into a maintained evaluation harness that can guide model, prompt, and launch decisions.
- Situation
- You need AI evaluation evidence that reflects real user work, not only demos, benchmarks, or generic quality checks.
- Goal
- Create evaluation evidence that helps the team decide whether the AI system works for the tasks users actually need.
- Do not use when
- the feature is still a throwaway exploration
- Primary owner
- AI feature owner
- Roles involved
AI feature ownerML engineerproduct managerdomain expertquality leadsupport representative
Context
The situation
Deciding whether to reach for this playbook: when it fits, and when it doesn't.
Use when
Conditions where this playbook is the right tool.
- An AI feature is moving beyond prototype
- Model or prompt choices are being made with weak evidence
- Production failures are not reflected in evaluation
- Stakeholders are relying on demos or benchmark scores
Stakes
Why this matters
What this playbook protects against, and why skipping or half-running it tends to be expensive.
AI systems often fail in the gap between impressive examples and repeated task performance. A task-grounded harness makes that gap visible before users do.
Quality bar
What good looks like
The observable qualities of a team or system that is actually doing this well. Not just going through the motions.
Signs of the playbook done well
- The eval is organized around real tasks
- Known production failures become eval cases
- Results are reviewed by task slice, not only aggregate score
- Eval ownership and refresh cadence are explicit
- Launch decisions cite eval limits as well as eval wins
Preparation
Before you start
What you need available and true before running the procedure. Skipping this is the most common reason playbooks fail.
Inputs
Material you'll want to gather first.
- Task inventory
- Support or user feedback
- Production traces or examples
- Known failure taxonomy
- Model or prompt change history
Prerequisites
Conditions that should be true for this to work.
- Named target tasks
- Access to realistic examples
- Agreement on what failures matter
- Owner for eval maintenance
Procedure
The procedure
Each step carries its purpose (why it exists), its actions (what you do), and its outputs (what you produce). Read the purpose. It's what keeps the step from degenerating into checklist theatre.
Name the real tasks
Anchor evaluation in work users actually need done.
Actions
- List the top user or operator tasks the system must support
- Group tasks by risk and frequency
- Mark which tasks are launch-critical
Outputs
- Task inventory
- Launch-critical task list
Define expected behavior
Make pass, fail, and acceptable uncertainty visible.
Actions
- Write expected outcomes for each task
- Capture unacceptable failure modes
- Identify cases that require human review
Outputs
- Task expectations
- Failure taxonomy
Seed the harness with real cases
Avoid evaluating only against invented or convenient examples.
Actions
- Pull examples from support, discovery, logs, and domain experts
- Include edge cases and recent failures
- Label each case with its source and task
Outputs
- Eval case set
- Source traceability map
Report by task slice
Prevent aggregate scores from hiding important weaknesses.
Actions
- Run evaluations by task, risk tier, and failure class
- Separate regression checks from exploratory metrics
- Document known blind spots
Outputs
- Task-sliced report
- Known eval limits
Feed production learning back in
Keep evaluation from going stale after launch.
Actions
- Review production failures on a regular cadence
- Add representative failures to the harness
- Retire cases that no longer reflect real behavior
Outputs
- Eval refresh cadence
- Production feedback loop
Judgment
Judgment calls and pitfalls
The places where execution actually diverges: decisions that need thought, questions worth asking, and mistakes that recur regardless of good intent.
Decision points
Moments where judgment and trade-offs matter more than procedure.
- Which tasks are launch blockers?
- Which failures require human review rather than automated scoring?
- How much production feedback is needed before expanding rollout?
- Who can change the eval set?
Questions worth asking
Prompts to use on yourself, the team, or an AI assistant while running the procedure.
- Which real user tasks are missing from this eval set?
- Which production failures should become regression cases?
- What does this eval prove, and what does it not prove?
Common mistakes
Patterns that surface across teams running this playbook.
- Using generic benchmark tasks instead of product tasks
- Reporting one aggregate score
- Failing to add production misses to the harness
- Letting the same team that optimizes prompts silently own eval criteria
Warning signs you are doing it wrong
Signals that the playbook is being executed but not landing.
- The eval has many cases but nobody can name task coverage
- Scores improve while user satisfaction does not
- Failed production cases are discussed but never added
- The eval set has no freshness owner
Outcomes
Outcomes and signals
What should exist after the playbook runs, how you'll know it worked, and what to watch for over time.
Artifacts to produce
Durable outputs the playbook should leave behind.
- Task inventory
- Failure taxonomy
- Eval case set
- Task-sliced report
- Eval refresh plan
Success signals
Observable changes that mean the playbook landed.
- Launch decisions cite task-level evidence
- Production failures become eval cases quickly
- Model and prompt changes are compared against stable task slices
- Stakeholders understand what the eval does not prove
Follow-up actions
Moves that keep the playbook's effects compounding after it finishes.
- Connect eval results to rollout decisions
- Review drift signals against the harness
- Update task coverage after major product changes
Metrics or signals to watch
Longer-horizon indicators that the underlying problem is receding.
- Task coverage
- Failure recurrence
- Production failures added to eval
- Score by risk tier
- Age of eval cases
AI impact
AI effects on this playbook
How AI-assisted and AI-driven workflows help execution, and the ways they can make it worse.
AI can help with
Where AI tooling genuinely reduces the cost of running this playbook well.
- Clustering failures
- Drafting eval cases from real examples
- Summarizing task coverage gaps
- Comparing model outputs across versions
AI can make worse by
Distortions AI introduces that make the underlying problem harder to see.
- Generating many plausible but unrepresentative cases
- Smoothing hard failure distinctions into generic labels
- Making the harness look mature before domain review
Relationships
Connected playbooks
Failure modes this playbook tends to address, decisions behind the situation, red flags that motivate running it, and neighboring playbooks.