Measure durable outcomes, not artifact volume
Shift measurement from activity and artifact volume toward evidence that user, system, team, or operational outcomes are actually improving.
- Situation
- A team is producing many tickets, documents, features, prompts, tests, or AI-generated artifacts, but confidence in real progress is low.
- Goal
- Make progress measurement reflect durable effects rather than visible production volume.
- Do not use when
- the team has not yet defined the intended outcome
- Primary owner
- engineering manager
- Roles involved
engineering managerproduct managertech leaddata analystteam representatives
Context
The situation
Deciding whether to reach for this playbook: when it fits, and when it doesn't.
Use when
Conditions where this playbook is the right tool.
- Output is high but confidence is low
- Teams are rewarded for shipping or producing artifacts
- AI tools have increased artifact volume
- Leaders cannot name which work changed outcomes
Stakes
Why this matters
What this playbook protects against, and why skipping or half-running it tends to be expensive.
Artifact volume is easy to increase and easy to game. Durable outcomes are harder to fake and more useful for decision-making.
Quality bar
What good looks like
The observable qualities of a team or system that is actually doing this well. Not just going through the motions.
Signs of the playbook done well
- Teams can name the outcome each major work item is meant to move
- Measurement includes lagging and leading signals
- Artifact volume is contextual, not the main success claim
- AI-generated output is reviewed for effect, not just quantity
- Features, tests, or documents can be retired when they do not help
Preparation
Before you start
What you need available and true before running the procedure. Skipping this is the most common reason playbooks fail.
Inputs
Material you'll want to gather first.
- Current metrics
- Business or user outcomes
- Support feedback
- Delivery history
- AI usage patterns
Prerequisites
Conditions that should be true for this to work.
- A named outcome area
- Visibility into current measurement
- Willingness to challenge existing success metrics
Procedure
The procedure
Each step carries its purpose (why it exists), its actions (what you do), and its outputs (what you produce). Read the purpose. It's what keeps the step from degenerating into checklist theatre.
Inventory current success claims
Reveal where the organization is currently equating output with progress.
Actions
- List the metrics, artifacts, and status claims used today
- Mark which ones measure activity, output, confidence, or outcome
- Identify metrics that can improve while the user outcome does not
Outputs
- Measurement inventory
- Proxy-risk list
Name durable outcomes
Clarify what should improve if the work is actually valuable.
Actions
- Define user, system, business, or team outcomes
- Connect each major workstream to one intended outcome
- Separate learning outcomes from delivery outcomes
Outputs
- Outcome map
- Work-to-outcome links
Choose evidence signals
Move from artifact counts to signals that indicate real effect.
Actions
- Choose a small set of leading and lagging signals
- Include qualitative evidence where numbers are weak
- Define what would falsify the success claim
Outputs
- Evidence signal set
- Falsification criteria
Review and retire weak measures
Prevent old volume metrics from continuing to dominate behavior.
Actions
- Downgrade output metrics to context where useful
- Remove measures that reward artifact volume alone
- Review whether decisions change when new evidence is used
Outputs
- Retired metric list
- Measurement review cadence
Judgment
Judgment calls and pitfalls
The places where execution actually diverges: decisions that need thought, questions worth asking, and mistakes that recur regardless of good intent.
Decision points
Moments where judgment and trade-offs matter more than procedure.
- Which output metrics are still useful as context?
- Which outcome signals are strong enough to guide decisions?
- What evidence would make us stop or change direction?
- How should AI-generated artifact volume be interpreted?
Questions worth asking
Prompts to use on yourself, the team, or an AI assistant while running the procedure.
- Which of these metrics can improve while user outcomes do not?
- What durable effect is this artifact meant to create?
- What evidence would make us stop producing more of this?
Common mistakes
Patterns that surface across teams running this playbook.
- Replacing one vanity metric with another
- Choosing outcomes so broad nobody can act on them
- Keeping artifact volume in executive summaries as the headline
- Treating AI output count as productivity evidence
Warning signs you are doing it wrong
Signals that the playbook is being executed but not landing.
- Teams still celebrate artifact count first
- Nobody can name which features moved key metrics
- AI adoption reports measure usage but not effect
- Work continues after evidence shows weak outcome movement
Outcomes
Outcomes and signals
What should exist after the playbook runs, how you'll know it worked, and what to watch for over time.
Artifacts to produce
Durable outputs the playbook should leave behind.
- Measurement inventory
- Outcome map
- Evidence signal set
- Retired metric list
- Measurement review cadence
Success signals
Observable changes that mean the playbook landed.
- Status conversations discuss effects before volume
- Teams stop or reshape work based on outcome evidence
- Artifact counts no longer stand alone as progress claims
- AI-generated output is judged by durability and usefulness
Follow-up actions
Moves that keep the playbook's effects compounding after it finishes.
- Review measures after one delivery cycle
- Add outcome evidence to roadmap reviews
- Audit whether incentives still reward volume
Metrics or signals to watch
Longer-horizon indicators that the underlying problem is receding.
- Decisions changed by evidence
- Work stopped due to weak outcome movement
- Artifact count vs outcome divergence
- User or support signal movement
AI impact
AI effects on this playbook
How AI-assisted and AI-driven workflows help execution, and the ways they can make it worse.
AI can help with
Where AI tooling genuinely reduces the cost of running this playbook well.
- Classifying metrics
- Summarizing user feedback
- Finding output/outcome mismatches
- Drafting falsification questions
AI can make worse by
Distortions AI introduces that make the underlying problem harder to see.
- Generating more artifacts to satisfy output-based reporting
- Creating polished summaries that make weak evidence sound decisive
- Optimizing for metrics that are easy to produce
Relationships
Connected playbooks
Failure modes this playbook tends to address, decisions behind the situation, red flags that motivate running it, and neighboring playbooks.