Define source trust model
Classify sources by authority, freshness, allowed use, and conflict handling so retrieval behavior is grounded in material humans actually trust.
- Situation
- A RAG or knowledge-backed AI system uses sources whose authority, freshness, and conflict rules are not explicit.
- Goal
- Make source trust explicit enough that retrieval quality can be governed, evaluated, and maintained.
- Do not use when
- the system is not source-grounded
- Primary owner
- AI feature owner
- Roles involved
AI feature ownerdomain expertknowledge ownerML engineerproduct managerrisk or compliance owner
Context
The situation
Deciding whether to reach for this playbook: when it fits, and when it doesn't.
Use when
Conditions where this playbook is the right tool.
- RAG uses sources nobody fully trusts
- Source lists are informal or stale
- Documents conflict with each other
- Citations appear but users still do not trust answers
Stakes
Why this matters
What this playbook protects against, and why skipping or half-running it tends to be expensive.
Retrieval systems inherit the quality and politics of their sources. Without a trust model, citations can create false confidence instead of grounded behavior.
Quality bar
What good looks like
The observable qualities of a team or system that is actually doing this well. Not just going through the motions.
Signs of the playbook done well
- Sources are grouped by authority and allowed use
- Freshness expectations are explicit
- Conflict rules exist for disagreeing sources
- Source ownership is named
- Evaluation checks source trust, not only answer fluency
Preparation
Before you start
What you need available and true before running the procedure. Skipping this is the most common reason playbooks fail.
Inputs
Material you'll want to gather first.
- Source inventory
- Domain authority rules
- Freshness requirements
- Known source conflicts
- User trust feedback
Prerequisites
Conditions that should be true for this to work.
- Inventory of candidate sources
- Domain expert availability
- Owner for source maintenance
Procedure
The procedure
Each step carries its purpose (why it exists), its actions (what you do), and its outputs (what you produce). Read the purpose. It's what keeps the step from degenerating into checklist theatre.
Inventory sources and owners
Make the corpus visible before judging retrieval quality.
Actions
- List source types and repositories
- Name the owner or steward for each source
- Mark sources with unclear authority
Outputs
- Source inventory
- Source ownership map
Define trust tiers
Separate authoritative sources from merely available sources.
Actions
- Classify sources by authority, freshness, and allowed use
- Identify sources that should never answer certain tasks
- Define what makes a source stale
Outputs
- Source trust tiers
- Freshness rules
Define conflict behavior
Avoid silent blending of contradictory sources.
Actions
- Identify common source conflicts
- Define precedence rules
- Decide when the system should refuse, escalate, or cite uncertainty
Outputs
- Conflict rules
- Escalation behavior
Connect trust to evaluation
Ensure source quality is tested, not only response quality.
Actions
- Add source trust checks to eval cases
- Test stale and conflicting-source scenarios
- Review source failures after production incidents
Outputs
- Source-trust eval cases
- Source failure review loop
Judgment
Judgment calls and pitfalls
The places where execution actually diverges: decisions that need thought, questions worth asking, and mistakes that recur regardless of good intent.
Decision points
Moments where judgment and trade-offs matter more than procedure.
- Which sources are authoritative for each task?
- What freshness is required for safe use?
- What should happen when trusted sources conflict?
- Who can add or retire sources?
Questions worth asking
Prompts to use on yourself, the team, or an AI assistant while running the procedure.
- Which sources are authoritative for this task?
- What should the system do when two trusted sources disagree?
- Which indexed sources should never be used for user-facing answers?
Common mistakes
Patterns that surface across teams running this playbook.
- Treating all indexed content as equally trustworthy
- Using citations as proof of authority
- Forgetting source freshness
- Letting model behavior resolve source conflicts implicitly
Warning signs you are doing it wrong
Signals that the playbook is being executed but not landing.
- Sources are cited but not trusted
- Source owners are unclear
- Users know which docs are wrong but the system does not
- Conflicting sources produce confident answers
Outcomes
Outcomes and signals
What should exist after the playbook runs, how you'll know it worked, and what to watch for over time.
Artifacts to produce
Durable outputs the playbook should leave behind.
- Source inventory
- Trust-tier model
- Freshness rules
- Conflict rules
- Source-trust eval cases
Success signals
Observable changes that mean the playbook landed.
- Answers prefer authoritative sources
- Stale or low-trust sources are handled explicitly
- Source conflicts are visible
- Corpus changes have owners and review
Follow-up actions
Moves that keep the playbook's effects compounding after it finishes.
- Review source trust after corpus changes
- Add production source failures to evals
- Curate or retire low-trust sources
Metrics or signals to watch
Longer-horizon indicators that the underlying problem is receding.
- Unowned sources
- Stale-source answer rate
- Conflict cases detected
- Source trust failures
- User trust feedback
AI impact
AI effects on this playbook
How AI-assisted and AI-driven workflows help execution, and the ways they can make it worse.
AI can help with
Where AI tooling genuinely reduces the cost of running this playbook well.
- Classifying sources
- Finding conflicting documents
- Summarizing source freshness
- Drafting source-trust eval cases
AI can make worse by
Distortions AI introduces that make the underlying problem harder to see.
- Making weak sources sound authoritative
- Hiding source conflicts in fluent synthesis
- Generating trust tiers without domain review
Relationships
Connected playbooks
Failure modes this playbook tends to address, decisions behind the situation, red flags that motivate running it, and neighboring playbooks.