The Retrieval Pipeline
Optimization should target the stage where knowledge fails, not the surface where the failure becomes visible.
On this pageThe five stages⌄
The five stages
The stages are useful because the same visible symptom can come from different causes.
A source may be absent because the page cannot be accessed. It may be accessible but not selected. It may be selected and misunderstood. It may inform the answer but receive no citation. Each failure requires a different response.
1. Candidate Discovery
Candidate discovery asks whether a system can encounter the source or a relevant passage at all.
On the open web, this can depend on crawl access, indexing eligibility, internal links, URL stability, rendering, response codes, canonical handling, page availability, and whether the content is exposed in a format the system can access.
In a private retrieval system, it may depend on connector coverage, permissions, ingestion, chunking, metadata, synchronization, and query matching.
A failure here means later content improvements may have no effect because the source never becomes a candidate.
Questions to investigate
- Can the page be accessed without a login or blocked script path?
- Does the canonical URL return a stable successful response?
- Can the relevant text be extracted from the rendered page?
- Is the page internally linked from pages that systems already encounter?
- Is the content eligible for indexing on the target surface?
- For private systems, did ingestion complete and preserve the right permissions?
2. Source Selection
Discovery creates candidates. Selection determines which candidates are used.
A source can be discoverable and still lose to a better candidate. Another source may be more direct, current, authoritative, specific to the question, easier to interpret, or better aligned with the requested geography or period.
Selection is not always page level. A system may select a passage, data field, product feed, knowledge base item, or tool response. It may combine several sources.
Questions to investigate
- Does the page answer the exact user question or only a related topic?
- Is there a more direct primary source?
- Is the relevant information current and visibly dated?
- Does the source clearly establish who published the information?
- Does the answer require a local, regulated, or specialized source?
- Are competing sources easier to extract or more specific?
A common selection problem is weak differentiation. Ten pages may repeat the same public definition. A system has little reason to select one consistently.
Original data, clear methodology, first hand analysis, useful tools, or direct access to a primary source create a stronger basis for selection than additional wording around the same facts.
3. Interpretation
Interpretation asks whether the selected knowledge keeps its meaning.
This is where entity ambiguity, missing units, mixed periods, unclear labels, unexplained calculations, conflicting page sections, and weak source relationships cause damage.
Consider a page that displays two numbers: a trailing price to earnings ratio and a forward price to earnings ratio. If the labels are visually close but structurally weak, extraction may preserve the values and lose the distinction. The result can be a confident answer with the wrong number.
Questions to investigate
- Is the entity named clearly in the relevant section?
- Can the statement stand alone without a vague reference such as "this" or "it"?
- Are units, currencies, periods, and estimate labels explicit?
- Are related metrics clearly distinguished?
- Does the structured representation agree with the visible content?
- Can a reviewer trace a derived value to its source and method?
Better writing can help, but interpretation is broader than copy. It can require changes to data modeling, template structure, semantic HTML, labeling, source display, and Knowledge Object composition.
4. Answer Use
A system can discover, select, and understand knowledge without including it in the final answer.
The answer may prioritize brevity, combine several sources, omit information outside the requested scope, avoid a claim with insufficient support, or choose a different angle. Some interfaces provide links without repeating all retrieved facts. Others produce a direct answer and show only a subset of sources.
Teams should distinguish between retrieval and final answer inclusion. A missing mention does not prove that the source was never considered.
Questions to investigate
- Does the prompt require the fact that the source provides?
- Is the requested answer broad enough that many sources could satisfy it?
- Does the interface prioritize concise output or a fixed number of links?
- Is the fact material to the answer or only supporting context?
- Does a more specific follow up question surface the source?
5. Attribution
Attribution asks whether the source receives visible credit.
Attribution behavior varies by product and answer type. Some systems provide inline citations. Some show a source list. Some link selected statements. Some provide no visible citation.
A source can influence an answer without receiving a visible link. A cited link can also point to a secondary page rather than the original source.
Because attribution is an external product decision, publishers can improve source clarity but cannot guarantee citation.
Questions to investigate
- Does the page identify the publisher and author where relevant?
- Is there a stable canonical URL for the specific information?
- Does the source relationship remain clear when the passage is extracted?
- Is the cited page the original source, a summary, or a data intermediary?
- Does the answer cite a source that actually supports the statement?
A failure matrix
| Observed outcome | Likely stage to investigate first | Useful next step |
|---|---|---|
| Page never appears in search or cited answers | Candidate Discovery | Check access, indexing, rendering, links, and stable URLs |
| Competitor appears for the same exact fact | Source Selection | Compare specificity, freshness, source authority, and directness |
| Brand appears but the fact is wrong | Interpretation | Review labels, periods, source conflicts, and extraction context |
| Source appears for narrow prompts but not broad prompts | Answer Use | Separate retrieval visibility from final response prioritization |
| Fact appears but another site receives credit | Attribution | Review canonical source clarity and whether another source is more direct |
| Old value appears after the page was updated | Discovery, Selection, or Interpretation | Check recrawl timing, duplicate values, timestamps, and historical labels |
Example: a current financial metric
Suppose a user asks for the current market capitalization of a company.
A publisher may have a correct value on a company page and still fail at several stages.
Discovery failure
The value renders only after a client side request that the retrieval system does not process.
Selection failure
An exchange or primary market source provides a more direct answer.
Interpretation failure
The page shows several currencies and the selected passage omits the currency label.
Answer use difference
The system answers with a rounded range and omits the exact source value.
Attribution difference
The answer cites the data provider rather than the publisher presentation.
The correct response differs at each stage. Adding more paragraphs does not fix rendering. Adding schema does not make a secondary source primary. Rewriting a definition does not control citation policy.
How to run a pipeline review
Step 1: Define an evaluation set
Use real user questions. Include factual, comparative, explanatory, current, historical, branded, and unbranded forms. Record the exact prompt, date, interface, and location where relevant.
Step 2: Record observable outputs
Capture whether the source was visible, cited, correctly interpreted, current, and useful. Do not infer hidden retrieval solely from the final answer.
Step 3: Classify the first likely failure
Use the five stages. Start with the earliest plausible failure because later stages depend on earlier ones.
Step 4: Select one intervention
Change the source model, presentation, access, timestamp, linking, or explanatory context based on the diagnosis. Avoid several simultaneous changes when learning matters.
Step 5: Retest after a meaningful interval
External systems update on different schedules. Document the interval and avoid treating one immediate response as conclusive.
What the pipeline does not prove
- It does not prove that every AI system uses these five literal internal stages.
- It does not prove that a source was retrieved when only the final answer is visible.
- It does not turn a correlation between a page change and an answer change into proven causation.
- It does not guarantee ranking, citation, inclusion, or traffic.
- It does not replace official platform documentation or direct technical testing.
The model has a narrower purpose. It gives teams a shared language for investigating where information is lost or distorted between a user question and an answer.
Why the model is useful
The pipeline prevents teams from compressing every problem into "content quality."
Content quality matters, but discoverability also depends on access, identity, source suitability, temporal context, system behavior, and interface decisions.
By naming the stages, teams can assign the right owner.
Engineering
Access, rendering, ingestion, delivery, and source availability.
Product
Object boundaries, failure states, presentation rules, and user value.
Data
Identity, provenance, freshness, source authority, and calculations.
Content
Clarity, definitions, qualifications, and extractable explanation.
Search
Evaluation sets, canonical structure, linking, and observable discovery behavior.
The result is not control over external systems. It is a more disciplined way to decide what the organization can improve.
Scope
The Retrieval Pipeline is an analytical model. It does not claim that every search engine or AI product uses the same proprietary architecture or executes these stages in a fixed linear order.
Its purpose is to help publishing and product teams diagnose observable discovery failures without guessing about hidden internals.
Framework in one minute
- Useful knowledge can fail during candidate discovery, source selection, interpretation, answer use, or attribution.
- The visible symptom often appears later than the underlying failure.
- The earliest plausible failure should guide the first intervention.
- Publishers can improve source clarity and accessibility, but cannot guarantee final answer inclusion or citation.
- The pipeline is valuable because it gives different teams one shared diagnostic language.