A record can fail to exist at every step of the process, and the steps fail in different ways. None of these pathways is currently quantified (that is what the work described under Validation is for), but all of them are known to be possible, and a reader assessing coverage should have them in view.
- The search never surfaces the page. The evidence exists but the queries did not reach it.
- The wrong site is identified as the institution’s own. Everything that follows then examines the wrong domain.
- The site-restricted search misses the page even when the right domain was identified.
- The eligibility filter removes a relevant address. The rules are deliberately narrow, but they are rules, and a rule can misfire.
- Triage judges a relevant page not worth reading.
- Retrieval fails or returns incomplete content: the page is unreachable, blocked from automated access, or returns too little text to read.
- Extraction does not recognise the evidence on a page that was read successfully.
- Consolidation rejects a record, or a run does not reach the step that would have written it to the published tables.
These pathways compound: an institution whose official site was misidentified early is unlikely to be recovered later, because every subsequent step is working from that mistake.
Where a record can be wrong
The opposite failure (a record that exists but should not, or that says something the evidence does not support) has its own distinct pathways. Some involve model judgement and some do not.
- An irrelevant page survives triage and is read as though it were evidence.
- Extraction misreads the page: attributing an activity to the wrong institution, or reading a description of something else as a description of the institution.
- Ordinary automation is recorded as generative AI. A rules-based system, a decision tool or a chatbot built without a language model can be described in language that resembles generative-AI adoption.
- Speculative or conditional language is recorded as confirmed activity. A page discussing what an institution might do, or is consulting on, can be read as a statement of what it does.
- Provenance is incomplete or attached to the wrong evidence, so a record cannot be checked against the page that produced it.
- Consolidation does not catch an unsupported reading when it reconciles several pages for one institution.
It would be convenient to attribute all of these to the model calls, and it would be wrong. A page that reaches extraction because retrieval returned a redirect or a boilerplate landing page is a retrieval failure; a record that survives because several pages repeat one press release is a consolidation limit. The distinction matters because the remedies differ.
Error classes we already know to look for
In June 2026 a manual audit examined a sample of records against their sources by hand. It identified four recurring kinds of error. That audit examined an earlier version of the collection pipeline, which has since been retired and replaced: the version it studied assembled pre-collected spreadsheets, whereas the current pipeline performs its own search and extraction. Its findings are therefore not measurements of the system these pages describe.
- Ordinary automation coded as generative AI.
- Speculative “might adopt” language recorded as confirmed activity.
- A skew in retrieved material toward 2024.
- Missed evidence on non-English-language sites.
We publish them because they describe failure modes of the task, not of one implementation, and each remains plausible for the current pipeline. Three of the four map onto known characteristics of the current system: the definitional boundary between automation and generative AI, and between confirmed and speculative activity, is enforced in the extraction and consolidation instructions; discovery applies no date control; and discovery runs in English.
These are the error classes our validation work is designed to measure. They are not a resolved checklist, they are not current error rates, and their appearance here is not evidence that the current pipeline does or does not exhibit them. Whether any of them affects the present system, and to what extent, is an open empirical question.
Comparing one run with another
Search results and retrieved pages are stored locally and reused when the same query or address comes up again. This keeps the pipeline affordable and reduces the load placed on the websites it reads, but it has a consequence a reader comparing outputs should know about.
Two runs over the same institutions may be reading the same stored copies of external material rather than consulting the web afresh. Where that happens, agreement between the runs partly reflects the stored inputs they shared, not only the stability of the pipeline. The reverse holds too: if the stored material has been cleared or has expired, a second run consults a web that has moved on in the meantime, and disagreement can reflect the world changing rather than the pipeline behaving inconsistently.
Any claim about how consistently this pipeline reproduces its own results therefore has to record what stored material was in use at the time. That is a precondition of the stability work now underway, and it is the reason no such claim appears on this page yet.