G3O Global Government GenAI Observatory

1. What this document is

This is the technical companion to the Collection Pipeline methodology page. That page describes, for a general reader, how a record is produced and how it can go wrong. This document goes one level deeper: it names each processing step, states what the step consumes and produces, whether it is decided by fixed rules or by a language model, how work can be lost at that step, and what the step has been measured to process. It is a companion of the same kind as the coverage codebook is to the registry page.

2. The pipeline at a glance

A collection run works forward from an institution drawn from the registry in nine operational steps. Three of the nine belong to one discovery stage that runs in separated parts, which is why the pipeline is usually described as having seven numbered stages. The final step, persistence, is invoked separately from the rest.

1a  discovery, general web        find the institution and candidate evidence pages
 2  official-site identification  pick the institution's own website        [model]
1b  discovery, site-restricted    search within that website
1c  eligibility filter            drop structurally non-content addresses
 3  triage                        decide which pages are worth reading      [model]
 4  scrape                        retrieve and render the kept pages
 5  extract                       per-page extraction to structured claims  [model]
 6  validate                      per-institution consolidation and checks  [model]
 7  persist                       write the three published tables

Step 2 sits between the two discovery passes because the second pass needs the first pass's output to have been interpreted before it can be aimed: pass 1a finds the institution, step 2 decides which returned address is its own website, and pass 1b searches within that website. Where step 2 finds no official site, pass 1b does not run for that institution — a common and expected condition, not a failure.

Step 7 sits outside the orchestrated run on purpose: steps 1a–6 work institution by institution and can resume from disk after an interruption; step 7 is a whole-run pass that assembles every institution's consolidated result into the published tables, and it refuses to overwrite its own output. In the current default configuration a run stops after step 5; steps 6 and 7 are invoked separately. The existence of extracted material for an institution therefore does not by itself mean the institution has been through consolidation or persistence — the distinction is recorded in each record's provenance.

3. Where rules end and model judgement begins

Four of the nine steps are decided by a large language model; the other five are rule-based. The four model steps all use the same model (gpt-5-nano, submitted through a batch interface) and are constrained to return data in a fixed, machine-checked shape. That constraint governs the form of an answer, not its content: a response can be perfectly well-formed and still be a misreading of the page it describes.

"Rule-based" licenses no claim about reproducibility of a run. The rule-based steps are deterministic in a narrow sense — identical inputs and settings produce identical output — but their inputs are not stable: web searches run against an index that changes without notice, and pages are edited, moved and withdrawn. The model steps add a further, different source of variation. Whether two runs over the same institutions produce the same records is a question under active study and not yet answered; see §8.

4. The nine steps, one by one

1a — Discovery, general web

One constructed query per institution asks the open web for the institution and its official website, using the institution's name, country, and disambiguating detail from the registry. Discovery currently runs in English: running it in another language requires a curated, separately approved set of search terms for that language, and adding one is a research decision, not a configuration change. The languages actually searched for each institution are recorded in that institution's provenance, derived from the run's configuration so the field cannot overstate what was searched.

How work is lost here: a failed search request is recorded in the run's loss ledger; a search that runs and returns nothing leaves an empty result, recoverable from the run's own files. Evidence published only in a language other than English is systematically less likely to be found.

2 — Official-site identification model step

A model reads the returned addresses and decides which, if any, is the institution's own website. Everything downstream of a wrong pick examines the wrong domain, which is why this is structurally the most consequential judgement in the pipeline. Where no site is identified, the site-restricted pass (1b) is skipped for that institution.

1b — Discovery, site-restricted

A second query searches only within the identified website for generative-AI material, so the institution's own domain is examined directly rather than only through what the open web happened to surface. Its results are merged with pass 1a's, with duplicates removed.

1c — Eligibility filter

A fixed set of pattern rules drops addresses that cannot carry evidence — login pages, site maps, machine-readable feeds, on-site search results, link shorteners, and bare social-media profile pages (social-media posts pass, because a post may be the only public record of an announcement). The rules are deliberately narrow: they judge from the address alone, they accept misses rather than risk false drops, and an address the rules cannot parse passes rather than being dropped. In the current default the filter runs in a record-only ("shadow") mode: its decisions are written to the run's files but nothing is actually removed, pending sign-off on the rule list. Each run records the exact rule-set version that produced its decisions, so past decisions remain traceable as the rules evolve.

3 — Triage model step

A model decides which of the surviving addresses are worth retrieving. This is the last screen before the pipeline begins retrieving pages. Addresses it drops are recorded in the step's own output file.

4 — Scrape

The kept pages are fetched and rendered to text. Retrieval respects sites' automation policies (robots.txt); a disallowed page, a failed fetch, and a page that returns too little text to read are each recorded in the loss ledger. Pages that render nearly empty never reach the model.

5 — Extract model step

A model reads each retrieved page and extracts structured claims under a fixed, versioned output contract: what activity is described, what kind of tool, how far adoption has progressed, what governance measures are documented, and what exactly the page establishes. Very long pages are truncated at a fixed cap, and the truncation is recorded. Where a required field comes back malformed, a documented fallback value may be imputed and the imputation is recorded, so an assumed value remains distinguishable from an observed one. A response that cannot be parsed at all is recorded in the loss ledger and that page's claims are lost.

6 — Validate model step

All of one institution's extracted claims are consolidated in a single pass: duplicates across pages are merged, disagreements between sources are resolved with regard to source credibility, and uncertainty is carried forward rather than smoothed away. This is a consistency check on what extraction produced — it compares nothing to an independently established answer, and it cannot detect an error that extraction and consolidation share.

7 — Persist

The consolidated results are validated against the published schema and written to the three published tables (§7). An institution whose consolidated result fails validation is omitted from all three tables and counted as a load failure — nothing is imputed. The step always writes all three tables, with headers, even when a table has zero rows.

5. Operational benchmarks — measured

Every figure in this section is a measurement on a live run, with its sample size and date. Two samples are used: 200 institutions (2026-08-01, discovery stages) and 100 institutions (run of 2026-08-02/03, all stages). Both samples were drawn from the subset of the registry that carries a reference website — about 2% of the registry, weighted toward national institutions — so these are national-institution figures, not registry figures, and they must not be projected onto a full sweep of the registry (§6).

Material handled

Model workload

Time

6. The measurement frame — read before generalising

Benchmarks need a reference set, and the only reference available today is the registry's own website column. Its coverage bounds what any measurement of this pipeline can currently claim.

The consequence is a generalisation gap, and it is stated rather than hidden: everything measured against this frame is measured on roughly 2% of the registry, weighted toward national institutions, while the other 98% is overwhelmingly United States local government. No figure measured on this frame — including every figure in §5 — should be projected onto the registry as a whole. Building a hand-verified reference set for subnational institutions is on the project's measurement agenda precisely to close this gap.

7. What a record carries — the three published tables

The pipeline's output is not free text. Step 7 writes three related tables under a fixed, versioned, machine-checked schema.

Every row carries provenance: the run that produced it, the date, the model and software used, the languages the search actually ran in, and identifiers linking each finding to its supporting sources — the mechanism by which any record can be traced back to the page it came from and checked. Where a value was supplied by a documented fallback rather than read from a page, that is recorded too. Absence of a value is encoded as the literal sentinel _NA_, never as an empty cell, and anyone quoting the tables must preserve it. Records are scoped to the run that produced them; identity that persists across runs belongs to the institution registry.

8. Caching, run comparability, and what is not yet claimed

Search responses and retrieved pages are stored locally and reused when the same query or address comes up again. This reduces the load placed on the websites the pipeline reads — and it means run-to-run comparisons must be interpreted with care.

Two runs over the same institutions may be reading the same stored copies of external material rather than consulting the web afresh: agreement between them then partly reflects the shared stored inputs, not only the pipeline's stability. Conversely, if the stored material has been cleared or has expired, a second run consults a web that has moved on, and disagreement can reflect the world changing rather than the pipeline behaving inconsistently. Any claim about how consistently the pipeline reproduces its own results must therefore record what stored material was in use — which is a precondition of the stability work now underway, and the reason no such claim appears here.

Losses are accounted for in two complementary run-level instruments: an append-only loss ledger recording what was dropped, where, and why, and a per-institution report recording how far each institution travelled through the pipeline and with what outcome. Neither alone is a complete account; loss analysis reads both.

Validation status. The pipeline contains one internal checking step (§4, step 6), and it is a consistency mechanism, not a comparison to ground truth. Measuring the pipeline against independently established answers — step-by-step estimates of missed evidence and unsupported findings against hand-coded records, and a comparison against a straightforward AI-assisted web search — is separate work, currently in design. Until that work is complete and its results approved, G3O publishes no figure of any kind about the pipeline's accuracy, consistency, or error rates, and readers should treat any such figure attributed to G3O as unverified. Validation results will be added here and to the methodology page when they are complete and approved.