World Gov domain names

This repository collects the government domain names of 197 States and entities, so we can measure how much governments contribute to publicly available software archived by Software Heritage.

Coverage

197 jurisdictions, each tagged with its UN status:

• 193 UN member states
• 2 UN observer states: Holy See, Palestine
• 2 non-UN entities: Taiwan, Kosovo (kept for context, but excluded from any UN-facing aggregation).

Scope

We sort government domains into levels:

• central – the central government: ministries, national agencies (including national research bodies: CNRS, CNR…) and the root domains that stand for them (gouv.fr, gov.uk).
• central-1 – the first administrative tier below: state, Land, region… (sp.gov.br, nouvelle-aquitaine.fr).
• the rest – everything lower (municipalities, counties, parishes…), kept out of the lists and regexes.

Universities are out of scope: the strictly public ones (Wikidata public / public-research / national university classes) are kept as a parked dataset in data/public-universities.csv, refreshed by scripts/detect-universities.clj (bb universities), for a possible later re-exploration. Nothing in the pipeline reads that file.

Method

We proceed by lists of domain names and subdomains, one country at a time. The situation varies from country to country: where each level has its own roots, the lists are enough; but where everything under a single root like gov.br would also include local levels, a curated list (sources/registry/excluded-labels.csv) excludes the local levels when needed.

What you get

Consolidated lists

Three ready-to-use CSV files, covering UN members and observers only:

• data/public-sector-domains.csv – every government host we found, apex domains and subdomains (e.g. gouv.fr, beta.gouv.fr).
• data/public-sector-domains-central.csv – one row per central-government root domain (e.g. gouv.fr, insee.fr). A root already stands for all its subdomains, so this is the short list of email domains to watch.
• data/public-sector-domains-central+.csv – the central roots (level central) plus the bodies of the first administrative tier below (level central-1: Land, state, region…). Central-1 rows only enter once confirmed, mirroring the manual gate of the central level: either through a curated registry (Brazilian/Australian states, UK devolved bodies; sources is registry) or through the per-country curation channel (sources/curated/central_admin.csv; sources includes curated). Unconfirmed central-1 candidates stay in candidates-local.csv – that file is the curation worklist.

Each row carries metadata: UN status, region, languages and GDP

Per-country data (countries/<ISO3>_<slug>/)

One directory per country (FRA_france, USA_united_states, …), with:

• subdomains.csv – all known subdomains and their HTTP status.
• candidates.csv – central administration domains not confirmed yet, scored 0 to 10 by how many sources agree.
• candidates-local.csv – same for domains attributed to subnational administrations; the level column separates central-1 (bodies of a first-level subdivision: Land, state, region…) from local (the rest). Central-1 rows enter the central+ scope once curated.
• summary.md – a human-readable cross-check report.
• sources/ – the raw data behind the above, one folder per source. sources/linkgraph/indegree.csv (EU + neighbours only) counts, for each domain, how many distinct public-sector domains of the same country link to it, from the crawl published by eu-plus-government-scans (bb indegree refreshes it); well-linked hosts enter candidates.csv with a strong score bonus.

Master data (data/)

• world-governments.csv – the master list of the 197 countries.
• forge-candidates.csv – hosts that look like software forges or source-code catalogs (git.*, gitlab.*, forge.*, code.*, …), extracted from the harvested subdomains; leads for the swh-sopc-data-sources catalog list.
• known-forges.csv – curated list of self-hosted government forges (GitLab, Forgejo…), seeded from the swh-sopc-data-sources catalog list.
• known-catalogs.csv – curated list of government source-code catalogs (portals pointing at code hosted elsewhere), same columns; kept for reference, never searched.
• github-gov-orgs.csv – government GitHub organizations from GitHub’s community-maintained governments.yml, plus the local additions of github-gov-orgs-extra.csv (orgs not merged upstream yet).
• forge-unknown-swh.csv – forge targets (harvested + curated) that the Software Heritage archive does not know yet (checked against its origin-search API): candidates for archival. Each target is also probed over HTTPS: forge_type is the forge software sniffed from the homepage (gitlab, gitea, cgit…, or github for redirects), http_status the final HTTP code (000 when unreachable) and note a short diagnostic (DNS failure, timeout, WAF, default page, cross-host redirect). bb forges forges-probe refreshes these columns without re-querying the SWH API. The whole forge/catalog channel lives in scripts/detect-forges.clj: it shares this repository’s harvested data but serves the Software Heritage catalog goal, not the domain-regex goal.
• domain-regexes.csv (and public-sector-domains-central+-regex.txt) – per-country (and global) email-domain regexes over the central+ scope (central government plus the first subdivision tier, i.e. public-sector-domains-central+.csv), ready for the domain_regex setting of swh-institutional-analysis. On mixed suffixes (gov.uk, gov.br=…), subtrees below the first tier are kept out via curated exclusions (see =scripts/README.org).
• A few generated mapping files linking country names to source IDs.

Where the data comes from

• crt.sh – subdomains seen in public TLS certificates (main harvest).
• HTTPS probes – check that a host actually serves content.
• Wikidata – ministries, parliaments, central banks, courts and their official websites.
• IANA, UN/DESA, CIA World Factbook, OECD – other official sources used to cross-check candidates.
• Official directories of public bodies (bb directories harvest, declarative specs in scripts/detect-from-directories.clj) – the per-country directories listed in swh-sopc-data-sources. When the directory’s central scoping is authoritative (Norway’s Enhetsregisteret, sector 6100) its domains enter the registry channel directly; when levels or types are mixed (Germany’s Behördenwegweiser) its hosts feed candidates.csv with a strong score bonus and the directory source tag, and curation decides.

See swh-sopc-data-sources for a list of all data sources used in the final State of Public Code.

Running the pipeline

Everything is driven by Babashka through bb.edn tasks.

# Install Babashka (Linux / macOS)
bash <(curl -s https://raw.githubusercontent.com/babashka/babashka/master/install)

# From the repository root:
bb pipeline help     # list all sub-commands
bb pipeline all      # full run: collect + enrich + report

See scripts/README.org for the full command reference, scoring rules and environment variables.

AI policy

See AI_policy.md.

Contributing

This repository uses Conventional Commits.

Please don’t push changes directly: create tickets and merge requests or send me patches.

When in doubt, please get in touch: bastien.guerry@inria.fr

Checking the classifications for your country

If you know the public administration of a country, you can help verify that our inclusions and exclusions are right:

• data/public-sector-domains-central+.csv – are the central and central-1 rows really the central government and the first tier below it?
• countries/<ISO3>_<slug>/candidates.csv and summary.md – did we miss or misclassify domains? candidates-local.csv holds what we set aside as subnational.
• countries/<ISO3>_<slug>/sources/registry/excluded-labels.csv – for shared roots like gov.br: is the list of excluded labels complete and correctly tagged (central-1 vs local)? Corrections go in excluded-labels-extra.csv, which generated files never overwrite.
• First-tier bodies hosted inside an excluded subtree (a state agency such as educacao.sp.gov.br) match nothing until listed explicitly: telling us about them is especially useful.

Please email me at bastien.guerry@inria.fr if you have questions and suggestions that don’t fit in a ticket or merge request.

License

Data are published under the CC0 1.0 license; code under the EPL 2.0 license.