Operating contract for autonomous and semi-autonomous coding agents working in this repository. Goal: safe, reproducible contributions with minimal human back-and-forth.
A FastAPI service that turns a GitHub URL (repository / user / org) into
JSON-LD aligned with Open Pulse Ontology v2.1.2. The service runs the
input through a multi-stage pipeline that combines deterministic rules,
provider lookups (GitHub REST, ROR, ORCID, Infoscience), and optional LLM
agents to produce a graph of schema:SoftwareSourceCode,
schema:Person, org:Organization, org:Membership,
pulse:Contribution, and schema:ScholarlyArticle entities.
The legacy v1 API was removed in 3.0.0 (repo-split release). All work
targets V2 under git_metadata_extractor/; docs/migration-v1-to-v2.md maps the removed
endpoints for old consumers.
git_metadata_extractor/
app.py # FastAPI app: mounts the /v2 router + /docs UI
api/ # the /v2 HTTP surface (URL prefix is /v2 — public contract)
_router.py # the single APIRouter
_helpers.py # env gates, stage constants, app-state resolution
extract.py # GET/POST /extract + _run_extract_job + assembly
auto_ingest.py # post-extract index write-through (opt-in flags)
jobs.py # /jobs/{id}, cancel, /crawl
system.py # /cache/clear, /health
jobs.py # async job store backing POST /v2/extract
config.py # config knobs
dependencies.py # provider wiring, cache resolver
log_context.py # request-id logging context
observation/query_log.py # per-request external-query log
agents/ # production runtimes only
models.py # AgentResult, ProviderSet, TypedEntityBuckets
registry.py # runtime → runner table
llm/ # LLM-backed agents
_payload_helpers.py # force_server_uuid + shared post-LLM stamps
_verdict_cache.py # per-agent result cache
model_config.py # LLM provider/model profiles + credentials
<kind>/agent.py # per-entity LLM agents (one Pydantic AI run each)
agent_tools/ # Tool factories (selenium, ROR, ORCID, etc.)
# *_rag.py: per-index Qdrant search tools
# — see docs/v2-rag-tools.md
rule_based/
<kind>_agent.py # deterministic counterparts (no LLM)
refiners/ # hybrid-runtime LLM refiners — propose targeted
<kind>/agent.py # patches over rule-based output, whitelisted fields only
providers/ # the provider/READ layer (was "ingest" pre-split)
cache.py # ProviderCache (SQLite, WAL)
github_provider.py etc. # github / ror / orcid / infoscience clients
# *_rag.py: async Qdrant-backed RAG providers
gimie_api_client.py # gimie sidecar client (TTL → JSON-LD bridge)
gimie_extract.py # extract_gimie intermediate (sidecar/in-process seam)
github_accounts/ # GitHub user/org GraphQL parsers + models
detection/ # GitHub URL classifier
pipeline/
orchestrator.py # stage runner, fan-out concurrency, retries
stages/ # the actual stages — see "Pipeline" below
schema/ # JSON Schemas (agent + strict) + JSON-LD context
validation/ # strict-schema + SHACL validators
canonicalization/ # ID resolution, string normalisation
experimental/ # not production: pi terminal-agent PoC
terminal/ terminal_subagent/ skills/
# NOTE: the RAG index layer (formerly src/index/ + src/module/ + the
# /v2/indices/* + /v2/manifest API) lives in a separate repo/service:
# https://github.com/sdsc-ordes/open-pulse-sources
# The v2 read-side providers import it as the `open_pulse_sources`
# library — a declared, tag-pinned dependency in pyproject.toml (see
# "Cross-repo version pin" below). This service only READS the indices
# (Qdrant + DuckDB under data/index/); ingest/embed/reset happen in the
# open-pulse-sources service. config/index/*.yaml stays here because
# the library resolves config/data paths CWD-relative.
tests/v2/ # default test target
There are two sequencers, not one — a distinction worth holding onto, because "add a stage" means different work in each:
PLAN_BY_TYPE(pipeline/orchestrator.py:120) — the agent-generation plan. Three plans, one per detected type; the orchestrator runs them with fan-out concurrency and retries. This is the only part that varies with the input kind._run_extract_job(api/extract.py) — everything after the agents: reconciliation, ROR resolvers, validation, inference, output. A flat sequence of direct calls, not a plan object. Stage banners in that file (# === <name> stage ===) and theSTAGE_*constants inapi/_helpers.pyare the naming source of truth.
/v2/extract runs the same post-agent sequence regardless of
agent_runtime; the runtime selects agent implementations and gates the
[LLM] stages below.
Phase 1 — agent generation (orchestrator, PLAN_BY_TYPE):
repository: context_gather -> repo_agent -> person_agents -> org_agents
-> article_agents -> membership_agents -> contribution_agents
user: context_gather -> person_agent -> repo_agents -> org_agents
-> article_agents -> membership_agents -> contribution_agents
organization: context_gather -> org_agent -> person_agents -> repo_agents
-> article_agents -> membership_agents -> contribution_agents
classify_urlruns in the API layer before the orchestrator, not as a plan stage.context_summary_agentis not a plan entry: it runs insidecontext_gather(orchestrator.py:377), LLM runtime only, and fails open with a warning — the pipeline continues without a compiled summary.- The root stage per type comes from
ROOT_STAGE_BY_DETECTED_TYPE.
Phase 2 — post-agent sequence (api/extract.py, in execution order):
permissive_validation per-entity permissive schema pass
llm_dedup [LLM] cross-bucket entity dedup + ID remap (fail-open)
reconcile_entities deterministic ID canonicalisation + linkage
(also anonymises emails via stages/privacy.py)
resolve_company_to_ror _company -> ROR: Membership + Org stub
resolve_bio_to_ror bio/blog/email -> ROR: Membership + Org stub
resolve_bio_to_ror_llm [LLM] LLM long-tail affiliation resolver
resolve_placeholder_orgs_to_ror upgrade placeholder Orgs to real ROR entities
llm_critic [LLM, gated] drop-suggestion stage (off by default)
refine_with_llm [hybrid] whitelisted-field patches over rule-based output
guarantee_repo_author stamp github-owner as schema:author when empty
strict_validation per-entity strict JSON Schema check
assemble_output split graph into root + related + excluded
link_veracity [LLM, gated] verify every URL via Selenium fetch + LLM;
on failure promote_failed_id_entities +
apply_link_pruning_to_assembled_output
validate_articles drop placeholder/sentinel-DOI articles
validate_author_classes drop schema:author refs whose target is missing
or not a schema:Person (SHACL class shape)
validate_ownership strip mismatched pulse:owns
infer_owners stamp pulse:owns / pulse:ownedBy from handles;
coerces bare-login strings (e.g. "luzpaz") to
{"@id": "https://github.com/luzpaz"} so SHACL
never sees a <file:///CWD/...> URI
validate_ownership SECOND pass — catches inverse edges that
infer_owners just added
prune_dangling_refs drop refs whose target left the graph
infer_github_handle_parents fuzzy-search ROR for each github org's parent;
add ROR org entities, stamp unitOf
org_relationships [LLM] whole-graph LLM call to refine unitOf edges
infer_org_units deterministic name-token fallback for unitOf
demote_github_props_to_units move github-only props off ROR-identified orgs
emit_fork_parent_stubs materialise the upstream repo of a fork
infer_article_source_organization attribute articles to a source org
concept_tagging [gated] EPFL Graph concepts/keywords/disciplines from
README + GitHub description onto the root repo
as internal _concepts/_keywords/_disciplines
(off by default: V2_CONCEPT_TAGGING_ENABLED)
tag_rule_based_disciplines deterministic discipline fallback
build_jsonld_output final JSON-LD graph; strips redundant pulse:ror
from any org:Organization whose @id is already
the ROR (closed-shape violation fix)
shacl_gate SHACL validation — warning-only (see below)
compute_stats response counters/timings
Verify this list against the code before trusting it — it was reconstructed
from PLAN_BY_TYPE, the stage banners, and the STAGE_* constants on
2026-07-29. A published, human-facing version lives in
docs/v2-pipeline.md.
Gates:
- Stages tagged
[LLM]only run inagent_runtime=llm. link_veracityis[LLM]-only too: inagent_runtime=rule_basedit is always skipped (rule-based mode is guaranteed LLM-free).agent_runtime=hybridruns the rule-based generators (stages 4-9) and an LLM refiner stage (refine_with_llm, between reconciliation andguarantee_repo_author). The refiner proposes whitelisted-field patches per entity type: organizations (pulse:OrganizationType), repositories (pulse:discipline,pulse:repositoryTypeonly when current ispulse:Other), and persons (schema:nameonly when current looks like a GitHub handle). LLM-only stages (llm_dedup,llm_critic,link_veracity,org_relationships) are skipped in hybrid mode. Toggle withV2_HYBRID_REFINER_ENABLED(defaulttrue).llm_criticis off by default. SetV2_APPLY_CRITIC_PRUNING=trueto enable (LLM mode only).link_veracityis on by default in LLM mode. SetV2_LINK_VERACITY_ENABLED=falseto skip even in LLM mode (recommended for batch runs).concept_taggingis off by default. SetV2_CONCEPT_TAGGING_ENABLED=trueto opt in. Backends are pluggable viaV2_CONCEPT_TAGGING_BACKEND∈ {epfl_graph(default, calls graphai),wikipedia(credential-free MediaWiki opensearch),llm(pydantic-ai)}. Stamps_concepts/_keywords/_disciplinesas internal_*metadata (stripped before JSON-LD output and strict validation). Optional OpenAlex enrichment per discipline viaV2_CONCEPT_TAGGING_OPENALEX_RELATED_ENABLED=true(publications, people, units). Full reference atdocs/concept-tagging.md.
Hallucination guards baked into agents:
force_server_uuidoverwrites whatever UUID the LLM emitted with a server-generated one inidentifiers.uuidonly — never as a top-level field (additionalProperties violations).- Repository agent post-LLM: stars/forks from GitHub REST (deterministic, not LLM-derived); discipline fallback
wd:Q428691(computer engineering) when LLM emits empty/null;pulse:repositoryTypekeyword heuristic in rule-based mode. - Contribution agent post-LLM: stamps
schema:author = target_person.idandpulse:contributionTo = target_repository.idfrom the orchestrator's authoritative pair, regardless of what the LLM emits. - Article agent post-LLM: drops the entity if
schema:identifieris a placeholder DOI (10.0000/...) or sentinel string (UNKNOWN,N/A,TBD, etc.) and nopulse:infoscienceArticleIdentifieris present. - Article agent (rule-based) defaults to repo-name-only Infoscience queries; opt in to the wider
include_person_queries=True/include_organization_queries=Trueblend only when over-attribution risk is low. - Membership agents (both rule-based and LLM): swap
time:hasBeginning/time:hasEndwhen ORCID returns the pair inverted, sohasBeginning <= hasEndalways holds.
SHACL conformance auto-fixes (warning-only shacl_gate, but the graph is fixed in place):
The SHACL gate emits violations as result.warnings rather than aborting,
so the responsibility for producing a SHACL-clean graph lives in the
upstream stages and agents. The four most common violations are
addressed deterministically:
pulse:ownedByIRI shape —infer_ownersrewrites bare-login strings to{"@id": "https://github.com/{handle}"}. Without this, SHACL resolves the bare token against the working-directory base URI (<file:///workspaces/project/luzpaz>) and the closed-shape check onschema:Person | org:Organizationfails.pulse:rorredundancy —build_jsonld_outputstrips the field on anyorg:Organizationwhose@idis already the ROR. The Organization shape issh:closedand rejectspulse:ror; the@idalready carries that information.Membershipdate order — both membership agents swap inverted dates (above).schema:authorclass —validate_author_classesfilters refs whose target is missing from the graph or not typedschema:Person. Catches Membership / Contribution ids leaking into author lists.
Orchestrator fanout filtering:
_filter_person_work_itemsskips a queued person fanout when the GitHub handle resolves totype=Organization(cachedprovider.github.get_user(login)lookup)._filter_org_work_itemsmirrors the rule for org fanouts: skips handles whose GitHubtypeisUser. This prevents the 4× retry loop inorg_agentfor personal handles encoded intoorg:hasMembershipcomposite ids._person_fanout_contextsmaterialises a User-account repo owner as a Person when missing fromcontributors(abandoned repos, empty repos). Prevents the owner from leaking as a bare-string ref inpulse:ownedBy/schema:authorwith no backing entity.
GET /v2/health— health check (open, no auth)POST /v2/extract— async job. Body:{source_url, agent_runtime?, output_format?, include_context_summary?}. Returns{job_id, status, status_url}; pollGET /v2/jobs/{job_id}.GET /v2/jobs/{job_id}— job status + result when completeGET /v2/extract/{full_path:path}— synchronous extract (single repo)GET /docs— Swagger UI with auto/manual dark-mode toggle (override persisted inlocalStorage)
Auth: /v2/extract and /v2/jobs/{id} require
Authorization: Bearer <API_TOKEN> (see the API_TOKEN row below). /,
/docs, and /v2/health are open. The dependency lives in
git_metadata_extractor/auth.py::verify_token.
| Var | Default | Purpose |
|---|---|---|
GME_GITHUB_TOKEN |
— | required for live GitHub provider |
API_TOKEN |
— | bearer token guarding /v2/extract and /v2/jobs/{id}. Fails closed: missing → 503 (no dev bypass). /, /docs, /v2/health stay open. Generate with python -c "import secrets; print(secrets.token_urlsafe(32))". |
RCP_TOKEN / OPENAI_API_KEY / OPENROUTER_API_KEY |
— | one is required for LLM mode |
INFOSCIENCE_TOKEN |
unset | only for protected Infoscience routes |
SELENIUM_REMOTE_URL |
unset | enables Selenium-backed link veracity + selenium-fetch tool |
V2_AGENT_RUNTIME_DEFAULT |
llm |
default runtime when /v2/extract omits agent_runtime |
V2_USE_MOCK_PROVIDERS |
true |
swap in mock GitHub/ORCID/Infoscience/ROR providers |
V2_LINK_VERACITY_ENABLED |
true |
turn off to skip the link-veracity stage in LLM mode (rule-based mode skips unconditionally) |
V2_CONTEXT_SUMMARY_SCOUT_MODE |
false |
upgrade context_summary LLM stage to scout mode: broad RAG-search toolkit (orcid/ror/infoscience/openalex/zenodo/ethz/huggingface/renkulab/snsf/epfl_graph + selenium_fetch) on top of the legacy grep_repository_corpus + DuckDuckGo pair, plus a structured-brief prompt with explicit People / Organizations / Articles / Affiliations / Caveats sections. Per-entity LLM agents (person, org, article, membership, contribution) automatically benefit since they already consume the summary_markdown. Trade-off: heavier upfront LLM call, but per-entity calls send less context and duplicate ORCID/ROR lookups across entities collapse into the scout's shared brief. |
V2_INFOSCIENCE_RAG_ENABLED |
true |
enables the Infoscience RAG agent tools (Qdrant-backed semantic search + on-demand chunk/record fetch). Construction degrades gracefully when Qdrant or RCP is unreachable. |
V2_ETHZ_RESEARCH_COLLECTION_RAG_ENABLED |
true |
enables the ETH Research Collection RAG agent tools (DSpace-backed sister index to Infoscience for ETHZ research outputs). Same shape: search + fetch_chunks + fetch_records. |
V2_HUGGINGFACE_RAG_ENABLED |
true |
enables the HuggingFace Hub RAG search tool (collections: hf_models, hf_datasets, hf_spaces, hf_orgs). |
V2_OPENALEX_RAG_ENABLED |
true |
enables the OpenAlex RAG search tool (collections: works, authors, institutions, sources, topics, concepts). |
V2_ZENODO_RAG_ENABLED |
true |
enables the Zenodo RAG search tool (collection: zenodo_records). |
V2_ORCID_RAG_ENABLED |
true |
enables the ORCID RAG search tool (entities: persons, employments, educations; collections namespaced by scope). |
V2_ROR_RAG_ENABLED |
true |
enables the ROR RAG search tool (scopes: epfl_ethz, switzerland, europe, worldwide). |
V2_SWISSUBASE_RAG_ENABLED |
true |
enables the SWISSUbase RAG search tool (collection: swissubase_entities; entities: studies, datasets, persons, institutions). Ingest is Selenium-driven; default scope embeds only EPFL/ETHZ/SDSC-affiliated studies. |
V2_RENKULAB_RAG_ENABLED |
true |
enables the RenkuLab RAG search tool (renkulab.io). One Qdrant collection per entity type: renkulab_projects, renkulab_groups, renkulab_users, renkulab_data_connectors. The single tool searches across all four by default; the entity_types argument scopes to a subset. |
RENKULAB_TOKEN |
unset | optional; without it the indexer can still ingest public projects/groups/data_connectors and harvest users via /search/query?q=type:User. With it, set on https://renkulab.io/api/data for richer user records. |
V2_EPFL_GRAPH_RAG_ENABLED |
true |
enables the EPFL Graph disciplines RAG search tool (search_epfl_graph_disciplines). Single Qdrant collection epfl_graph_disciplines over the curated EPFL Graph academic-discipline ontology (~2226 categories, depth 1..5, embeddings built from name + canonical Wikipedia lead-section + top anchor concept names). Wired into the repository, person, organization, and article LLM agents. Refresh with just epfl-graph-{ingest,enrich-wikipedia,embed}. See docs/epfl-graph-disciplines.md. |
EPFL_GRAPH_USERNAME, EPFL_GRAPH_PASSWORD |
unset | required by the epfl-graph-ingest recipe (the auth handshake against graphai.epfl.ch). Not needed at runtime once the index is hydrated — search_epfl_graph_disciplines only hits Qdrant + RCP. |
INDEX_QDRANT_URL |
http://qdrant:6333 (yaml default) |
Qdrant endpoint for every RAG index. Inside the devcontainer use http://gme-qdrant:6333. |
V2_APPLY_CRITIC_PRUNING |
false |
turn on to enable critic drop suggestions |
V2_MAX_CONCURRENT_AGENTS |
6 |
per-stage fan-out concurrency |
V2_PROVIDER_CACHE_PATH |
.cache/v2/providers.db |
SQLite path for the provider+verdict+pipeline cache. Use a different path per run profile (e.g. LLM vs rule-based) for isolation. |
V2_PROVIDER_CACHE_TTL_DAYS |
30 |
TTL for cached entries |
V2_PROVIDER_CACHE_ENABLED |
true |
when false, every external lookup is fresh |
V2_PIPELINE_CACHE_ENABLED |
true |
when false, every /extract re-runs the full pipeline |
V2_QUERY_LOG_DIR |
logs/v2_queries |
per-request external-query log destination |
LOG_LEVEL |
INFO |
DEBUG/INFO/WARNING/ERROR |
Required environment for serving requests: GME_GITHUB_TOKEN and at least
one LLM credential (when LLM mode is the default).
Rules:
- Never print, log, or commit secrets.
- Never modify
.env,.env2, or other secret-bearing files unless explicitly asked. - Fail fast and report the missing variable name (no value) when a required var is absent.
The justfile is the source of truth. Prefer just <recipe> over ad-hoc shell.
| Recipe | Purpose |
|---|---|
just install-dev |
install with dev extras |
just setup |
install + scaffold .env |
just serve-dev |
uvicorn + auto-reload (watches only src/**/*.py) |
just serve-gunicorn |
gunicorn with 4 workers (production-shape) |
just serve-stop |
stop whatever's bound to :$PORT |
just test |
fast tests via testmon |
just test-full |
full deterministic test run |
just test-file <path> |
one file |
just lint / just type-check / just check |
quality gates |
just v2-models-generate |
regenerate Pydantic models from strict schemas |
just v2-models-check |
assert generated models are in sync |
For batch extractions over many repos: scripts/v2/batch_extract.sh reads
a hardcoded URL list and runs them through /v2/extract with configurable
parallelism. Resumable: skips repos whose result file already exists with
a non-running status.
The open_pulse_sources library version lives in exactly one place: the
open-pulse-sources @ git+https://github.com/sdsc-ordes/open-pulse-sources@<tag>
entry in pyproject.toml dependencies. Every install path inherits it —
just install-dev, just install, CI, and tools/image/Dockerfile.
- Never add a second
pip install "open-pulse-sources @ git+…@<tag>"anywhere;tests/v2/test_open_pulse_sources_pin.pyfails on hardcoded second pins. - The
gme-sourcesimage tag intools/deploy/docker-compose.ymlmust match the library pin (same test enforces it). The library reads and that service writes the same DuckDB/Qdrant stores — skew corrupts shared state. - Pins must be immutable: a release tag (
vX.Y.Z) or a full commit SHA.main/latestdefaults fail the test. - Bumping the child = edit the pyproject pin + the compose image tag + add a README compatibility-matrix row.
OPEN_PULSE_SOURCES_REF(Dockerfile build arg) defaults to empty and is an override-only escape hatch for testing unreleased child revisions.- For cross-repo development,
just install-devre-installs a checkout at./open-pulse-sourcesor../open-pulse-sourcesas editable.
Three caches share a single SQLite DB (path: V2_PROVIDER_CACHE_PATH):
- Provider cache — gimie payloads, GitHub REST responses, ROR / ORCID / Infoscience hits. Deterministic, content-addressed.
- Agent verdict cache — LLM agent results keyed on agent name + identity. Skips a repeat LLM call for a known-good payload.
- Pipeline cache — full
/v2/extractresponse for a given source URL.
Set V2_PROVIDER_CACHE_PATH to a different file per run profile (e.g.,
.cache/v2-rule-based/providers.db for rule-based runs) to keep them
isolated and independently invalidatable.
- Keep diffs minimal and scoped to the requested task.
- Preserve existing code style, project conventions, and import patterns.
- Do not rename or move public modules unless explicitly requested.
- Do not modify secret-bearing files unless explicitly requested.
- Do not run destructive git/file operations unless explicitly requested.
- If unrelated local changes exist, do not revert them — work around them and report context.
JSON Schemas live in three byte-identical copies that must stay in sync:
git_metadata_extractor/schema/json/{type}/{entity}.schema.json(source)dev/ontology-v2-json-response/a-001/json-schema/{type}/pulse_{Entity}Shape.schema.json(promoted)tests/v2/fixtures/schema/{type}/{entity}.schema.json(test fixture)
After any schema edit: copy to all three and run just v2-models-generate
to regenerate Pydantic models in git_metadata_extractor/schema/models/. just v2-models-check
in CI catches drift.
The check regenerates and compares byte-for-byte, so the codegen toolchain
is part of the contract: datamodel-code-generator and ruff are pinned
exactly in the dev extra. Bump them deliberately and regenerate in the same
commit — a range would let any upstream formatting change turn the gate red on
an unrelated PR.
- Person:
https://orcid.org/{orcid}when ORCID known, elsehttps://github.com/{login}, elseurn:pulse:{uuid} - Organization:
https://ror.org/{id}when ROR known, elsehttps://github.com/{handle}, elseurn:pulse:{uuid} - Repository:
https://github.com/{owner}/{name} - Article:
https://doi.org/{doi}when DOI known - Membership:
{person_id}__{org_id}composite (double underscore) - Contribution:
{person_id}__{repo_id}composite (double underscore)
Composites are built with __ (see rule_based/membership_agent.py:431,
contribution_agent.py:355, ownership_check.py:1801,
resolve_bio_to_ror_llm.py:209). _extract_composite_pair still accepts a
single _ when parsing, for back-compat with graphs produced before the
convention changed — never emit that form in new code.
identifiers.uuid is always a server-generated UUIDv4 (via
git_metadata_extractor/agents/models.py::generate_uuid()); the LLM never controls it.
Fields whose names start with _ (e.g. _person_ref on Memberships) are
internal pipeline metadata. They are stripped before strict
validation, JSON-LD output, RDF serialisation, and any external
artefact. Never expose _-prefixed fields in API responses.
Completion reports must include:
- Files changed
- Behaviour change
- Commands run + key results
- Risks / follow-ups
No vague "done". Reports must include verifiable evidence.
- Requested scope implemented
- Relevant tests/checks passed (or blockers explicitly documented)
- No secret leakage
- No unrelated mutations
- No undocumented behaviour changes