Skip to content

fix(ontology): prevent duplicate Ontology nodes from concurrent ingestion - #596

Merged
aaronsb merged 1 commit into
mainfrom
fix/588-duplicate-ontology-list
Oct 2, 2026
Merged

aaronsb merged 1 commit into
mainfrom
fix/588-duplicate-ontology-list

Conversation

@aaronsb

@aaronsb aaronsb commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Refs #588 (the existing duplicate nodes still need repair; follow-ups in #597)

Root cause

It's a data duplicate, not a query fan-out. Read-only Cypher on the dev graph shows two :Ontology nodes named operating-models:

id ontology_id creation_epoch
2533274790395938 ont_a5198155-… 6
2533274790395939 ont_367a4350-… 6

The two ids are consecutive, both nodes were created at epoch 6, and each has 11 incoming SCOPED_BY edges from Sources. Edge creation matches the ontology by name, so it links each Source to both nodes. The list and scores endpoints just return every node.

The cause is ensure_ontology_exists: it checks and then CREATEs, and AGE has no uniqueness constraint. Two ingestion workers starting on a new ontology both see it as missing and both create it. The existing "race" handler only runs if CREATE raises, and CREATE never raises here. The nodes date from epoch 6, so the epoch-22 CLEAVE suggested in the issue comment isn't the cause.

Fix

  • ensure_ontology_exists takes a per-name session advisory lock (pg_advisory_lock(200, hashtext(name))) on one connection. Under that lock it re-checks, CREATEs, and commits before it unlocks. If the unlock fails, the connection is discarded rather than returned to the pool.
  • _execute_cypher (and get_ontology_node / create_ontology_node) accept an optional caller-held conn.

Not in this PR: repairing existing data

The duplicate node is still in the graph. A repair would keep one node (the lower id), move the SCOPED_BY edges and anything keyed by the other ontology_id (scores, annealing proposals) onto it, then DETACH DELETE the loser. That is destructive, so I've left it for a separate decision. A later uniqueness guard (an expression index on Ontology name) can only go in after the repair.

Test plan

  • New TestEnsureOntologyExistsConcurrency: two threads ensure the same new ontology. On pre-fix code it fails (2 nodes created); with the fix it passes. It also checks that the lock is released when CREATE fails.
  • pytest tests/unit/: 881 passed (run in kg-api-dev on a copy of this branch)
  • pytest tests/api/ -k "ontolog or batch": 103 passed, 5 skipped
  • lint_queries.py: clean
  • Not exercised against real concurrent ingestion on Postgres

AGE has no uniqueness constraints, so the check-then-CREATE in
ensure_ontology_exists let two concurrent ingestion workers both see a
new ontology as missing and both create it. That left two Ontology nodes
named operating-models, which ontology list and scores then show twice.

Run the re-check and the CREATE on one connection holding a per-name
session advisory lock, committing before unlock so the next waiter sees
the node. _execute_cypher gains an optional caller-held conn for this.

Refs #588
@aaronsb
aaronsb merged commit e4ef8f2 into main Oct 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant