test(rdf): record the canonicalizer gaps against the extracted library - #25
Open
jdsika wants to merge 3 commits into
Open
test(rdf): record the canonicalizer gaps against the extracted library#25jdsika wants to merge 3 commits into
jdsika wants to merge 3 commits into
Conversation
RDFC-1.0 canonicalization already makes RDF output deterministic:
isomorphic graphs always serialize identically. It does not make output
diffable. Blank nodes are numbered `c14nN` in a single global order, so
inserting one class can renumber every blank node after it and rewrite
most of the file. A one-line semantic change lands as a whole-file diff,
which makes generated OWL/SHACL hard to review and noisy to keep under
version control.
Add a `diff_stable` argument to `canonicalize_rdf_graph()` and a
`--diff-stable/--no-diff-stable` flag to the four RDF generators. When
enabled, blank-node labels are derived from each node's own neighbourhood
via Weisfeiler-Lehman refinement, so an edit relabels only the blank
nodes it actually touches.
Measured churn on a real schema (add one class, count changed lines):
generator default --diff-stable
owlgen 2091 17
shexgen 796 50
shaclgen 291 13
rdfgen 115 25
Output stays deterministic and isomorphic either way; only the choice of
label changes. Off by default, because enabling it relabels existing
output.
The refinement itself lives in `diffable-rdf`, whose only dependencies
(rdflib, pyoxigraph) are already linkml-runtime dependencies at higher
versions, so this adds no new transitive dependencies.
…-opping Bump the floor to diffable-rdf 0.3.0 and add the missing uv.lock entry: the dependency was declared in pyproject.toml but never locked, so "uv lock --check" and the "uv sync --frozen" anti-malware gate would both have failed CI. 0.3.0 also fixes two defects in the Weisfeiler-Lehman labelling this feature relies on. Disconnected blank-node components now converge independently, so an edit in one region no longer relabels an unrelated one. And the suffix used to tell structurally indistinguishable nodes apart was assigned in c14nN *text* order, so c14n10 sorted between c14n1 and c14n2 -- adding a tenth tied blank node relabelled eight of the nine already there, the exact opposite of what this labelling is for. Separately, diff_stable=True was silently ignored whenever pyoxigraph refused the graph and canonicalize_rdf_graph degraded to rdflib. Weisfeiler-Lehman refinement consumes canonical pyoxigraph quads, and that path exists precisely because there are none, so the argument could not be honoured -- but the caller was never told. "shaclgen --include-annotations --diff-stable" reaches it, via the literal predicate an annotation tag without a ':' produces, and returned output byte-identical to --no-diff-stable. It now warns, with a regression test asserting the warning and the byte-identical output that makes silence misleading.
rdf_canonicalize.py and diffable_rdf.canonicalize_rdf_graph are the same code, split when the maintainers asked for the implementation to live outside linkml (linkml#3295). The copies have diverged. This adds a conformance file that asserts each correctness property against both and marks the side that does not hold it xfail(strict=True), so a fix on either side makes its case pass and the suite fail until the mark is dropped. Nine gaps are in the local copy, two of them silent data corruption: - a base IRI ending in "#" relativizes every term to <#a>, which rdflib reads back as ...d##a, and the output parses cleanly - a shared rdf:List tail is written once per referencing list, so nine triples in become eleven out - N-Triples output for a graph pyoxigraph refused is written anyway and fails on line 1 - fallback sorting uses str.splitlines(), which breaks on U+2028 and five other characters that N-Triples permits raw inside a literal - degraded RDF/XML, degraded Turtle with unbound namespaces, and json-ld are each non-deterministic across processes - trailing newlines differ by format - a Dataset is accepted and flattened, dropping the graph names One gap runs the other way: the library drops @base on its rdflib fallback, which linkml preserves. rdflib_dumper.dumps(prefix_map={"@base": ...}) hits exactly that path, because the metamodel's bare "status: testing" produces a relative IRI that forces the fallback. The library is therefore not yet a drop-in replacement, and no behaviour changes here. Delegating the default path was tried and reverted for that reason. It fixes all nine local gaps and leaves the four RDF generators byte-identical, but regresses test_loaders_no_namespace. It becomes viable once diffable-rdf carries base_iri, and also needs diff_stable so the flag added in the previous commit stops running on the unfixed local pipeline.
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Record the RDF canonicalizer gaps against the extracted library
This changes no behaviour. It adds one test file and nothing else.
Why
linkml_runtime/utils/rdf_canonicalize.pyanddiffable_rdf's canonicalizerare the same code. The module was extracted into a standalone library at the
maintainers' request (linkml#3295), and the two copies have since
diverged.
While integrating the library I compared them property by property. The
divergence turned out to be worth writing down, so this adds a conformance file
that asserts each property against both implementations and marks whichever
side fails it
xfail(strict=True). Strict, so the file is a ratchet in bothdirections: when a fix lands on either side its case starts passing and the
suite fails until the mark is removed.
Nothing is asserted about the library that is not also asserted about linkml,
so the library is not taken on trust.
What it finds
Nine gaps in linkml's copy. Two are silent data corruption, which is the part
worth a second look — the output parses cleanly and says something the input
never said, so nothing reports it:
#rewrites every term. Relativizinghttp://ex.org/d#aagainst basehttp://ex.org/d#gives<#a>, correct perRFC 3986, but rdflib's parser resolves a fragment reference by concatenation
and reads it back as
http://ex.org/d##a.rdf:Listtail is written twice. rdflib's Turtle writer renders( … )collection syntax per list, so a tail referenced from two listsbecomes two separate blank nodes. Nine triples in, eleven out.
The rest fail loudly or produce unusable output:
fails on line 1. The fallback is reached because a term is non-standard,
and for N-Triples that term is always one N-Triples cannot write either.
str.splitlines(), which breaks on U+2028, U+2029,U+0085, U+000B, U+000C and U+001C–1E. N-Triples permits those raw inside a
quoted literal, so one statement becomes two lines, the halves sort
independently, and the document no longer parses.
unbound namespaces gets
ns1/ns2names allocated in traversal order, andjson-ldis absent from the format map so it falls through to rdflib. Allthree vary across processes — tested with four
PYTHONHASHSEEDvalues.Datasetis aGraphsubclass, so it passes the type check and isflattened, dropping the graph names.
One gap runs the other way
diffable-rdfdrops@baseon its own rdflib fallback, and linkml does not.rdflib_dumper.dumps(..., prefix_map={"@base": ...})hits exactly that path,because the metamodel's bare
status: testingon auriorcurieslotserializes to a relative
<testing>that forces the fallback.That is asserted here too, as an xfail on the library side. It is a bug in the
library, it is ours to fix, and it is the reason this PR does not simply
replace linkml's implementation with the library's.
Why not just delegate
I tried. Replacing the default path with
diffable_rdf.canonicalize_rdf_graphfixes all nine gaps and leaves all four RDF generators byte-identical — but it
regresses
test_loaders_no_namespace, because of the@basegap above.Delegation becomes viable once the library carries
base_iri. It also wants adiff_stableparameter, so the flag in linkml#3985 stops running on theunfixed local pipeline. Both are on the library's side; I'll raise them there.
Until then this PR just states the facts and leaves the code alone, so
maintainers can decide whether to fix them in place, wait for delegation, or
accept them as known.
Related
Follows from: linkml#3295 · Module added by: linkml#3407 ·
Hardened by: linkml#3524, linkml#3703 · Known fallback issue:
linkml#3803 · Nondeterminism leaks: linkml#3516