llm-link-provenance

A URL that resolves is not the same as a URL that was grounded.

Models routinely emit links that are real, reachable, and completely invented — pulled from parametric memory rather than the context you supplied. Existence checks pass those. An HTTP 200 tells you the page exists; it tells you nothing about whether your model was given it.

This package checks the other thing: is this URL in the set of URLs I actually handed the model? Pure string comparison, no network, no model, no dependencies.

Model output:  "The project is open source on [GitHub](https://github.com/Acme-io/Acme)."
You supplied:  https://github.com/acme-io/acme-pro
                              ↓
Result:        "The project is open source on GitHub."
               removedLinks: [{ anchor: "GitHub", destination: "https://github.com/Acme-io/Acme" }]

Acme stands in for a real organisation here, and that is the whole problem: put an actual name in its place and the invented URL returns 200, because it is somebody's repository. It is still not the one you gave the model, and no amount of liveness checking will tell you so.

Where this sits

Approach	Verifies	Cost
Liveness checks (Wayback, HTTP HEAD)	The URL exists	Network round-trip per link
Guardrails AI ProvenanceLLM, NLI grounding	The prose is supported by context	A model call
llm-link-provenance	The URL literal is in your input set	A Set.has()

These compose. Run this first — it is free and catches the class that liveness checks structurally cannot — then send whatever survives to semantic verification.

Measured stakes: research puts 3–13% of LLM-emitted citation URLs at pure hallucination (no Wayback record, likely never existed), with 5–18% non-resolving overall. The subset that resolves and was never grounded is the one this package is for — and it is the larger one. Sampling 400 real citation URLs, 84.5% answer an HTTP request today: a liveness gate passes every one of them, whether or not the model was ever shown them.

Install

npm install llm-link-provenance

Use

import { buildAllowlist, enforceLinkProvenance } from 'llm-link-provenance'

// Build the allowlist from exactly what the model received.
const allowlist = buildAllowlist([sourceDocument, retrievedChunks, userPrompt])

const { text, removedLinks } = enforceLinkProvenance(modelOutput, allowlist)

if (removedLinks.length) {
  console.warn('ungrounded links removed', removedLinks)
}

Be precise about what you pass to buildAllowlist. Anything not in it is ungrounded by definition — pass the retrieved chunks, not the whole corpus they came from.

Site-relative links

By default only absolute http(s) links are governed; relative links pass through untouched. To govern your own internal paths, supply a pattern:

const options = { internalPathPattern: /\/posts\/\d+(?:-[^\s#]+)?(?:#[^\s]+)?/ }

const allowlist = buildAllowlist([source], options)
enforceLinkProvenance(modelOutput, allowlist, options)

Write it as a complete matcher for the whole path. It gets anchored for destination checks and given the g flag for extraction; flags on the RegExp you pass are ignored.

API

Function	Purpose
enforceLinkProvenance(text, allowlist, options?)	Flatten ungrounded Markdown links → { text, removedLinks }
buildAllowlist(inputs, options?)	Model inputs → canonical URL set
canonicalizeUrl(value)	One URL → its identity string, or null if unvouchable
extractUrlCandidates(text, options?)	Pull every URL out of a string

Design decisions worth knowing

Degrade, don't delete

An ungrounded link becomes its anchor text. The sentence still reads, a human reviewer can still see what the model meant, and removedLinks is your audit trail. Deleting the anchor would silently rewrite the prose on top of dropping the link — a second failure to fix the first.

URL identity is not string equality

Five things vary without changing which source is meant, so they are normalized:

Edge	Input	Identity
Percent-encoding spelling	https://ex.com/%7euser	https://ex.com/~user
Protocol upgrade	http://ex.com/a	https://ex.com/a
Fragment	https://ex.com/a#section	https://ex.com/a
Trailing slash	https://ex.com/a/	https://ex.com/a
HTML entity	https://ex.com/a?x=1&y=2	https://ex.com/a?x=1&y=2

Host, path and query stay strict — those genuinely change the source. https://ex.com/docs and https://ex.com/doc are different sources, and treating them as one is how a provenance checker starts laundering hallucinations.

Two cases return null, meaning unvouchable, which callers must treat as ungrounded rather than as "no opinion": malformed percent escapes (/50%off) and non-http(s) schemes (javascript:, mailto:).

If you write this yourself in twenty lines, percent-encoding and the http→https rule are the two you will miss, and you will find out when a legitimate source gets flagged as a hallucination in production. Measured on 11,091 real citation URLs: exact string matching calls every one of these rewrites a hallucination, and lowercase-plus-trailing-slash — the version you get after the first bug report — still misses the protocol upgrade (10,174 URLs affected), the appended fragment (11,081), and percent-encoded unreserved characters (2,473). See bench/README.md.

Fenced code is content, not citation

Code blocks pass through byte-for-byte. A URL in a shell example is being shown, not claimed. Conversely, extractUrlCandidates deliberately does look inside fenced code when building the allowlist — the model received those bytes, so they count as supplied.

CJK punctuation terminates a URL

In Chinese and Japanese prose a URL is routinely followed by 。 or ） with no whitespace. A whitespace-only terminator set swallows them into the URL and every match fails. ，。；：！？、〈〉） are treated as boundaries.

Images and escapes are out of scope

![alt](url) is an image, \[text\] is escaped literal text. Neither is a citation, and neither is touched.

Measured on real corpora

Behaviour is measured on documents nobody wrote for this test: 2,598 GitHub READMEs (68.5 MiB, including Chinese and Japanese repos) and 11,091 citation URLs from ExpertQA.

Property	Result
Grounded links preserved when the allowlist comes from the document itself	267,795 links, 3 false flags — all three malformed in the source
Inline links governed, against markdown-it as oracle	99.977% of 267,415
False hallucinations across 35,530 realistic URL rewrites	0
End-to-end on 2,149 real retrieval-augmented answers	100.000% of grounded links kept, 99.985% of ungrounded removed

The corpus found five defects that hand-written fixtures did not, including a one-line inline code span that opened a Markdown fence and silently disabled governance for the rest of the document. Each is now a regression test. Full method and numbers: bench/README.md.

What this does not do

• Does not check that a URL exists. Use a liveness checker for that; they answer different questions.
• Does not verify that the linked page supports the claim. That is semantic grounding — use an NLI-based validator downstream.
• Does not rewrite or repair links. It flattens or it leaves alone.

Those three are deliberate: another tool answers each of them better. The next section is different — it is a gap, not a boundary.

Known limitations

Only inline links are governed. [a](b) — with angle destinations, titles, balanced parens and nested emphasis in labels — plus fenced-code exclusion. Three other forms carry a URL and none of them is governed:

Form	Count in the README corpus	Governed
Inline [a](url)	267,415	✅ 99.977%
Reference [a][ref] + [ref]: url	4,792	❌
Autolink <https://…>	597	❌
Bare URL in prose	uncounted	❌

Why this one matters more than the count suggests. Every defect the benchmark found failed conservatively — a grounded link got flattened, which is visible and annoying. This gap fails in the opposite direction: an ungrounded link in one of these forms passes through untouched and ships to the reader. A hallucinated citation is exactly what you installed this to catch.

Two mitigations until it is fixed:

• The allowlist side is safe. A source document written in reference style still contributes its URLs — [ref]: https://… puts the URL in prose, where extraction finds it. Only enforcement on the output is affected.
• Constrain the output format. If you control the prompt, require inline links. Long-form authored Markdown uses reference style freely; models asked for [description](URL) overwhelmingly comply.

Tracked as a known issue — if your model emits reference-style or bare-URL citations, this package governs none of them today.

License

MIT