Per-chunk text "breadcrumb" / metadata hook — `chunking_func` is silently bypassed on `/documents/scan` & `/documents/upload`

[geraldtscharre]

Summary

There is no working way — through the REST API — to prepend per-chunk metadata (e.g. source filename, path, document as-of date) to the chunk text that the extraction LLM sees. The one documented customization point for per-chunk text, chunking_func, is silently bypassed on the /documents/scan and /documents/upload paths, with no error and no log line. This makes per-chunk provenance/temporal grounding impossible for API-driven ingestion.

The problem (silent bypass)

chunking_func is described in lightrag/lightrag.py as a "Legacy chunking-function customization point," and its docstring is explicit:

"If process_options explicitly contains a chunking selector char (F/R/V/P), the dispatcher routes to a chunker that follows the new file-chunker contract … This chunking_func is NOT called in that case — it is a legacy escape hatch and is intentionally bypassed when the user opted into a specific strategy."

Meanwhile, in lightrag/api/routers/document_routes.py, the scan/upload enqueue path does:

api_process_options = process_options or PROCESS_OPTION_CHUNK_FIXED

So both endpoints always name a strategy (F by default) → chunking_func is never invoked. The only way to reach the no-selector path where chunking_func actually runs is the SDK (ainsert) with an empty/non-chunking process_options — not the HTTP API most deployments use.

Net effect: a user who supplies a chunking_func that, say, prepends a SOURCE / PATH / DATE breadcrumb to every chunk gets 0 of their chunks augmented, silently. The chunks are created, entities extracted, the graph builds — and the customization simply never ran.

Why this matters (use case)

The extraction LLM sees only chunk text, never chunk metadata (file_path, headings, created_at). For multi-document corpora this loses information that matters at extraction time, not query time:

• Temporal grounding: a document's as-of date (often only present in the filename or a header line) lands in chunk 1 and is lost on chunks 2..N. Entities extracted from later chunks get no date grounding, so the graph can't self-document when a fact was attested.
• Source provenance: which document a chunk came from — useful for any multi-doc corpus, essential when the same entity is mentioned across many documents with different roles/statuses.
• Section / table disambiguation: a chunk cut mid-table or mid-section loses the heading that scopes it (e.g. a membership roster whose column headers landed in a different chunk).

A per-chunk prefix ("breadcrumb") carrying SOURCE / PATH / DATE closes all three, because it puts the context into the text the extractor reads. ENABLE_CONTENT_HEADINGS helps at query time but not at extraction time, so it is not a substitute.

What I tried

Supplying a custom chunking_func that prepends the breadcrumb. It works on the SDK ainsert path (no selector → legacy path → chunking_func runs), but is a complete no-op when ingesting via POST /documents/scan or /documents/upload, because those force PROCESS_OPTION_CHUNK_FIXED. There is no process_options value I can send through the API to opt out of all four strategies and reach the legacy path.

Proposed solution

The cleanest general fix is a strategy-agnostic per-chunk transform hook, decoupled from "how to split":

LightRAG(
    ...,
    chunking_func=...,                 # how to split (existing)
    chunk_transform_func=my_transform,  # NEW: decorate every chunk after splitting
)
# my_transform(chunk: dict, doc_meta: dict) -> dict
#   receives one chunk {"content": ..., ...} + {"file_path":..., "doc_date":..., ...}
#   and may rewrite chunk["content"] (e.g. prepend a breadcrumb).
#   runs AFTER the strategy chunker (F/R/V/P) on every chunk, on every ingest path.

Why a new hook rather than extending chunking_func:

• It works for all four strategies (F/R/V/P), not just the fixed-size one — the bypass is the same for R/V/P.
• It keeps "how to split" and "what to decorate" orthogonal; a user can combine the paragraph-semantic strategy with a breadcrumb transform.
• It runs on both ingest paths (SDK and HTTP scan/upload), since it's applied after the strategy chunker regardless of how the chunker was selected.
• It composes with the recent chunking perf work (fix(pipeline): take chunking off the event loop and bound the separator cascade #3547/feat(tokenizer): safe encode-verified split/truncate contract, replacing #3559 #3565) without competing with it.

As a smaller interim, the F-branch dispatch in lightrag/pipeline.py could delegate to self.chunking_func when it is a non-default callable (identity-gated: is not chunking_by_token_size), via the legacy 6-arg signature — this restores chunking_func on the scan default path without a new API. We have this working locally (with regression tests). It doesn't help R/V/P, but it unblocks the scan default immediately. I'm happy to open a PR for either.

At minimum, the docstring / docs should call out that chunking_func is a no-op on the HTTP scan/upload paths, so users don't spend time wiring a customization that can't run there.

Minimal reproduction

calls = {"n": 0}
def my_chunking_func(tokenizer, content, split_by_character, split_by_character_only, overlap, size):
    calls["n"] += 1
    chunks = chunking_by_token_size(tokenizer, content, split_by_character, split_by_character_only, overlap, size)
    for c in chunks:
        c["content"] = "SOURCE: my-doc\n---\n" + c["content"]
    return chunks

rag = LightRAG(working_dir=..., chunking_func=my_chunking_func, ...)

# SDK path: works — chunks carry "SOURCE: my-doc"
await rag.ainsert(open("doc.md").read())

# HTTP path: my_chunking_func is NEVER called; chunks have no breadcrumb
# POST /documents/scan  →  process_options defaults to PROCESS_OPTION_CHUNK_FIXED
#                         → F-branch bypasses chunking_func → 0 augmented chunks

Environment

• LightRAG main (as of Aug 2026).
• Python 3.11, Postgres + Neo4j backends.
• Ingestion via /documents/scan (a supervisor polling inputs/<workspace>/).

[danielaskdd]

Thanks for the detailed writeup — the core claim checks out. Below is what we verified against main, a few corrections/additions, and the design direction we are currently discussing. Nothing is implemented yet; this comment is for discussion.

1. Verification: the bypass is real

Dispatch is driven by doc_process_opts.chunking_explicit (lightrag/pipeline.py:4924); only the else branch (pipeline.py:5058) calls self.chunking_func. The API forces a selector at document_routes.py:2408 (scan/upload) and document_routes.py:2644 (/documents/text, /texts).

Reproduced end-to-end with a chunking_func that prefixes every chunk:

process_options	chunking_func calls	chunks augmented
F (what the API forces)	0	0
"" (SDK ainsert)	1	1
i (non-chunking option)	1	1

No WARNING is emitted. The only signal is Chunking F: vs Chunking F(legacy): at INFO (pipeline.py:5049 / 5089) and doc_status.metadata.chunk_method = fixed_token vs legacy_chunking_func (pipeline.py:5164). The behaviour is intentional and pinned by tests/chunker/test_chunking_raw_lightrag_parity.py:501 — so this is a design/reachability gap, not a regression.

2. Corrections and additions

There is an escape hatch on scan/upload (contrary to "no process_options value can reach the legacy path"). resolve_parser_directives supplies the options, and or PROCESS_OPTION_CHUNK_FIXED only fires when they are empty. So any non-chunking option char reaches the legacy path:

a.txt          hint=''   -> api='F'  chunking_explicit=True
a.[-F].txt     hint='F'  -> api='F'  chunking_explicit=True    # explicit F == default, no difference
a.[legacy].txt hint=''   -> api='F'  chunking_explicit=True
a.[-i].txt     hint='i'  -> api='i'  chunking_explicit=False   # <- chunking_func runs

a.[-].txt (empty options) is rejected by _validate_filename_hint_for_resolution (routing.py:1028), so a non-chunking char is mandatory. On a legacy/raw text document i is effectively inert (pipeline.py:6371 returns early with no sidecars), but on a mineru/docling-parsed document it triggers real VLM analysis — so it is a workaround with cost, not a fix. /documents/text and /texts have no escape hatch at all, since TextChunkingStrategy only maps to the four strategies.

The packaged server never passes chunking_func (lightrag/api/lightrag_server.py:2238). So for anyone using the official server there is no supported way to set it in the first place — arguably the more fundamental gap than the silent bypass.

The legacy 6-arg signature carries no document identity. pipeline.py:5100 passes (tokenizer, content, split_by_character, split_by_character_only, overlap, size) — no file_path, no doc_id, no date. This matters for your use case: even on the SDK path a chunking_func cannot build a per-file breadcrumb (your repro hardcodes "SOURCE: my-doc"). By contrast the newer file-chunker contract already passes context — chunking_by_paragraph_semantic(..., blocks_path=..., doc_id=...) (chunker/paragraph_semantic.py:2048). Any real fix needs the metadata plumbed in, not just the hook re-enabled.

Partially covered already: extraction-time section context exists — ---Section Context--- is built from the chunk's heading breadcrumb and injected into the extraction prompt (operate.py:3929). It is empty under F, because the fixed-token chunker emits no headings; P/structured parsing does. Your point about ENABLE_CONTENT_HEADINGS being query-time only is correct.

Stale docstring: lightrag/lightrag.py:576-577 still says R/V are "not yet implemented and fall back to F". Both are implemented and dispatched (pipeline.py:5003 / 5029).

3. Direction under discussion

Rather than a separate chunk_transform_func, the option we favour is a custom chunking strategy selector backed by a chunker registry, mirroring the existing parser extension mechanism (lightrag/parser/registry.py + lightrag/parser/plugins.py, documented in docs/ThirdPartyParser.md):

• A ChunkerSpec table with register_chunker() and a lightrag.chunkers entry-point group, loaded in create_app() next to load_third_party_parsers() (lightrag_server.py:1269). Operators opt in by installing a package or calling register_chunker(...) in their own startup script — deliberately not an env var holding an import path, so configuration data never becomes a code-loading surface.
• Request-side input (filename hint, LIGHTRAG_PARSER rule, API body) can only name an already-registered key, never a module path — the same closed-set rule _validate_filename_hint_for_resolution already applies to parser engines.
• Addressing: chunking selectors are single chars (F/R/V/P), which cannot host arbitrary third-party names. The existing chunk-parameter block syntax already solves this without new grammar — report.[native-C(name=breadcrumb)].md, i.e. C selects "registered chunker" and name is the registry key (one new param in param_schema.py:142).
• Custom chunkers would follow the new file-chunker contract (receiving doc_id / file_path / blocks_path), which is a strict superset of the legacy 6-arg signature — that is what actually unblocks the breadcrumb use case.
• Hard requirement: selecting an unregistered chunker must fail loudly (422 at enqueue, or the document FAILED with a clear error_msg). Silently falling back to F is exactly the failure mode this issue is about.
• Open question on overriding: register_parser allows replacing built-in engines by name. For chunkers, silently replacing F would change chunking for every default document, so overriding built-ins should probably be refused or at least warned about at startup.

Your interim proposal (delegate from the F branch when chunking_func is not chunking_by_token_size) we would rather avoid: it gives the same F selector two different meanings depending on instance state, conflicts with the contract the tests pin, and still only covers F.

4. Question back to you

Where should the breadcrumb land? Rewriting chunk["content"] means it is persisted, embedded, returned as citation text at query time, and changes chunk IDs (utils_pipeline.py:164, :415). If the goal is purely extraction-time grounding, an additional prompt block alongside the existing ---Section Context--- (operate.py:3929) would be cleaner and cost nothing at retrieval time. Which behaviour do you actually need — decorated stored text, or extraction-only context? That answer changes the contract, so it is worth settling before any implementation.

Independently of that, the two cheap items look worth doing regardless: fix the stale R/V docstring, and emit a warning when a non-default chunking_func is configured but a selector bypasses it.

---

Generated by Claude Code

[geraldtscharre]

Thanks for the thorough verification and the repro — and for the two corrections, both of which check out. The no-identity one is the crux, and it's worth expanding on, because it's really the heart of the problem, not a wiring detail.

Why the breadcrumb matters

A chunk is not just a span of text — it inherits meaning from the document it came from, and that meaning is not evenly distributed across the document body. Three things live in the document's envelope, not in its prose:

• The filename frequently encodes a document type, a sequence/ID number, or a meeting reference. None of that is in the body.
• The file path encodes provenance — custody, category, which collection the document belongs to. Also not in the body.
• The creation/document date stamps the temporal validity of every fact in the document. A balance figure, a membership roster, a contract term — each is only true as of that date.

The default fixed-token chunker lands all of that context on chunk 1 only (or on no chunk at all, if the metadata sits in a front-matter block the chunker strips). Chunks 2..N of a large document are then silently stripped of provenance and of their as-of date. The cost isn't a crash — it's a quiet degradation: entities and relations get extracted from chunks that no longer know when they were true or where they came from, and that ambiguity gets baked into the graph. For a corpus where the same entity appears across years of documents, losing the date is exactly the class of bug that produces "still current?" answers and misattributed facts. That's why this isn't just an extraction nicety — it's provenance and temporal grounding, and it has to reach every chunk, not just the first.

Confirming the two corrections

• Escape hatch via a non-chunking option char: confirmed. For raw-text documents i is inert (no VLM cost), so it would technically run a custom chunker — but it's the wrong semantic: using an image-analysis selector to mean "run my chunker" is exactly the kind of implicit-overload that causes the silent-bypass class of bug in the first place, so we didn't adopt it.
• Legacy 6-arg signature carries no document identity: confirmed, and this is the deeper issue. Our chunker currently recovers source/path/date by parsing a YAML front-matter block that the converter bakes into the document text — i.e. we sidestep the missing identity by having the upstream converter embed it in the content the chunker receives. That works, but it's fragile and it silently shifts the contract onto the converter: the chunker only "knows" the file because the converter pre-stamped it into the bytes. The new file-chunker contract (doc_id / file_path / blocks_path) removes that dependency, which is the real unblock for this use case — and the reason we'd migrate to it rather than keep the front-matter workaround.

Landing question: decorated text vs extraction-only context

Our view: both have a legitimate place, and the choice should be a property of the registered chunker, not a contract-level either/or.

• Extraction-only context (a block alongside the existing ---Section Context---) is the cleaner default — no chunk-ID changes, no embedding pollution, no citation-text change. For use cases where the breadcrumb is purely extraction-time grounding, this is the right answer and costs nothing at retrieval.
• Decorated stored text is legitimately needed when the source/provenance/date must also be visible at retrieval and answer time, not only at extraction. The intrinsic meaning of path/filename/date is needed by the answer model and by the user when a chunk is cited — "as of date", "from source" — and if that context lives only in an extraction-time prompt block, it's gone by query time and the cited chunk loses its provenance again. For deployments that surface citations or let the answer model reason about freshness, decorating the stored content is the point, not a side effect.

One nuance worth stating: date filtering can also be driven off file_path independently of chunk content, so extraction-only context can suffice for retrieval when the path is queryable — but whether the answer model needs to see the date/source in the cited text is a per-deployment requirement, which is why it shouldn't be foreclosed by the contract. Suggest the registry let a chunker declare whether it mutates content, and treat chunk-ID/embedding/citation implications as consequences of that flag rather than as global behavior.

Registry direction

+1. Closed-set naming (request-side names an already-registered key, never a module path — so configuration data never becomes a code-loading surface), loud failure on an unregistered chunker (no silent F fallback — that's the root cause this issue exists), and the new-contract plumbing are exactly right. On overriding built-ins: silently replacing F would change chunking for every default document, so warning loudly at startup (at minimum) is the right call.

Two cheap items

+1 to both. The warning — emit when a non-default chunking_func is configured but a selector bypasses it — would have caught this immediately in our case; that alone would have prevented a chunk of the confusion. Happy to test any of this against a real multi-thousand-document run if useful.

[danielaskdd]

Thanks — that's a useful reply, and the framing of "envelope vs prose" is the right one. Below: the one thing we'd push back on, one correction to my own earlier comment, and a fact that I think changes where the remaining gap actually is.

1. On "let the chunker declare whether it mutates content"

Agreed that both behaviours are legitimate and neither should be foreclosed. But we would not add the flag, on enforcement grounds rather than taste: nothing can verify the declaration, and nothing would consume it. A chunker returns a list of chunk dicts, and chunk["content"] is the stored text — chunk keys, embeddings, citation text and token budgets are all computed from it unconditionally. A boolean that the pipeline cannot check and does not branch on is documentation wearing a contract's clothes, and it would invite the assumption that setting mutates_content=False protects you from something.

So the contract stays: what you return is what is stored, embedded and cited. Decorating content needs no permission — it is what returning that content means. The registry docs state the consequences instead. Nothing is foreclosed by that.

2. Correction to my own earlier comment: chunk IDs

I said decorating content "changes chunk IDs". That is too broad. On the normal chunker path the key is positional — f"{doc_id}-chunk-{order:03d}" from chunk_order_index (utils_pipeline.py:161-162); the content hash at :164 only fires when the chunk carries neither a chunk_id nor an integer order, and make_custom_chunk_id (:405) belongs to the ainsert_custom_chunks path, not to chunking. So a chunker that emits chunk_order_index keeps stable IDs while decorating content.

The costs that do stand: the decoration is embedded (an identical prefix on every chunk of a document pulls those vectors together and raises the similarity floor in naive mode), it is paid again in tokens on every retrieval of that chunk, it reaches the reranker, and it cannot be turned off without re-ingesting.

3. The non-mutating channel already exists — for any chunker, today

This is the part I don't think came across in my earlier comment. build_chunks_dict_from_chunking_result keeps every field a chunker returns, verbatim:

stored_chunk = {k: v for k, v in dp.items() if k != "_source_span"}   # utils_pipeline.py:184

and the pipeline upserts that dict into text_chunks (pipeline.py:5325), where it lands in full; chunks_vdb gets the same dict but stores only its whitelisted meta_fields (lightrag.py:1466), which is why the query stage backfills headings from text_chunks rather than from the vector hit. There is one call site (pipeline.py:5293), after the strategy branch — so this holds for F/R/V/P and the legacy chunking_func path alike.

Concretely: a chunker that returns heading (nested {"level","heading","parent_headings"} or the legacy flat triple — chunk_schema.normalize_chunk_heading accepts both) gets, with no content mutation and no ID churn:

• extraction time, on every chunk: the ---Section Context--- block (operate.py:3929, via format_heading_context);
• query time: content_headings on the chunk JSON when ENABLE_CONTENT_HEADINGS is on (operate.py:5472-5474 → _attach_content_headings).

That already covers your third bullet (section/table disambiguation on chunks 2..N) and can carry a source breadcrumb as well, without touching stored text.

4. Provenance at answer time is already served; the date is not

The file_path rides in the chunk JSON handed to the answer model (operate.py:5433/5447/5461) and in the reference list (:5589). So "if the breadcrumb lives only in an extraction-time block, the cited chunk loses its provenance again" is not accurate on main — the source is there at answer time regardless of chunker.

What is absent everywhere is the document date:

• create_time on a chunk record is the ingestion timestamp (json_kv_impl.py:358-359, LIGHTRAG_DOC_CHUNKS in PG), not the document's as-of date;
• there is no caller-supplied per-document metadata channel at all — ainsert(...) takes ids / file_paths / track_id and nothing else, the upload/text request models carry no metadata field, and doc_status.metadata is internal pipeline bookkeeping (the API even strips internal keys before returning it).

So today the as-of date can only be recovered by the chunker parsing it out of the filename, the path, or the body — which is exactly the converter front-matter arrangement you described, and the fragility you correctly called out. A chunker registry does not fix that; it only relocates the parsing. Even with doc_id / file_path / blocks_path plumbed in, the date still has to be guessed from a string, because nobody ever told LightRAG what it is.

That suggests the date belongs in a separate change: a user-supplied per-document metadata channel (set at ainsert / upload / text, persisted with full_docs, rendered into both the extraction prompt block and the query-time chunk JSON), so the value reaches every chunk without any custom chunker at all.

Question back: would a user-supplied document_date (plus arbitrary key/values), rendered at both extraction and query time, fully replace your front-matter workaround? If yes, that is the more valuable half of this issue for most users, and it is independent of the registry — worth its own issue so the two don't block each other.

5. Scope

• Now: the stale R/V docstring, and the warning when a non-default chunking_func is configured but a selector bypasses it.
• Next: the chunker registry as agreed — closed-set naming, loud failure on an unregistered key, new file-chunker contract, and refuse-or-warn on overriding a built-in.
• Separate issue: the per-document metadata / date channel above.

The offer to test against a real multi-thousand-document run is very welcome — most useful against the bypass warning and the registry PR, and (if we get there) against the metadata channel, where the interesting question is whether rendering the date at both stages actually changes extraction quality on a multi-year corpus.

---

Generated by Claude Code

[geraldtscharre]

Hi — thanks for engaging with this so carefully. Let me try to state the underlying problem as plainly as I can, because I think it's simpler than the chunker framing makes it sound, and the proposed solution follows directly from it.

The real problem

When LightRAG ingests a document, the only "identity" it keeps for each chunk is a file_path — and in practice that file_path is just the filename of the converted text file, sitting in a single flat input directory. The original document usually lived somewhere much more meaningful: inside a nested folder structure that says which collection, category, or custody chain it belongs to, and under its original filename. All of that is lost when the source is converted and staged flat. The flattened name does retain some of the hierarchy (a year, a date, a document type) because the converter folded it into the filename — so it's not nothing — but it's a lossy collapse, not the real path, and not the real source filename.

Worse, the document's as-of date — the date the document is actually from, which is the temporal stamp on every fact inside it — has nowhere to live at all. LightRAG records a create_time, but that's the moment I ingested the document (say, 2026), not the moment the document is about (say, a meeting in October 2018). Those are completely different things, and only the second one lets you tell apart "true in 2018" from "true in 2023."

So today, to get the true source path and the document date in front of both the extractor and the answering model, I have to do something fairly awkward: have my converter bake them into the document text as a header block, and then have my chunker parse that header back out and stamp it onto every chunk. It works — but it's a fragile round-trip. It shifts the contract onto my converter, it breaks on edge cases, it costs tokens on every retrieval, and it's just the kind of workaround that shouldn't be necessary in the first place. The information is known at ingest time; the system just has no way to be told it.

Why it matters — a concrete example

Take a corpus of board documents spanning many years, and a question like "how has the size of the board changed — today, five years ago, ten years ago?" The same entity ("the board") appears in documents from 2014, 2019, and 2024, and each document states a membership count that is only true as of that document's date.

For the answering model to build that timeline, it has to see, on each chunk it retrieves, the date that chunk's document is from — not just have that date filterable somewhere, but actually see it in the evidence it's citing. If the date lives only at extraction time, it's gone by the time the model composes the answer, and the model can't reliably assemble "2014: 14 members → 2019: 12 → 2024: 11." If the only path the model sees is a flattened filename with no real folder structure, it also can't tell the user where a fact came from in any meaningful way.

What I'd propose

Let the operator tell LightRAG, at ingest time, a small set of per-document metadata — above all the document's as-of date, and optionally the original source path and filename, plus any other key/values a deployment cares about. LightRAG would persist that metadata with the document and carry it to every chunk, rendering it in two places:

• into the extraction prompt, so the extractor stamps each fact with its as-of date and its real source; and
• into the context the answering model sees at query time, so the model can reason about freshness, build timelines, and cite "as of October 2018, from the minutes collection" rather than stating a stale fact as current.

This needs no custom chunker, and it replaces the bake-then-parse round-trip entirely. The date is the single highest-value piece; the original path/filename and arbitrary key/values are a natural extension that costs little once the channel exists.

I think this could be the more valuable half of the breadcrumb discussion for most users, and it's independent of the chunker registry — so it needn't block on that work. Would you like me to open this as its own issue so it can move on its own track, or would you rather keep it here? Either way, I'm happy to test it against a multi-year corpus where the same entities recur — that's exactly the setting where the date reaching both stages should show a measurable difference.

[geraldtscharre]

Thanks — this is a really useful reply, and the reframing of where the gap actually is landed. Below: the concession, two things I verified against my own tree, a correction to something I overstated, and an answer to your question. (I've also posted a plain-language restatement of the underlying problem and the proposal as a separate comment — same point, written for a human rather than point-by-point.)

On "let the chunker declare whether it mutates content"

You're right, and I withdraw the suggestion. Nothing can verify the declaration and nothing would consume it — chunk["content"] is the stored text, and embeddings, citation text, and token budgets all flow from it unconditionally. A boolean the pipeline can't check and doesn't branch on is documentation wearing a contract's clothes, and it would invite the assumption that mutates_content=False protects you from something it doesn't. The contract "what you return is what is stored, embedded, and cited — consequences documented" is the cleaner position. Decorating content needs no permission; it's what returning that content means.

On the chunk-ID correction — confirmed on my side too

I checked my own implementation against this. My chunker only mutates content and tokens and leaves chunk_order_index untouched, so my chunk keys are the positional {doc_id}-chunk-{order:03d} form — stable across the decoration, no content-hash fallback, no ID churn. So the costs that do stand are exactly the ones you list, and I own them: the decoration is embedded (an identical prefix on every chunk of a document pulls those vectors together and raises the similarity floor in naive mode), it's paid again in tokens on every retrieval, it reaches the reranker, and it can't be turned off without re-ingesting. I accepted those knowingly — they're the price of getting provenance and the as-of date in front of the answer model on a deployment with no metadata channel — but they're real, and your ledger is accurate.

On the non-mutating channel and file_path at answer time — verified

Two of your claims I went and checked in my code, because if they held they changed my picture:

• The heading channel. Confirmed: format_heading_context feeds the ---Section Context--- block at extraction, and _attach_content_headings backfills content_headings onto the query-time chunk JSON when enable_content_headings is on — and that flag defaults to True in my deployment. So a chunker that returns a heading field gets section grounding on chunks 2..N with no content mutation and no ID churn. I'm not currently feeding that field, which is on me. This covers my "section/table disambiguation on chunks 2..N" bullet more cleanly than content decoration would.
• file_path at answer time. Confirmed. The source rides in the chunk JSON handed to the answer model and in the reference list regardless of chunker. (One thing I'd note here, which the plain-language comment expands on: in practice that file_path is just the filename of the converted text file in a flat input directory — the original nested source path is already lost by the time LightRAG sees it. So provenance reaches the model, but in a lossy, flattened form.)

Which means I owe you a correction: in my earlier comment I said that if the breadcrumb lives only in an extraction-time block, "the cited chunk loses its provenance again" at query time. That is not accurate for the source — file_path already carries provenance to the answer model on main, independent of any chunker. The piece that is genuinely absent everywhere is the one you then put your finger on: the document date. create_time is the ingestion timestamp, there is no caller-supplied per-document metadata channel, and doc_status.metadata is internal bookkeeping. So today the as-of date can only be recovered by the chunker parsing it out of the filename, the path, or the body — which is precisely the converter-bakes-it-into-the-body-then-the-chunker-parses-it-back-out arrangement I described, and the fragility I flagged. You're right that a chunker registry does not fix that; it only relocates the parsing. Even with doc_id / file_path / blocks_path plumbed in, the date still has to be guessed from a string, because nobody ever told LightRAG what it is.

Answer to your question

Would a user-supplied document_date (plus arbitrary key/values), rendered at both extraction and query time, fully replace your front-matter workaround?

Yes — for the date, which is the part I'm actually contorting for. The source is already served by file_path; the date is the only piece with nowhere to live. A per-document metadata channel that takes a document_date (and arbitrary key/values) at ainsert / upload / text, persists it with full_docs, and renders it into both the extraction prompt block and the query-time chunk JSON would replace the front-matter round-trip entirely, and would reach every chunk with no custom chunker at all.

One nuance I'd want to be sure is covered, and which your proposal already seems to: the date has to be visible to the answer model when it's composing, not only filterable at retrieval. The failure mode I care about is the answer model stating a stale fact as present, or a user asking "is this still current?" — and that requires the model to see the as-of date in the cited chunk's context, not just for the retriever to be able to filter on it. Rendering the date into the query-time chunk JSON (as you describe) covers that point.

On spinning this out

The metadata / document_date channel is the more valuable half of this for most users and it's independent of the registry, so it shouldn't block on it — but rather than open it unilaterally, I've asked in the plain-language comment whether you'd prefer it as its own issue on a separate track. Happy to open and link it back here if you'd like.

Scope and the test offer

+1 to the three-tier scope (now: docstring + bypass warning; next: the registry; separate: the metadata channel). The bypass warning alone would have caught my case immediately — that's the highest-leverage cheap item.

The test offer stands and is most useful against, in order: the bypass warning, the registry PR, and then the metadata channel — where the genuinely interesting question is whether rendering the document date at both extraction and query time actually changes extraction quality on a multi-year corpus where the same entities recur. That's exactly the corpus I have to point at, so I'm happy to run it and report numbers.

[danielaskdd]

Thanks again for the detailed investigation and the follow-up discussion.

To keep the next design and implementation work in one place, I opened RFC #3664: Add C strategy for Server-injected custom chunking callbacks.

The RFC proposes an explicit C selector for the existing chunking_func mechanism, trusted Server-side registry injection, a context-aware callback contract based on durable sidecar_location, legacy six-argument compatibility, and an explicit warned C -> F fallback when no custom callback was injected.

Please continue design discussion on #3664. Contributors interested in implementing the RFC are very welcome to comment there with the area they would like to take on.

[danielaskdd]

Partial fix merged — #3665 landed on main (merge commit f791fa4), preceded by #3662 which removed the "silent" half of this report.

What is now available. Instead of identity-gating the F branch (the interim you offered), main has an explicit chunking selector C that invokes LightRAG.chunking_func with its unchanged six-argument legacy contract. Your my_chunking_func from the reproduction should work as written — no signature change, no new hook to implement. Async callbacks are awaited; a raising callback fails the document with a located RuntimeError rather than falling back silently.

C is reachable from every ingest path, including the two that used to bypass the callback:

• /documents/scan (your supervisor polling inputs/<workspace>/): set a LIGHTRAG_PARSER rule, e.g. LIGHTRAG_PARSER=*:legacy-C or per-suffix md:legacy-C;*:legacy-R. The process_options or PROCESS_OPTION_CHUNK_FIXED line you cited is still there, but it only fills in an empty options string — a rule- or hint-supplied C is preserved.
• /documents/upload: a filename hint, doc.[-C].md.
• /documents/text and /documents/texts: chunking.strategy="custom".

Parameters come from the fixed_token snapshot (those four fields are exactly what populate the legacy signature), and per-file overrides work: *:legacy-C(chunk_ts=800,chunk_ol=80) or doc.[-C(chunk_ts=800)].md.

Guards, so a persisted C document can never become a silent no-op:

• Synchronous request boundaries with a caller to correct — upload and the text APIs — return 422 when chunking_func is still the default callback.
• Scan and reprocessing have no such caller, so they accept a persisted C, warn once per attempt, and use an exact fixed-token fallback.
• fix: warn when selectors bypass custom chunking #3662 covers the inverse case you reported: an explicit F/R/V/P that bypasses a non-default chunking_func now logs a warning instead of silently discarding the customization.

Could you test this against your deployment? That is the main reason I am writing. Two things would be especially useful to hear:

1. Whether your breadcrumb callback runs and produces the chunk text you expect on the /documents/scan path with LIGHTRAG_PARSER=*:legacy-C.
2. Whether the observability is enough to diagnose it without reading logs. The document's doc_status metadata carries chunk_method, which distinguishes custom_chunking_func (your callback ran) from custom_chunking_fallback_fixed_token (callback absent, fell back), alongside a chunk_opts string with the parameters actually used. The matching log lines are Chunking C(custom): … and Chunking C(fallback F): ….

One deliberate consequence worth knowing before you test: sidecar source-span backfill is disabled for C custom output, because a callback that rewrites chunk text (exactly what a breadcrumb does) cannot be located back in blocks.jsonl. C's fixed-token fallback keeps backfill, since it emits exact source spans. If per-chunk source spans matter for your provenance work, please say so — that trade-off is worth revisiting separately.

Keeping this issue open, because the general request in it is not delivered: C restores the legacy splitting extension point on the HTTP paths, but it does not give you a strategy-agnostic chunk_transform_func that composes with R/V/P — you still cannot combine paragraph-semantic chunking with a breadcrumb transform. Your argument for keeping "how to split" and "what to decorate" orthogonal still stands on its own; feedback from testing C will help decide the shape of that hook.

[geraldtscharre]

Thanks — #3665 + #3662 are exactly the right first slice. The bypass warning (#3662) is the cheap item that would have caught my case immediately; C as an explicit selector rather than overloading F is the cleaner call.

On the test ask: I'm mid-ingestion on my local F-branch workaround right now (the very bypass this issue documents), so I haven't exercised upstream C against my corpus yet — I'll report numbers once this run closes rather than claim results I don't have. When I do, I'll confirm the two things you flagged: that the breadcrumb callback runs and produces the expected chunk text on /documents/scan with LIGHTRAG_PARSER=*:legacy-C, and that the doc_status chunk_method distinction (custom_chunking_func vs custom_chunking_fallback_fixed_token) is enough to diagnose without reading logs. I'll also check the one deliberate consequence you called out — sidecar source-span backfill disabled for C custom output — to see whether it costs me anything in practice; I don't think I consume source spans today, but I'll confirm rather than assume.

On the date question you raised: I've filed #3706 for the per-document as-of-date / metadata channel, per your read that it's the more valuable half for most users and shouldn't block on the registry. #3603's remaining open item — the strategy-agnostic chunk_transform_func that composes with R/V/P, which C doesn't deliver — is the part of the original ask I still need; I've put my design input on that contract over on #3664 so it doesn't fragment this thread. Keeping #3603 open for that transform hook reads correct to me: it's the undelivered half, and the date channel shrinks (but doesn't eliminate) it.

[danielaskdd]

Scope update, so the remaining threads are easy to follow.

The document date is settled and is being done separately. The discussion here and on #3706 converged on: the as-of date is the one piece of envelope metadata with nowhere to live; it belongs at the document level, not in chunk text; and it must reach both the extraction prompt and the query-time chunk context. That is now split across two tracks:

• feat: add document-level date context #3805 (under review) — the document_date field on ainsert and the REST text / upload endpoints, full_docs persistence, the ---Document Context--- extraction block, and query-time attachment to chunk context.
• Document date for scan / upload: filename-hint document-level parameter block (date=...) and LIGHTRAG_DOC_DATE_PATTERN #3830 (new) — where the date comes from when the server processes files: a document-level parameter block in the filename hint (minutes.[(date=2018-10-01)].pdf) and an operator-declared LIGHTRAG_DOC_DATE_PATTERN for corpora whose filenames already carry a date. Covers /documents/scan and hint-carrying uploads, which feat: add document-level date context #3805 does not.

Please take date-related design comments to those two issues rather than here.

The rest of the breadcrumb (source path, section, arbitrary key/values) is deliberately not being implemented yet. Provenance at answer time is already served by file_path in the reference list, and section context by the heading channel, as established above. What is still unclear is how other per-document or per-chunk metadata should be stored (document level vs. chunk level, which backends, which fields the vector store keeps), which of it the extraction and answer models should see, and how it interacts with the strategy-agnostic transform hook that is the remaining open item of this issue. Until that is thought through, no implementation is planned for it — the risk of baking in a chunk-level channel that the later design has to undo is higher than the cost of waiting. This issue stays open for that design work and for feedback from testing the C selector.

---

Generated by Claude Code