You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 56d2aaa
Browse filesBrowse the repository at this point in the historyBrowse files
fix: raise on an empty HTTP-200 split chunk in NDJSON mode (0.46.1) (#348)
## The bug
In NDJSON elements-file mode, a split-PDF chunk that returned HTTP 200
with an empty body was logged and skipped, so the combined
`elements_file` was silently short by that chunk's pages while the call
still returned 200.
Nothing downstream could detect it: `combine_chunk_files_to_ndjson`
hands back only the combined path, and `split_pdf_allow_failed=False`
does not cover the case because an empty 200 counts as a *successful*
chunk. The buffered path fails outright on the same response
(`res.json()` raises `JSONDecodeError`), so enabling NDJSON mode
converted a hard failure into silent truncation.
## The fix
Recombination now raises `EmptyChunkResponseError` (a `ValueError`,
matching where the buffered path's `JSONDecodeError` lands) instead of
skipping.
Emptiness is judged against the chunk's own `Content-Type`, because the
two formats disagree about what an empty body means:
- **JSON** has no empty document — a chunk with no elements is `[]` — so
an empty body is malformed and raises.
- **`application/x-ndjson`** encodes zero records as zero lines, so an
empty body is well formed. It contributes nothing and does not fail the
partition; otherwise a split whose pages are blank would break once a
server honors the accept header.
- An **unknown or missing** media type is read as JSON. That is what the
deployed API returns, and guessing NDJSON would reinstate the silent
truncation.
`combine_chunk_files_to_ndjson` takes an optional `media_types` list,
positionally matched to `chunk_paths`; omitting it keeps the strict
reading. `_elements_from_task_responses` collects each chunk's
`Content-Type` before the cached branch overwrites the body with a
temp-file path — the header survives both cache branches, so the cached
case carries a real media type too.
## Tests
`_test_unstructured_client/unit/test_ndjson_elements_file.py`:
- empty JSON chunk raises, over 3 empty-ish bodies × `application/json`
/ unset / with-charset
- empty NDJSON chunk is zero records, over 3 bodies × 3 media-type
spellings (parameters, casing)
- every chunk empty yields an empty output rather than an error
- `ValueError` parity with the buffered path, and the strict default
when `media_types` is omitted
- hook level, both cache modes: an empty JSON chunk fails the operation
and leaves no partial, spilled, or combined file behind; an empty NDJSON
chunk is accepted with the surrounding chunks' elements intact
Mutating `_is_ndjson_media_type` to `return False` fails 12 of these,
including both hook-level cache-mode tests, so the coverage is
load-bearing.
271 unit + 64 contract tests pass; pylint 10/10 on both changed modules.
## Release
Bumped to 0.46.1 with CHANGELOG and RELEASES entries. Consumers that pin
`unstructured-client >=0.46.0` for NDJSON elements-file mode should
raise the floor to `>=0.46.1`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CHANGELOG.md
+5Lines changed: 5 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,8 @@
1
+
## 0.46.1
2
+
3
+
### Fixes
4
+
* Fail loudly when a split-PDF chunk returns HTTP 200 with an empty JSON body in NDJSON elements-file mode. The chunk used to be logged and skipped, so the combined `elements_file` was silently short by those pages while the call still returned 200 — and `split_pdf_allow_failed=False` did not catch it, because an empty 200 counts as a successful chunk. Recombination now raises `EmptyChunkResponseError` (a `ValueError`), matching the buffered path, which raises `JSONDecodeError` on the same response. Emptiness is judged against the chunk's own `Content-Type`: JSON has no empty document (a chunk with no elements is `[]`), while `application/x-ndjson` encodes zero records as zero lines, so an empty NDJSON chunk is well formed and still contributes nothing.
0 commit comments