Repository navigation
feat: NDJSON elements-file mode for partition (0.46.0) #347
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
fce575a
7210084
c01a6d1
5fc5a14
3080cf6
1ec2e25
358e21a
ba3e7f4
140ced0
19ca812
735b840
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -427,6 +427,37 @@ req = operations.PartitionRequest( | |
| ) | ||
| ``` | ||
|
|
||
| ### Streaming elements to a file instead of memory | ||
|
|
||
| For very large documents, the parsed element list can dominate the client's memory: the split-PDF path holds a list per chunk, a flattened list, a serialized blob and the SDK's re-parse of that blob. Request `application/x-ndjson` to skip all of it. The chunk responses are concatenated on disk and you get back a path in `elements_file` instead of a list in `elements`, which keeps peak memory at roughly one chunk. | ||
|
|
||
| **You own the returned file and are responsible for deleting it.** | ||
|
|
||
| > [!NOTE] | ||
| > The memory saving applies to the split-PDF path, i.e. a PDF with `split_pdf_page=True` (the default). For unsplit inputs — a non-PDF file, or `split_pdf_page=False` — the response body is still read fully into memory before being written to disk, so peak memory can reach roughly twice the body size. You still get `elements_file` either way. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This sentence isn't true against the current API — see the comment on That makes the example below worse than a plain failure: If the contract gets made uniform in
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Softening this one too — see the reply on The narrow version still stands: if If
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The contract was made uniform in I also corrected the caveat itself, which was wrong in a way you did not flag: it said "non-PDF, or The double-TypeError is gone with the contract fix, and the example uses Worth noting the same defect then turned up in my own tests — a later pass caught bare |
||
|
|
||
| Example: | ||
| ```python | ||
| import json | ||
| from pathlib import Path | ||
|
|
||
| from unstructured_client.general import PartitionAcceptEnum | ||
|
|
||
| res = client.general.partition( | ||
| request=req, | ||
| accept_header_override=PartitionAcceptEnum.APPLICATION_X_NDJSON, | ||
| ) | ||
|
|
||
| try: | ||
| with open(res.elements_file, encoding="utf-8") as f: | ||
| for line in f: | ||
| element = json.loads(line) | ||
| ... | ||
| finally: | ||
| # missing_ok so a failure to open the file isn't masked by the cleanup. | ||
| Path(res.elements_file).unlink(missing_ok=True) | ||
| ``` | ||
|
|
||
| <!-- Start File uploads [file-upload] --> | ||
| ## File uploads | ||
|
|
||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.