Skip to content

Commit 2e56042

Browse files
committed
Merge mem/content-spooling into mem/combined
Combine the three memory-reduction branches: - mem/pipeline-objects : stream raw objects one at a time (getObjectsStream) - mem/stream-sections : chunked getSectionsText() - mem/content-spooling : opt-in spooling of decoded stream content to disk Peak memory was measured by parsing each sample PDF and running getText() in a fresh PHP process (memory_get_peak_usage(true), memory_limit=-1), in three configs: master, the merged branch with spooling off, and with spooling on (Config::setContentSpooling(true)). Extracted text is byte-identical across all three configs. sample (size) master off spool d-spool bugs/PullRequest457.pdf (16 MiB) 150.9 94.9 57.4 -62% DocumentWithLotsOfObjects.pdf (6.1 MiB) 101.9 65.9 39.9 -61% bugs/Issue356.pdf (5.9 MiB) 95.9 77.9 63.6 -34% bugs/Issue267_array_access.pdf (2.0 MiB) 14.0 8.0 6.0 -57% bugs/Issue391.pdf (0.9 MiB) 16.0 14.0 12.0 -25% ImproperFontFallback.pdf (0.7 MiB) 24.0 10.0 10.0 -58% bugs/Issue585.pdf (0.5 MiB) 22.0 10.0 10.0 -55% (figures in MiB; d-spool = spool-on vs master) The in-memory streaming changes alone (off) already cut peak memory by roughly 35-60% on large documents, by never holding the full raw-object array or all decoded streams at once. Spooling reclaims most of the remaining decoded-stream footprint, roughly halving it again on the largest files. On documents whose peak is dominated by something other than retained decoded streams (e.g. ImproperFontFallback, Issue585) the streaming pass captures the win and spooling adds nothing further. Documents below ~0.4 MiB stay within the allocator's granularity (2-4 MiB) and show no measurable change. bugs/Issue104a.pdf (108 KiB) peaks at ~203 MiB in all three configs; its cost is in glyph/text handling, not stream decoding, and is unaffected by these changes.
2 parents 5cb2201 + bfa950c commit 2e56042

10 files changed

Lines changed: 361 additions & 4 deletions

File tree

doc/CustomConfig.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ The `Config` class has the following options:
2626
| `setPdfWhitespaces` | String | `\0\t\n\f\r ` | |
2727
| `setPdfWhitespacesRegex` | String | `[\0\t\n\f\r ]` | |
2828
| `setRetainImageContent` | Boolean | `true` | If parsing fails due to memory exhaustion, you can set the value to `false`. This will reduce memory usage, although it will no longer retain image content. |
29+
| `setContentSpooling` | Boolean | `false` | If parsing large documents fails due to memory exhaustion, set this to `true` to spool decoded stream content to a temporary file instead of keeping it in memory. Lowers peak memory usage at the cost of some extra disk I/O. |
2930

3031

3132
## option setDecodeMemoryLimit + setRetainImageContent (manage memory usage)
@@ -41,6 +42,26 @@ $config->setDecodeMemoryLimit(1000000);
4142
$parser = new \Smalot\PdfParser\Parser([], $config);
4243
```
4344

45+
## option setContentSpooling (manage memory usage)
46+
47+
When parsing large documents, the bulk of the memory a parsed document keeps
48+
alive is the decoded content of its stream objects. Enabling content spooling
49+
writes that content to a single temporary file as objects are parsed and reads
50+
it back on demand, so the full set of decoded streams no longer has to reside in
51+
memory at once. This noticeably lowers peak memory usage for large files in
52+
exchange for a small amount of disk I/O. The extracted text and document details
53+
are identical with the option on or off.
54+
55+
```php
56+
$config = new \Smalot\PdfParser\Config();
57+
// Spool decoded stream content to a temporary file instead of memory
58+
$config->setContentSpooling(true);
59+
$parser = new \Smalot\PdfParser\Parser([], $config);
60+
```
61+
62+
The temporary file is created in the system temp directory and removed
63+
automatically once the parsed document is no longer referenced.
64+
4465
## option setHorizontalOffset
4566

4667
When words are broken up or when the structure of a table is not preserved, you can use `setHorizontalOffset`.

src/Smalot/PdfParser/Config.php

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,15 @@ class Config
8989
*/
9090
private $ignoreEncryption = false;
9191

92+
/**
93+
* Whether decoded object stream content is spooled to a temporary file
94+
* instead of being kept in memory. Trades disk I/O for a lower peak memory
95+
* footprint when parsing large documents.
96+
*
97+
* @var bool
98+
*/
99+
private $contentSpooling = false;
100+
92101
public function getFontSpaceLimit()
93102
{
94103
return $this->fontSpaceLimit;
@@ -172,4 +181,14 @@ public function setIgnoreEncryption(bool $ignoreEncryption): void
172181
{
173182
$this->ignoreEncryption = $ignoreEncryption;
174183
}
184+
185+
public function getContentSpooling(): bool
186+
{
187+
return $this->contentSpooling;
188+
}
189+
190+
public function setContentSpooling(bool $contentSpooling): void
191+
{
192+
$this->contentSpooling = $contentSpooling;
193+
}
175194
}
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
<?php
2+
3+
/**
4+
* @file
5+
* This file is part of the PdfParser library.
6+
*
7+
* @author Andreas Gohr <gohr@cosmocode.de>
8+
*
9+
* @date 2026-06-18
10+
*
11+
* @license LGPLv3
12+
*
13+
* @url <https://github.com/smalot/pdfparser>
14+
*
15+
* PdfParser is a pdf library written in PHP, extraction oriented.
16+
* Copyright (C) 2017 - Sébastien MALOT <sebastien@malot.fr>
17+
*
18+
* This program is free software: you can redistribute it and/or modify
19+
* it under the terms of the GNU Lesser General Public License as published by
20+
* the Free Software Foundation, either version 3 of the License, or
21+
* (at your option) any later version.
22+
*
23+
* This program is distributed in the hope that it will be useful,
24+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
25+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
26+
* GNU Lesser General Public License for more details.
27+
*
28+
* You should have received a copy of the GNU Lesser General Public License
29+
* along with this program.
30+
* If not, see <http://www.pdfparser.org/sites/default/LICENSE.txt>.
31+
*/
32+
33+
namespace Smalot\PdfParser;
34+
35+
/**
36+
* Temporary on-disk store for decoded stream content.
37+
*
38+
* When content spooling is enabled (see Config::setContentSpooling()), the
39+
* decoded content of each parsed object is appended to a single temporary file
40+
* instead of being held in memory, and read back on demand. This trades a small
41+
* amount of disk I/O for a markedly lower peak memory footprint, since the full
42+
* set of decoded streams - usually the largest thing a parsed Document keeps
43+
* alive - no longer has to reside in RAM all at once.
44+
*
45+
* The backing temporary file is created lazily on first use and removed
46+
* automatically once the spool (and the Document owning it) is destroyed.
47+
*
48+
* @internal
49+
*/
50+
class ContentSpool
51+
{
52+
/**
53+
* @var resource|null
54+
*/
55+
private $handle;
56+
57+
/**
58+
* Current size of the spool file, i.e. the offset at which the next chunk
59+
* of content will be written.
60+
*
61+
* @var int
62+
*/
63+
private $size = 0;
64+
65+
/**
66+
* Append a chunk of content to the spool.
67+
*
68+
* @return array{0: int, 1: int}|null [offset, length] locating the stored
69+
* content, or null if it could not be
70+
* stored (the caller should then keep
71+
* the content in memory)
72+
*/
73+
public function store(string $content): ?array
74+
{
75+
$length = \strlen($content);
76+
if (0 === $length) {
77+
return null;
78+
}
79+
80+
$handle = $this->handle();
81+
if (null === $handle) {
82+
return null;
83+
}
84+
85+
$offset = $this->size;
86+
if (0 === fseek($handle, $offset) && fwrite($handle, $content) === $length) {
87+
$this->size += $length;
88+
89+
return [$offset, $length];
90+
}
91+
92+
return null;
93+
}
94+
95+
/**
96+
* Read back content previously stored via store().
97+
*
98+
* @param int $offset offset returned by store()
99+
* @param int $length length returned by store()
100+
*/
101+
public function fetch(int $offset, int $length): string
102+
{
103+
if (!\is_resource($this->handle) || $length <= 0) {
104+
return '';
105+
}
106+
107+
// Seeking before reading also flushes any pending write buffer, which
108+
// is required when reading data that was just written to the handle.
109+
if (0 !== fseek($this->handle, $offset)) {
110+
return '';
111+
}
112+
113+
$content = '';
114+
$remaining = $length;
115+
while ($remaining > 0 && !feof($this->handle)) {
116+
$chunk = fread($this->handle, $remaining);
117+
if (false === $chunk || '' === $chunk) {
118+
break;
119+
}
120+
$content .= $chunk;
121+
$remaining -= \strlen($chunk);
122+
}
123+
124+
return $content;
125+
}
126+
127+
/**
128+
* Lazily open the backing temporary file.
129+
*
130+
* @return resource|null
131+
*/
132+
private function handle()
133+
{
134+
if (null === $this->handle) {
135+
// tmpfile() opens a binary-safe read-write handle and removes the
136+
// underlying file automatically when the handle is closed.
137+
$handle = tmpfile();
138+
$this->handle = false !== $handle ? $handle : null;
139+
}
140+
141+
return $this->handle;
142+
}
143+
144+
/**
145+
* Ensure the backing file is closed and removed when the spool is destroyed.
146+
*/
147+
public function __destruct()
148+
{
149+
if (\is_resource($this->handle)) {
150+
fclose($this->handle);
151+
}
152+
}
153+
}

src/Smalot/PdfParser/Document.php

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,11 +74,29 @@ class Document
7474
*/
7575
protected $details;
7676

77+
/**
78+
* Optional on-disk store for decoded object stream content, shared by all
79+
* objects of this document. Only set when content spooling is enabled.
80+
*
81+
* @var ContentSpool|null
82+
*/
83+
protected $contentSpool;
84+
7785
public function __construct()
7886
{
7987
$this->trailer = new Header([], $this);
8088
}
8189

90+
public function getContentSpool(): ?ContentSpool
91+
{
92+
return $this->contentSpool;
93+
}
94+
95+
public function setContentSpool(?ContentSpool $contentSpool): void
96+
{
97+
$this->contentSpool = $contentSpool;
98+
}
99+
82100
public function init()
83101
{
84102
$this->buildDictionary();

src/Smalot/PdfParser/PDFObject.php

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,19 @@ class PDFObject
6565
protected $header;
6666

6767
/**
68-
* @var string
68+
* @var string|null
6969
*/
7070
protected $content;
7171

72+
/**
73+
* Location of this object's content within the document's ContentSpool,
74+
* as [offset, length], when the content has been spooled to disk instead
75+
* of being kept in $content. Null when the content lives in memory.
76+
*
77+
* @var array{0: int, 1: int}|null
78+
*/
79+
protected $contentRef;
80+
7281
/**
7382
* @var Config|null
7483
*/
@@ -130,9 +139,45 @@ public function getDetails(bool $deep = true): array
130139

131140
public function getContent(): ?string
132141
{
142+
// Content has been spooled to disk; read it back on demand.
143+
if (null !== $this->contentRef) {
144+
$spool = $this->document->getContentSpool();
145+
146+
return null !== $spool
147+
? $spool->fetch($this->contentRef[0], $this->contentRef[1])
148+
: null;
149+
}
150+
133151
return $this->content;
134152
}
135153

154+
/**
155+
* Move this object's in-memory content to the document's ContentSpool, if
156+
* one is configured, freeing the in-memory copy. The content is transparently
157+
* read back from disk by getContent() when needed.
158+
*
159+
* @internal
160+
*/
161+
public function spoolContent(): void
162+
{
163+
if (null !== $this->contentRef
164+
|| null === $this->content
165+
|| '' === $this->content) {
166+
return;
167+
}
168+
169+
$spool = $this->document->getContentSpool();
170+
if (null === $spool) {
171+
return;
172+
}
173+
174+
$ref = $spool->store($this->content);
175+
if (null !== $ref) {
176+
$this->contentRef = $ref;
177+
$this->content = null;
178+
}
179+
}
180+
136181
/**
137182
* Creates a duplicate of the document stream with
138183
* strings and other items replaced by $char. Formerly
@@ -706,7 +751,7 @@ public function getTextArray(?Page $page = null): array
706751
$marked_stack = [];
707752
$last_written_position = false;
708753

709-
$sections = $this->getSectionsText($this->content);
754+
$sections = $this->getSectionsText($this->getContent());
710755
$current_font = $this->getDefaultFont($page);
711756
$current_font_size = 1;
712757
$current_text_leading = 0;

src/Smalot/PdfParser/Parser.php

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,13 @@ public function parseContent(string $content): Document
114114
$document = new Document();
115115
$this->objects = [];
116116

117+
// When content spooling is enabled, give the document an on-disk store
118+
// so each object's decoded content can be moved out of memory as soon
119+
// as the object is built (see parseObject()).
120+
if ($this->config->getContentSpooling()) {
121+
$document->setContentSpool(new ContentSpool());
122+
}
123+
117124
// Stream the raw objects one at a time instead of building the whole
118125
// raw object graph up front. Each structure is turned into a PDFObject
119126
// and then goes out of scope before the next is parsed, so the largest
@@ -246,7 +253,12 @@ protected function parseObject(string $id, array $structure, ?Document $document
246253
}
247254

248255
if (!isset($this->objects[$id])) {
249-
$this->objects[$id] = PDFObject::factory($document, $header, $content, $this->config);
256+
$object = PDFObject::factory($document, $header, $content, $this->config);
257+
// Free the just-decoded content from memory by moving it to the
258+
// document's on-disk spool (no-op unless content spooling is on).
259+
// The local $content copy is released when this method returns.
260+
$object->spoolContent();
261+
$this->objects[$id] = $object;
250262
}
251263
}
252264

src/Smalot/PdfParser/XObject/Form.php

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ class Form extends Page
4444
public function getText(?Page $page = null): string
4545
{
4646
$header = new Header([], $this->document);
47-
$contents = new PDFObject($this->document, $header, $this->content, $this->config);
47+
$contents = new PDFObject($this->document, $header, $this->getContent(), $this->config);
4848

4949
return $contents->getText($this);
5050
}

0 commit comments

Comments
 (0)