Skip to content

Commit 7b7dff1

Browse files
jamie-lemonjamie-lemon
committed
Docs: Adds section on compressing files
1 parent 00e563d commit 7b7dff1

3 files changed

Lines changed: 337 additions & 4 deletions

File tree

‎docs/compressing-files.rst‎

Lines changed: 316 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,316 @@
1+
.. include:: header.rst
2+
3+
.. _CompressingFiles:
4+
5+
==============================
6+
Compressing Files
7+
==============================
8+
9+
10+
PyMuPDF & Compression
11+
-------------------------
12+
13+
There are various ways to reduce the size of a PDF file and a variety of options for doing so with the :meth:`Document.save()` method parameters.
14+
15+
By using the `mupdf_explored <https://mupdf.readthedocs.io/en/1.28.2/cookbook/mupdf-explored.html>`_ PDF as a control file let's look at the effect of various save-time parameters on the file size of a PDF document.
16+
17+
The original file is 1.8 MB in size. Let's see what happens when we save it with various optional parameters defined.
18+
19+
20+
`use_objstms`
21+
~~~~~~~~~~~~~~~~~~~~~~~~~
22+
23+
This boolean option packs object definitions into compressible streams.
24+
25+
**Example**
26+
27+
.. code-block:: python
28+
29+
import pymupdf
30+
31+
doc = pymupdf.open("mupdf_explored.pdf")
32+
33+
doc.save(
34+
"output.pdf",
35+
use_objstms=True, # pack object definitions into compressible streams
36+
)
37+
doc.close()
38+
39+
40+
The result is a file size of 1.5 MB. However, we can achieve a better result by using the `deflate` parameter to compress the uncompressed streams.
41+
42+
43+
44+
`deflate`
45+
~~~~~~~~~~~~~~~~~~~~
46+
47+
By setting this option we can compress uncompressed streams.
48+
49+
Availabe options are:
50+
51+
.. list-table::
52+
:header-rows: 1
53+
54+
* - **Value**
55+
- **Meaning**
56+
* - `0`
57+
- No compression (this is the default)
58+
* - `1`
59+
- Use standard `Flate compression <https://en.wikipedia.org/wiki/Deflate>`_
60+
* - `2`
61+
- Use `Brotli compression <https://en.wikipedia.org/wiki/Brotli>`_ (slowest, smallest output, but experimental and unsupported in many tools and viewers) [1]_
62+
63+
64+
Used in combination with `use_objstms` to pack object definitions into compressible streams we can achieve even better results.
65+
66+
Flate
67+
"""""""""""""""
68+
69+
**Flate Example**
70+
71+
.. code-block:: python
72+
73+
import pymupdf
74+
75+
doc = pymupdf.open("mupdf_explored.pdf")
76+
77+
doc.save(
78+
"output.pdf",
79+
use_objstms=True, # pack object definitions into compressible streams
80+
deflate=1, # compress uncompressed streams with Flate compression
81+
)
82+
doc.close()
83+
84+
The result is a file size of 903 KB.
85+
86+
87+
.. note::
88+
89+
:meth:`Document.ez_save()` automatically applies these paramters for you, so the same result can be achieved with:
90+
91+
.. code-block:: python
92+
93+
doc.ez_save("output.pdf")
94+
95+
Brotli
96+
"""""""""""""""
97+
98+
**Brotli Example**
99+
100+
.. warning::
101+
102+
Brotli is experimental and unsupported in many tools and viewers. [1]_
103+
104+
105+
.. code-block:: python
106+
107+
import pymupdf
108+
109+
doc = pymupdf.open("mupdf_explored.pdf")
110+
111+
doc.save(
112+
"output.pdf",
113+
use_objstms=True, # pack object definitions into compressible streams
114+
deflate=2, # compress uncompressed streams with Brotli compression
115+
)
116+
doc.close()
117+
118+
The result is a file size of 863 KB.
119+
120+
Brotli is expected to **yield meaningful differences** on text- and vector-heavy documents, and close to
121+
**none** on scanned or photo-heavy ones.
122+
123+
`compression_effort`
124+
'''''''''''''''''''''
125+
126+
When using Brotli (`deflate=2`) by setting a further parameter `compression_effort` we can control how hard MuPDF works when compressing stream data. It trades CPU time for file size and *never changes what the document looks like*.
127+
128+
.. code-block:: python
129+
130+
import pymupdf
131+
132+
doc = pymupdf.open("mupdf_explored.pdf")
133+
134+
doc.save(
135+
"output.pdf",
136+
use_objstms=True, # pack object definitions into compressible streams
137+
deflate=2, # compress uncompressed streams with Brotli compression
138+
compression_effort=100, # ask Brotli to work hard at it
139+
)
140+
doc.close()
141+
142+
The result now is a file size of 834 KB.
143+
144+
145+
`compression_effort`: What it does and does not affect
146+
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
147+
148+
It is an `int`, not a `bool`!
149+
150+
.. list-table::
151+
:header-rows: 1
152+
153+
* - **Value**
154+
- **Meaning**
155+
* - `0`
156+
- Zero effort (this is the default)
157+
* - `1`
158+
- Minimum effort — fastest, largest output
159+
* - `100`
160+
- Maximum effort — slowest, smallest output
161+
162+
163+
This is the single most common mistake:
164+
165+
.. code-block:: python
166+
167+
# WRONG — True == 1 == minimum effort
168+
doc.save("out.pdf", use_objstms=True, deflate=2, compression_effort=True)
169+
170+
# CORRECT - maximum effort defined
171+
doc.save("out.pdf", use_objstms=True, deflate=2, compression_effort=100)
172+
173+
Because Python's `True` is `1`, passing a boolean silently selects an effort setting of `1`.
174+
No warning is raised, and the file simply comes out larger than
175+
you expected.
176+
177+
Artifex published effort guidance in `Brotli is Here! <https://artifex.com/blog/brotli-is-here>`_
178+
for the equivalent `-e` flag on `mutool clean`:
179+
180+
- **40** — roughly the speed of default Flate compression.
181+
- **75** — roughly the speed of maximum Flate compression.
182+
- **100** — squeezes out everything available, at a noticeably longer runtime.
183+
184+
That guidance is written in the context of Brotli compression, so treat the exact
185+
numbers as indicative.
186+
187+
188+
`compression_effort` tunes the compressor applied to stream objects as the file is
189+
written. It does not decide *whether* streams are compressed using Brotli — that is `deflate=2`
190+
— and it does not repack object definitions — that is `use_objstms=True`. Ensures that you have correctly set those two parameters before you start tuning `compression_effort`.
191+
192+
It has an effect on:
193+
194+
- content streams (text and vector graphics),
195+
- font programs and other uncompressed binary streams,
196+
- object streams produced by `use_objstms=True`.
197+
198+
It has little or no effect on:
199+
200+
- images already stored in a compressed format (JPEG, JPEG2000, JBIG2). Re-running
201+
a general-purpose compressor over them buys nothing. Use
202+
`Document.rewrite_images()` for those.
203+
- documents whose bulk is unreferenced objects. Use `garbage=3` or `garbage=4`.
204+
205+
206+
`garbage`
207+
~~~~~~~~~~~~~~~~~~~~
208+
209+
Documents with unreferenced objects can be reduced in size by setting the `garbage` parameter to `3` or `4`. This will de-duplicate and drop unreferenced objects.
210+
211+
.. list-table::
212+
:header-rows: 1
213+
214+
* - **Value**
215+
- **Meaning**
216+
* - `0`
217+
- none (default)
218+
* - `1`
219+
- remove unused (unreferenced) objects.
220+
* - `2`
221+
- in addition to 1, compact the :data:`xref` table.
222+
* - `3`
223+
- in addition to 2, merge duplicate objects.
224+
* - `4`
225+
- in addition to 3, check :data:`stream` objects for duplication. This may be slow because such data are typically large.
226+
227+
228+
How much is saved here depends on the document, and how much "garbage" it may contain. For example, if we set `garbage=4` we only save a few KB:
229+
230+
231+
.. code-block:: python
232+
233+
import pymupdf
234+
235+
doc = pymupdf.open("mupdf_explored.pdf")
236+
237+
doc.save(
238+
"output.pdf",
239+
use_objstms=True, # pack object definitions into compressible streams
240+
deflate=2, # compress uncompressed streams with Brotli compression
241+
compression_effort=100, # ask Brotli to work hard at it
242+
garbage=4 # drop unreferenced objects, compact xref table, merge duplicate objects, check streams for duplication
243+
)
244+
doc.close()
245+
246+
247+
Now we have a resulting PDF with a file size of 831 KB.
248+
249+
250+
Results Summary
251+
--------------------
252+
253+
.. list-table::
254+
:header-rows: 1
255+
256+
* - **Description**
257+
- **File size result**
258+
- **Comments**
259+
* - Original file
260+
- 1.8 MB
261+
-
262+
* - `use_objstms=True`
263+
- 1.5 MB
264+
-
265+
* - `use_objstms=True`, `deflate=1`
266+
- 903 KB
267+
-
268+
* - `use_objstms=True`, `deflate=2`
269+
- 863 KB
270+
- Warning: Brotli is experimental and unsupported in many tools and viewers [1]_
271+
* - `use_objstms=True`, `deflate=2`, `compression_effort=100`
272+
- 834 KB
273+
- Warning: Brotli is experimental and unsupported in many tools and viewers [1]_
274+
* - `use_objstms=True`, `deflate=2`, `compression_effort=100`, `garbage=4`
275+
- 831 KB
276+
- Warning: Brotli is experimental and unsupported in many tools and viewers [1]_
277+
278+
PyMuPDF & Compression Benchmarking
279+
----------------------------------------
280+
281+
Gains are corpus-dependent. Measure before paying the CPU cost in a batch job:
282+
283+
.. code-block:: python
284+
285+
import pathlib
286+
import time
287+
288+
import pymupdf
289+
290+
src = "input.pdf"
291+
for effort in (0, 40, 75, 100):
292+
doc = pymupdf.open(src)
293+
out = f"out_{effort}.pdf"
294+
start = time.perf_counter()
295+
doc.save(out, garbage=4, deflate=1, use_objstms=True,
296+
compression_effort=effort)
297+
elapsed = time.perf_counter() - start
298+
doc.close()
299+
size = pathlib.Path(out).stat().st_size
300+
print(f"effort={effort:>3} {size/1e6:6.2f} MB {elapsed:5.2f}s")
301+
302+
See also
303+
--------------------
304+
305+
- :meth:`Document.save()`
306+
- :meth:`Document.ez_save()`
307+
- :meth:`Document.rewrite_images()`
308+
- :meth:`Document.subset_fonts()`
309+
- `Brotli is Here! <https://artifex.com/blog/brotli-is-here>`_
310+
311+
312+
313+
.. [1] If you've used Brotli to compress your PDFs and found that they don't work in your favorite viewer, please try `MUPDF GL <https://mupdf.readthedocs.io/en/1.28.2/tools/mupdf-gl.html>`_. If you still find a problem viewing the file then please file an issue on our `Github tracker <https://github.com/pymupdf/PyMuPDF/issues>`_.
314+
315+
316+
.. include:: footer.rst

‎docs/document.rst‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1301,7 +1301,8 @@ For details on **embedded files** refer to Appendix 3.
13011301

13021302
:arg bool clean: Clean and sanitize content streams [#f1]_. Corresponds to "mutool clean -sc".
13031303

1304-
:arg bool deflate: Deflate (compress) uncompressed streams.
1304+
:arg int deflate: Deflate (compress) uncompressed streams. See: :ref:`Compressing Files <CompressingFiles>` for full details on this parameter.
1305+
13051306
:arg bool deflate_images: *(new in v1.18.3)* Deflate (compress) uncompressed image streams [#f4]_.
13061307
:arg bool deflate_fonts: *(new in v1.18.3)* Deflate (compress) uncompressed fontfile streams [#f4]_.
13071308

@@ -1332,19 +1333,26 @@ For details on **embedded files** refer to Appendix 3.
13321333

13331334
:arg int use_objstms: *(new in v1.24.0)* compression option that converts eligible PDF object definitions to information that is stored in some other object's :data:`stream` data. Depending on the `deflate` parameter value, the converted object definitions will be compressed -- which can lead to very significant file size reductions.
13341335

1335-
.. warning:: The method does not check, whether a file of that name already exists, will hence not ask for confirmation, and overwrite the file. It is your responsibility as a programmer to handle this.
1336-
1336+
See: :ref:`Compressing Files <CompressingFiles>` for full details on this parameter.
1337+
13371338
:arg int compression_effort:
13381339

13391340
* 0 for default
13401341
* 1 for minimum effort.
13411342
* 100 for maximum effort.
1343+
1344+
See: :ref:`Compressing Files <CompressingFiles>` for full details on this parameter.
13421345

13431346
:arg bool raise_on_repair: *(new in v1.27.0)* If true we raise an exception if the save caused a repair.
13441347
This is useful because repairs can cause changes to be lost.
13451348

13461349
Also see `Document.repair()`.
13471350

1351+
.. warning::
1352+
1353+
The method does not check, whether a file of that name already exists, will hence not ask for confirmation, and overwrite the file. It is your responsibility as a programmer to handle this.
1354+
1355+
13481356
.. note::
13491357

13501358
**File size reduction**
@@ -1354,12 +1362,15 @@ For details on **embedded files** refer to Appendix 3.
13541362

13551363
2. "Lossy" file size reduction in essence must give up something with respect to images, like (a) remove all images (b) replace images by their grayscale versions (c) reduce image resolutions. Find examples in the `PyMuPDF Utilities "replace-image" folder <https://github.com/pymupdf/PyMuPDF-Utilities/tree/master/examples/replace-image>`_.
13561364

1365+
See: :ref:`Compressing Files <CompressingFiles>` for more.
1366+
1367+
13571368

13581369
.. method:: ez_save(*args, **kwargs)
13591370

13601371
* New in v1.18.11
13611372

1362-
PDF only: The same as :meth:`Document.save` but with changed defaults `deflate=True, garbage=3, use_objstms=1`.
1373+
PDF only: The same as :meth:`Document.save` but with changed defaults `deflate=1, garbage=3, use_objstms=True`.
13631374

13641375
.. method:: saveIncr()
13651376

‎docs/recipes.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@
1818

1919
----
2020

21+
.. toctree::
22+
23+
compressing-files.rst
24+
25+
----
26+
2127
.. toctree::
2228

2329
recipes-ocr.rst

0 commit comments

Comments
 (0)