Reducing ZIM file size: analysis, tiers, and two RFC PRs

[soloturn]

Summary

Investigated whether the ZIM format can be made substantially smaller. Short answer: not by half, not from format changes alone — but there's real, verified headroom in metadata structures that are currently stored completely uncompressed. This issue is the tracking/RFC home for that investigation.

Pull requests from this investigation

• feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120 — Creator::configCompactIndexStructures(): compressed path pointer list + compressible title listing. Opt-in, off by default.
• feat(writer): compress the dirent table itself, chunked like clusters #1123 (builds on feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120's branch; diff shows both commits combined until feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120 merges) — dirent table itself, chunked and compressed the same way clusters are. The larger of the two pieces.

Both are draft/RFC, not asking for a merge yet — opened to get real, measured numbers in front of maintainers rather than just a written proposal.

Does this require a new ZIM format version? What does that mean in practice?

Yes. Both PRs above bump Fileheader::minorVersion (new zimMinorVersionCompactIndex = 4, current released format is minor version 3) when their opt-in flag is enabled. Concretely:

• A file written without the new flag is byte-for-byte what libzim already produces today — no change, no version bump, reads on every existing libzim release.
• A file written with the flag set is a real, incompatible format extension: the path pointer list and dirent table are no longer flat arrays of raw offsets, they're compressed blobs with a different internal layout. Any libzim release that predates this change cannot open such a file — not "reads it but slower," it will misinterpret the bytes or throw, depending on where in the read path it fails.
• Because it's strictly opt-in, nothing forces this on anyone: existing ZIM files, existing readers, and existing writers that don't call the new config method are entirely unaffected. This is deliberate — it lets the format extension exist and be tested in the wild before anyone proposes making it a new default.
• Making it a default later is a separate, much bigger decision: every consumer (libkiwix, kiwix-tools, kiwix-android, kiwix-desktop, kiwix-apple, any third-party reader) would need a libzim release supporting minor version 4 before ZIM files built with the new default could be opened by users on older app versions. Given ~100TB of already-published ZIM content and an ecosystem of independent readers, that's an ecosystem coordination question for this issue to surface, not something either PR presumes to answer.

Where the bytes go

Measured on wikipedia_en_climate_change_mini_2024-06.zim (8.56MB, 20,568 entries):

region	bytes	share
compressed clusters (html/css/js)	2,737,195	30.7%
Xapian indexes (raw)	2,498,560	28.0%
media, webp/png (raw, correctly so)	2,290,887	25.7%
dirent table (raw)	1,217,297	13.6%
path pointer list (raw)	164,536	1.8%
title listing (raw)	15,284	0.2%
header + cluster ptrs + checksum	2,208	0.0%

Only 30% of a ZIM file is compressed article content, and that part is already well-tuned (0.093 ratio on English Wikipedia HTML — close to what generic compression can do). 43% is metadata and index structures stored with zero compression. That's where the recoverable space is.

Tiers (cumulative, same test file)

tier	cumulative size	reduction	status
baseline	8,926,398	—	—
cluster size 2MB→32MB (config-only)	8,403,391	-5.9%	not done — see caveat below
+ compress dirent table	7,453,702 (est.)	-16.5% (est.)	done & verified, see below
+ compress path ptr list & title listing	7,290,680 (est.)	-18.3% (est.)	done & verified, see below
+ compress Xapian index	6,048,282 (est.)	-32.2% (est.)	blocked upstream, see below
+ dedup paths in Xapian title index	5,947,790 (est.)	-33.4% (est.)	not attempted

Implemented and independently re-verified

• feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120 (Creator::configCompactIndexStructures(), opt-in, new minor version): delta+varint+zstd-compressed path pointer list, plus the title listing routed through the existing compressible-cluster mechanism instead of a hardcoded false. Real measured effect on the same test file: 8,926,398 → 8,765,662 bytes (-1.80%) — smaller than the table above's estimate for that slice, because the shipped title-listing implementation deliberately reuses generic cluster zstd (no delta/varint pre-transform) for lower risk, rather than the more aggressive bespoke encoding the original estimate assumed. The path-pointer-list piece matches the estimate closely.
• feat(writer): compress the dirent table itself, chunked like clusters #1123 (dirent table itself, chunked like clusters, same flag): dirent table alone measured 1,217,297 → 267,520 bytes (-78.02%), matching the estimate above almost exactly. Combined with feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120: 8,926,398 → 7,816,094 bytes (-12.44%) total, real and independently reproduced from a clean build. Note this doesn't include the 32MB-cluster-size step, so it's not directly comparable to the table's "-16.5%" row (which stacks on top of the cluster-size change) — it's the dirent+pointer-list contribution in isolation, on the default 2MiB cluster size.
• Both are opt-in and require a minor version bump; old readers cannot parse a file written with either enabled.

Not yet done

• Cluster size 32MB: config-only (Creator::configClusterSize()), no format change, reader-transparent. Real cost, measured separately: cluster decompression time scales with cluster size (0.59ms at 2MiB → 6.79ms at 32MiB on an M-series Mac, 5-8x worse on a low-end phone), and CLUSTER_CACHE_SIZE (default 16MiB) needs raising in step or you trade file size for repeated decompression. 8MiB clusters + a bigger cache looked like the defensible point in initial testing; 32MiB did not.
• Xapian index compression: the big remaining opportunity (28% of the file), and genuinely blocked — not by ZIM, by Xapian. libzim's getDbFromAccessInfo() (src/tools.cpp) hands Xapian a raw fd; Xapian's glass backend does its own direct pread/mmap I/O on that fd with no pluggable block-I/O layer a transparent decompression shim could sit underneath. This is the same root issue as Allow extern getFd() to get file descriptors from Java for Kiwix Android #852 and Unable to host books with FileDescriptor on Server kiwix/libkiwix#1015 (the "external getFd()" discussion) — solving it is a multi-quarter Xapian-side project or a search-index-format replacement, not something fixable from libzim alone.
• Xapian title-index path dedup (~12.5% of that structure): src/writer/xapianIndexer.cpp:143-149 already has a TODO for this. Real, small, self-contained, not attempted here.

Ruled out (tested, not worth it)

• Content dedup: 0.02% of content bytes are exact duplicates. Nothing to gain.
• Trained zstd dictionaries: looked like -15.6% on a naive test, but that was overfit (dictionary trained on the same data being compressed). On a proper held-out split: only -2.1% at 2MiB clusters, and worse than no dictionary at 8MiB clusters. Real value of a dictionary here is different: it buys "big-cluster compression ratio at small-cluster latency," a latency/memory tradeoff, not a size win.
• xz/lzma instead of zstd: -2.8% ratio at equal cluster size, for a large decompression-speed cost. The 2019 zstd migration was the right call.
• Similarity-based content reordering: sorting/clustering articles by similarity before grouping into clusters bought at most -4.6% (2MiB clusters) to nothing (-0.5% with a minhash-shingle approach at 8MiB). HTML boilerplate is already fully exploited within a cluster; reordering doesn't add much.
• bzip3 (tested separately, not part of the original analysis): best ratio of anything tried (-17.6% vs zstd-19 on real content), but ~10x slower to decompress. Disqualifying for content read repeatedly on a phone.
• Brotli (tested separately, not part of the original analysis): consistently 8-8.5% smaller than zstd-19 across every content type tested (HTML, dirent text, even pure-binary delta-varint data), for ~5-6% slower decompression at ZIM's real ~2MiB cluster granularity (compression itself is 1.4-3.7x slower depending on quality level, irrelevant for an offline write). This is a real, verified, broadly-applicable opportunity not captured in the tiers above since it's a codec swap, not a structural change — worth its own proposal, potentially bigger than the metadata-compression tiers combined once applied to the 30%-of-file content clusters.

Bottom line

"Half" was never plausible from format changes. What's real and now partially shipped:

• ~1.8% (feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120, open, not yet merged): path pointer list + title listing compression.
• ~12.4% (feat(writer): opt-in compact index structures (smaller path pointer list + compressible title listing) #1120 + dirent-table follow-up): adds dirent-table chunking, the actual big piece. Verified twice over on real Wikipedia content.
• ~28%+ potential remains locked in the Xapian index, blocked on upstream Xapian I/O flexibility.
• ~8%+ additional, independent of all the above: switching cluster compression from zstd to brotli. Untested at scale, no dependency currently, real integration cost, but the numbers are good enough to be worth a proposal.

Caveats

• All numbers above are from one 8.56MB "mini" test file. A full-size Wikipedia ZIM (media- or index-dominated differently) needs its own measurement before any of these percentages are treated as commitments at scale — dirent-table share in particular shrinks proportionally on a huge archive (more articles doesn't mean proportionally more dirent bytes per article once you're past the metadata overhead per entry).
• Every structural format change here needs a minor/major version bump and is an ecosystem-wide coordination problem: libkiwix, kiwix-tools, kiwix-android, kiwix-desktop, kiwix-apple, and any third-party reader all need to understand the new bits before it's safe to make any of this the default.

cc @mgautierfr @kelson42 @veloman-yunkan

[kelson42]

@soloturn How all these compressed things should be used.... if they are compressed? Do you imply to decompress them all each time before to do anything with the ZIM?