|
2 | 2 |
|
3 | 3 | # difflib-fast |
4 | 4 |
|
5 | | -**The exact `difflib` similarity ratio — at suffix-automaton speed.** |
| 5 | +**The exact `difflib` similarity ratio — up to `8,500×` faster.** |
6 | 6 |
|
7 | 7 | [](https://www.rust-lang.org/) |
8 | 8 | [](https://opensource.org/licenses/MIT) |
9 | 9 | [](https://docs.python.org/3/library/difflib.html) |
| 10 | +[](#python-package) |
| 11 | + |
| 12 | +The **same number** Python's `difflib` gives you — byte-for-byte, no `autojunk` approximation. A corpus |
| 13 | +stdlib `difflib` would chew on for **~30 minutes** clusters in **~0.2 seconds**. |
10 | 14 |
|
11 | 15 | </div> |
12 | 16 |
|
13 | | ---- |
| 17 | +```python |
| 18 | +import difflib_fast |
| 19 | + |
| 20 | +difflib_fast.ratio("the quick brown fox", "the quick brown dog") # 0.8947368421052632 (== difflib) |
| 21 | +difflib_fast.ratio(pairs) # list[float] — computed across every core inside Rust, GIL released |
| 22 | +``` |
| 23 | + |
| 24 | +<div align="center"> |
| 25 | + |
| 26 | +| from Python · real corpus · 12 cores | throughput | vs stdlib `difflib` | |
| 27 | +|---|---|---| |
| 28 | +| `ratio(a, b)` — one call | 2.4k pairs/s | **104×** | |
| 29 | +| `ratio(pairs)` — batch, all cores | 15k pairs/s | **628×** | |
| 30 | +| `cluster_canonicals(corpus)` — the real workload | 199k pairs/s | **8,541×** | |
| 31 | + |
| 32 | +*23 pairs/s → 199,000 pairs/s on the same task. Same answer. ([how ↓](#how-it-works))* |
| 33 | + |
| 34 | +</div> |
14 | 35 |
|
15 | | -`difflib.SequenceMatcher.ratio()` tells you how similar two strings are — the way a human diff sees |
16 | | -it. It is the right metric, and it is **slow**. `difflib-fast` computes the *exact same number*, |
17 | | -byte-for-byte, with a suffix automaton — so it stays linear exactly where `difflib` falls apart. |
| 36 | +And a pure-Rust crate, with **zero** Python dependency by default: |
18 | 37 |
|
19 | 38 | ```rust |
20 | 39 | use difflib_fast::ratio; |
21 | | - |
22 | 40 | // bit-for-bit identical to difflib.SequenceMatcher(None, a, b, autojunk=False).ratio() |
23 | 41 | assert_eq!(ratio("the quick brown fox", "the quick brown dog"), 0.8947368421052632); |
24 | 42 | ``` |
@@ -180,22 +198,14 @@ pairs = [(a, b) for a in corpus for b in corpus] |
180 | 198 | difflib_fast.ratio(pairs) # list[float], one per pair — fanned out over all cores |
181 | 199 | ``` |
182 | 200 |
|
183 | | -This matters because Python *can't* parallelize the stdlib version: `difflib` in a thread pool stays |
184 | | -GIL-bound and doesn't speed up at all. The batch form sidesteps that entirely — the parallelism lives |
185 | | -in Rust, not in Python threads. Measured on real canonicalized Python (mypy, M3 Pro, 12 threads, |
186 | | -`benchmarks/bench_python.py`): |
187 | | - |
188 | | -| from Python | throughput | vs stdlib `difflib` | |
189 | | -|---|---|---| |
190 | | -| `ratio(a, b)` — one call | 2.4k pairs/s | **104×** | |
191 | | -| `ratio(pairs)` — batch, all cores | 15k pairs/s | **628×** | |
192 | | -| `cluster_canonicals(corpus)` — the real workload | 199k eff. pairs/s | **8,541×** | |
193 | | -| `difflib` in a `ThreadPoolExecutor` | 23 pairs/s | 1.0× — *threads don't help* | |
| 201 | +This matters because Python *can't* parallelize the stdlib version: `difflib` in a `ThreadPoolExecutor` |
| 202 | +stays GIL-bound — **23 → 23 pairs/s, zero speedup**. The batch form sidesteps that entirely: the |
| 203 | +parallelism lives in Rust, not in Python threads, so it just scales (numbers up top; full harness in |
| 204 | +[`benchmarks/bench_python.py`](benchmarks/bench_python.py), measured on real canonicalized Python, |
| 205 | +M3 Pro, 12 threads). |
194 | 206 |
|
195 | 207 | Clustering wins biggest because each string's automaton is built **once** and reused across the whole |
196 | | -`n²` join (and dissimilar pairs early-exit) — so it's not 12× the per-call speed, it's a different |
197 | | -algorithm. The number you'd actually feel: a corpus whose all-pairs comparison would take stdlib |
198 | | -`difflib` **~30 minutes** clusters in **~0.2 s**. |
| 208 | +`n²` join (dissimilar pairs early-exit) — it's not 12× the per-call speed, it's a different algorithm. |
199 | 209 |
|
200 | 210 | The package is **typed** (`py.typed` + `.pyi` stubs — pyright/mypy see the overloads), gated behind the |
201 | 211 | `python` cargo feature so the pure-Rust crate keeps **zero** Python dependency by default. Built with |
|
0 commit comments