Skip to content

Commit 87b3e95

Browse files
README: lead with the speedup — 8,500x in the hero, at-a-glance table up top
1 parent 7993a71 commit 87b3e95

1 file changed

Lines changed: 30 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 30 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,41 @@
22

33
# difflib-fast
44

5-
**The exact `difflib` similarity ratio — at suffix-automaton speed.**
5+
**The exact `difflib` similarity ratio — up to `8,500×` faster.**
66

77
[![Rust 2021](https://img.shields.io/badge/rust-2021-orange.svg)](https://www.rust-lang.org/)
88
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
99
[![exact difflib](https://img.shields.io/badge/difflib-byte--for--byte-blue.svg)](https://docs.python.org/3/library/difflib.html)
10+
[![vs difflib: up to 8,500×](https://img.shields.io/badge/vs%20difflib-up%20to%208%2C500%C3%97-brightgreen.svg)](#python-package)
11+
12+
The **same number** Python's `difflib` gives you — byte-for-byte, no `autojunk` approximation. A corpus
13+
stdlib `difflib` would chew on for **~30 minutes** clusters in **~0.2 seconds**.
1014

1115
</div>
1216

13-
---
17+
```python
18+
import difflib_fast
19+
20+
difflib_fast.ratio("the quick brown fox", "the quick brown dog") # 0.8947368421052632 (== difflib)
21+
difflib_fast.ratio(pairs) # list[float] — computed across every core inside Rust, GIL released
22+
```
23+
24+
<div align="center">
25+
26+
| from Python · real corpus · 12 cores | throughput | vs stdlib `difflib` |
27+
|---|---|---|
28+
| `ratio(a, b)` — one call | 2.4k pairs/s | **104×** |
29+
| `ratio(pairs)` — batch, all cores | 15k pairs/s | **628×** |
30+
| `cluster_canonicals(corpus)` — the real workload | 199k pairs/s | **8,541×** |
31+
32+
*23 pairs/s → 199,000 pairs/s on the same task. Same answer. ([how ↓](#how-it-works))*
33+
34+
</div>
1435

15-
`difflib.SequenceMatcher.ratio()` tells you how similar two strings are — the way a human diff sees
16-
it. It is the right metric, and it is **slow**. `difflib-fast` computes the *exact same number*,
17-
byte-for-byte, with a suffix automaton — so it stays linear exactly where `difflib` falls apart.
36+
And a pure-Rust crate, with **zero** Python dependency by default:
1837

1938
```rust
2039
use difflib_fast::ratio;
21-
2240
// bit-for-bit identical to difflib.SequenceMatcher(None, a, b, autojunk=False).ratio()
2341
assert_eq!(ratio("the quick brown fox", "the quick brown dog"), 0.8947368421052632);
2442
```
@@ -180,22 +198,14 @@ pairs = [(a, b) for a in corpus for b in corpus]
180198
difflib_fast.ratio(pairs) # list[float], one per pair — fanned out over all cores
181199
```
182200

183-
This matters because Python *can't* parallelize the stdlib version: `difflib` in a thread pool stays
184-
GIL-bound and doesn't speed up at all. The batch form sidesteps that entirely — the parallelism lives
185-
in Rust, not in Python threads. Measured on real canonicalized Python (mypy, M3 Pro, 12 threads,
186-
`benchmarks/bench_python.py`):
187-
188-
| from Python | throughput | vs stdlib `difflib` |
189-
|---|---|---|
190-
| `ratio(a, b)` — one call | 2.4k pairs/s | **104×** |
191-
| `ratio(pairs)` — batch, all cores | 15k pairs/s | **628×** |
192-
| `cluster_canonicals(corpus)` — the real workload | 199k eff. pairs/s | **8,541×** |
193-
| `difflib` in a `ThreadPoolExecutor` | 23 pairs/s | 1.0× — *threads don't help* |
201+
This matters because Python *can't* parallelize the stdlib version: `difflib` in a `ThreadPoolExecutor`
202+
stays GIL-bound — **23 → 23 pairs/s, zero speedup**. The batch form sidesteps that entirely: the
203+
parallelism lives in Rust, not in Python threads, so it just scales (numbers up top; full harness in
204+
[`benchmarks/bench_python.py`](benchmarks/bench_python.py), measured on real canonicalized Python,
205+
M3 Pro, 12 threads).
194206

195207
Clustering wins biggest because each string's automaton is built **once** and reused across the whole
196-
`n²` join (and dissimilar pairs early-exit) — so it's not 12× the per-call speed, it's a different
197-
algorithm. The number you'd actually feel: a corpus whose all-pairs comparison would take stdlib
198-
`difflib` **~30 minutes** clusters in **~0.2 s**.
208+
`n²` join (dissimilar pairs early-exit) — it's not 12× the per-call speed, it's a different algorithm.
199209

200210
The package is **typed** (`py.typed` + `.pyi` stubs — pyright/mypy see the overloads), gated behind the
201211
`python` cargo feature so the pure-Rust crate keeps **zero** Python dependency by default. Built with

0 commit comments

Comments
 (0)