You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ca22c3f
Browse filesBrowse the repository at this point in the historyBrowse files
HTML API: Analyze WARC files picked at random from the latest crawl.
The analysis always read the first WARC file listed for the crawl, because
cc-analyzer took the listing in order and each run starts with an empty
workspace. Every run analyzed the same 21,642 documents until CommonCrawl
published its next crawl.
The fetch now picks its WARC files at random from the crawl's whole list,
around 100,000 files fetched over a couple of weeks. The seed which picked
them is shown in the report with the files read, and the new `seed` and
`crawl` inputs analyze the same documents again.
cc-analyzer.phar is rebuilt from cc-analyzer commit 1bca572, which adds
`--warc-order random` and `--seed`, and finds the latest crawl from the data
server's listing when index.commoncrawl.org cannot be reached, instead of
falling back to a months-old crawl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: tools/html-api-commoncrawl/README.md
+6-4Lines changed: 6 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# HTML API CommonCrawl analysis
2
2
3
-
Compares how the HTML API parses real-world documents with and without a pull request. It fetches a random sample of HTML documents from the latest [CommonCrawl](https://commoncrawl.org/) crawl, then replays the same documents through both checkouts to measure:
3
+
Compares how the HTML API parses real-world documents with and without a pull request. It fetches every HTML document in a WARC file picked at random from the latest [CommonCrawl](https://commoncrawl.org/) crawl, then replays the same documents through both checkouts to measure:
4
4
5
5
- how long the Tag Processor and the HTML Processor take to parse each document, and how long `wp_kses( $html, 'post' )` takes on it, for all documents and for small, medium, and large ones;
6
6
- which documents the HTML Processor does not support, and why it stopped on them, with a few example documents for each reason. Each example shows the token the HTML Processor stopped on, the token with up to 30 characters (extended grapheme clusters) of the document on either side, its byte offsets in the document, and the breadcrumbs of open elements at that point.
@@ -13,7 +13,7 @@ The report is left as a comment on the pull request. Each new analysis replaces
13
13
14
14
## Files
15
15
16
-
-`cc-analyzer.phar`: [cc-analyzer](#rebuilding-cc-analyzerphar), which fetches documents from CommonCrawl into a local batch, and replays a batch through an analysis script. It streams whole WARC files and keeps their HTML responses. That takes only a handful of requests: CommonCrawl blocks runners which send many, such as one per record. It honors CommonCrawl's rate limiting, and if a download still fails, it lists the failure and the analysis runs on the documents it has.
16
+
-`cc-analyzer.phar`: [cc-analyzer](#rebuilding-cc-analyzerphar), which fetches documents from CommonCrawl into a local batch, and replays a batch through an analysis script. It streams whole WARC files and keeps their HTML responses. That takes only a handful of requests: CommonCrawl blocks runners which send many, such as one per record. The files are picked at random from the crawl's whole list of around 100,000, fetched over a couple of weeks, so each analysis sees a different part of the web. One file already holds documents from nearly 20,000 hosts. It honors CommonCrawl's rate limiting, and if a download still fails, it lists the failure and the analysis runs on the documents it has.
17
17
-`bootstrap.php`: loads the HTML API, and what `wp_kses()` needs, from the `src/` directory in `HTML_API_BENCH_WP_SRC`, without loading WordPress. It presents a UTF-8 site at example.com. When a change adds a file to the HTML API, add it to the list of files here.
18
18
-`benchmark.php`: an analysis script which times the Tag Processor, the HTML Processor, and `wp_kses( $html, 'post' )` on each document and writes `benchmark.csv`.
19
19
-`analysis.php`: an analysis script which records whether the HTML Processor supports each document, and where it stopped on those it does not, and writes `analysis.csv`.
@@ -27,7 +27,7 @@ This needs PHP 8.5, as used by the workflow, with the `curl`, `pdo_sqlite`, and
27
27
# Fetch 1,000 documents, or as many as can be fetched in 5 minutes. The workflow uses --limit all instead, to keep every
28
28
# HTML document in the WARC files it downloads.
29
29
php -d memory_limit=1G tools/html-api-commoncrawl/cc-analyzer.phar --workspace /tmp/cc batch fetch corpus \
30
-
--resource-types html --limit 1000 --limit-warcs 1 --max-jobs 1 --http-retries 6 --duration 5m --progress text
30
+
--resource-types html --warc-order random --limit 1000 --limit-warcs 1 --max-jobs 1 --http-retries 6 --duration 5m --progress text
31
31
32
32
# Time each document three times with each checkout, and record support once.
33
33
forcheckoutin base merge;do
@@ -56,14 +56,16 @@ The workflow reads these environment variables. When it is run manually, each on
56
56
| Variable | Default | Meaning |
57
57
| --- | --- | --- |
58
58
|`HTML_API_BENCH_WARCS`|`1`| How many WARC files to download. Every HTML document in them is analyzed; each file holds around 20,000. |
59
+
|`HTML_API_BENCH_SEED`| random | Picks the WARC files. The report gives the seed it used; give it again, with the same crawl, to analyze the same documents. |
60
+
|`HTML_API_BENCH_CRAWL`|`latest`| The CommonCrawl crawl to read, such as `CC-MAIN-2026-43`. |
59
61
|`HTML_API_BENCH_FETCH_DURATION`|`30m`| Longest time to spend fetching. Fetching stops at whichever limit is reached first. |
60
62
|`HTML_API_BENCH_ROUNDS`|`2`| How many times to time each document with each checkout. Each document's time is its median. |
61
63
|`HTML_API_BENCH_SMALL_MAX_BYTES`|`32768`| Documents smaller than this are small. |
62
64
|`HTML_API_BENCH_LARGE_MIN_BYTES`|`262144`| Documents at least this large are large. Everything between is medium. |
63
65
64
66
## Rebuilding `cc-analyzer.phar`
65
67
66
-
`cc-analyzer.phar` was built from cc-analyzer commit `57bfbca` (branch `index-sampled-fetch`) with `tools/build-phar.php`. Before building, the default HTTP identity was removed so that requests carry only a generic `User-Agent` and no `From` header:
68
+
`cc-analyzer.phar` was built from cc-analyzer commit `1bca572` (branch `index-sampled-fetch`) with `tools/build-phar.php`. Before building, the default HTTP identity was removed so that requests carry only a generic `User-Agent` and no `From` header:
67
69
68
70
-`src/Http/CurlStreamClient.php`: `DEFAULT_USER_AGENT` set to `'cc-analyzer/0.1 PHP-cURL'` and `DEFAULT_CONTACT_HEADER` set to `''`.
69
71
-`src/Cli/Application.php`: the `--user-agent` and `--contact-email` defaults set to the same values, and the help text to match.
? sprintf( "%s picked at random with seed `%s`. Run the workflow manually with this seed and crawl `%s` to analyze the same documents again.\n\n", 1 === count( $warcs ) ? 'It was' : 'They were', $seed, $crawl[1] )
605
+
: ''
606
+
);
588
607
589
608
$report .= "### HTML Processor parse time\n\n";
590
609
$report .= "All documents, including those where parsing stopped early on unsupported markup:\n\n";
0 commit comments