Skip to content

Commit ca22c3f

Browse files
dmsnellclaude
andcommitted
HTML API: Analyze WARC files picked at random from the latest crawl.
The analysis always read the first WARC file listed for the crawl, because cc-analyzer took the listing in order and each run starts with an empty workspace. Every run analyzed the same 21,642 documents until CommonCrawl published its next crawl. The fetch now picks its WARC files at random from the crawl's whole list, around 100,000 files fetched over a couple of weeks. The seed which picked them is shown in the report with the files read, and the new `seed` and `crawl` inputs analyze the same documents again. cc-analyzer.phar is rebuilt from cc-analyzer commit 1bca572, which adds `--warc-order random` and `--seed`, and finds the latest crawl from the data server's listing when index.commoncrawl.org cannot be reached, instead of falling back to a months-old crawl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 2847156 commit ca22c3f

4 files changed

Lines changed: 48 additions & 6 deletions

File tree

‎.github/workflows/html-api-commoncrawl.yml‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,16 @@ on:
3232
required: false
3333
type: number
3434
default: 1
35+
seed:
36+
description: 'Picks the WARC files: leave empty for a new random choice, or give the seed from an earlier report to analyze the same documents again'
37+
required: false
38+
type: string
39+
default: ''
40+
crawl:
41+
description: 'CommonCrawl crawl to read, e.g. CC-MAIN-2026-43, or latest'
42+
required: false
43+
type: string
44+
default: 'latest'
3545
fetch-duration:
3646
description: 'Longest time to spend fetching documents, e.g. 90s, 30m, 1h'
3747
required: false
@@ -65,6 +75,8 @@ permissions: {}
6575
env:
6676
# The analysis keeps every HTML document in this many WARC files, unless the fetch duration passes first.
6777
HTML_API_BENCH_WARCS: ${{ inputs.warcs || 1 }}
78+
HTML_API_BENCH_SEED: ${{ inputs.seed || '' }}
79+
HTML_API_BENCH_CRAWL: ${{ inputs.crawl || 'latest' }}
6880
HTML_API_BENCH_FETCH_DURATION: ${{ inputs.fetch-duration || '30m' }}
6981
HTML_API_BENCH_ROUNDS: ${{ inputs.rounds || 2 }}
7082
HTML_API_BENCH_SMALL_MAX_BYTES: ${{ inputs.small-max-bytes || 32768 }}
@@ -76,7 +88,7 @@ jobs:
7688
# Performs the following steps:
7789
# - Checks out the pull request merged into its target branch, and the target branch beside it.
7890
# - Sets up PHP.
79-
# - Fetches a random sample of HTML documents from the latest CommonCrawl crawl.
91+
# - Fetches every HTML document in WARC files picked at random from the latest CommonCrawl crawl.
8092
# - Times the Tag Processor, the HTML Processor, and wp_kses() on each document with both checkouts, running the rounds
8193
# of both checkouts at the same time, one per vCPU, so that both share the same load.
8294
# - Records which documents each checkout supports, and why the others are unsupported, with both checkouts at once.
@@ -119,13 +131,21 @@ jobs:
119131
ini-values: memory_limit=1G
120132

121133
# Streams whole WARC files, a handful of requests in all: CommonCrawl blocks runners which send many requests, such as
122-
# one per record. Every HTML document in them is kept; the number of files only bounds the download. When CommonCrawl
134+
# one per record. Every HTML document in them is kept; the number of files only bounds the download. The files are
135+
# picked at random from the crawl's whole list, around 100,000 files fetched over a couple of weeks, so that each run
136+
# sees a different part of the web. The seed is reported, and running the workflow again with it reads the same files. When CommonCrawl
123137
# asks to slow down, with HTTP 429 or 503, no request is sent until its Retry-After or the backoff has passed. A download
124138
# which still fails is listed, and the analysis runs on the documents already fetched. The cache limit is raised so
125139
# that pruning never runs during the fetch.
126140
- name: Fetch documents from CommonCrawl
141+
id: fetch
127142
run: |
143+
seed="${HTML_API_BENCH_SEED:-$(( (RANDOM << 15) | RANDOM ))}"
144+
echo "seed=$seed" >> "$GITHUB_OUTPUT"
128145
php "$CC_ANALYZER" --workspace "$RUNNER_TEMP/cc-analyzer" batch fetch corpus \
146+
--crawl "$HTML_API_BENCH_CRAWL" \
147+
--warc-order random \
148+
--seed "$seed" \
129149
--resource-types html \
130150
--limit all \
131151
--limit-warcs "$HTML_API_BENCH_WARCS" \
@@ -187,6 +207,7 @@ jobs:
187207
HTML_API_BENCH_BASE_SHA: ${{ steps.checkouts.outputs.base-sha }}
188208
HTML_API_BENCH_MERGE_SHA: ${{ steps.checkouts.outputs.merge-sha }}
189209
HTML_API_BENCH_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
210+
HTML_API_BENCH_SEED: ${{ steps.fetch.outputs.seed }}
190211

191212
- name: Upload the report
192213
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

‎tools/html-api-commoncrawl/README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# HTML API CommonCrawl analysis
22

3-
Compares how the HTML API parses real-world documents with and without a pull request. It fetches a random sample of HTML documents from the latest [CommonCrawl](https://commoncrawl.org/) crawl, then replays the same documents through both checkouts to measure:
3+
Compares how the HTML API parses real-world documents with and without a pull request. It fetches every HTML document in a WARC file picked at random from the latest [CommonCrawl](https://commoncrawl.org/) crawl, then replays the same documents through both checkouts to measure:
44

55
- how long the Tag Processor and the HTML Processor take to parse each document, and how long `wp_kses( $html, 'post' )` takes on it, for all documents and for small, medium, and large ones;
66
- which documents the HTML Processor does not support, and why it stopped on them, with a few example documents for each reason. Each example shows the token the HTML Processor stopped on, the token with up to 30 characters (extended grapheme clusters) of the document on either side, its byte offsets in the document, and the breadcrumbs of open elements at that point.
@@ -13,7 +13,7 @@ The report is left as a comment on the pull request. Each new analysis replaces
1313

1414
## Files
1515

16-
- `cc-analyzer.phar`: [cc-analyzer](#rebuilding-cc-analyzerphar), which fetches documents from CommonCrawl into a local batch, and replays a batch through an analysis script. It streams whole WARC files and keeps their HTML responses. That takes only a handful of requests: CommonCrawl blocks runners which send many, such as one per record. It honors CommonCrawl's rate limiting, and if a download still fails, it lists the failure and the analysis runs on the documents it has.
16+
- `cc-analyzer.phar`: [cc-analyzer](#rebuilding-cc-analyzerphar), which fetches documents from CommonCrawl into a local batch, and replays a batch through an analysis script. It streams whole WARC files and keeps their HTML responses. That takes only a handful of requests: CommonCrawl blocks runners which send many, such as one per record. The files are picked at random from the crawl's whole list of around 100,000, fetched over a couple of weeks, so each analysis sees a different part of the web. One file already holds documents from nearly 20,000 hosts. It honors CommonCrawl's rate limiting, and if a download still fails, it lists the failure and the analysis runs on the documents it has.
1717
- `bootstrap.php`: loads the HTML API, and what `wp_kses()` needs, from the `src/` directory in `HTML_API_BENCH_WP_SRC`, without loading WordPress. It presents a UTF-8 site at example.com. When a change adds a file to the HTML API, add it to the list of files here.
1818
- `benchmark.php`: an analysis script which times the Tag Processor, the HTML Processor, and `wp_kses( $html, 'post' )` on each document and writes `benchmark.csv`.
1919
- `analysis.php`: an analysis script which records whether the HTML Processor supports each document, and where it stopped on those it does not, and writes `analysis.csv`.
@@ -27,7 +27,7 @@ This needs PHP 8.5, as used by the workflow, with the `curl`, `pdo_sqlite`, and
2727
# Fetch 1,000 documents, or as many as can be fetched in 5 minutes. The workflow uses --limit all instead, to keep every
2828
# HTML document in the WARC files it downloads.
2929
php -d memory_limit=1G tools/html-api-commoncrawl/cc-analyzer.phar --workspace /tmp/cc batch fetch corpus \
30-
--resource-types html --limit 1000 --limit-warcs 1 --max-jobs 1 --http-retries 6 --duration 5m --progress text
30+
--resource-types html --warc-order random --limit 1000 --limit-warcs 1 --max-jobs 1 --http-retries 6 --duration 5m --progress text
3131

3232
# Time each document three times with each checkout, and record support once.
3333
for checkout in base merge; do
@@ -56,14 +56,16 @@ The workflow reads these environment variables. When it is run manually, each on
5656
| Variable | Default | Meaning |
5757
| --- | --- | --- |
5858
| `HTML_API_BENCH_WARCS` | `1` | How many WARC files to download. Every HTML document in them is analyzed; each file holds around 20,000. |
59+
| `HTML_API_BENCH_SEED` | random | Picks the WARC files. The report gives the seed it used; give it again, with the same crawl, to analyze the same documents. |
60+
| `HTML_API_BENCH_CRAWL` | `latest` | The CommonCrawl crawl to read, such as `CC-MAIN-2026-43`. |
5961
| `HTML_API_BENCH_FETCH_DURATION` | `30m` | Longest time to spend fetching. Fetching stops at whichever limit is reached first. |
6062
| `HTML_API_BENCH_ROUNDS` | `2` | How many times to time each document with each checkout. Each document's time is its median. |
6163
| `HTML_API_BENCH_SMALL_MAX_BYTES` | `32768` | Documents smaller than this are small. |
6264
| `HTML_API_BENCH_LARGE_MIN_BYTES` | `262144` | Documents at least this large are large. Everything between is medium. |
6365

6466
## Rebuilding `cc-analyzer.phar`
6567

66-
`cc-analyzer.phar` was built from cc-analyzer commit `57bfbca` (branch `index-sampled-fetch`) with `tools/build-phar.php`. Before building, the default HTTP identity was removed so that requests carry only a generic `User-Agent` and no `From` header:
68+
`cc-analyzer.phar` was built from cc-analyzer commit `1bca572` (branch `index-sampled-fetch`) with `tools/build-phar.php`. Before building, the default HTTP identity was removed so that requests carry only a generic `User-Agent` and no `From` header:
6769

6870
- `src/Http/CurlStreamClient.php`: `DEFAULT_USER_AGENT` set to `'cc-analyzer/0.1 PHP-cURL'` and `DEFAULT_CONTACT_HEADER` set to `''`.
6971
- `src/Cli/Application.php`: the `--user-agent` and `--contact-email` defaults set to the same values, and the help text to match.
6.85 KB
Binary file not shown.

‎tools/html-api-commoncrawl/report.php‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
* HTML_API_BENCH_BASE_SHA Commit of the target branch, shown in the report.
1818
* HTML_API_BENCH_MERGE_SHA Commit of the PR merged into the target branch, shown in the report.
1919
* HTML_API_BENCH_RUN_URL Link to the workflow run, shown in the report.
20+
* HTML_API_BENCH_SEED Seed which picked the WARC files, shown in the report so that they can be read again.
2021
*
2122
* @package WordPress
2223
*/
@@ -573,6 +574,16 @@ static function ( $pair ) {
573574
$base_sha = getenv( 'HTML_API_BENCH_BASE_SHA' );
574575
$merge_sha = getenv( 'HTML_API_BENCH_MERGE_SHA' );
575576
$run_url = getenv( 'HTML_API_BENCH_RUN_URL' );
577+
$seed = getenv( 'HTML_API_BENCH_SEED' );
578+
579+
// The WARC files the documents came from, named by the part of each document's key before its record range.
580+
$warcs = array();
581+
foreach ( array_keys( $pairs ) as $key ) {
582+
if ( preg_match( '~/(crawl-data/[^#|]+\.warc\.gz)~', $key, $warc ) ) {
583+
$warcs[ $warc[1] ] = true;
584+
}
585+
}
586+
ksort( $warcs );
576587

577588
$report = HTML_API_BENCH_REPORT_MARKER . "\n";
578589
$report .= "## HTML API CommonCrawl analysis\n\n";
@@ -585,6 +596,14 @@ static function ( $pair ) {
585596
1 === $rounds ? 'Each document was timed once' : "Each time is the median of {$rounds} rounds",
586597
$run_url ? " [Workflow run]({$run_url})." : ''
587598
);
599+
$report .= sprintf(
600+
"<details><summary>%s</summary>\n\n%s\n%s</details>\n\n",
601+
1 === count( $warcs ) ? 'The WARC file read' : sprintf( 'The %d WARC files read', count( $warcs ) ),
602+
implode( '', array_map( static fn ( $path ) => "- `{$path}`\n", array_keys( $warcs ) ) ),
603+
false !== $seed && '' !== $seed
604+
? sprintf( "%s picked at random with seed `%s`. Run the workflow manually with this seed and crawl `%s` to analyze the same documents again.\n\n", 1 === count( $warcs ) ? 'It was' : 'They were', $seed, $crawl[1] )
605+
: ''
606+
);
588607

589608
$report .= "### HTML Processor parse time\n\n";
590609
$report .= "All documents, including those where parsing stopped early on unsupported markup:\n\n";

0 commit comments

Comments
 (0)