Skip to content

Extract portable PyPI metadata transformations - #732

Draft
simonchrz wants to merge 9 commits into
git-pkgs:mainfrom
simonchrz:feat/pypi-metadata
Draft

simonchrz wants to merge 9 commits into
git-pkgs:mainfrom
simonchrz:feat/pypi-metadata

Conversation

@simonchrz

Copy link
Copy Markdown
Contributor

Part of #501, the PyPI step you asked for there. Builds on #722, which changes the same PyPI filters; until that merges this PR shows its commits too, and the PyPI change is the last commit.

As in #612 for npm, the PyPI metadata transformations move out of internal/handler into internal/metadata/pypi:

  • RewriteSimpleHTML, RewriteDownloadHrefs and FilterSimpleHTMLLinks for the simple API's HTML, and RewriteSimpleJSON for PEP 691 JSON
  • RewriteJSON for the JSON API, package and version documents, with the denylist and cooldown it applies itself
  • ReleaseTimes, which reads each release's newest upload time
  • FilenameIdentity and the wheel, sdist, egg and Windows installer parsers, with MetadataSuffix

The package takes its inputs explicitly in Options: proxy and download URLs, the cooldown policy, the time cooldown is evaluated at, and a denylist check. It reports what the JSON API rewrite leaves out through OnFiltered and OnWithheld, which the handler logs and records as before. It doesn't depend on handler.Proxy.

What stays in the handler: fetching, the rewrite cache, choosing which versions the simple index filters (the JSON API lookup and versionInCooldown), downloads, and the streaming href rewrite for the root index (replacingReader). The handler's method names stay as thin wrappers, so the handler tests run unchanged against the real routes.

Other changes:

  • handler.PyPIFilenameIdentity is gone; internal/mirror calls pypi.FilenameIdentity. handler.PyPIMetadataSuffix stays, set from pypi.MetadataSuffix, for internal/server.
  • The simple index rewrites get only the URLs, so a handler built without a proxy, as TestPyPIHandlerRewritesConfiguredDownloadUpstream does, still works.
  • Cooldown is checked with Evaluate at one EvaluatedAt per rewrite instead of IsAllowed per release, as in internal/metadata/npm. The handler passes time.Now(), so results are the same unless a rewrite runs across a cooldown boundary.

Portability: GOOS=wasip1 GOARCH=wasm go build ./internal/metadata/pypi works. imports_test.go fails on any non-standard import outside an allowlist (cooldown, purl, vers, packageurl-go, internal/packageurl, and golang.org/x/net/html for the link filter) and on net, net/http, log/slog or database/sql.

Benchmarks (internal/handler, #722's head vs this branch, test binaries run alternately, n=8, medians):

base branch
PyPISimpleHTMLRequest 1.025 ms 1.026 ms +0.1%
PyPISimpleHTMLFilteredRequest 1.095 ms 1.100 ms +0.4%
PyPISimpleJSONRequest 0.977 ms 0.963 ms −1.5%
PyPIJSONAPIRequest 1.025 ms 1.022 ms −0.2%
PyPIRewriteSimpleHTML 251 µs 250 µs −0.4%
PyPIRewriteSimpleHTMLFiltered 72.6 ms 72.7 ms +0.2%

Bytes and allocations per op are unchanged within a few allocations.

New tests in internal/metadata/pypi cover the JSON API rewrite with cooldown evaluated at EvaluatedAt by a release's newest file, the denylist, both callbacks, a disabled policy and no policy, the simple index in HTML and JSON, release times and filename identity. They were mutation-checked. go test ./... and golangci-lint run ./... pass.

Part of git-pkgs#617. The package page lists a held back version only if the
versions table has it, and the metadata filters didn't write one, so a
version nobody downloaded wasn't shown, and a package nobody downloaded
had no page.

recordWithheld is the shared helper the filters call for each version
they hold back. It stores the publish time with
DB.RecordWithheldVersion, which adds the package row if it is missing
and fills published_at only where it is empty, in one transaction and
without reading first.

Called from the npm, Cargo, Composer, pub, RubyGems, Hex, Go, Maven,
NuGet and PyPI filters. Conda builds have their own times and Helm
charts are stored per repository, so neither records; nor does a Maven
named repository, whose times are kept under its own PURLs.
Routed npm and Composer packages keep their times apart from the default
registry's PURLs, and a Go module with capitals stores its times under the
escaped path, so recording them would put times where the cooldown checks
read the default registry's, or give one release two rows. A package
without a name is skipped too, and a NuGet registration leaf, which
carries no version, records the version its path names.
The PyPI JSON API filtered releases in cooldown without recording them, so
a client reading /pypi/{name}/json left nothing on the package page. The
release filter now hands back the upload time it decided on, and the
package and version documents record what they hold back.

A Go .info in cooldown was refused without storing anything; the version is
now recorded before the refusal.
A release's time is its newest file's, and a file can be added after the
release was first held back. The PyPI filters now record through
RecordWithheldVersionNewest, which replaces an earlier stored time. The
other ecosystems keep the first one.
storeUploadTime wrote the time it read unconditionally, so a download that
read an older document than one already recorded could set an earlier time
and end cooldown early. It now goes through AdvanceVersionPublishedAt, which
only moves the time forward, as the metadata filters do.
A download that read an upload time out of cooldown let the file through
even when a newer time inside cooldown was stored meanwhile. storeUploadTime
now returns the newer of the two, and versionInCooldown decides with it.
Move the simple API rewrites (HTML and JSON), the JSON API rewrite, the
release-time parser and the filename parsers out of internal/handler into
internal/metadata/pypi. The package takes its inputs explicitly (proxy and
download URLs, cooldown policy, evaluation time, a denylist check) and
reports what the JSON API rewrite holds back through callbacks the handler
logs and records. It imports no handler, database, HTTP or logging code, and
builds for wasip1.

Cooldown is checked with Evaluate at one EvaluatedAt per rewrite, as in
internal/metadata/npm.
@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant