Skip to content

Extract portable Composer metadata transformations - #733

Draft
simonchrz wants to merge 9 commits into
git-pkgs:mainfrom
simonchrz:feat/composer-metadata
Draft

simonchrz wants to merge 9 commits into
git-pkgs:mainfrom
simonchrz:feat/composer-metadata

Conversation

@simonchrz

Copy link
Copy Markdown
Contributor

Part of #501, the Composer step after PyPI (#732). Builds on #722, which changes the Composer cooldown filter; until that merges this PR shows its commits too, and the Composer change is the last commit. It doesn't depend on #732, and the two touch different files.

As in #612 for npm, the Composer metadata transformations move out of internal/handler into internal/metadata/composer:

  • Rewrite: cooldown filtering, dist URLs pointing at the proxy, pushed versions in place of upstream ones, and the expansion of minified version lists it needs for that
  • FindDist, which reads a version's upstream dist for a download
  • ProxyDistURL, UpstreamDistURL, DistFilename, ShaQueryCommit, IsCommit and PublishTime
  • LocalDocument and PushedVersion for pushed versions

The package takes its inputs explicitly in Options: proxy URL, cooldown policy, the time cooldown is evaluated at, the pushed versions, and whether a package is routed. It reports versions cooldown holds back through OnCooldownFilter, which the handler logs and records, skipping routed packages as before. It doesn't depend on handler.Proxy. It uses internal/jsonscan, which #612 moved out for Composer already.

What stays in the handler: fetching, the rewrite cache, routes, downloads and their cooldown check, composerIntegrity and ComposerReleaseArchive (which internal/mirror calls), and the advisories endpoints. The handler's function names stay as thin wrappers, so the handler tests run unchanged against the real routes.

Cooldown in the rewrite is checked with Evaluate at one EvaluatedAt instead of IsAllowed per version, as in internal/metadata/npm. The handler passes time.Now(), so results are the same unless a rewrite runs across a cooldown boundary. The download path keeps IsAllowed.

Portability: GOOS=wasip1 GOARCH=wasm go build ./internal/metadata/composer works. imports_test.go fails on any non-standard import outside an allowlist (cooldown, purl, vers, packageurl-go, internal/jsonscan, internal/packageurl) and on net, net/http, log/slog or database/sql.

Benchmarks (internal/handler, #722's head vs this branch, test binaries run alternately, n=8, medians):

base branch
ComposerRewriteMetadata 9.89 ms 9.96 ms +0.7%
ComposerFindDownloadURL 3.42 ms 3.49 ms +2.2%

Bytes and allocations per op are unchanged within a few allocations. #612 saw differences of this size come and go between runs, so I read these as noise.

New tests in internal/metadata/composer cover cooldown evaluated at EvaluatedAt on a minified list, the callback, a disabled and a missing policy, pushed versions replacing upstream ones, routed packages, the dist lookup and the URL helpers. They were mutation-checked, as was the handler's wiring. go test ./... and golangci-lint run ./... pass.

Part of git-pkgs#617. The package page lists a held back version only if the
versions table has it, and the metadata filters didn't write one, so a
version nobody downloaded wasn't shown, and a package nobody downloaded
had no page.

recordWithheld is the shared helper the filters call for each version
they hold back. It stores the publish time with
DB.RecordWithheldVersion, which adds the package row if it is missing
and fills published_at only where it is empty, in one transaction and
without reading first.

Called from the npm, Cargo, Composer, pub, RubyGems, Hex, Go, Maven,
NuGet and PyPI filters. Conda builds have their own times and Helm
charts are stored per repository, so neither records; nor does a Maven
named repository, whose times are kept under its own PURLs.
Routed npm and Composer packages keep their times apart from the default
registry's PURLs, and a Go module with capitals stores its times under the
escaped path, so recording them would put times where the cooldown checks
read the default registry's, or give one release two rows. A package
without a name is skipped too, and a NuGet registration leaf, which
carries no version, records the version its path names.
The PyPI JSON API filtered releases in cooldown without recording them, so
a client reading /pypi/{name}/json left nothing on the package page. The
release filter now hands back the upload time it decided on, and the
package and version documents record what they hold back.

A Go .info in cooldown was refused without storing anything; the version is
now recorded before the refusal.
A release's time is its newest file's, and a file can be added after the
release was first held back. The PyPI filters now record through
RecordWithheldVersionNewest, which replaces an earlier stored time. The
other ecosystems keep the first one.
storeUploadTime wrote the time it read unconditionally, so a download that
read an older document than one already recorded could set an earlier time
and end cooldown early. It now goes through AdvanceVersionPublishedAt, which
only moves the time forward, as the metadata filters do.
A download that read an upload time out of cooldown let the file through
even when a newer time inside cooldown was stored meanwhile. storeUploadTime
now returns the newer of the two, and versionInCooldown decides with it.
Move the metadata rewrite (cooldown filtering, dist URLs, pushed versions,
the minified field expansion), the dist lookup for downloads and the dist
URL helpers out of internal/handler into internal/metadata/composer. The
package takes its inputs explicitly (proxy URL, cooldown policy, evaluation
time, pushed versions, a routed-package check) and reports versions cooldown
holds back through a callback the handler logs and records. It imports no
handler, database, HTTP or logging code, and builds for wasip1.

Cooldown in the rewrite is checked with Evaluate at one EvaluatedAt, as in
internal/metadata/npm.
@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant