Repository navigation
Conversation
Part of git-pkgs#617. The package page lists a held back version only if the versions table has it, and the metadata filters didn't write one, so a version nobody downloaded wasn't shown, and a package nobody downloaded had no page. recordWithheld is the shared helper the filters call for each version they hold back. It stores the publish time with DB.RecordWithheldVersion, which adds the package row if it is missing and fills published_at only where it is empty, in one transaction and without reading first. Called from the npm, Cargo, Composer, pub, RubyGems, Hex, Go, Maven, NuGet and PyPI filters. Conda builds have their own times and Helm charts are stored per repository, so neither records; nor does a Maven named repository, whose times are kept under its own PURLs.
Routed npm and Composer packages keep their times apart from the default registry's PURLs, and a Go module with capitals stores its times under the escaped path, so recording them would put times where the cooldown checks read the default registry's, or give one release two rows. A package without a name is skipped too, and a NuGet registration leaf, which carries no version, records the version its path names.
The PyPI JSON API filtered releases in cooldown without recording them, so
a client reading /pypi/{name}/json left nothing on the package page. The
release filter now hands back the upload time it decided on, and the
package and version documents record what they hold back.
A Go .info in cooldown was refused without storing anything; the version is
now recorded before the refusal.
A release's time is its newest file's, and a file can be added after the release was first held back. The PyPI filters now record through RecordWithheldVersionNewest, which replaces an earlier stored time. The other ecosystems keep the first one.
storeUploadTime wrote the time it read unconditionally, so a download that read an older document than one already recorded could set an earlier time and end cooldown early. It now goes through AdvanceVersionPublishedAt, which only moves the time forward, as the metadata filters do.
A download that read an upload time out of cooldown let the file through even when a newer time inside cooldown was stored meanwhile. storeUploadTime now returns the newer of the two, and versionInCooldown decides with it.
Move the metadata rewrite (cooldown filtering, dist URLs, pushed versions, the minified field expansion), the dist lookup for downloads and the dist URL helpers out of internal/handler into internal/metadata/composer. The package takes its inputs explicitly (proxy URL, cooldown policy, evaluation time, pushed versions, a routed-package check) and reports versions cooldown holds back through a callback the handler logs and records. It imports no handler, database, HTTP or logging code, and builds for wasip1. Cooldown in the rewrite is checked with Evaluate at one EvaluatedAt, as in internal/metadata/npm.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #501, the Composer step after PyPI (#732). Builds on #722, which changes the Composer cooldown filter; until that merges this PR shows its commits too, and the Composer change is the last commit. It doesn't depend on #732, and the two touch different files.
As in #612 for npm, the Composer metadata transformations move out of
internal/handlerintointernal/metadata/composer:Rewrite: cooldown filtering, dist URLs pointing at the proxy, pushed versions in place of upstream ones, and the expansion of minified version lists it needs for thatFindDist, which reads a version's upstream dist for a downloadProxyDistURL,UpstreamDistURL,DistFilename,ShaQueryCommit,IsCommitandPublishTimeLocalDocumentandPushedVersionfor pushed versionsThe package takes its inputs explicitly in
Options: proxy URL, cooldown policy, the time cooldown is evaluated at, the pushed versions, and whether a package is routed. It reports versions cooldown holds back throughOnCooldownFilter, which the handler logs and records, skipping routed packages as before. It doesn't depend onhandler.Proxy. It usesinternal/jsonscan, which #612 moved out for Composer already.What stays in the handler: fetching, the rewrite cache, routes, downloads and their cooldown check,
composerIntegrityandComposerReleaseArchive(whichinternal/mirrorcalls), and the advisories endpoints. The handler's function names stay as thin wrappers, so the handler tests run unchanged against the real routes.Cooldown in the rewrite is checked with
Evaluateat oneEvaluatedAtinstead ofIsAllowedper version, as ininternal/metadata/npm. The handler passestime.Now(), so results are the same unless a rewrite runs across a cooldown boundary. The download path keepsIsAllowed.Portability:
GOOS=wasip1 GOARCH=wasm go build ./internal/metadata/composerworks.imports_test.gofails on any non-standard import outside an allowlist (cooldown, purl, vers, packageurl-go,internal/jsonscan,internal/packageurl) and onnet,net/http,log/slogordatabase/sql.Benchmarks (
internal/handler, #722's head vs this branch, test binaries run alternately, n=8, medians):Bytes and allocations per op are unchanged within a few allocations. #612 saw differences of this size come and go between runs, so I read these as noise.
New tests in
internal/metadata/composercover cooldown evaluated atEvaluatedAton a minified list, the callback, a disabled and a missing policy, pushed versions replacing upstream ones, routed packages, the dist lookup and the URL helpers. They were mutation-checked, as was the handler's wiring.go test ./...andgolangci-lint run ./...pass.