Repository navigation
Conversation
Part of git-pkgs#617. The package page lists a held back version only if the versions table has it, and the metadata filters didn't write one, so a version nobody downloaded wasn't shown, and a package nobody downloaded had no page. recordWithheld is the shared helper the filters call for each version they hold back. It stores the publish time with DB.RecordWithheldVersion, which adds the package row if it is missing and fills published_at only where it is empty, in one transaction and without reading first. Called from the npm, Cargo, Composer, pub, RubyGems, Hex, Go, Maven, NuGet and PyPI filters. Conda builds have their own times and Helm charts are stored per repository, so neither records; nor does a Maven named repository, whose times are kept under its own PURLs.
Routed npm and Composer packages keep their times apart from the default registry's PURLs, and a Go module with capitals stores its times under the escaped path, so recording them would put times where the cooldown checks read the default registry's, or give one release two rows. A package without a name is skipped too, and a NuGet registration leaf, which carries no version, records the version its path names.
The PyPI JSON API filtered releases in cooldown without recording them, so
a client reading /pypi/{name}/json left nothing on the package page. The
release filter now hands back the upload time it decided on, and the
package and version documents record what they hold back.
A Go .info in cooldown was refused without storing anything; the version is
now recorded before the refusal.
A release's time is its newest file's, and a file can be added after the release was first held back. The PyPI filters now record through RecordWithheldVersionNewest, which replaces an earlier stored time. The other ecosystems keep the first one.
storeUploadTime wrote the time it read unconditionally, so a download that read an older document than one already recorded could set an earlier time and end cooldown early. It now goes through AdvanceVersionPublishedAt, which only moves the time forward, as the metadata filters do.
A download that read an upload time out of cooldown let the file through even when a newer time inside cooldown was stored meanwhile. storeUploadTime now returns the newer of the two, and versionInCooldown decides with it.
Move the simple API rewrites (HTML and JSON), the JSON API rewrite, the release-time parser and the filename parsers out of internal/handler into internal/metadata/pypi. The package takes its inputs explicitly (proxy and download URLs, cooldown policy, evaluation time, a denylist check) and reports what the JSON API rewrite holds back through callbacks the handler logs and records. It imports no handler, database, HTTP or logging code, and builds for wasip1. Cooldown is checked with Evaluate at one EvaluatedAt per rewrite, as in internal/metadata/npm.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #501, the PyPI step you asked for there. Builds on #722, which changes the same PyPI filters; until that merges this PR shows its commits too, and the PyPI change is the last commit.
As in #612 for npm, the PyPI metadata transformations move out of
internal/handlerintointernal/metadata/pypi:RewriteSimpleHTML,RewriteDownloadHrefsandFilterSimpleHTMLLinksfor the simple API's HTML, andRewriteSimpleJSONfor PEP 691 JSONRewriteJSONfor the JSON API, package and version documents, with the denylist and cooldown it applies itselfReleaseTimes, which reads each release's newest upload timeFilenameIdentityand the wheel, sdist, egg and Windows installer parsers, withMetadataSuffixThe package takes its inputs explicitly in
Options: proxy and download URLs, the cooldown policy, the time cooldown is evaluated at, and a denylist check. It reports what the JSON API rewrite leaves out throughOnFilteredandOnWithheld, which the handler logs and records as before. It doesn't depend onhandler.Proxy.What stays in the handler: fetching, the rewrite cache, choosing which versions the simple index filters (the JSON API lookup and
versionInCooldown), downloads, and the streaming href rewrite for the root index (replacingReader). The handler's method names stay as thin wrappers, so the handler tests run unchanged against the real routes.Other changes:
handler.PyPIFilenameIdentityis gone;internal/mirrorcallspypi.FilenameIdentity.handler.PyPIMetadataSuffixstays, set frompypi.MetadataSuffix, forinternal/server.TestPyPIHandlerRewritesConfiguredDownloadUpstreamdoes, still works.Evaluateat oneEvaluatedAtper rewrite instead ofIsAllowedper release, as ininternal/metadata/npm. The handler passestime.Now(), so results are the same unless a rewrite runs across a cooldown boundary.Portability:
GOOS=wasip1 GOARCH=wasm go build ./internal/metadata/pypiworks.imports_test.gofails on any non-standard import outside an allowlist (cooldown, purl, vers, packageurl-go,internal/packageurl, andgolang.org/x/net/htmlfor the link filter) and onnet,net/http,log/slogordatabase/sql.Benchmarks (
internal/handler, #722's head vs this branch, test binaries run alternately, n=8, medians):Bytes and allocations per op are unchanged within a few allocations.
New tests in
internal/metadata/pypicover the JSON API rewrite with cooldown evaluated atEvaluatedAtby a release's newest file, the denylist, both callbacks, a disabled policy and no policy, the simple index in HTML and JSON, release times and filename identity. They were mutation-checked.go test ./...andgolangci-lint run ./...pass.