Skip to content

Commit cb566d5

Browse files
authored
Expire cooldown rewrites when a withheld version becomes available (#638)
A cached rewrite made with cooldown on stays right until a version it held back leaves cooldown, so it now expires then instead of after a fixed time. Handlers ask cooldown through Proxy.cooldown, which reports each withheld version to the rewrites running at that moment. A rewrite that held nothing back is kept for the shortest cooldown, at most a day.
1 parent ce9ab47 commit cb566d5

21 files changed

Lines changed: 413 additions & 92 deletions

‎docs/configuration.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1144,7 +1144,7 @@ Helm uses each release's `created` time from the repository's `index.yaml`. Rele
11441144

11451145
RubyGems, Hex and PyPI keep their publish times outside the metadata clients read: the RubyGems versions API, the Hex HTTP API (`upstream.hex_api`) and the PyPI JSON API. The proxy fetches them alongside each compact index, package or simple page it filters, and for PyPI downloads. Cooldown fails closed: when it applies to a package and those times can't be fetched or parsed, or the metadata can't be rewritten, the request fails with 502, or with 503 and `Retry-After` while the upstream is rate limiting the proxy, instead of serving the metadata unfiltered. The same goes for npm, Composer and pub.dev metadata that can't be rewritten, and for npm tarball downloads whose publish time can't be read. A PyPI file or npm tarball that is already cached is still served when its publish time can't be read; only a download that would have to be fetched fails. A PyPI release's time is the upload time of its newest file. The proxy stores it when a download looks it up, and serves files of that release that are already cached against the stored time without reading the JSON API again. A file that isn't cached yet is always looked up, since it may have been added to the release since, and the newer time is stored. Packages exempted with a duration of `0` are served as before and don't need the times. A PyPI package the JSON API returns 404 for has no times to go by and is served unfiltered.
11461146

1147-
For RubyGems and Hex, both documents go through the metadata cache. With [`cache_metadata`](#metadata-caching) they are served from it within `metadata_ttl` without asking the upstream, revalidated with their ETags after it, and filtered from the last cached copies, marked stale, while the upstream is down; without it they are still fetched once for concurrent requests. Each document has its own TTL, so the compact index or Hex package can be newer than the cached times and list a version they don't have. The times are then fetched again before filtering, and if that fails, such versions are left out until the times can be read. The filtered index or package carries an ETag of its own, so Bundler and Mix revalidate it with `304 Not Modified`. Publish times are compared with the current time on each request, but the filtered result is kept for up to an hour (see [rewritten metadata cache](#rewritten-metadata-cache)), so a version leaving cooldown can show up that much later; a newly published version shows up within `metadata_ttl`, as with other cached metadata. Upstream errors are answered as for other cached metadata (404, or 502) rather than relayed.
1147+
For RubyGems and Hex, both documents go through the metadata cache. With [`cache_metadata`](#metadata-caching) they are served from it within `metadata_ttl` without asking the upstream, revalidated with their ETags after it, and filtered from the last cached copies, marked stale, while the upstream is down; without it they are still fetched once for concurrent requests. Each document has its own TTL, so the compact index or Hex package can be newer than the cached times and list a version they don't have. The times are then fetched again before filtering, and if that fails, such versions are left out until the times can be read. The filtered index or package carries an ETag of its own, so Bundler and Mix revalidate it with `304 Not Modified`. Publish times are compared with the current time on each request, and the filtered result is kept until the first version it held back leaves cooldown (see [rewritten metadata cache](#rewritten-metadata-cache)); a newly published version shows up within `metadata_ttl`, as with other cached metadata. Upstream errors are answered as for other cached metadata (404, or 502) rather than relayed.
11481148

11491149
With cooldown on, the proxy asks npm for full packuments, since the abbreviated ones it uses otherwise carry no publish times. Abbreviated packuments cached before cooldown was turned on are fetched again in full rather than served, including as a stale copy when the upstream is down. When `latest` is in cooldown, the packument tags the newest version out of cooldown instead, and `npm dist-tag ls` shows the same tags.
11501150

@@ -1423,7 +1423,7 @@ Or via environment variable: `PROXY_ARTIFACT_MAX_SIZE=2GB`.
14231423

14241424
### Rewritten metadata cache
14251425

1426-
The npm, Composer, PyPI and Helm handlers rewrite every metadata document they serve, so that download URLs point at the proxy. That means decoding the whole document and encoding it again, and Composer's minified documents are also expanded, which together cost milliseconds per typical package and far more for very large ones; a large Helm `index.yaml` takes over a hundred milliseconds. The proxy keeps rewritten documents in memory, so each distinct upstream document is rewritten once and requests arriving while it is being rewritten wait for that rewrite. New bytes from upstream are rewritten again. A PyPI simple page is cached per representation (HTML or JSON) and per set of versions filtered out of it. With version cooldown enabled, a cached rewrite expires, because cooldown filtering depends on the current time: it is kept for a 24th of the shortest cooldown configured, at least a minute and at most an hour, so a version shows up at most that long after its cooldown ends. With a cooldown of a day or more that is an hour; with `"2h"` somewhere, five minutes. Cached documents carry an `ETag` of their own, and are dated like any other, so clients that revalidate with `If-None-Match` or `If-Modified-Since`, such as Composer, get `304 Not Modified`. The RubyGems compact index and Hex packages filtered by cooldown are kept the same way; they are never served gzip-compressed.
1426+
The npm, Composer, PyPI and Helm handlers rewrite every metadata document they serve, so that download URLs point at the proxy. That means decoding the whole document and encoding it again, and Composer's minified documents are also expanded, which together cost milliseconds per typical package and far more for very large ones; a large Helm `index.yaml` takes over a hundred milliseconds. The proxy keeps rewritten documents in memory, so each distinct upstream document is rewritten once and requests arriving while it is being rewritten wait for that rewrite. New bytes from upstream are rewritten again. A PyPI simple page is cached per representation (HTML or JSON) and per set of versions filtered out of it. With version cooldown enabled, a cached rewrite expires when the first version it held back leaves cooldown, since versions it let through stay allowed and a new upstream version changes the document. A rewrite that held nothing back is kept for the shortest cooldown configured, at least a minute and at most a day. Cached documents carry an `ETag` of their own, and are dated like any other, so clients that revalidate with `If-None-Match` or `If-Modified-Since`, such as Composer, get `304 Not Modified`. The RubyGems compact index and Hex packages filtered by cooldown are kept the same way; they are never served gzip-compressed.
14271427

14281428
Each cached document is kept gzip-compressed, when that makes it smaller, and served with `Content-Encoding: gzip` to clients that accept it, as npm, pnpm, Yarn and Composer do. A large packument is compressed once rather than on every request, by the proxy or by a reverse proxy in front of it. Only the compressed copy is kept, typically an eighth of the document, and it is decompressed for the clients that don't accept gzip. Documents rewritten outside the cache are served uncompressed.
14291429

‎internal/handler/cargo.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ func (h *CargoHandler) applyCooldownFiltering(downstreamResponse http.ResponseWr
171171

172172
cratePURL := canonicalPackagePURL("cargo", crate.Name)
173173

174-
if !h.proxy.Cooldown.IsAllowed("cargo", cratePURL, publishedAt) {
174+
if !h.proxy.cooldown().IsAllowed("cargo", cratePURL, publishedAt) {
175175
h.proxy.Logger.Info("cooldown: filtering cargo version",
176176
"crate", crate.Name, "version", crate.Version,
177177
"published", crate.PublishTime)

‎internal/handler/composer.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -394,7 +394,7 @@ func (h *ComposerHandler) shouldFilterVersion(packagePURL, packageName, version
394394
return false
395395
}
396396

397-
if !h.proxy.Cooldown.IsAllowed("composer", packagePURL, publishedAt) {
397+
if !h.proxy.cooldown().IsAllowed("composer", packagePURL, publishedAt) {
398398
h.proxy.Logger.Info("cooldown: filtering composer version",
399399
"package", packageName, "version", version)
400400
return true
@@ -731,7 +731,7 @@ func (h *ComposerHandler) serveCachedRelease(w http.ResponseWriter, r *http.Requ
731731
if err != nil || ver == nil || !ver.PublishedAt.Valid {
732732
return false
733733
}
734-
if !h.proxy.Cooldown.IsAllowed("composer", canonicalPackagePURL("composer", packageName), ver.PublishedAt.Time) {
734+
if !h.proxy.cooldown().IsAllowed("composer", canonicalPackagePURL("composer", packageName), ver.PublishedAt.Time) {
735735
h.proxy.Logger.Info("cooldown: withholding composer archive",
736736
"package", packageName, "version", version)
737737
http.Error(w, "version not found", http.StatusNotFound)

‎internal/handler/conda.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -493,7 +493,7 @@ func (f *repodataFilter) inCooldown(filename string, raw []byte) bool {
493493
return false
494494
}
495495
publishedAt := time.Unix(int64(entry.Timestamp)/condaTimestampDivisor, 0)
496-
decision := f.h.proxy.Cooldown.Evaluate("conda", canonicalPackagePURL("conda", entry.Name), publishedAt, f.now)
496+
decision := f.h.proxy.cooldown().Evaluate("conda", canonicalPackagePURL("conda", entry.Name), publishedAt, f.now)
497497
if decision.Allowed {
498498
return false
499499
}
@@ -517,7 +517,7 @@ func (h *CondaHandler) recordInCooldown(filename, name, version string, timestam
517517
return false
518518
}
519519
publishedAt := time.Unix(int64(timestamp)/condaTimestampDivisor, 0)
520-
if h.proxy.Cooldown.IsAllowed("conda", canonicalPackagePURL("conda", name), publishedAt) {
520+
if h.proxy.cooldown().IsAllowed("conda", canonicalPackagePURL("conda", name), publishedAt) {
521521
return false
522522
}
523523
h.proxy.Logger.Info("cooldown: filtering conda package",
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
package handler
2+
3+
import (
4+
"sync"
5+
"time"
6+
7+
"github.com/git-pkgs/cooldown"
8+
)
9+
10+
// A rewrite made with cooldown on stays right until a version it held back
11+
// leaves cooldown: versions it let through stay allowed as time passes, and
12+
// new versions upstream change the document and so the cache key. So the
13+
// cached rewrite expires when the earliest version it withheld becomes
14+
// available, instead of after a fixed time.
15+
//
16+
// Handlers ask cooldown through Proxy.cooldown, which tells every rewrite
17+
// running at that moment when a withheld version becomes available. A
18+
// rewrite may hear of versions another rewrite withheld at the same time;
19+
// that only makes it expire earlier than it has to, never later.
20+
21+
// maxCooldownRewriteTTL is the longest a rewrite made with cooldown on is
22+
// served, even when it held nothing back.
23+
const maxCooldownRewriteTTL = 24 * time.Hour
24+
25+
// minCooldownRewriteTTL is the shortest cap on a rewrite made with cooldown
26+
// on, however short the cooldown.
27+
const minCooldownRewriteTTL = time.Minute
28+
29+
// cooldownRewriteTTL returns the longest a rewrite made with cooldown on is
30+
// served: the shortest cooldown configured, between a minute and a day. It
31+
// only matters when a rewrite held nothing back, or held back a version
32+
// whose availability is unknown.
33+
func (p *Proxy) cooldownRewriteTTL() time.Duration {
34+
ttl := p.ShortestCooldown
35+
switch {
36+
case ttl <= 0 || ttl > maxCooldownRewriteTTL:
37+
return maxCooldownRewriteTTL
38+
case ttl < minCooldownRewriteTTL:
39+
return minCooldownRewriteTTL
40+
}
41+
return ttl
42+
}
43+
44+
// cooldownWatch holds the rewrites running with cooldown on.
45+
type cooldownWatch struct {
46+
mu sync.Mutex
47+
active map[*cooldownRecorder]struct{}
48+
}
49+
50+
// cooldownRecorder collects when the versions withheld during one rewrite
51+
// become available.
52+
type cooldownRecorder struct {
53+
// earliest is when the first withheld version becomes available, or
54+
// zero if none was withheld.
55+
earliest time.Time
56+
// unknown is set when a version was withheld without a time it becomes
57+
// available.
58+
unknown bool
59+
}
60+
61+
// watchCooldown starts recording the versions cooldown holds back for a
62+
// rewrite about to run. unwatchCooldown ends it.
63+
func (p *Proxy) watchCooldown() *cooldownRecorder {
64+
rec := &cooldownRecorder{}
65+
w := &p.cooldownWatch
66+
w.mu.Lock()
67+
defer w.mu.Unlock()
68+
if w.active == nil {
69+
w.active = make(map[*cooldownRecorder]struct{})
70+
}
71+
w.active[rec] = struct{}{}
72+
return rec
73+
}
74+
75+
func (p *Proxy) unwatchCooldown(rec *cooldownRecorder) {
76+
w := &p.cooldownWatch
77+
w.mu.Lock()
78+
defer w.mu.Unlock()
79+
delete(w.active, rec)
80+
}
81+
82+
// cooldownExpiry returns when a rewrite that started at start and recorded
83+
// into rec should expire: when the first version it held back leaves
84+
// cooldown, and no later than cooldownRewriteTTL after start.
85+
func (p *Proxy) cooldownExpiry(rec *cooldownRecorder, start time.Time) time.Time {
86+
w := &p.cooldownWatch
87+
w.mu.Lock()
88+
earliest, unknown := rec.earliest, rec.unknown
89+
w.mu.Unlock()
90+
91+
limit := start.Add(p.cooldownRewriteTTL())
92+
if unknown || earliest.IsZero() || earliest.After(limit) {
93+
return limit
94+
}
95+
return earliest
96+
}
97+
98+
// withheld tells the running rewrites that a version is held back until
99+
// availableAt, zero when that is unknown.
100+
func (p *Proxy) withheld(availableAt time.Time) {
101+
w := &p.cooldownWatch
102+
w.mu.Lock()
103+
defer w.mu.Unlock()
104+
for rec := range w.active {
105+
switch {
106+
case availableAt.IsZero():
107+
rec.unknown = true
108+
case rec.earliest.IsZero() || availableAt.Before(rec.earliest):
109+
rec.earliest = availableAt
110+
}
111+
}
112+
}
113+
114+
// cooldown returns the proxy's cooldown policy, telling the running rewrites
115+
// about each version it holds back. Handlers use it rather than p.Cooldown.
116+
func (p *Proxy) cooldown() CooldownPolicy { //nolint:ireturn // nil when cooldown is not configured, as p.Cooldown is
117+
if p.Cooldown == nil {
118+
return nil
119+
}
120+
return recordingCooldown{p: p}
121+
}
122+
123+
// recordingCooldown is the proxy's cooldown policy, reporting the versions it
124+
// holds back to the running rewrites.
125+
type recordingCooldown struct {
126+
p *Proxy
127+
}
128+
129+
func (r recordingCooldown) IsAllowed(ecosystem, packagePURL string, publishedAt time.Time) bool {
130+
if r.p.Cooldown.IsAllowed(ecosystem, packagePURL, publishedAt) {
131+
return true
132+
}
133+
r.p.withheld(r.availableAt(ecosystem, packagePURL, publishedAt))
134+
return false
135+
}
136+
137+
func (r recordingCooldown) Evaluate(ecosystem, packagePURL string, publishedAt, evaluatedAt time.Time) cooldown.Decision {
138+
d := r.p.Cooldown.Evaluate(ecosystem, packagePURL, publishedAt, evaluatedAt)
139+
if !d.Allowed {
140+
r.p.withheld(d.AvailableAt)
141+
}
142+
return d
143+
}
144+
145+
func (r recordingCooldown) For(ecosystem, packagePURL string) time.Duration {
146+
return r.p.Cooldown.For(ecosystem, packagePURL)
147+
}
148+
149+
func (r recordingCooldown) Enabled() bool {
150+
return r.p.Cooldown.Enabled()
151+
}
152+
153+
// availableAt is when a version IsAllowed held back leaves cooldown, or zero
154+
// when that is unknown.
155+
func (r recordingCooldown) availableAt(ecosystem, packagePURL string, publishedAt time.Time) time.Time {
156+
if publishedAt.IsZero() {
157+
return time.Time{}
158+
}
159+
d := r.p.Cooldown.For(ecosystem, packagePURL)
160+
if d <= 0 {
161+
return time.Time{}
162+
}
163+
return publishedAt.Add(d)
164+
}

0 commit comments

Comments
 (0)