Spec for a market plugin that raises the origin cache window from seconds to an hour or more and keeps it correct by purging entries when the content behind them changes.
Companion to CACHING.md (the app↔proxy contract, which this does not change) and
CLOUDFLARE.md (the edge layer). shopclass-plugin-cloudflare stays a separate plugin
and is the model for this one — it already solves purge-on-change for the edge, and
this repeats that shape against the origin. Where they overlap, follow it rather than
invent: same hook set, same URL construction (Purge::itemUrls()), same retry queue.
The origin micro-cache holds a page for 30 seconds (public_cache_max_age). That is
the whole staleness budget, and it is short because nothing can invalidate an entry — time
is the only eviction. Raising it without invalidation would mean an edited listing showing
its old title for a day, and a newly posted one missing from the homepage for a day.
Measured on the dev harness, not assumed:
-
With the reference config, an expiring entry is served stale while it refreshes behind the request —
X-Cache: STALE, thenUPDATING, both answered in well under a millisecond.fastcgi_cache_use_stale updating+background_update ondo what they say. Reproduced on a controlled probe and on a real listing page, with and withoutmust-revalidate, and with and withoutfastcgi_cache_valid.(An earlier run of this measurement reported
EXPIREDand a synchronous re-render. It did not reproduce in four subsequent runs and no config difference accounts for it, so it is treated as unreliable rather than explained away.)So the saving from a longer window is origin renders, not visitor latency — visitors already get cache-speed responses at either setting. One background re-render per URL per hour instead of per thirty seconds is the whole benefit, and on a long tail of listings that is real but it is CPU, not page speed. Worth being honest about when justifying the work.
-
nginx honours the app's
s-maxageoverfastcgi_cache_valid. An entry expired after 35s withfastcgi_cache_valid 200 1dset. The TTL is app-controlled, so raising it is a filter, not an nginx change.
In: TTL tiering, purge on the events that change a page, admin configuration with a help page, Docker preconfiguration, an nginx build carrying the purge module.
Out: edge/CDN purge (see CLOUDFLARE.md), purging search or category pages (§4),
caching for logged-in visitors (the cookie contract in CACHING.md is unchanged).
shopclass-plugin-cloudflare already does the hard half. It purges on posted_item,
edited_item, after_delete_item, the seven state-change hooks (enable_item,
disable_item, activate_item, deactivate_item, item_premium_on, item_premium_off,
item_expiration_updated), plus add_category, after_delete_category, edit_page and
after_delete_page — all verified to fire in core. Purge::itemUrls() already builds the
right URL set for an item: homepage, sitemap, category search URL, the seller's public
profile, the canonical item URL and one per locale, with a non-friendly fallback when the
title is missing. Failed purges retry from cron_hourly.
This plugin should mirror all of it and differ only in transport. Two behaviours worth copying exactly: purging the nameable aggregate pages (home, that category, that profile) rather than trying to enumerate search, and queueing a failed purge for retry instead of dropping it.
osc_invalidate_item_cache($itemId) is already wired to every event that makes an
item's rendered page wrong (hCache.php:296-325):
| Event | Hook |
|---|---|
| listing edited | edited_item |
| image uploaded | uploaded_file |
| image deleted | delete_resource |
| listing deleted | after_delete_item |
| image offloaded to remote storage | direct call from StorageWorker::invalidateOwnerCaches() |
That last one is why this matters: offload rewrites the image URL, so a cached page keeps pointing at a local file that no longer exists. It is not a hook today — it is a function call — so no plugin can observe it. The Cloudflare plugin cannot either: it listens to none of the storage hooks, so an edge-cached listing page keeps its pre-offload image URLs until something else changes the listing. The core hook below fixes both plugins at once, which is most of the argument for putting it in core rather than working around it here.
Rather than add five hooks, core fires one inside the function:
function osc_invalidate_item_cache($itemId)
{
// … existing object-cache clearing …
osc_run_hook('invalidate_item_cache', (int) $itemId);
}One line. Every caller above — offload included — becomes observable, and the hook means exactly "this item's rendered output is now wrong", which is precisely the purge trigger.
Static pages and categories already have their own hooks (§8); only the offload seam is missing.
Slug nginx-cache, bundled. Three jobs:
- Raise TTLs through
public_cache_max_age, per page type (§4). - Purge on
invalidate_item_cacheand the page hook, by issuing one HTTP request per affected URL to nginx's purge location. - Configure and explain — admin page plus a help page that prints the exact nginx stanza an install needs, since a standalone install has to add it by hand.
The transport is behind a filter (nginx_cache_purge_urls) so another plugin can send the
same URL list somewhere else — a CDN, Varnish — without this one changing.
The spike proved this works on the current image:
nginx-modules/ngx_cache_purgebuilds as a dynamic module against nginx 1.31.3,--with-compat, unpatched. ~3s compile, 137 KB.so.- Two-stage Dockerfile: build against the pinned
nginx:<version>-alpine, copy the.sointo the stock runtime,load_module. Not OpenResty; stock nginx plus one module. - Verified end to end against real php-fpm:
MISS → HIT,PURGE → 200(cache file deleted),MISS,HIT.
Long TTLs are only safe where the set of URLs holding the content is enumerable, because that is what can be purged.
The line is not "listing pages vs the rest" but nameable vs not. The Cloudflare plugin already purges the homepage, a category page and a seller's profile by name; only search with arbitrary parameters cannot be enumerated.
| Pages | Default | Why |
|---|---|---|
| Item pages, static pages | 3600 | nameable, and change rarely |
| Home, category, user profile | 3600 | nameable — purged on every post/edit/delete, as the CF plugin already does |
| Search with parameters | 30s (core default, untouched) | every keyword, filter, sort and page number is its own entry; not enumerable |
| Any URL with a query string | 30s | added after implementation: /?utm_source=x is the home page by every helper core has, and ?comments-page=2 is the listing, but each is its own entry that no purge names. Page type is not enough — the URL has to be the one a purge will name |
| Everything, with permalinks off | 30s | the canonical URL of each page is then a query URL itself, indistinguishable from the row above without matching parameter order against a rebuilt URL |
Default 3600, not 86400. A day maximises the saving, but the damage from a purge gap nobody notices scales with it, and this is a first release against a module we have run for an afternoon. An hour still cuts renders per URL by 120×, which is most of the win. The setting is per-tier, so a site that trusts it can raise it.
Search staying short is what keeps a newly posted listing findable.
The cache key is $scheme$request_method$host$request_uri. A purge must reproduce it
exactly. In the spike, purging via the container name returned HTTP 412 because $host
was spike-web while the entry held localhost.
So the plugin must send the purge to the internal address (the origin, never the public
URL — that is the hairpin that silently broke auto-cron behind a proxy) while presenting
the public hostname in a Host: header. Both halves are required, and neither is
obvious. This gets a test, not a comment.
One listing is more than one cache entry:
- the canonical URL,
osc_item_url_from_item($item) - one per enabled locale,
osc_item_url_from_item($item, $locale), where locale-prefixed permalinks are in use - trailing-slash variant, if the permalink structure allows both
?comments-page=Nvariants — not enumerable; accepted as a known gap, and the reason comment pagination should be excluded from the long tier
The list is built by a filter (nginx_cache_item_urls) so a theme with extra item routes
can extend it.
The dangerous state is a long TTL with a purge that silently does not work — a day of staleness instead of 30 seconds. So:
The plugin serves core's default TTL unless a purge self-test has passed.
The admin page has a Test purge button that primes a known URL, purges it, and checks the entry is gone.
It must not prime with the host it purges with. Doing so makes the test self-consistent
and blind: a purge_host naming something no visitor ever sends creates an entry under
that name, deletes it again, and reports success while every real purge misses. So the
entry is created the way a visitor creates one — the site's own host — and removed with
the configured settings. The scheme has the same problem and cannot be tested the same way,
because the prime is forced to use the endpoint's; it is checked instead against
REQUEST_SCHEME, which is nginx's own $scheme for real traffic. Its result is stored; the TTL filter returns the raised value only when
that flag is set, and the flag is cleared whenever the endpoint or host setting changes.
A misconfigured install therefore behaves exactly as it does today rather than worse.
A purge is sent synchronously, in the request that changed the content. That is the whole feature — a listing edit is visible on the next request, not on the next cron tick. Nothing about correctness may depend on a scheduler.
The URLs are collected through the request and sent once at shutdown, which is coalescing, not deferral: one save can fire several of the hooks in §3.1 for the same listing, and each would otherwise be its own round trip. The send still happens in that same request, before the process ends.
It is cheap enough to do inline. The endpoint is the loopback origin, a purge is a bare
HTTP request with no body, and a listing produces a handful of them — single-digit
milliseconds against a save that already writes rows and processes images. Blocking the
response for that is the right trade: it guarantees the entry is gone before the seller's
next request, and the seller bypasses the cache anyway (oc_cache_bypass), so nobody is
waiting on it to see their own change.
The queue catches only what that attempt could not deliver — the origin down, a reload
mid-request, a refused connection. It is a fallback, and it should be sized like one: a
bounded list, retried on the next cron tick. It does not need StorageQueue's worker
locking, eight-step exponential backoff or dead-letter ceiling, because it is not a work
pipeline; it is a short list of things that briefly failed.
One consequence to get right: the retry must run more often than the shortest TTL it
protects. Retrying on cron_hourly against a 3600s window means the entry usually
expires on its own before the retry fires, which makes the fallback pointless. Hang it on
the generic cron hook (auto-cron fires at most every five minutes) rather than the hourly
tier.
A cached page carries the token minted when it was cached, and a token stops being accepted
Csrf::TOKEN_LIFETIME (7200s) after it was issued. So a page held for T seconds hands out
tokens up to T seconds old, and every form on it fails once T approaches 7200 — contact
seller, report listing, comment — with "Your session has expired, please reload the page".
That is a hard ceiling on this whole design, and it is why the default is 3600 rather than the day this began as: at 86400 the token is expired for twenty-two hours out of every twenty-four. The self-test in §5.3 does not catch it, because purging works perfectly while the forms quietly do not.
Two things follow:
ttl_item/ttl_page/ttl_aggregatemust stay meaningfully below 7200. The admin page should refuse a larger value, or warn in terms of what breaks rather than a number.- Going past it requires taking the token out of the HTML — fetched by JavaScript after render, so the page itself is token-free and cacheable indefinitely.
Note the tokens are already bucketed onto Csrf::ISSUE_BUCKET, which is what makes two
renders of a page identical. That helps the validator, not the ceiling: it changes the
granularity of issue, not the lifetime.
It is the only route past §5.5, and the objection to it is narrower than it first appears: a form already protected by a captcha needs JavaScript to submit at all, so moving its token to JavaScript costs that form nothing.
But that is a per-site, per-form condition, not a property of the software:
- captcha is off by default and gated on
osc_captcha_enabled(), with per-form switches (osc_recaptcha_items_enabled,_comments_,_reports_) — three forms, opt-in; - the item contact form, send-to-friend, register and login have no per-form switch at all, so on a site with captcha off they are plain HTML forms that would stop working without JavaScript. On a classifieds site, contact-seller is the form the site exists for.
So the decision can be made per response — token in the HTML when the response is not cacheable or the form has no captcha, fetched otherwise — but that couples the cache window to an unrelated setting, and someone will be surprised by it. If long windows are wanted, the honest framing is that the site requires JavaScript to submit forms, decided on its own merits rather than arrived at as a caching side effect.
Preferences under section nginx_cache:
| Key | Type | Default | Meaning |
|---|---|---|---|
purge_endpoint |
STRING | env, else http://127.0.0.1/purge |
internal base URL of the purge location |
purge_host |
STRING | env, else host from WEB_PATH |
Host: header the key was built with |
ttl_item |
INTEGER | 3600 | item pages |
ttl_page |
INTEGER | 3600 | static pages |
ttl_aggregate |
INTEGER | 3600 | home, category, profile |
verified |
BOOLEAN | 0 | set by the self-test; gates the raised TTLs |
Docker presets these from environment variables at install, so the bundled image comes up working with no admin visit. Standalone installs get the defaults plus a help page that prints the nginx stanza to add and the module to build; until the self-test passes they keep today's behaviour.
| Case | Behaviour |
|---|---|
| purge endpoint unreachable | log once per request, keep serving; TTL already gated on the self-test |
| purge returns 412 (key mismatch) | surface it in the admin page as a configuration error, not a transient one — it means purge_host is wrong |
module absent (unknown directive) |
nginx will not start, so this is caught at deploy, not runtime; the help page says so |
| multi-domain install | one purge per configured host; out of scope for v1, documented as a limitation |
| purge storm on bulk edit | coalesce per request — collect URLs, de-duplicate, send once at shutdown |
| origin refused or unreachable | queue for retry (§5.4); the page is wrong until then, which is the cost of the longer window |
osc_run_hook('invalidate_item_cache', $itemId) inside osc_invalidate_item_cache() —
that is the only core change required. Every other hook this plugin needs already
exists and fires — all fourteen the Cloudflare plugin listens on were checked, including
edit_page, after_delete_page, add_category and after_delete_category. (add_page
and edit_category have no hook, but neither is needed: a new page is not yet cached, and
a renamed category is covered by add_category's sibling paths.)
Verified: osc_invalidate_item_cache() fires no hook today, and hCache.php fires none at
all.
- Unit: URL list for an item across locales; key composition; TTL filter returns core's
default while
verifiedis 0 and the raised value when 1. - Integration (harness, the shape the spike used): prime → HIT → edit a listing → assert MISS on the next request. Same for an image upload, a delete, and an offload, which is the case with no test coverage anywhere today.
- Guard: the purge must carry a
Host:header — a scan test, since dropping it fails silently with a 412 that nothing surfaces.
Status: 1 to 3 are done; the plugin is verified end to end against nginx 1.31.3 with
ngx_cache_purge (prime → HIT → purge → MISS, on a listing and its three aggregates, plus
the queue filling on a refused origin and draining on retry), and both admin pages are
driven in a browser. 4 and 5 remain — the image and the docs page.
Two things the admin work changed: the TTL ceiling is enforced at 3600 rather than warned about (§5.5), and the self-test treats STALE/UPDATING/EXPIRED as "the page is cached" — requiring HIT failed the test whenever it happened to run while an entry was refreshing behind a request, which is a false alarm on the one control that switches the feature on.
- Core: the one hook. Independently useful — it is what lets the existing Cloudflare plugin notice a storage offload, which it cannot today. Ships in core, no plugin needed.
- Plugin skeleton: TTL tiering behind the self-test gate, purge transport, the hook
set copied from the Cloudflare plugin, retry queue on
cron_hourly. - Admin: settings page, help page printing the nginx stanza, Test purge button.
- Image: two-stage Dockerfile carrying the module; compose wiring; env presets.
- Docs: a
docs/site/page for the standalone setup path.
Step 1 lands in this repo; steps 2-5 in a new shopclass-plugin-nginx-cache repo,
registered in shopclass-plugins the way the Cloudflare plugin is.
- Default TTL 3600, per tier, raisable. Conservative first ship; the self-test gate (§5.3) is what would make a longer default defensible later.
- Market plugin, not bundled — core stays lean, and an install with no purge module has no reason to carry it. Distributed like the storage adapters and the Cloudflare plugin.
- Cloudflare stays separate.
shopclass-plugin-cloudflarealready owns the edge; this owns the origin. They compose: a site can run both, edge purge and origin purge firing from the same core hooks. No shared code beyond those hooks.