Merge pull request #39 from PureStackStudio/perf-cache-for-plugins

Reuse highlighted code across builds through a shared persistent cache.

Author: koculu

CONTRIBUTING.md

@@ -53,6 +53,25 @@ enforce branch protection. See GitHub's [ruleset documentation](https://docs.git
 Dependabot opens weekly pull requests for GitHub Actions and dependencies. Review
 them through the same CI and pull request process as other changes.
 
+To publish one package manually, log in to npm once and run the command from the
+repository root:
+
+```sh
+npm login --registry=https://registry.npmjs.org
+yarn publish-package ts-images
+```
+
+Both short names (`ts-images`) and full names (`@purestack/ts-images`, `purestack`)
+are accepted. The command rebuilds and verifies all distributable bundles, packs
+only the selected workspace with Yarn to resolve its `workspace:` dependencies,
+then publishes its current version publicly to npm with the `latest` tag. npm's
+authentication prompts remain interactive. It does not bump versions or publish
+dependencies; the package's dependency versions must already be available on npm.
+
+Use `yarn publish-package ts-images --dry-run` to build, pack, and inspect npm's
+publish preview without uploading anything. An existing name/version cannot be
+published again; update the version through the normal release process first.
+
 The existing **Publish new npm packages** workflow remains a manual operation.
 It only publishes packages missing from npm; it is not a version-release workflow.
 Use its default dry run first and review the registry results before publishing.

frontend/purestack.studio/guides/extending/plugins.mdx

@@ -185,7 +185,7 @@ This keeps the layouts and components while omitting automatic social metadata,
 
 An explicitly listed standard plugin replaces the automatic instance by name and keeps its initialization position. You can also use `createDefaultPlugins()` to obtain the complete set. In programmatic builds, use `options.defaultPlugins` alongside `options.plugins`.
 
-The highlighting plugin selects the configured backend. Configure native backend factories through `highlightPlugin({ createShiki, createHighlightJs })`, or supply `markdown.createHighlighter(config)` for a different engine. The generic provider returns `{ codeToHtml(code, lang?) }`. Ordinary providers override the default without constructing it; per-language `markdown.resolveHighlighter` callbacks retain precedence.
+The highlighting plugin selects the configured backend. Configure native backend factories through `highlightPlugin({ createShiki, createHighlightJs })`, or supply `markdown.createHighlighter({ mdx, cache })` for a different engine. The generic provider returns `{ codeToHtml(code, lang?) }`. Ordinary providers override the default without constructing it; per-language `markdown.resolveHighlighter` callbacks retain precedence.
 
 The Pagefind plugin supplies a `search` provider. To use another engine, replace `purestack:pagefind` or disable the default set and install your own provider. The builder supplies excluded output paths and the rendered-page list for lazy requests, and manages invalidation on edits and deletions. Sitemap and robots generation remain built in.
 
@@ -236,6 +236,8 @@ export const brandPlugin = definePlugin({
 
 `resolveSiteConfig(input, { plugins, defaultPlugins })` composes the supplied plugin instances with the standard plugins by default. This synchronous helper does not apply JSON plugin selection (`plugins` or `defaultPlugins`) or load `purestack.config.ts`. Plugin-owned sections (`style`, `docLayout`, `pageToc`, `logo`, `auth`, `consent`, `analytics`, `pagefind`, and `preview`) are optional in `SiteConfig`: an omitted plugin does not supply defaults or validate its unused input. A replacement plugin reads its settings from `input` and `file`, then contributes normalized config. Analytics also resolves explicitly configured consent when its browser runtime is omitted, so enabled consent is never bypassed. Configuration contributions are synchronous and may run whenever a project configuration is resolved; keep file writes and other side effects in lifecycle hooks.
 
+`config.cache` is `false` when a run uses `--no-cache` or `options.cache: false`. A plugin that keeps a persistent cache outside the output directory should neither read nor write it then. Site settings cannot change this flag, and it never invalidates incremental output.
+
 Builder style state is isolated across asynchronous operations, including concurrent sites. Its owned theme options remain linked to `context.config.style.theme`, so in-place config-hook changes and `themes.setOptions` update page colors, CSS, and generated favicons consistently. Head, metadata, template, Markdown, provider, and lifecycle errors identify their owning plugin and preserve the original cause. Markdown wrappers preserve unified presets, duplicate-plugin option merging, and callback transforms.
 
 ## Optimize images
@@ -617,6 +619,22 @@ When you publish the package:
 - **Ship JavaScript.** Packages load without bundling, so publish compiled JavaScript with type declarations, not TypeScript source.
 - **Share the site's PureStack.** List `purestack`, and any `@purestack/*` package the plugin imports, under `peerDependencies`, not `dependencies`. The plugin then uses the site's copy instead of installing its own.
 
+## Cache build-time computations
+
+Plugins and build-time components can use `PersistentCache` from
+`@purestack/ts-util-node` to reuse expensive results across builds. Create one
+cache per factory, use an owner namespace and implementation version, and include
+all output-affecting inputs in its keys. Pass `enabled: config.cache` to honor
+`--no-cache`.
+
+Only cache computed values: dependency tracking, styles, and asset emission must
+still run when a value is reused. The cache batches small values into JSON shards
+and keeps bounded memory. Await `cache.flush()` after the computation batch;
+periodic development flushing does not keep Node alive. Highlighters can return
+an optional `flush()` method, which core awaits after each build or development
+render batch, including failures. See the
+[cache API and component example](https://github.com/PureStackStudio/PureStack/tree/main/packages/ts-util-node#persistent-build-cache).
+
 ## Build from a script
 
 `buildSite` and `startDevServer` run the same build as the CLI, for a project that needs its own build script. They do not look for `purestack.config.ts`; pass plugins in `options.plugins`:

frontend/purestack.studio/guides/getting-started/purestack-cli.mdx

@@ -115,6 +115,10 @@ yarn purestack build --content ./content --clean
 
 `--clean` removes the configured `outDir` before rebuilding it. Use it when you need an output folder containing only files from the current build. Without `--clean`, PureStack keeps the directory and rewrites current outputs.
 
+`--no-cache` bypasses reads and writes to persistent computation caches kept outside the output directory, such as highlighted code and optimized images. It works with `build`, `serve`, and `publish`. Use it to compare uncached rendering or investigate a stale cached result. Existing cache files stay in place; a cold build with caching enabled also spends time populating its cache.
+
+During `serve`, pages still render on request and reuse their rendered output until invalidated. `--no-cache` affects computations when a page renders; it does not force rendering on every request. To rebuild into an empty output directory with persistent caches bypassed, combine `--clean` and `--no-cache` (`publish` already cleans its output directory).
+
 ### `serve`: preview and edit
 
 ```sh
@@ -136,6 +140,7 @@ These options belong to `serve`:
 | `--host <host>` | Listen on a particular interface, for example `127.0.0.1`. |
 | `--port <port>` | Listen on a different port. |
 | `--clean` | Remove `outDir` before setup and when config changes reset shared setup. |
+| `--no-cache` | Skip persistent caches, such as highlighted code and optimized images. |
 | `--full-render` | Build every page, asset, and the complete search index once at startup. Watched edits remain incremental. |
 | `--no-watch` | Keep the server running without watching the content directory. Pages still render on request, or all at startup with `--full-render`. |
 | `--no-reload` | Keep watching and updating the site, but do not inject the browser live reload script. |
@@ -150,7 +155,7 @@ yarn purestack publish --content ./content
 
 `publish` uses `publishDir` instead of `outDir`. It cleans that directory before building, minifies generated HTML and browser scripts, disables pretty printed CSS, and treats asset build errors as failures. It creates the same site content and optional search and sitemap outputs as `build`, according to `siteConfig.json`.
 
-Point your deployment process at `publishDir` after a successful run. `publish` prepares files locally; it does not upload them. It accepts `--content`, with no separate `--clean` flag because cleaning is built in.
+Point your deployment process at `publishDir` after a successful run. `publish` prepares files locally; it does not upload them. It accepts `--content` and `--no-cache`, with no separate `--clean` flag because cleaning is built in.
 
 ## Choose the public URL
 

frontend/purestack.studio/guides/getting-started/site-config.mdx

@@ -190,7 +190,11 @@ Place this in `content/guides/_nav.json`. A custom item's `url` is relative to t
 }
 ```
 
-`mdx.highlighter` accepts `highlightjs` (the default and faster option) or `shiki`. `disableHighlighter` turns highlighting off. `compileMdAsMdx` defaults to `true`, so `.md` files use the MDX/Regor markup pipeline; `.mdx` files use that pipeline in either case. The [Regor guide](/guides/extending/regor/) explains the supported markup and components.
+`mdx.highlighter` accepts `highlightjs` (the default) or `shiki`. The standard Shiki backend caches highlighted HTML across builds in the OS temporary directory. Cache keys include the code, language, themes, loaded languages, and Shiki version. On a warm build, cached blocks reuse their HTML, and Shiki initializes only when a block needs highlighting. Output from a custom `createShiki` factory is not cached.
+
+Use the CLI's [`--no-cache` flag](/guides/getting-started/purestack-cli/#build-generate-static-files) to bypass persistent caches for a run. This is a run option, not a `siteConfig.json` setting.
+
+`disableHighlighter` turns highlighting off. `compileMdAsMdx` defaults to `true`, so `.md` files use the MDX/Regor markup pipeline; `.mdx` files use that pipeline in either case. The [Regor guide](/guides/extending/regor/) explains the supported markup and components.
 
 ## Search
 

package.json

@@ -52,6 +52,7 @@
     "bump-version": "yarn tsx --no-warnings scripts/bump-version.ts",
     "bundle": "yarn tsx --no-warnings scripts/bundle.ts",
     "cleanup": "yarn tsx --no-warnings scripts/cleanup.ts",
+    "publish-package": "yarn tsx --no-warnings scripts/publish-package.ts",
     "publish-new-packages": "yarn tsx --no-warnings scripts/publish-new-packages.ts",
     "test": "vitest",
     "test:ci": "vitest run",

packages/purestack/README.md

@@ -103,6 +103,8 @@ Run these commands from the project root:
 
 `publish` prepares files for a static host; it does not upload them. Use
 `--clean` with `build` or `serve` to clear the output directory before a build.
+Use `--no-cache` with any command to skip persistent caches, such as
+highlighted code and optimized images, for a fully cold build.
 Use `--host` and `--port` with `serve` to choose its address. Every command
 needs `--content` pointing to a directory with `siteConfig.json`.
 

packages/ts-common/src/code-highlighter.ts

@@ -1,4 +1,6 @@
 /** Converts source code to highlighted pre/code HTML. */
 export interface MdxCodeHighlighter {
   codeToHtml: (code: string, lang?: string) => string | Promise<string>
+  /** Persist buffered computation results after a render batch. Await producers before calling. */
+  flush?: () => void | Promise<void>
 }

packages/ts-common/src/site-config-types.ts

@@ -22,6 +22,11 @@ export interface CoreSiteConfig {
   sitemap: SitemapConfig
   i18n: I18nConfig
   mdx: SiteMdxConfig
+  /**
+   * Whether plugins may read and write persistent caches outside the output
+   * directory. Set per run (`--no-cache`, `options.cache`), never by site settings.
+   */
+  cache: boolean
 }
 
 export interface SiteConfig extends CoreSiteConfig {
@@ -156,7 +161,9 @@ export type SiteStyleConfigInput = DeepPartial<
   theme?: ThemeOptionsInput
 }
 
-export type SiteConfigInput = DeepPartial<Omit<SiteConfig, 'style'>> & {
+export type SiteConfigInput = DeepPartial<
+  Omit<SiteConfig, 'style' | 'cache'>
+> & {
   style?: SiteStyleConfigInput
 }
 

packages/ts-components/package.json

@@ -56,7 +56,7 @@
     "@purestack/ts-style": "workspace:^",
     "@purestack/ts-svg-icons": "workspace:^",
     "@purestack/ts-util": "workspace:^",
-    "regor": "^1.7.5"
+    "regor": "^1.7.6"
   },
   "regorComponents": [
     "AlertBox",

packages/ts-highlight/README.md

@@ -26,7 +26,23 @@ unused factory is never called. The Shiki options contain the default
 languages and themes. For highlight.js, prefer a new instance with explicitly
 registered grammars to avoid changing the shared registry.
 
-`createMdxHighlighter`, `createHljsHighlighter`, `DEFAULT_MDX_CODE_LANGS`, and
+The default Shiki backend caches highlighted HTML in the OS temporary
+directory (`purestack-cache/shiki`), keyed by Shiki's version, the themes, the loaded languages, and the
+code. Repeat builds reuse it, and Shiki loads only when a block is not cached.
+Builds run with `--no-cache` skip it. Output from a custom `createShiki`
+factory is never cached, because its grammars are unknown.
+
+Storage uses `PersistentCache` from `@purestack/ts-util-node`: 32 lazy JSON shards,
+buffered writes merged under shard locks, bounded memory, and cleanup of entries
+unused for 30 days. Large entries stay in separate files. PureStack flushes after
+build and development render batches. Standalone users should await
+`highlighter.flush?.()` after rendering their batch; the periodic flush timer
+does not keep Node alive.
+Other build-time plugins and components can use the same utility with their own
+namespace, version, and input keys.
+
+`createMdxHighlighter(themes?, langs?, { create?, cache? })`,
+`createHljsHighlighter`, `DEFAULT_MDX_CODE_LANGS`, and
 `DEFAULT_MDX_CODE_THEMES` are exported for standalone use. The shared
 `MdxCodeHighlighter` contract is exported by `ts-common` and `ts-ssg`.