Skip to content

Commit 04ab4ea

Browse files
authored
Merge pull request #225 from KubaO/staging
`shoot_docs` retakes 161 IDE pictures, and the help add-in's project file is published as a download
2 parents 5feb6d0 + 63e9dda commit 04ab4ea

314 files changed

Lines changed: 10865 additions & 1432 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/actions/run-gates/action.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,14 @@ runs:
103103
- name: Unit-test the harness's port claims (test/ports.test.mjs)
104104
shell: bash
105105
run: node --test test/ports.test.mjs
106+
# Unit tests for builder/addin-project.mjs, which packs add-in/ into the
107+
# help add-in's project file the build publishes: the project holds exactly
108+
# the files git tracks there, never Resources/HELP/, and the same bytes from
109+
# a CRLF checkout as from an LF one. Needs git. No browser, no built tree,
110+
# about 1 s.
111+
- name: Unit-test the help add-in's project file (test/addin-project.test.mjs)
112+
shell: bash
113+
run: node --test test/addin-project.test.mjs
106114
# A regex that backtracks exponentially does not fail a build, it stops
107115
# one: the corpus passes until a page happens to contain the trigger, and
108116
# then a render worker sits inside String.replace forever. Nothing else

‎.github/workflows/checks.yml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -98,10 +98,11 @@ jobs:
9898
# step fails on the exit code: 1 when the check found a problem,
9999
# 2 when the build could not run.
100100
#
101-
# --no-help-archive: a local build ends by zipping the offline tree
102-
# for the IDE help add-in. CI has no use for that archive and must
103-
# not spend time on it.
104-
run: node builder/tbdocs.mjs --src docs --no-fetch-assets --no-help-archive --check-audit-index
101+
# The build also zips the offline tree for the IDE help add-in and
102+
# packs the zip into the add-in's project file, the download the site
103+
# offers. Never pass --no-help-archive here: the download would lose
104+
# the archive (check_ci_workflows.mjs refuses the flag).
105+
run: node builder/tbdocs.mjs --src docs --no-fetch-assets --check-audit-index
105106
# The gates both workflows run, in one list: see
106107
# .github/actions/run-gates/action.yml, which check_ci_workflows.mjs
107108
# checks against test.bat and check.bat. Each gate is its own group in

‎.github/workflows/tbdocs-gh-pages.yml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -93,10 +93,11 @@ jobs:
9393
# online tree gets a base path -- the offline tree's links are all
9494
# relative after the rewrite.
9595
#
96-
# --no-help-archive: a local build ends by zipping the offline tree
97-
# for the IDE help add-in. CI has no use for that archive and must
98-
# not spend time on it.
99-
run: node builder/tbdocs.mjs --src docs --url '${{ steps.pages.outputs.origin }}' --baseurl '${{ steps.pages.outputs.base_path }}' --no-fetch-assets --no-help-archive --check-audit-index
96+
# The build also zips the offline tree for the IDE help add-in and
97+
# packs the zip into the add-in's project file, the download the site
98+
# offers. Never pass --no-help-archive here: the download would lose
99+
# the archive (check_ci_workflows.mjs refuses the flag).
100+
run: node builder/tbdocs.mjs --src docs --url '${{ steps.pages.outputs.origin }}' --baseurl '${{ steps.pages.outputs.base_path }}' --no-fetch-assets --check-audit-index
100101
# The gates both workflows run, in one list: see
101102
# .github/actions/run-gates/action.yml, which check_ci_workflows.mjs
102103
# checks against test.bat and check.bat. checks.yml runs only on pull

‎WIP.Build.md‎

Lines changed: 107 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,10 @@ keys through .NET, because `reg.exe` mangles names outside the console code page
129129
`lib/` — modules that every other tooling folder may import, and that import none of them.
130130
`builder/` may not import `scripts/`, so code that both need lives here;
131131
[lib/README.md](lib/README.md) states the rule, and `biome.jsonc` refuses an import that
132-
breaks either one.
132+
breaks either one. The rule has one exception, named in `biome.jsonc`:
133+
`scripts/impexp.mjs`, a published standalone tool that must stay one dependency-free
134+
file, which the build packs the help add-in's project file with (see [The help add-in's
135+
project file](#the-help-add-ins-project-file)).
133136

134137
`wisdom/` — Discord knowledge-harvesting tool (three-phase: export → process → extract). Plans in `wisdom/PLAN-{1,2,3}.md`; implementation under `wisdom/`. Uses only Node.js built-in APIs. Running it is [WIP.Wisdom.md](WIP.Wisdom.md).
135138

@@ -215,7 +218,7 @@ Unlike the link check, **a finding here aborts the build**. A broken link still
215218
leaves a tree worth inspecting; a tree with a private key in it is a tree nobody
216219
should be one `upload-pages-artifact` away from publishing.
217220

218-
Three details of the policy are essential:
221+
Four details of the policy are essential:
219222

220223
- **`SOURCE_EXTENSIONS` and `BUILD_EXTENSIONS` are separate sets, and must stay
221224
separate.** The build emits `.xml` and `.json`; a contributor has no business
@@ -235,6 +238,14 @@ Three details of the policy are essential:
235238
ends spelled out, which is what makes them shippable. The same extension
236239
anywhere else still fails; otherwise declaring one entry would bless a whole
237240
type.
241+
- **`addin_project` is exempt by path too, and on the tree surface only.** The
242+
help add-in's `.twinproj` is written by the build, not found in `docs/`, so
243+
`publishPolicyFor` puts its `dest` in a set of its own, `generated`, which only
244+
`unpublishableTreePaths` reads. A `.twinproj` found in `docs/`, even at that
245+
path, is refused at source: the build writes the file there, and a copy in the
246+
source tree could only be a stale one. `check_publish_policy.mjs` asserts all
247+
three: the path publishes from a tree, a `.twinproj` elsewhere does not, and
248+
one at that path in `docs/` does not.
238249

239250
**A clean build says only that nothing in `docs/` is currently refused, which is
240251
also what an allowlist widened until it refuses nothing says.** No build over a
@@ -392,7 +403,7 @@ node scripts/check_tree_fresh.mjs --tree docs/_site-pdf --marker book.html
392403
**`--marker` is what makes that work on this tree.** The script identifies a tree
393404
by its `index.html`, which every output tree has *except* `_site-pdf/` --- that one
394405
holds a single `book.html`. Exit codes are the script's: **2** when the tree is
395-
absent, **1** when it is older than `docs/`, `builder/` or `lib/`. The renderer that runs after it
406+
absent, **1** when it is older than one of its sources (`DEFAULT_SOURCES`). The renderer that runs after it
396407
has no 1, so `book.bat`'s 1 means a stale tree (or a failed `npm install`) and nothing about the render.
397408

398409
> **Batch detail:** `%ERRORLEVEL%` inside a parenthesised `if errorlevel 1 (...)`
@@ -401,11 +412,14 @@ has no 1, so `book.bat`'s 1 means a stale tree (or a failed `npm install`) and n
401412
> `goto :fail` and captures outside the block, which is the same shape
402413
> `test.bat` uses, and for the same reason.
403414
404-
**Known false positive.** `DEFAULT_SOURCES` is `["docs", "builder", "lib"]` and
405-
does not distinguish code from notes, so editing a `builder/PLAN-*.md` or
406-
`REVIEW-*.md` marks every tree stale even though nothing in the build reads those
407-
files. It errs toward refusing, which is the safe direction, and a rebuild is
408-
~4 s --- but a pure note edit blocks a `book.bat` render until you rebuild.
415+
**Known false positive.** `DEFAULT_SOURCES` is `docs`, `builder`, `lib`, `add-in`
416+
and `scripts/impexp.mjs`, and does not distinguish code from notes, so editing a
417+
`builder/PLAN-*.md` or `REVIEW-*.md` marks every tree stale even though nothing in
418+
the build reads those files; so does a file under `add-in/` that git does not track,
419+
which the add-in's project file leaves out. It errs toward refusing, which is the
420+
safe direction, and a rebuild is ~4 s --- but a pure note edit blocks a `book.bat`
421+
render until you rebuild. `add-in/` and `impexp.mjs` are there because they decide
422+
the bytes of the [help add-in's project file](#the-help-add-ins-project-file).
409423

410424
**Which folders under `docs/` are outputs comes from one list.** A build given
411425
`--dest docs/_site-basepath` writes `_site-basepath-offline` and
@@ -421,7 +435,11 @@ prefix list (`_site`, `_serve`, `_pdf`) the markdown walk uses --- and keeps onl
421435
`page-baseline.json` and `symbol-baseline.json` (`IGNORED_FILES`) are written
422436
after the tree, so counting them would mark every tree that added a page or a
423437
heading stale on the next `check.bat`. `package-api.json` is an input: the build
424-
reads it and it decides the bytes of `tB/symbols.json`.
438+
reads it and it decides the bytes of `tB/symbols.json`. Under `add-in/` two paths are
439+
skipped by their place in the repository (`IGNORED_PATHS`): `Resources/HELP/`, where
440+
the build writes the help archive, and `Resources/SYMBOLS/symbols.json`,
441+
the copy of the index the build rewrites. Neither is an input, since the project file is
442+
packed with the index the build wrote rather than with that copy.
425443

426444
### The code-region gate
427445

@@ -642,6 +660,86 @@ members are placed before the rest**, so a member's URL is the heading under
642660
Properties and not a prose section of the same name above it (`Screen.Fonts` is
643661
`#fonts-1`, its `### Fonts`, not a prose `#fonts`).
644662

663+
### The help add-in's project file
664+
665+
Every build of `docs`, CI's included, writes `tB/IDE/AddIns/downloads/tbDocsHelp.twinproj`
666+
into the online tree: the `addinProject` task
667+
([builder/addin-project.mjs](builder/addin-project.mjs)), declared by `addin_project` in
668+
`_config.yml`, so the test fixtures' configs get none. The Help Add-In page offers it.
669+
The decisions, each with what it rules out:
670+
671+
- **What goes in is `git ls-files` of `add-in/`, read from the working folder.** A plain
672+
walk of the folder packs whatever lies there: a stale `Resources/HELP/site.zip`
673+
(gitignored, ~26 MB, from some earlier build), a `Build/` folder an IDE wrote, a
674+
scratch module. Reading git's blobs instead would leave a local build's download one
675+
commit behind the add-in being edited. `Resources/HELP/` is also refused by name, in
676+
case it is ever added. A file git does not track is named in the build's output.
677+
- **The archive in the project is the one this build wrote, handed over in memory.**
678+
The `helpArchive` task returns the zip it wrote, and `addinProject` waits for it and
679+
packs those bytes as `Resources/HELP/site.zip`. Reading the file back from
680+
`add-in/Resources/HELP/` could pick up the archive of an earlier build whenever this
681+
one wrote none (`--no-help-archive`, `--serve`, another `--dest`, a fixture): that
682+
case packs no archive, and the summary line says `without the offline archive`.
683+
The archive is made **before** the Gantt injection, so it holds `BuildInfo.html`
684+
without the chart: the chart holds the build's timings, and a download that held it
685+
would differ from one build of a tree to the next. `recheckHelpArchive` checks at
686+
the end of `runBuild` that the offline tree still lists the archive's files and
687+
differs from it in `BuildInfo.html` and `assets/images/gantt.svg` alone.
688+
`gantt.svg` is a tracked 105-byte placeholder in `docs/assets/images/`, copied into
689+
both trees as a static file, so it is in the archive's file list; the injection
690+
overwrites it in the trees and the archive keeps the placeholder.
691+
- **The task waits for every writer of the offline tree.** `flushJoin` (the render
692+
lanes write each page's offline HTML in their flush), `scss` (the offline
693+
stylesheet) and `writeOffline` (statics, theme assets, redirect stubs, `nav.js`,
694+
`search-data.js`). Nothing else writes there before the injection; a writer added
695+
later and missed here fails the build in `recheckHelpArchive`, which is why that
696+
check compares every file's content.
697+
- **The offline tree has no copy, and its link goes to the website.** The project
698+
carries the offline tree's archive, so a copy in that tree would be inside its own
699+
archive. `deriveTreeRels` takes the path as an online-only file. The page links to
700+
it relatively, and the offline rewrite (`computeWebsiteUrl` in `offline-rewrite.mjs`)
701+
writes any URL the offline tree lacks and the online tree holds as `site.url` + base
702+
path + path, without counting a miss. The offline link check would flag that URL
703+
(the `forbid` prefix is the site's origin), so its worker env carries `forbidExempt`
704+
(`isWebsiteOnlyLink`), which excuses a website URL only when the online tree holds
705+
the path and the offline tree does not: a live-site link to a page the offline tree
706+
holds is still a missed rewrite. `scripts/check_links.mjs`, which reads trees from
707+
disk, makes the same exemption when `--online-root` names the online tree, and
708+
`check_links_diff.mjs`'s two offline cases pass it.
709+
- **Line endings are git's, not the checkout's.** `.gitattributes` checks
710+
`add-in/Resources/` out `-text`, but not `Settings`, `Sources/*.twin` or the `.tbform`:
711+
under `core.autocrlf` a Windows checkout holds them with CRLF and CI's Linux one with
712+
LF, and `impexp` stores `Settings` and the `.tbform` byte for byte. So CRLF becomes LF
713+
outside `Resources/` before packing, and `impexp` then gives the code files CRLF. Without
714+
it the published file (built on Linux) and a local build differ, and so do
715+
`compare_trees` sides built on different machines.
716+
- **The index is the build's, not the committed copy.** `syncAddinIndex` never writes in
717+
CI, so packing `add-in/Resources/SYMBOLS/symbols.json` would publish the committed copy,
718+
one build behind whenever a page changed. The task packs `symbolIndex`'s `json`, so
719+
the download holds the index of the pages it is published with; that is also why it
720+
runs after `symbolIndex` and why `checkReport` waits for it.
721+
- **The path is in the online tree from `dispatch` on.** It goes into the list
722+
`deriveTreeRels` takes (`onlineOnlyRels`), so the link check resolves the page's link to
723+
it and the publish sweep sees it. If the task stops writing,
724+
`--check-audit-index` reports the file as indexed but not on disk.
725+
- **The pack is `impexp.mjs`'s `importProject`, in process.** `builder/` may not import
726+
`scripts/` (`biome.jsonc`), because `check_tree_fresh.mjs` would not see a change to
727+
what the build imports. `impexp.mjs` is the one exception, by name in the rule: it
728+
imports nothing, the build already publishes it through `bundle_extra`, and
729+
`check_tree_fresh.mjs` now watches it and `add-in/`. Not the compiler executable,
730+
which CI does not have, and not `impexp.py`, which CI's Node job need not have.
731+
- **`--serve` writes it, `--dry-run` does not.** The preview has the page that links to
732+
it, and packing takes about 100 ms; the help archive is skipped under `--serve` because
733+
it takes seconds and writes into `add-in/`, which this writes nowhere near. The `--serve`
734+
download holds no archive.
735+
- **Deterministic by construction.** `impexp` writes no time, numbers every revision with
736+
a constant and sorts every folder by code point; `git ls-files` is sorted too.
737+
`test/addin-project.test.mjs` packs the real `add-in/` and fixture repositories:
738+
exactly the tracked files, no `Resources/HELP/` file from the disk, the given archive
739+
byte for byte (and none when none is given), a CRLF and an LF checkout giving the
740+
same bytes, and the file in exactly the trees it is given; and `checkHelpArchive`
741+
against small trees.
742+
645743
### Build-time counts as named values
646744

647745
`{{tbdocs:pages}}` in a page renders as the number of pages the build

0 commit comments

Comments
 (0)