Skip to content

test(webapi): harden runtime and release validation [Codex] - #397

Draft
jderochervlk wants to merge 11 commits into
mainfrom
agent/alpha-tooling-release
Draft

jderochervlk wants to merge 11 commits into
mainfrom
agent/alpha-tooling-release

Conversation

@jderochervlk

@jderochervlk jderochervlk commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

The alpha readiness review found that runtime fixtures could silently remain compile-only, the generated-JavaScript drift check inspected ignored outputs, the rescript@>=13 peer range excluded the intentionally supported ReScript 13 alpha, and release versions were generated from commit hashes rather than a numbered alpha series.

This branch adds explicit compile-only, documentation-example, Node-runtime, and Chromium-runtime fixture categories with registration checks; meaningful assertions for the existing Node-compatible Fetch and URL fixtures; and a native Chromium runner that imports compiled ReScript modules and reports fixture-specific async failures. Negative consumer compilations verify intended diagnostics rather than accepting arbitrary compiler failures.

The packed-consumer check installs the actual npm tarball with ReScript 13.0.0-alpha.5, compiles default, Fetch, and SafeFetch consumers, runs a Node smoke test, verifies rejection of unsupported compiler versions and Fetch-only SafeFetch, and checks package contents. The release machinery uses package.json/package-lock.json version 1.0.0-alpha.1, validates the explicit alpha version and registry availability, exercises the same packed artifact, and separates no-OIDC validation/dry-run from a main-only publication job. Only that publish job receives id-token: write; it verifies the downloaded artifact name/version and publishes with the alpha dist-tag.

Closes #384.
Closes #396.
Refs #383 and #385. The binding-specific Chromium and negative type-contract fixtures are being added in the stacked bindings PR. The npm trusted-publisher account configuration cannot be confirmed from repository access; #385 remains release-gated until a maintainer verifies owner rescript-lang, repository experimental-rescript-webapi, workflow filename release.yml, environment unset, and that direct npm publish is allowed. This PR does not publish or merge.

@jderochervlk

Copy link
Copy Markdown
Collaborator Author

Independent review of exact head 77b59836d6ef1838795f78496f7dd89cd98816de against live issue bodies #383/#384/#385/#396 (base ebe9cca601fb85e6064c786367e546ded9b07d05). No source edits or approval submitted.

One P2 finding: tests/browser/native-api.spec.mjs:24–28: the failure formatter can itself throw. In Chromium, injecting an awaited module rejection shaped as {RE_EXN_ID: "Review_bigint", _1: 1n, Error: new Error("original")} produces only page.evaluate: TypeError: Do not know how to serialize a BigInt; a cyclic payload similarly produces the circular-JSON error. These replace the original exception and lose fixture/source-stack diagnostics. This is not a false pass, but violates the requested robust failure reporting. Use a nonthrowing formatter that always retains fixture, exception identity and original stack; cover these cases. Ordinary async ReScript Assert_failure diagnostics work.

Independent evidence: fresh npm ci, formatting, build, feature checks, packed consumers, negative consumers, Node tests, Chromium tests, docs build, release check, full/prod audits, and npm publish <same consumer-validated tarball> --dry-run --access public --tag alpha all passed. Artifact SHA-256 was unchanged. Local toolchain: Node 24.16.0/npm 11.15.0; GitHub CI and Release also passed on the configured Node 26 (release pins npm 11.5.1), with actual publication skipped. In-memory probes confirmed Node assertion and registration-omission failures, empty browser-registry failure, useful ordinary async assertion diagnostics, and main/event/published-version/fail-closed registry guards.

Acceptance still open: #383 needs the formatter fix above; binding-specific native/unsafe/nullable cases are intentionally stacked, not missing foundation dependencies. #384 installation/artifact behavior passes, but supported/tested compiler-version prose is absent from this head. #385 machinery passes inspection/execution, but release/tag/release-note/changelog/dist-tag-verification instructions are absent here, and external npm configuration remains unverified: owner rescript-lang, repository experimental-rescript-webapi, workflow release.yml, environment unset, direct npm publish allowed. Keep #385 open. Documentation gaps can be owned by the docs stack; the intentionally deferred docs-link gate is not a foundation defect. #396 passes: narrow patched overrides, clean install, zero full/prod audit findings, no added runtime dependencies.

@jderochervlk

Copy link
Copy Markdown
Collaborator Author

Independent agent follow-up review — exact head 1a6aceb398a22d2a38ebf4bc6a73ba092b302e45

Both P2 findings from the review at 858d87f remain resolved. No new concrete code defect found. This is a separate follow-up verdict, not a relabeling of the completed review of e6d5adf42e6e7e8b88b05d01a367f725cf640291. Base main remains ebe9cca601fb85e6064c786367e546ded9b07d05.

Reviewed delta

Checked the immutable GitHub compare and local Git trees: e6d5adf…1a6aceb contains exactly one commit and one added line, in the expected checkout fixture: publishConfig: { access: "public", provenance: true }. Production code, workflow, package metadata/lockfile, docs, consumers, and browser harness are byte-for-byte unchanged. The malformed packed-config cases now explicitly run against valid checkout configuration; the helper must still inspect the packed manifest.

Previous findings

  1. Lifecycle hooks under OIDC — resolved. The publish job directly invokes node scripts/check-release.mjs --publish, so candidate precheck:release / postcheck:release hooks are not run. Real publication and validation dry-run both use --ignore-scripts. No candidate dependency installation/build/pack/test/npm-script command executes in the privileged job; the global npm installation only bootstraps the toolchain. The no-OIDC validation job and needs: validate + main-only/manual-dispatch/non-dry-run publication guard remain intact.
  2. Checkout-versus-packed configuration — resolved. The workflow verifier extracts package/package.json from the downloaded tarball and passes packed to the helper. Only public access and boolean provenance are allowed in publishConfig; every other key, including registry/scoped-registry/tag overrides and latest, is rejected before publication. Lifecycle scripts are rejected too. The default download is scoped to the same workflow run's named artifact; the verified archive path is the published path.

Evidence and limits

Rerun on an isolated archive of exact 1a6aceb…, Node 26.3.0 / npm 11.5.1: node --test scripts/validate-release-artifact.test.mjs (6/6), npm run check:release (6/6 plus successful live unpublished-version check), and git diff --check all passed. GitHub CI for this head succeeded: https://github.com/rescript-lang/experimental-rescript-webapi/actions/runs/42323173253 . Release validation was still in progress at inspection: https://github.com/rescript-lang/experimental-rescript-webapi/actions/runs/42323173275 .

Carried forward transparently from the unchanged-code e6d5adf review, not rerun here: fresh install, formatting, build, all 28 feature checks, packed supported/unsupported compiler and default/Fetch/SafeFetch scenarios, negative consumers, Node runtime, Chromium, docs, full/prod audits (zero vulnerabilities), exact archive consumer validation and dry-run, actual workflow-verifier malformed-archive probes, direct-guard lifecycle-hook sentinel probes, and harness failure-path probes. That earlier review used Node 26.3.0 / npm 11.5.1 too. No need to repeat those unrelated checks for this test-only addition.

Acceptance gaps unchanged

No source edits, merge, actual publication, account/settings changes, issue closure, or formal approval performed. This same-account agent comment is not an independent maintainer approval or a release-ready claim.

@jderochervlk

Copy link
Copy Markdown
Collaborator Author

Independent agent follow-up — exact head 6fa3016a58700c1986acc705f7ff851efc54709e

Both previous P2 release findings remain resolved; no new concrete code defect found. This verdict separately reviews the delta after 1a6aceb398a22d2a38ebf4bc6a73ba092b302e45; it does not silently relabel earlier reviews.

The immutable GitHub comparison and local Git trees confirm one commit modifying only package.json's two formatter scripts. Each replaces the three explicit current browser .mjs paths with ./tests/browser/*.mjs. Parsed-package comparison confirmed no other metadata or script changes. The glob still includes all three current modules and automatically includes additional top-level browser helpers; it does not require importing stacked binding fixtures into this foundation PR. No source/workflow/dependency change accompanies it.

Rerun on an isolated archive of exact 6fa3016…, Node 26.3.0 / npm 11.5.1:

  • Fresh npm ci passed (installation audit reported zero vulnerabilities).
  • Exact npm run format:check passed.
  • Exact npm run check:release passed: six validator/guard tests plus the live unpublished-version check.
  • git diff --check passed.
  • A scratch-only additional malformed browser-helper .mjs made the exact npm run format:check fail and identify that helper, confirming the intended newly included coverage. The probe was removed; the worktree was not edited.

All release workflow/validator and runtime/consumer/harness code is unchanged from the prior reviewed SHAs. Their earlier checks are carried forward, not rerun here: direct Node guard avoids candidate pre/postcheck hooks; publish/dry-run suppress lifecycle scripts; no candidate installation/build/test runs in the privileged job; main/manual/non-dry-run gating and no-OIDC validation remain intact; actual packed-manifest validation rejects registry/tag overrides before publishing the same-run archive. Earlier exhaustive build/feature/consumer/Node/Chromium/docs/audit/exact-artifact dry-run evidence remains applicable to those unchanged paths. GitHub CI and Release runs for this new SHA were still in progress at inspection, not claimed successful.

Unchanged acceptance boundaries: #384/#396 remain supported by the prior evidence. #383 binding-specific native/unsafe-widening/nullable-contract regressions remain intentionally stacked, not demonstrated by the current URL smoke/generic negative fixture. #385 release/tag/release-notes/changelog/configuration/manual-dist-tag runbook remains absent (docs-stack gap). External npm trusted-publisher configuration remains unverified: owner rescript-lang, repo experimental-rescript-webapi, workflow filename release.yml, environment unset, direct npm publish allowed; successful provenance-bearing publication remains unproven.

No source edits, merge, actual publication, settings changes, issue closure, or formal approval. This same-account agent comment is not independent maintainer approval or a release-readiness claim.

@jderochervlk

Copy link
Copy Markdown
Collaborator Author

Acceptance status at 6fa3016a58700c1986acc705f7ff851efc54709e:

  • Establish reliable runtime regression tests for the alpha #383 — Harness foundation in this PR: tests/fixtures.json explicitly categorizes compile-only fixtures, documentation examples, Node runtime modules, and browser runtime modules; missing/duplicate registration fails. Existing Node Fetch/URL smoke tests assert behavior, the ignored-generated-JS drift guard is removed, and Chromium imports compiled ReScript modules with fixture-specific async error details. Negative consumer checks require both compilation failure and the expected diagnostic. The binding-specific browser regressions and unsafe/nullable contract fixtures remain in stacked PR fix(webapi): correct native binding contracts [Codex] #399, so Establish reliable runtime regression tests for the alpha #383 is not fully closed by this foundation PR.
  • Accept supported ReScript 13 prereleases and verify tarball installation #384 — Packed compiler consumers: development compiler and peer range are aligned to tested ReScript 13.0.0-alpha.5 (>=13.0.0-alpha.5 <13.0.0). npm run check:consumer installs the real tarball with the compiler, compiles default/Fetch/SafeFetch consumers, runs a Node smoke assertion, checks Fetch-only SafeFetch rejection, rejects unsupported compiler versions with the peer diagnostic, and validates the 343-file artifact allowlist. The exact packed artifact also passed npm publish dry-run with --tag alpha; no package was published.
  • Publish explicit numbered v1.0.0 alpha versions safely #385 — Release machinery: package/lock/docs metadata use 1.0.0-alpha.1; CI validates and dry-runs the same artifact, rejects already-published versions, and gates real publication to workflow dispatch on main with the alpha tag. Validation has no OIDC permission; only the guarded publish job has id-token: write, verifies the downloaded tarball metadata, and uses --ignore-scripts. Release/tag/changelog operator instructions are being added by the stacked docs PR. npm Trusted Publishing account configuration remains unverified (owner rescript-lang, repo experimental-rescript-webapi, workflow filename release.yml, no environment, direct npm publish allowed); no real publish occurred. Publish explicit numbered v1.0.0 alpha versions safely #385 remains open.
  • Update vulnerable documentation-tooling transitive dependencies #396 — Dependency advisories: compatible patched transitive versions are locked; fresh npm ci, npm audit, and npm audit --omit=dev pass with zero vulnerabilities and no added runtime dependency.

Independent review at this exact head found both previously reported P2 release issues resolved and no new code defect: #397 (comment). GitHub CI build and Release validate passed; docs deployment and package publication were skipped as intended. This does not claim the full alpha batch is release-ready; the #383 binding stack, #385 runbook, and account-level trusted-publisher confirmation remain outstanding.

@jderochervlk

jderochervlk commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

Update: local browser isolation has now been implemented in stacked bindings PR #399 at head d938c719b016e439aeeec3da2d185dcd0b4e714f: WEBAPI_TEST_PORT drives server/baseURL, reuseExistingServer is false, and invalid/occupied ports fail closed. Its author reports local default/4192 runs and occupied/invalid-port rejection; GitHub build and Release validate pass at that head. This code is in #399, not foundation #397, and its final independent review remains pending, so #383 acceptance is a stack-level outcome rather than claim that #397 alone includes the fix.

@jderochervlk

Copy link
Copy Markdown
Collaborator Author

Coordinator verification of the current foundation head 779568e28c1caef5554b6ebbe9292fc0e9daa0dd:

  • Its complete Git tree is identical to independently reviewed 6fa3016a58700c1986acc705f7ff851efc54709e: both tree IDs are dfd9e21de46e3c757793692e05abb00ccb12585c, and the GitHub comparison contains zero changed files.
  • The intervening downstream-only browser-port documentation addition was restored out of this foundation. That guidance will accompany the actual isolation feature in fix(webapi): correct native binding contracts [Codex] #399, so each PR remains accurate on its own.
  • GitHub CI build and Release validate passed on the current 779568e head; deployment and real publication were skipped.

The earlier independent source-review evidence and author acceptance report carry forward by exact tree identity; this is not a claim that every earlier local command was rerun, nor a maintainer approval or landing instruction. #383 still awaits the corrected binding-stack acceptance/review, and #385 still has the docs-runbook and external npm configuration gates. No merge, actual publication, settings change, or manual issue closure was performed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update vulnerable documentation-tooling transitive dependencies Accept supported ReScript 13 prereleases and verify tarball installation

1 participant