raw.json

{"summary":"The subset defines strong fail-closed scanner, artifact, credential, and verification contracts, but its advertised CI entry point permits Bash startup-code execution before its trust-boundary cleanup. Cypress artifact retrieval also leaves host/HOME handling weaker than the Playwright equivalent, and its media-download guidance does not match the helper's publication path.","scores":{"semantic_correctness":92,"false_positive_control":95,"security_trust_boundaries":74,"verification_design":87,"scope_contract_consistency":86,"docs_usability":88},"findings":[{"severity":"H","category":"security_trust_boundaries","file":"scripts/ci/ci-local.sh","line":1,"title":"CI gate processes Bash startup hooks before cleanup","evidence":"The entry point uses `#!/bin/bash` rather than privileged mode. For the documented `bash scripts/ci/ci-local.sh` invocation, noninteractive Bash can process BASH_ENV and imported functions before lines 9-14 reset PATH, unset BASH_ENV, and remove functions, so untrusted startup code executes before the claimed command trust boundary is established.","recommendation":"Enter through `/bin/bash -p` and reject non-privileged execution, or use a minimal privileged wrapper that sanitizes the environment before loading this script; update the documented invocation accordingly."},{"severity":"M","category":"security_trust_boundaries","file":"skills/cypress-debugger/scripts/download-cypress-reports.py","line":179,"title":"Cypress downloader does not harden HOME like the Playwright downloader","evidence":"`gh_environment()` begins by accepting the inherited HOME and later forwards it unchanged, without canonicalizing it or rejecting a repository-controlled location. The included Playwright downloader performs both checks before credential-bearing `gh` calls.","recommendation":"Require an absolute HOME, canonicalize it, reject a value resolving inside the workspace, and pass only that canonical value to `gh`."},{"severity":"M","category":"security_trust_boundaries","file":"skills/cypress-debugger/scripts/download-cypress-reports.py","line":334,"title":"Cypress GitHub API calls do not pin the hostname","evidence":"The bounded `gh api` invocation supplies method and endpoint but no `--hostname github.com`; the Playwright helper explicitly pins every API call to github.com. Consequently repository or gh configuration participates in host selection for a credential-bearing operation.","recommendation":"Define a fixed GitHub hostname and add `--hostname github.com` to run-metadata, artifact-listing, and artifact-download calls."},{"severity":"M","category":"scope_contract_consistency","file":"skills/cypress-debugger/SKILL.md","line":115,"title":"CI download guidance overstates media availability","evidence":"The guidance says the fixed CI artifact download supplies screenshots and videos for Phase 3, but the included helper publishes the entire archive only as `cypress/reports`, while Phase 3 requires media under sibling roots `cypress/screenshots` and `cypress/videos`. Safe ZIP validation also prevents archive traversal into those siblings.","recommendation":"Either narrow the guidance to report files and document a separate validated media-artifact path, or change the helper to validate and atomically publish an explicitly structured parent artifact containing reports, screenshots, and videos at their documented roots."}],"limitations":["Packet-only static review; no commands, tests, scanners, parsers, or framework probes were executed.","Omitted holdouts, benchmark reports, scorecards, repository history, and excluded implementation files were not used or assessed.","This subset review does not establish full product coverage, runtime success, skill accuracy, independent ground truth, or remote model attestation."],"verdict":"FAIL"}