Expand Cairo audit corpus with public reports #381
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Quality Gate | |
| on: | |
| pull_request: | |
| push: | |
| branches: | |
| - main | |
| jobs: | |
| quality: | |
| permissions: | |
| contents: read | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 | |
| - name: Setup Python | |
| uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | |
| with: | |
| python-version: "3.12" | |
| - name: Setup Ruff | |
| uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1 (v3) | |
| - name: Install Python deps | |
| run: python -m pip install -r requirements.txt | |
| - name: Python compile check | |
| run: python -m py_compile scripts/audit-pipeline/*.py scripts/quality/*.py scripts/site/*.py | |
| - name: Validate SKILL contract | |
| run: python scripts/quality/validate_skills.py | |
| - name: Validate plugin marketplace metadata | |
| run: python scripts/quality/validate_marketplace.py | |
| - name: Ruff lint | |
| run: ruff check scripts/audit-pipeline scripts/audit-extraction scripts/quality scripts/site | |
| - name: Shellcheck | |
| run: shellcheck scripts/audit-extraction/*.sh | |
| - name: Validate normalized audit metadata JSON | |
| run: | | |
| python scripts/audit-pipeline/validate_json.py \ | |
| --schema datasets/normalized/audit.schema.json \ | |
| --glob 'datasets/normalized/audits/*.json' | |
| - name: Validate findings JSONL | |
| run: | | |
| for file in datasets/normalized/findings/*.jsonl; do | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema datasets/normalized/finding.schema.json \ | |
| --jsonl "$file" | |
| done | |
| - name: Validate benchmark case JSONL | |
| run: | | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/cases/benchmark-case.schema.json \ | |
| --jsonl evals/cases/cairo_auditor_benchmark.jsonl | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/cases/benchmark-case.schema.json \ | |
| --jsonl evals/cases/cairo_auditor_realworld_benchmark.jsonl | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/cases/contract-benchmark-case.schema.json \ | |
| --jsonl evals/cases/contract_skill_benchmark.jsonl | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/cases/contract-generation-case.schema.json \ | |
| --jsonl evals/cases/contract_skill_generation_eval.jsonl | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/cases/benchmark-case.schema.json \ | |
| --jsonl evals/heldout/cairo_auditor_llm_eval_cases.jsonl | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/reports/data/external-triage-label.schema.json \ | |
| --jsonl evals/reports/data/external-repo-scan-low-profile-rerun-2026-03-09-v5.labels.jsonl | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema evals/reports/data/manual-19-gold.schema.json \ | |
| --jsonl evals/reports/data/manual-19-gold.jsonl | |
| - name: Validate vuln-db parity and attack-vector coverage | |
| run: | | |
| python scripts/quality/check_vulndb_parity.py \ | |
| --cases evals/cases/cairo_auditor_benchmark.jsonl \ | |
| --cases evals/cases/cairo_auditor_realworld_benchmark.jsonl | |
| python scripts/quality/check_attack_vector_coverage.py \ | |
| --min-vectors 120 | |
| python scripts/quality/check_semgrep_vector_coverage.py \ | |
| --core-min 1 \ | |
| --core-max 120 | |
| - name: Manual-19 gold recall smoke | |
| run: | | |
| python scripts/quality/check_manual_gold_recall.py \ | |
| --gold evals/reports/data/manual-19-gold.jsonl \ | |
| --findings evals/reports/data/external-repo-scan-low-profile-rerun-2026-03-09-v5.findings.jsonl \ | |
| --output-md /tmp/manual-19-gold-recall.md \ | |
| --output-json /tmp/manual-19-gold-recall.json \ | |
| --min-recall 0.80 \ | |
| --min-class-recall 0.60 | |
| - name: Validate manifest JSONL | |
| run: | | |
| python scripts/audit-pipeline/validate_jsonl.py \ | |
| --schema datasets/manifests/audit-manifest.schema.json \ | |
| --jsonl datasets/manifests/audits.jsonl | |
| python scripts/audit-pipeline/check_unique_ids.py \ | |
| --jsonl datasets/manifests/audits.jsonl \ | |
| --keys audit_id source_url raw_path extracted_path raw_sha256 | |
| - name: Segment smoke tests | |
| run: | | |
| cat > /tmp/segment_audit_a.txt <<'EOF' | |
| L-01 Example finding | |
| File(s): contract.cairo | |
| Description: Example issue. | |
| Recommendation: Example fix. | |
| EOF | |
| cat > /tmp/segment_audit_b.txt <<'EOF' | |
| 1.2.3 Example section | |
| Body line | |
| EOF | |
| python scripts/audit-pipeline/segment_text.py \ | |
| --audit-id smoke_audit_a \ | |
| --input /tmp/segment_audit_a.txt \ | |
| --output /tmp/csc.segmented.jsonl | |
| python scripts/audit-pipeline/segment_text.py \ | |
| --audit-id smoke_audit_b \ | |
| --input /tmp/segment_audit_b.txt \ | |
| --output /tmp/erim.segmented.jsonl | |
| test -s /tmp/csc.segmented.jsonl | |
| test -s /tmp/erim.segmented.jsonl | |
| - name: Generate manifest smoke test | |
| run: | | |
| mkdir -p datasets/audits/raw datasets/audits/extracted /tmp | |
| printf 'dummy-raw' > datasets/audits/raw/smoke_ci.pdf | |
| printf 'dummy-extracted' > datasets/audits/extracted/smoke_ci.txt | |
| cat > /tmp/manifest.seed.json <<'EOF' | |
| [ | |
| { | |
| "audit_id": "smoke_ci_2026_03", | |
| "project": "Smoke", | |
| "auditor": "CI", | |
| "date": "2026-03-08", | |
| "source_url": "https://example.com/smoke.pdf", | |
| "source_type": "direct_pdf", | |
| "raw_path": "datasets/audits/raw/smoke_ci.pdf", | |
| "extracted_path": "datasets/audits/extracted/smoke_ci.txt", | |
| "source_sha256": "1c3ed0f82d0f18c7388e94efe32df725b1ba3d51eca7fc8e5ab2fdbe0fc67bdf", | |
| "license": "unknown", | |
| "usage_rights": "public_reference_only", | |
| "redaction_status": "none", | |
| "extractor_version": "quality-smoke@v1" | |
| } | |
| ] | |
| EOF | |
| python scripts/audit-pipeline/generate_manifest.py \ | |
| --seed /tmp/manifest.seed.json \ | |
| --output /tmp/audits.generated.jsonl | |
| test -s /tmp/audits.generated.jsonl | |
| - name: Held-out leakage check | |
| run: python scripts/audit-pipeline/check_no_heldout_leak.py | |
| - name: Verify website snapshots are fresh | |
| run: | | |
| python scripts/site/build_site.py --domain starkskills.org | |
| git diff --exit-code -- \ | |
| website/index.html \ | |
| website/vuln-cards/index.html \ | |
| website/data/site-data.json \ | |
| website/CNAME |