CI #701
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # This workflow will build a .NET project | |
| # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-net | |
| name: CI | |
| on: | |
| push: | |
| branches: [ "master" ] | |
| pull_request: | |
| branches: [ "master" ] | |
| workflow_dispatch: | |
| schedule: | |
| # Nightly 06:00 UTC: the heavy tiers (Docker / browser / LLM / MCP / perf). | |
| - cron: '0 6 * * *' | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v6 | |
| - name: Setup .NET | |
| uses: actions/setup-dotnet@v5 | |
| with: | |
| dotnet-version: 10.0.x | |
| - name: Restore dependencies | |
| run: dotnet restore | |
| - name: Build | |
| run: dotnet build --no-restore | |
| - name: Test | |
| run: dotnet test WebReaper.Tests/WebReaper.UnitTests/WebReaper.UnitTests.csproj --no-build --verbosity normal | |
| # ADR-0022: distributed-adapter coverage. ubuntu-latest ships Docker, so | |
| # Testcontainers spins an ephemeral Redis (free on public-repo runners — | |
| # unlimited Actions minutes). Scoped to ONLY RedisDistributedAdapterTests: | |
| # the slow, network-flaky live-site tests in the same project stay out of | |
| # the gate. | |
| - name: Integration tests (Redis adapters via Testcontainers) | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --verbosity normal --filter "FullyQualifiedName~RedisDistributedAdapterTests" | |
| # Deterministic integration tier: the in-process Kestrel site (extraction, | |
| # crawl/follow, retry, SQLite adapter) — no Docker, browser, keys, or | |
| # network. Fast and stable enough for the gate. Excludes Container/Browser/ | |
| # Llm/Mcp/Perf/LiveSite by trait. | |
| - name: Integration tests (LocalServer tier) | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --verbosity normal --filter "Category=LocalServer" | |
| # CLI end-to-end: the real CLI binary as a subprocess against the local | |
| # site (scrape md/json, map, version, exit codes). | |
| - name: CLI end-to-end tests | |
| run: dotnet test WebReaper.Tests/WebReaper.Cli.Tests/WebReaper.Cli.Tests.csproj --no-build --verbosity normal --filter "Category=Cli" | |
| # ADR 0008 step 4: the AOT smoke test. Publishes the Newtonsoft-free | |
| # production path with PublishAot; the project scopes the IL2026/IL3050 | |
| # family to build errors, so any trim/AOT regression fails CI here. Native | |
| # AOT on Linux needs the clang toolchain + zlib headers. | |
| - name: Install Native AOT prerequisites | |
| run: sudo apt-get update && sudo apt-get install -y clang zlib1g-dev | |
| - name: AOT smoke test (Newtonsoft-free path) | |
| run: | | |
| dotnet publish WebReaper.Tests/WebReaper.AotSmokeTest/WebReaper.AotSmokeTest.csproj -c Release -r linux-x64 | |
| ./WebReaper.Tests/WebReaper.AotSmokeTest/bin/Release/net10.0/linux-x64/publish/WebReaper.AotSmokeTest | |
| # ADR-0084: the WebReaper.AI satellite is baked into the AOT CLI, so it must | |
| # publish clean under Native-AOT with the same IL warning set promoted to | |
| # errors. Roots both LLM JSON paths (the extractor and the tool-call args) | |
| # via a stub chat client; no network. | |
| - name: AOT smoke test (WebReaper.AI satellite, ADR-0084) | |
| run: | | |
| dotnet publish WebReaper.Tests/WebReaper.AI.AotSmokeTest/WebReaper.AI.AotSmokeTest.csproj -c Release -r linux-x64 | |
| ./WebReaper.Tests/WebReaper.AI.AotSmokeTest/bin/Release/net10.0/linux-x64/publish/WebReaper.AI.AotSmokeTest | |
| # ADR-0084 piece 2: the OpenAI-compatible IChatClient must also publish | |
| # clean under Native-AOT (raw HttpClient + System.Text.Json source-gen). | |
| # Exercises a request serialize + response deserialize against a stub | |
| # handler; no network. | |
| - name: AOT smoke test (WebReaper.AI.Http client, ADR-0084) | |
| run: | | |
| dotnet publish WebReaper.Tests/WebReaper.AI.Http.AotSmokeTest/WebReaper.AI.Http.AotSmokeTest.csproj -c Release -r linux-x64 | |
| ./WebReaper.Tests/WebReaper.AI.Http.AotSmokeTest/bin/Release/net10.0/linux-x64/publish/WebReaper.AI.Http.AotSmokeTest | |
| # ADR-0043: the CLI is the headline AOT artifact ("the single-binary | |
| # publish is the headline AOT artifact"). Its csproj promotes the same IL | |
| # warning set to errors as WebReaper.AotSmokeTest, so a Native-AOT publish | |
| # is the only way to catch a regression in the *bundled* graph (CLI + | |
| # WebReaper core) before a real consumer hits it on `dotnet tool install` | |
| # day. The smoke test above covers core in isolation; this covers the | |
| # composed AOT publish. | |
| - name: AOT publish — WebReaper.Cli (the headline single-binary) | |
| run: | | |
| dotnet publish WebReaper.Cli/WebReaper.Cli.csproj -c Release -r linux-x64 | |
| ls -lh WebReaper.Cli/bin/Release/net10.0/linux-x64/publish/WebReaper.Cli | |
| # ADR-0055 invariant smoke: every CLI subcommand's bundled-graph | |
| # path is exercised against the AOT'd binary. A regression that | |
| # only manifests post-AOT (e.g. a reflection-driven JSON case | |
| # surviving the static `dotnet build` but failing the AOT publish | |
| # graph) surfaces here. | |
| WEBREAPER=./WebReaper.Cli/bin/Release/net10.0/linux-x64/publish/WebReaper.Cli | |
| $WEBREAPER version | |
| $WEBREAPER help | |
| $WEBREAPER browser list | |
| $WEBREAPER stealth list | |
| # Heavy tiers kept out of the per-PR gate: Docker (Testcontainers), real | |
| # browsers, the MCP subprocess, the 500-page scale test, and the real-LLM | |
| # tests. Nightly + manual (workflow_dispatch) only. | |
| extended: | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v6 | |
| - name: Setup .NET | |
| uses: actions/setup-dotnet@v5 | |
| with: | |
| dotnet-version: 10.0.x | |
| - run: dotnet restore | |
| - run: dotnet build --no-restore | |
| # Testcontainers spins ephemeral Redis + MongoDB (Docker ships on the runner). | |
| - name: Container tier (Redis + Mongo sinks/schedulers) | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --filter "Category=Container" | |
| # Playwright needs its matching browser revision; the runner has pwsh. | |
| - name: Install Playwright browsers | |
| run: pwsh WebReaper.Tests/WebReaper.IntegrationTests/bin/Debug/net10.0/playwright.ps1 install chromium | |
| - name: Browser tier (Playwright + CDP render JS) | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --filter "Category=Browser" | |
| - name: MCP tier (scrape/map/extract over stdio) | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --filter "Category=Mcp" | |
| - name: Perf scale tier (500-page exact-count) | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --filter "Category=Perf" | |
| # LLM tier: passes vacuously when the secret is absent (the test's own | |
| # no-key guard), so this is safe even on forks without the secret. | |
| - name: LLM tier (real key if present, else vacuous) | |
| env: | |
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | |
| OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | |
| run: dotnet test WebReaper.Tests/WebReaper.IntegrationTests/WebReaper.IntegrationTests.csproj --no-build --filter "Category=Llm" |