|
| 1 | +# End-to-end (E2E) compatibility test |
| 2 | + |
| 3 | +This document explains the E2E CI added for the Flutter plugin. |
| 4 | + |
| 5 | +## Why |
| 6 | + |
| 7 | +Issue #680 reported that after adding `Invoke-Expression "$(vfox activate pwsh)"` to |
| 8 | +the PowerShell profile, a subsequent `vfox use --global flutter@<version>` stopped |
| 9 | +putting `flutter` on `PATH`. That is a *runtime* interaction between vfox's shell |
| 10 | +hook and a plugin's `EnvKeys`/`PreInstall` output, so no amount of offline Lua unit |
| 11 | +testing (see `tests/hooks_test.lua`) could catch it. The E2E suite therefore drives a |
| 12 | +real vfox binary, loads the real plugin from the working tree, installs a real Flutter |
| 13 | +SDK, and reproduces the failing flow in a fresh shell. |
| 14 | + |
| 15 | +## What runs |
| 16 | + |
| 17 | +The workflow [`.github/workflows/e2e.yml`](../.github/workflows/e2e.yml) builds one |
| 18 | +container image per OS and, inside it, exercises both vfox variants: |
| 19 | + |
| 20 | +| OS runner | Container | Script | vfox variants | |
| 21 | +|-----------|-----------|--------|---------------| |
| 22 | +| `ubuntu-latest` (`ubuntu:26.04`) | Linux container | `tests/e2e/run.sh` (bash) | `latest` + `main` | |
| 23 | +| `windows-2025` (`servercore:ltsc2025`) | Windows container | `tests/e2e/run.ps1` (PowerShell 7) | `latest` + `main` | |
| 24 | + |
| 25 | +The image is a **pinned toolchain only**: |
| 26 | +- `Dockerfile` — `ubuntu:26.04` + `curl`, `git`, `zip`, `unzip`, `xz-utils`. |
| 27 | +- `Dockerfile.windows` — `mcr.microsoft.com/windows/servercore:ltsc2025`, plus a build-time |
| 28 | + install of PowerShell 7 (`PowerShell-<ver>-win-x64.zip`, pinned via `ARG PWSH_VERSION`, |
| 29 | + default `7.6.6`) and a minimal Git for Windows (`MinGit`, resolved from the |
| 30 | + git-for-windows `releases/latest` API — its asset names carry a patch suffix that is not |
| 31 | + derivable, so the API `browser_download_url` is used rather than a hand-built URL). |
| 32 | + |
| 33 | +**Go is intentionally *not* in either image.** The scripts install it on demand, and only |
| 34 | +for the `vfox main` leg (`ensure_go` / `Ensure-Go`): they resolve the current stable from |
| 35 | +`https://go.dev/VERSION?m=text` (falling back to a pinned `GO_FALLBACK`/`$GoFallback`) and |
| 36 | +extract it into the scratch dir. The `latest` leg never needs a compiler, so it stays lean. |
| 37 | + |
| 38 | +vfox itself, the plugin, and the Flutter SDK are all installed at **runtime** by the |
| 39 | +scripts, so the `latest`/`main` pair shares one image and always reflects the working-tree |
| 40 | +plugin. |
| 41 | + |
| 42 | +`vfox latest` is the newest published release binary; `vfox main` is built from the |
| 43 | +tip of <https://github.com/version-fox/vfox> so the plugin is also validated against |
| 44 | +unreleased vfox changes. |
| 45 | + |
| 46 | +The workflow runs on every `pull_request` (no path filter, no scheduled or manual |
| 47 | +trigger). Because it downloads a real Flutter SDK and builds a container image per OS, |
| 48 | +each leg is a few minutes and roughly a gigabyte; `concurrency` cancels superseded runs |
| 49 | +on the same PR. `actions/checkout` is pinned to a full commit SHA |
| 50 | +(`3d3c42e5aac5ba805825da76410c181273ba90b1` = v7), matching `test.yml`. |
| 51 | + |
| 52 | +## Output and failure handling |
| 53 | + |
| 54 | +The scripts are **silent on success**: every external command (vfox, `git`, `curl`, the |
| 55 | +Flutter download, and each fresh-shell child) has its stdout/stderr appended to a scratch |
| 56 | +`e2e.log`, and nothing is written to the console unless a check fails. On the first failure |
| 57 | +the run aborts immediately and prints the failing command plus the last 200 lines of |
| 58 | +`e2e.log`, which keeps a real error legible without drowning the run in progress bars. The |
| 59 | +one intentional non-fatal line is the `WARN:` emitted when the #680 ordering check fails on |
| 60 | +a non-strict (latest) leg — see below. |
| 61 | + |
| 62 | +## The two checks |
| 63 | + |
| 64 | +Each script provisions the plugin and a concrete Flutter version, then runs two |
| 65 | +activated-shell checks. |
| 66 | + |
| 67 | +1. **`verify_session_ordering` — the literal #680 flow.** A fresh shell runs |
| 68 | + `vfox activate` *first* (global scope is still empty), then `vfox use --global` |
| 69 | + *in that same session*, then a PATH refresh, and finally requires `flutter`/`dart` |
| 70 | + to resolve and run. If vfox regresses so that `use --global` does not take effect |
| 71 | + after activation, this check catches it. |
| 72 | + |
| 73 | +2. **`verify_persistence` — the "day-2" flow.** Global selection is already set, so a |
| 74 | + brand-new shell that only runs `vfox activate` must expose the SDK with no further |
| 75 | + commands. This is the plain plugin↔vfox `EnvKeys`/global-scope compatibility gate |
| 76 | + and is enforced on every leg. |
| 77 | + |
| 78 | +Ordering matters: the ordering check runs first, while global state is clean, so the |
| 79 | +"not present before `use`" transition is observable; it then leaves the global |
| 80 | +selection in place for the persistence check. |
| 81 | + |
| 82 | +### `STRICT_SESSION` |
| 83 | + |
| 84 | +The same-session refresh bug from #680 was reported against **PowerShell on Windows**; |
| 85 | +the released vfox on Linux reproduces it as a pass. To keep the job meaningful without |
| 86 | +being permanently red on a vfox-side issue that is not yet released, the scripts treat |
| 87 | +the ordering check as: |
| 88 | + |
| 89 | +- a **hard failure** when `STRICT_SESSION=true` (used for the `vfox main` leg, where |
| 90 | + any fix must land), and |
| 91 | +- a **warning** otherwise (the `vfox latest` leg), so it reports the state of the |
| 92 | + released binary without blocking the plugin. |
| 93 | + |
| 94 | +`verify_persistence` is always a hard gate regardless. |
| 95 | + |
| 96 | +## How the reproduction stays deterministic |
| 97 | + |
| 98 | +Two vfox behaviors make a naïve script flaky in CI; both are handled explicitly: |
| 99 | + |
| 100 | +- **`vfox use` requires the activate hook.** vfox 1.x refuses `vfox use` unless the |
| 101 | + current shell has run `vfox activate` (it prints |
| 102 | + "vfox requires hook support..."). So every `use` happens *inside* an activated |
| 103 | + child shell — never in the harness process itself. |
| 104 | +- **The hook refreshes PATH from the interactive prompt.** `vfox activate` installs a |
| 105 | + `PROMPT_COMMAND`/`prompt` hook that calls `vfox env -s <shell>` on each prompt. A |
| 106 | + non-interactive CI shell never displays a prompt, so the scripts reproduce exactly |
| 107 | + one refresh by manually running `eval "$("vfox" env -s bash)"` |
| 108 | + (`Invoke-Expression (& vfox env -s pwsh)`) after `use --global`. This is the same |
| 109 | + command the next interactive prompt would run. |
| 110 | + |
| 111 | +## Environment variables |
| 112 | + |
| 113 | +The scripts are parameterised through env vars (consumed by the workflow): |
| 114 | + |
| 115 | +| Variable | Values | Default | Meaning | |
| 116 | +|----------|--------|---------|---------| |
| 117 | +| `VFOX_VERSION` | `latest` \| `main` | `latest` | Which vfox runtime to install. | |
| 118 | +| `FLUTTER_VERSION` | concrete release, e.g. `3.44.0` | `3.44.0` | Exact Flutter SDK to install (a concrete version, not a channel, so `use` and the installed dir match). | |
| 119 | +| `STRICT_SESSION` | `true` \| `false` | `false` | Promote the #680 ordering check to a hard failure. | |
| 120 | +| `E2E_WORK_DIR` | path | temp dir | Scratch dir for the vfox binary, plugin zip and caches. | |
| 121 | + |
| 122 | +`FLUTTER_VERSION` defaults to a real stable release referenced in `README.md`; bump the |
| 123 | +`FLUTTER_VERSION` env in `.github/workflows/e2e.yml` if that version is ever retired. |
| 124 | + |
| 125 | +## Running locally |
| 126 | + |
| 127 | +Linux container: |
| 128 | + |
| 129 | +```bash |
| 130 | +docker build -f tests/e2e/Dockerfile -t vfox-flutter-e2e:linux . |
| 131 | +docker run --rm -e VFOX_VERSION=main -e STRICT_SESSION=true vfox-flutter-e2e:linux |
| 132 | +``` |
| 133 | + |
| 134 | +Windows container (PowerShell, on a Windows host with the Docker daemon in Windows-container mode): |
| 135 | + |
| 136 | +```powershell |
| 137 | +docker build -f tests/e2e/Dockerfile.windows -t vfox-flutter-e2e:windows . |
| 138 | +docker run --rm -e VFOX_VERSION=main -e STRICT_SESSION=true vfox-flutter-e2e:windows |
| 139 | +``` |
| 140 | + |
| 141 | +Each run downloads a full Flutter SDK and lets `flutter --version` bootstrap the Dart |
| 142 | +tool, so expect a few minutes and roughly a gigabyte of download per leg. |
| 143 | + |
| 144 | +## Windows container caveats |
| 145 | + |
| 146 | +- The base image and the runner's OS must match, so the job uses `windows-2025` and |
| 147 | + `Dockerfile.windows` pins `mcr.microsoft.com/windows/servercore:ltsc2025`. (Microsoft |
| 148 | + publishes no `powershell` image for `ltsc2025`, hence PowerShell 7 is layered on top of |
| 149 | + the plain server-core base at build time.) |
| 150 | +- PowerShell 7 is required (the script uses `Invoke-WebRequest -SkipHttpErrorCheck` and |
| 151 | + runs its fresh-shell children with `pwsh`); the build-time install supplies `pwsh.exe`. |
| 152 | +- The runner's Docker daemon is switched to Windows containers by a best-effort step; on a |
| 153 | + native Windows-container engine that step is a harmless no-op. |
| 154 | +- This leg is validated only by a real `windows-2025` run; the Linux leg is fully |
| 155 | + reproducible locally with Docker. |
0 commit comments