test(e2e): multi-worker server + robust compare wait — kills full-suite flakiness (#221)

The full e2e suite (e2e-full.yml, self-hosted with models) ran the 13 Playwright
tests against the single-worker demo app, so back-to-back heavy requests queued and
timed out. Fixes:
- e2e-full.yml starts a dedicated 4-worker uvicorn on :8011 (isolated from the live
  :8000 demo), waits for readiness, runs the suite, and tears it down.
- test_compare_loads_and_loaders_clear waited for the FIRST card image then a fixed
  500ms before asserting BOTH images visible; the second card lags under load. Now
  polls for two non-loading cards with images (no fixed sleep).

Validated locally against a 4-worker server: 13/13 passed, twice, ~30s. Closes #212.

Signed-off-by: Austin L. <86896075+rvnminers-A-and-N@users.noreply.github.com>

Author: rvnminers-A-and-N

.github/workflows/e2e-full.yml

@@ -25,7 +25,22 @@ jobs:
         run: |
           python -m pip install --quiet pytest playwright
           python -m playwright install chromium || true
-      - name: Full suite against the running app (all models loaded)
+      - name: Start a dedicated multi-worker test server
+        # A dedicated 4-worker server (the box is 32-core) so the 13 back-to-back Playwright
+        # tests don't queue behind a single worker and time out. Isolated on :8011 so it never
+        # disturbs the live demo app on :8000. Each worker loads its own copy of the models.
+        working-directory: training
+        run: |
+          nohup python -m uvicorn app:app --host 127.0.0.1 --port 8011 --workers 4 \
+            > /tmp/flavor-e2e-app.log 2>&1 &
+          for i in $(seq 1 60); do
+            curl -sf http://127.0.0.1:8011/api/studio_terms >/dev/null && { echo "app up"; break; }
+            sleep 2
+          done
+      - name: Full suite against the multi-worker server (all models loaded)
         env:
-          FLAVORMANCER_URL: http://127.0.0.1:8000
+          FLAVORMANCER_URL: http://127.0.0.1:8011
         run: pytest tests/e2e -q
+      - name: Stop the test server
+        if: always()
+        run: pkill -f "uvicorn app:app.*8011" || true

tests/e2e/test_workbench_e2e.py

@@ -71,8 +71,11 @@ def test_formulation_studio_starter(page, base_url, needs_models):
 def test_compare_loads_and_loaders_clear(page, base_url, needs_models):
     _open(page, base_url)
     page.locator(".cmp-ex", has_text="vanillin vs").click()
-    page.wait_for_selector("#cmpResults .cmp-card:not(.loading) img", timeout=30000)
-    page.wait_for_timeout(500)
+    # wait for BOTH comparison cards to finish loading (not just the first) — the second can lag
+    # under load, so poll for the pair of non-loading cards with images rather than a fixed sleep
+    page.wait_for_function(
+        "document.querySelectorAll('#cmpResults .cmp-card:not(.loading) img').length === 2",
+        timeout=30000)
     assert page.locator("#cmpResults .cmp-card img:visible").count() == 2
     assert page.locator("#cmpResults .cmp-load:visible").count() == 0, "loaders must clear after load"