Add Aspire-hosted Playwright test harness by chabiss · Pull Request #10954 · NuGet/NuGetGallery

chabiss · 2026-08-25T20:54:27Z

Summary

Adds an Aspire-hosted Playwright harness that lets an agent or developer run the supported Gallery browser tests with one command. The test process now owns the AppHost lifecycle, Gallery readiness, functional-data seeding, test configuration, and teardown instead of requiring several manually coordinated scripts.

What changed

Adds Aspire.Hosting.Testing to the existing NuGetGallery.FunctionalTests project and references the Gallery AppHost.
Exposes the Gallery health probe through Aspire and waits for the named gallery-http endpoint before testing.
Adds a shared xUnit collection fixture that starts one AppHost for the Playwright suite, seeds data once, initializes GalleryConfiguration, and restores process state during disposal.
Marks the 10 browser tests supported by the minimal ci-gallery profile with Category=PlaywrightTests.
Keeps statistics-service and read-only-mode browser tests on their existing specialized execution paths.
Adds tests\Scripts\RunGalleryPlaywrightTests.ps1 as the stable local and agent entry point.
Separates Aspire-hosted Playwright execution from the externally hosted P0/P1/P2/Admin API path in GitHub Actions and Azure Pipelines, avoiding fixed-port conflicts.
Makes -TrustDevCert idempotent by reusing an already trusted certificate and reliably deleting temporary certificate/key files when import is required.
Makes the destructive Admin API soft-delete test repeatable by uploading a uniquely named package for each run.

Running the harness

From the repository root on Windows:

# One-time elevated machine setup
.\tools\Setup-DevEnvironment.ps1

# Build, start Aspire, seed, run the supported Playwright tests, and tear down
.\tests\Scripts\RunGalleryPlaywrightTests.ps1 -Configuration Release

The runner defaults to the ci-gallery profile. Use -AppHostProfile full only when the Azure Search prerequisites are available.

The existing non-Playwright functional-test path remains available:

.\tools\Run-AspireFunctionalTests.ps1 -Configuration $env:BuildConfiguration -TrustDevCert -IncludeAdminApiTests

Test scope and lifecycle

One AppHost startup and one seed pass are shared across the Playwright collection.
ConfigurationFilePath is set only after GalleryTools.exe seedfunctionaltests writes settings.CI.json, preventing premature initialization of the static Gallery configuration.
Startup and seeding have bounded timeouts with actionable process output on failure.
Environment variables and prior Gallery configuration are restored during teardown.
The Gallery continues to use fixed ports 80/443, so this harness must not run concurrently with another Gallery/AppHost instance on the same machine.

Validation

Playwright harness: 10 passed, 0 failed in Release.
Full Aspire functional-test command: 190 passed, 10 skipped, 0 failed across 200 tests.
Repeated startup, seeding, destructive Admin API testing, and teardown completed without stale fixture failures.
No Gallery, IIS Express, or AppHost process remained after the run.
PowerShell parser checks and git diff --check pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

Make Admin API soft-delete tests self-contained, prevent test-only auth bypass assemblies from surviving normal builds, and harden runner cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

This reverts commit 4700854. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

Reuse an already trusted development certificate without requiring elevation, clean up temporary certificate files reliably, and give the destructive Admin API soft-delete test a unique package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

…://github.com/NuGet/NuGetGallery into users/chabiss/aspire-playwright-agent-harness

…nch by an agent as well

joelverhagen · 2026-08-28T17:30:48Z

    - cron: '0 12 * * 1-5' # Weekdays at 8 AM Eastern (UTC 12:00)
  workflow_dispatch:

+permissions:


Is this necessary? #Resolved

joelverhagen · 2026-08-28T17:32:06Z

@@ -0,0 +1,82 @@
+# Plan: Integrate Aspire.Hosting.Testing for local NuGet End-to-End tests


Is this where we keep design docs? This seems like a point in time implementation plan not a long lived durable document. #Resolved

This needs to be removed, I forgot to clean it that was the old plan

joelverhagen · 2026-08-28T17:32:39Z

+            var options = base.ContextOptions();
+            if (IsAspireHarness)
+            {
+                options.IgnoreHTTPSErrors = true;


Why is this necessary? Don't we install the localhost dev certificate as trusted? #Resolved

joelverhagen · 2026-08-28T17:33:21Z

        [Fact]
        [Priority(0)]
        [Category("P0Tests")]
-        [Category("ReadOnlyModeTests")]


Why did we remove this category? #Resolved

joelverhagen · 2026-08-28T17:34:15Z

 namespace NuGetGallery.FunctionalTests.Playwright.BasicPages
 {
+	[Collection(AspirePlaywrightCollection.Definition)]
+	[Trait("Category", "PlaywrightTests")]


Would [Category("PlaywrightTests")]? Seems like there are two different approaches for categorizing tests now. #Resolved

What's wrong with having a category for for Playwrite test? We have CloudServiceTests, ReadOnlyModeTests?

I'm wondering why Trait and not Category attribute (which we have below in the file). What's the difference?

Ah I see, Trait is the actual xUnit feature. Category is just a commonly used Trait key ([Trait("Category", "...")]) that emulates NUnit/MSTest-style test categories.

I'll switch it to Category.

joelverhagen · 2026-08-28T17:37:43Z

+        [Fact]
+        [Priority(0)]
+        [Category("ReadOnlyModeTests")]
+        public async Task HomePageLoads_ContainsExpectedText()


Is this duplicating the test in tests/NuGetGallery.FunctionalTests/Playwright/BasicPages/HomePageValidationTest.cs? Why? #Resolved

joelverhagen · 2026-08-28T18:45:25Z

+.\tools\Setup-DevEnvironment.ps1
+$hostPid = .\tools\Start-AspireHost.ps1 -Configuration Release -UnsafeAdminApiAuthBypassForTesting
+try {
+    .\tools\Seed-FunctionalTestData.ps1 -Configuration Release


Is this needed since the test harness does it? #Resolved

Yes, it's still used by
Q:\src\NuGetGallery\tests\Scripts\RunAspireFunctionalTests.ps1

No, I mean does it need to be executed by the agent as part of copilot instructions, since the test harness is also doing this? Or am I missing something.

Not sure if this block is meaningful. I ran /Init in the CLI, that why the entire MD file was re-written.

…ses Aspire.Hosting.Testing

Make the existing required Aspire Functional Tests check run after Playwright and fail unless the Playwright job succeeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

chabiss requested a review from a team as a code owner August 25, 2026 20:54

chabiss and others added 2 commits August 26, 2026 15:25

Add Aspire-hosted Playwright test harness

6ce6aa3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

chabiss force-pushed the users/chabiss/aspire-playwright-agent-harness branch from 76bf13e to 4700854 Compare August 26, 2026 22:25

Revert "Stabilize Aspire functional test runner"

a43c436

This reverts commit 4700854. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 51555024-4670-44c9-96d2-dc67fa45e2c5

chabiss force-pushed the users/chabiss/aspire-playwright-agent-harness branch from bbee9d1 to a43c436 Compare August 27, 2026 01:11

chabiss added 3 commits August 26, 2026 22:13

Add Aspire-hosted Playwright test harness

9d143a5

Merge branch 'users/chabiss/aspire-playwright-agent-harness' of https…

f55662b

…://github.com/NuGet/NuGetGallery into users/chabiss/aspire-playwright-agent-harness

chabiss marked this pull request as draft August 27, 2026 18:46

Create RunAspireFunctionalTests.ps1 so that functional test can be lu…

8d1c156

…nch by an agent as well

chabiss marked this pull request as ready for review August 27, 2026 19:25

Copilot started work on behalf of chabiss August 27, 2026 20:38 View session

Copilot stopped work on behalf of chabiss due to an error August 27, 2026 20:47
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Copilot started work on behalf of chabiss August 27, 2026 21:12 View session

Copilot stopped work on behalf of chabiss due to an error August 27, 2026 21:24
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Some fixup

18fd0eb

chabiss force-pushed the users/chabiss/aspire-playwright-agent-harness branch from 521fa7f to 18fd0eb Compare August 28, 2026 00:52

joelverhagen reviewed Aug 28, 2026

View reviewed changes

Code Review Feedback

626ec2f

chabiss force-pushed the users/chabiss/aspire-playwright-agent-harness branch from 85cf4a3 to 626ec2f Compare August 28, 2026 22:19

chabiss added 2 commits August 28, 2026 17:42

Preventing Dual App Host initialization until entire FuntionalTests u…

32e084c

…ses Aspire.Hosting.Testing

		@@ -0,0 +1,82 @@
		# Plan: Integrate Aspire.Hosting.Testing for local NuGet End-to-End tests

Uh oh!

Conversation

chabiss commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Running the harness

Test scope and lifecycle

Validation

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chabiss Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chabiss Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

chabiss Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026 • edited by chabiss Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chabiss Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

joelverhagen Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

chabiss Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chabiss commented Aug 25, 2026 •

edited

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading

joelverhagen Aug 28, 2026 •

edited by chabiss

Loading