Skip to content

Commit 04c2332

Browse files
committed
fix: Configure LocalStack as GitHub Actions service for AWS integration tests
- Add LocalStack service container to Release-CI workflow - Configure health checks with 30 retries and 30s start period - Add verification step to ensure LocalStack is ready before tests - Update step numbering after adding LocalStack verification - Addresses timeout issues in GitHub Actions CI environment Related to bugfix spec: github-actions-localstack-timeout-fix
1 parent 06bd3f3 commit 04c2332

4 files changed

Lines changed: 149 additions & 45 deletions

File tree

‎.github/workflows/Master-Build.yml‎

Lines changed: 38 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,50 @@ on:
99
jobs:
1010
build:
1111
runs-on: ubuntu-latest
12+
13+
services:
14+
localstack:
15+
image: localstack/localstack:latest
16+
ports:
17+
- 4566:4566
18+
env:
19+
SERVICES: sqs,sns,kms,iam
20+
DEBUG: 1
21+
DOCKER_HOST: unix:///var/run/docker.sock
22+
options: >-
23+
--health-cmd "curl -f http://localhost:4566/_localstack/health || exit 1"
24+
--health-interval 10s
25+
--health-timeout 5s
26+
--health-retries 30
27+
--health-start-period 30s
28+
1229
steps:
1330
- uses: actions/checkout@v3
1431
- name: Setup .NET
1532
uses: actions/setup-dotnet@v3
1633
with:
1734
dotnet-version: 9.0.x
35+
36+
- name: Verify LocalStack is Ready
37+
run: |
38+
echo "Waiting for LocalStack to be fully ready..."
39+
max_attempts=30
40+
attempt=0
41+
while [ $attempt -lt $max_attempts ]; do
42+
if curl -f http://localhost:4566/_localstack/health 2>/dev/null; then
43+
echo "LocalStack is ready!"
44+
curl -s http://localhost:4566/_localstack/health | jq '.'
45+
break
46+
fi
47+
attempt=$((attempt + 1))
48+
echo "Attempt $attempt/$max_attempts - LocalStack not ready yet, waiting..."
49+
sleep 3
50+
done
51+
if [ $attempt -eq $max_attempts ]; then
52+
echo "ERROR: LocalStack did not become ready in time"
53+
exit 1
54+
fi
55+
1856
- name: Restore dependencies
1957
run: dotnet restore
2058
- name: Build
@@ -24,32 +62,6 @@ jobs:
2462
- name: Run Unit Tests
2563
run: dotnet test --no-build --verbosity normal --filter "Category=Unit"
2664

27-
# Start LocalStack container for integration tests
28-
- name: Start LocalStack Container
29-
run: |
30-
docker run -d \
31-
--name localstack \
32-
-p 4566:4566 \
33-
-e SERVICES=sqs,sns,kms,iam \
34-
-e DEBUG=1 \
35-
-e DOCKER_HOST=unix:///var/run/docker.sock \
36-
localstack/localstack:latest
37-
38-
# Wait for LocalStack to be ready (max 60 seconds)
39-
echo "Waiting for LocalStack to be ready..."
40-
timeout 60 bash -c 'until docker exec localstack curl -s http://localhost:4566/_localstack/health | grep -q "\"sqs\": \"available\""; do sleep 2; done' || echo "LocalStack startup timeout"
41-
42-
# Display LocalStack health status
43-
docker exec localstack curl -s http://localhost:4566/_localstack/health
44-
45-
# Configure AWS SDK to use LocalStack endpoints
46-
- name: Configure AWS SDK for LocalStack
47-
run: |
48-
echo "AWS_ACCESS_KEY_ID=test" >> $GITHUB_ENV
49-
echo "AWS_SECRET_ACCESS_KEY=test" >> $GITHUB_ENV
50-
echo "AWS_DEFAULT_REGION=us-east-1" >> $GITHUB_ENV
51-
echo "AWS_ENDPOINT_URL=http://localhost:4566" >> $GITHUB_ENV
52-
5365
# Run integration tests against LocalStack
5466
- name: Run Integration Tests with LocalStack
5567
run: dotnet test --no-build --verbosity normal --filter "Category=Integration&Category=RequiresLocalStack"
@@ -58,13 +70,6 @@ jobs:
5870
AWS_SECRET_ACCESS_KEY: test
5971
AWS_DEFAULT_REGION: us-east-1
6072
AWS_ENDPOINT_URL: http://localhost:4566
61-
62-
# Clean up LocalStack container
63-
- name: Stop LocalStack Container
64-
if: always()
65-
run: |
66-
docker stop localstack || true
67-
docker rm localstack || true
6873

6974
run-Lint:
7075
runs-on: ubuntu-latest

‎.github/workflows/Release-CI.yml‎

Lines changed: 45 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,23 @@ permissions:
1212
jobs:
1313
Build-Test-Publish:
1414
runs-on: ubuntu-latest
15+
16+
services:
17+
localstack:
18+
image: localstack/localstack:latest
19+
ports:
20+
- 4566:4566
21+
env:
22+
SERVICES: sqs,sns,kms,iam
23+
DEBUG: 1
24+
DOCKER_HOST: unix:///var/run/docker.sock
25+
options: >-
26+
--health-cmd "curl -f http://localhost:4566/_localstack/health || exit 1"
27+
--health-interval 10s
28+
--health-timeout 5s
29+
--health-retries 30
30+
--health-start-period 30s
31+
1532
env:
1633
working-directory: ${{ github.workspace }}
1734
github-token: '${{ secrets.GH_Packages }}'
@@ -49,42 +66,62 @@ jobs:
4966
with:
5067
dotnet-version: 9.0.x
5168

52-
- name: Step-06 Restore dependencies
69+
- name: Step-06 Verify LocalStack is Ready
70+
run: |
71+
echo "Waiting for LocalStack to be fully ready..."
72+
max_attempts=30
73+
attempt=0
74+
while [ $attempt -lt $max_attempts ]; do
75+
if curl -f http://localhost:4566/_localstack/health 2>/dev/null; then
76+
echo "LocalStack is ready!"
77+
curl -s http://localhost:4566/_localstack/health | jq '.'
78+
break
79+
fi
80+
attempt=$((attempt + 1))
81+
echo "Attempt $attempt/$max_attempts - LocalStack not ready yet, waiting..."
82+
sleep 3
83+
done
84+
if [ $attempt -eq $max_attempts ]; then
85+
echo "ERROR: LocalStack did not become ready in time"
86+
exit 1
87+
fi
88+
89+
- name: Step-07 Restore dependencies
5390
run: dotnet restore
5491
working-directory: '${{ env.working-directory }}'
5592

56-
- name: Step-07 Build Version (Pre-release)
93+
- name: Step-08 Build Version (Pre-release)
5794
if: ${{ env.is-release != 'true' }}
5895
run: dotnet build --configuration Release --no-restore -p:PackageVersion=${{ steps.gitversion.outputs.NuGetVersion }}
5996
working-directory: '${{ env.working-directory }}'
6097

61-
- name: Step-07 Build Version (Release)
98+
- name: Step-08 Build Version (Release)
6299
if: ${{ env.is-release == 'true' }}
63100
run: dotnet build --configuration Release --no-restore -p:PackageVersion=${{ steps.gitversion.outputs.MajorMinorPatch }}
64101
working-directory: '${{ env.working-directory }}'
65102

66-
- name: Step-08 Test Solution
103+
- name: Step-09 Test Solution
67104
run: dotnet test --configuration Release --no-build --no-restore --verbosity normal
68105
working-directory: '${{ env.working-directory }}'
69106

70-
- name: Step-09 Create NuGet Package (Pre-release)
107+
- name: Step-10 Create NuGet Package (Pre-release)
71108
if: ${{ env.is-release != 'true' }}
72109
run: dotnet pack --configuration Release --no-build --output ./packages -p:PackageVersion=${{ steps.gitversion.outputs.NuGetVersion }}
73110
working-directory: '${{ env.working-directory }}'
74111

75-
- name: Step-09 Create NuGet Package (Release)
112+
- name: Step-10 Create NuGet Package (Release)
76113
if: ${{ env.is-release == 'true' }}
77114
run: dotnet pack --configuration Release --no-build --output ./packages -p:PackageVersion=${{ steps.gitversion.outputs.MajorMinorPatch }}
78115
working-directory: '${{ env.working-directory }}'
79116

80-
- name: Step-10 Publish to Github Packages
117+
- name: Step-11 Publish to Github Packages
81118
if: ${{ env.is-release == 'true' }}
82119
run: |
83120
dotnet tool install gpr --global
84121
find ./packages -name "*.nupkg" -print -exec gpr push -k ${{ env.github-token }} {} \;
85122
working-directory: '${{ env.working-directory }}'
86123

87-
- name: Step-11 Publish to NuGet.org
124+
- name: Step-12 Publish to NuGet.org
88125
if: ${{ 'false' && env.is-release == 'true' && env.nuget-token != '' }}
89126
run: |
90127
find ./packages -name "*.nupkg" -print -exec dotnet nuget push {} --skip-duplicate --api-key ${{ env.nuget-token }} --source https://api.nuget.org/v3/index.json \;

‎.kiro/specs/github-actions-localstack-timeout-fix/design.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -158,8 +158,9 @@ Assuming our root cause analysis is correct:
158158
1. **Enhance External Instance Detection**: Improve `IsExternalLocalStackAvailableAsync` method
159159
- Increase timeout from 3 seconds to 10 seconds for CI environments
160160
- Add retry logic (3 attempts with 2-second delays)
161-
- Check not just for HTTP 200 but also parse JSON to verify services are "available"
162-
- Log detection results for diagnostics
161+
- **Lenient Detection**: Accept HTTP 200 from health endpoint even if services aren't fully initialized
162+
- Defer full service readiness validation to `WaitForServicesAsync`
163+
- Log detection results for diagnostics with service status details when available
163164

164165
2. **Improve Wait Strategy**: Modify `StartAsync` to add initial delay after container start
165166
- Add 5-second delay after `_container.StartAsync()` completes

‎docs/Cloud-Integration-Testing.md‎

Lines changed: 63 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -629,11 +629,20 @@ public async Task All_Configured_Resources_Should_Exist_After_Bootstrapping()
629629

630630
The test infrastructure includes specific optimizations for GitHub Actions CI environments:
631631

632+
**LocalStack Service Container Integration:**
633+
- **Pre-Started Container** - Release-CI workflow includes LocalStack as a service container
634+
- **Port Mapping** - LocalStack exposed on port 4566 for test access
635+
- **Service Configuration** - Configured with SQS, SNS, KMS, and IAM services
636+
- **Health Checks** - Container health validated before test execution begins
637+
- **Automatic Lifecycle** - GitHub Actions manages container startup and cleanup
638+
- **Resource Efficiency** - Single shared container across all test jobs
639+
632640
**LocalStack Timeout Handling:**
633641
- **Environment Detection** - Automatically detects GitHub Actions via `GITHUB_ACTIONS` environment variable
634642
- **Extended Timeouts** - Uses 90-second health check timeout in CI (vs 30 seconds locally) to accommodate slower container initialization
635-
- **Enhanced Retry Logic** - Increases retry attempts (20 vs 15) and delays (3 seconds vs 2 seconds) for CI environments
636-
- **External Instance Detection** - 10-second timeout (vs 3 seconds locally) to reliably detect pre-started LocalStack service containers
643+
- **Enhanced Retry Logic** - Increases retry attempts (30 vs 15) and delays (3 seconds vs 2 seconds) for CI environments
644+
- **External Instance Detection** - 10-second timeout (vs 3 seconds locally) with 3 retry attempts to reliably detect pre-started LocalStack service containers
645+
- **Lenient Detection** - Accepts HTTP 200 from health endpoint even if services aren't fully initialized, deferring full readiness validation to main wait loop
637646

638647
**Container Sharing:**
639648
- **xUnit Collection Fixtures** - `AwsIntegrationTestCollection` enforces shared `LocalStackTestFixture` across all test classes
@@ -645,6 +654,35 @@ The test infrastructure includes specific optimizations for GitHub Actions CI en
645654
- `LocalStackConfiguration.IsCI` - Property that detects GitHub Actions environment
646655
- `LocalStackManager.WaitForServicesAsync()` - Adaptive retry logic based on environment detection
647656

657+
**GitHub Actions Workflow Configuration:**
658+
659+
The Release-CI workflow includes LocalStack as a service container:
660+
661+
```yaml
662+
services:
663+
localstack:
664+
image: localstack/localstack:latest
665+
ports:
666+
- 4566:4566
667+
env:
668+
SERVICES: sqs,sns,kms,iam
669+
DEBUG: 1
670+
DOCKER_HOST: unix:///var/run/docker.sock
671+
options: >-
672+
--health-cmd "curl -f http://localhost:4566/_localstack/health || exit 1"
673+
--health-interval 10s
674+
--health-timeout 5s
675+
--health-retries 30
676+
--health-start-period 30s
677+
```
678+
679+
**Service Container Benefits:**
680+
- Container starts before test job begins
681+
- Health checks ensure services are ready before tests run
682+
- Automatic cleanup after job completion
683+
- No manual container management required in test code
684+
- Consistent environment across all CI runs
685+
648686
### Reporting and Analysis
649687
- **Comprehensive Reports** - Detailed test results with metrics and analysis
650688
- **Performance Trends** - Historical performance tracking and regression detection
@@ -664,6 +702,29 @@ The `AwsResourceManager` provides comprehensive automated resource lifecycle man
664702
- **Test Isolation** - Unique naming prevents conflicts in parallel test execution
665703

666704
### LocalStack Manager (Implemented)
705+
The `LocalStackManager` provides comprehensive container lifecycle management for AWS service emulation with enhanced features:
706+
707+
- **Smart Container Detection** - Automatically detects and reuses existing LocalStack instances (e.g., in CI/CD environments) to avoid redundant container creation
708+
- **Adaptive Timeout Configuration** - Automatically adjusts health check timeouts based on environment (90 seconds for CI, 30 seconds for local development)
709+
- **Enhanced External Instance Detection** - Accepts HTTP 200 responses from LocalStack health endpoint even if services aren't fully initialized, deferring full service readiness validation to the main wait loop
710+
- **Retry Logic** - Configurable retry attempts with delays for reliable external instance detection (3 attempts with 2-second delays)
711+
- **Port Management** - Automatic port conflict detection and resolution
712+
- **Service Validation** - Comprehensive AWS service emulation validation (SQS, SNS, KMS, IAM)
713+
- **Diagnostic Logging** - Detailed logging for troubleshooting container startup and service initialization issues
714+
715+
**External Instance Detection Behavior:**
716+
- Checks for existing LocalStack instances before starting new containers
717+
- Uses lenient detection: accepts HTTP 200 from health endpoint regardless of service status
718+
- Allows services to continue initializing after detection succeeds
719+
- Full service readiness validation occurs in `WaitForServicesAsync` with appropriate timeouts
720+
- Prevents port conflicts and reduces CI execution time by reusing pre-started containers
721+
722+
**CI/CD Optimizations:**
723+
- Detects GitHub Actions environment via `GITHUB_ACTIONS` environment variable
724+
- Uses extended timeouts (10 seconds vs 3 seconds) for external instance detection in CI
725+
- Increases retry attempts and delays for slower CI environments
726+
- Adds initial delay after container start (5 seconds in CI, 2 seconds locally) for initialization scripts
727+
667728
Enhanced LocalStack container management with comprehensive AWS service emulation:
668729

669730
- **Service Emulation** - Full support for SQS (standard and FIFO), SNS, KMS, and IAM

0 commit comments

Comments
 (0)