feat: Serve 10% of tag-page-rendering PROD traffic with ECS - #16746
Merged
Merged
Conversation
connoromalleyatwork
approved these changes
Sep 17, 2026
connoromalleyatwork
left a comment
Contributor
There was a problem hiding this comment.
Looks good!!
This is a no-op change. Co-authored-by: connoromalleyatwork <connoromalleyatwork@users.noreply.github.com>
We have performed a number of load tests against CODE and are happy with it's performance compared to EC2. Co-authored-by: connoromalleyatwork <connoromalleyatwork@users.noreply.github.com>
After load testing, we're confident that ECS can serve production traffic. 10% equates to 70-100 requests/minute. Co-authored-by: connoromalleyatwork <connoromalleyatwork@users.noreply.github.com>
akash1810
force-pushed
the
ecs-weighting
branch
from
September 17, 2026 10:19
732d37d to
a7188fc
Compare
🚀 Image pushed to AWS ECRImage digest: 🐛 Run the image locallyThe following can be used to run the image locally: # Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:a560de7b6a831fc522dab401b18cc7431ef55c4e0ce3ae4bebfc66ca522c2cfd"
# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-ecs-weighting"
# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-31292"
# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"
IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/artifacts --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"
# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY
# Pull the image
docker pull $IMAGE
# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE |
Jakeii
approved these changes
Sep 17, 2026
arelra
approved these changes
Sep 17, 2026
|
Seen on PROD (merged by @akash1810 9 minutes and 9 seconds ago) Please check your changes! |
Member
Author
|
This is live as of https://riffraff.gutools.co.uk/deployment/view/7824d0b4-1a12-404f-9fb7-5287e6c72194.
Average latency is up very slightly, the availability SLI remains at 100% though:
|
Member
Author
|
Metrics from ECS for the last 12 hours:
Looking pretty good! As a reminder, the EC2 infrastructure scales out at 400ms latency. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What does this change?
This change updates the traffic weighting configuration of tag-page-rendering:
Why?
Though we have load tested (see below), real production traffic is the best way to understand how the ECS infrastructure will perform.
Also, this is a step closer to removing EC2 infrastructure!
How has this change been tested?
In #16693 PROD was configured to run 9 tasks and scale at 20% CPU (up to 90 tasks). With this configuration, we've performed a load test (see #16452) with the following results:
Note
We think PROD ECS can serve 100% of traffic, however to be conservative, we'll start with 10%.
Co-authored-by: @connoromalleyatwork