Skip to content

feat: Serve 10% of tag-page-rendering PROD traffic with ECS - #16746

Merged
akash1810 merged 3 commits into
mainfrom
ecs-weighting
Sep 17, 2026
Merged

akash1810 merged 3 commits into
mainfrom
ecs-weighting

Conversation

@akash1810

@akash1810 akash1810 commented Sep 17, 2026

Copy link
Copy Markdown
Member

What does this change?

This change updates the traffic weighting configuration of tag-page-rendering:

  • In CODE serve 100% of traffic with ECS
  • In PROD serve 10% of traffic with ECS. This is roughly 70-100 requests / minute.

Why?

Though we have load tested (see below), real production traffic is the best way to understand how the ECS infrastructure will perform.

Also, this is a step closer to removing EC2 infrastructure!

How has this change been tested?

In #16693 PROD was configured to run 9 tasks and scale at 20% CPU (up to 90 tasks). With this configuration, we've performed a load test (see #16452) with the following results:

image image

Note

We think PROD ECS can serve 100% of traffic, however to be conservative, we'll start with 10%.


Co-authored-by: @connoromalleyatwork

@akash1810 akash1810 added the feature Departmental tracking: work on a new feature label Sep 17, 2026
@akash1810
akash1810 marked this pull request as ready for review September 17, 2026 09:57
@akash1810
akash1810 requested a review from a team as a code owner September 17, 2026 09:57
@akash1810 akash1810 added the run_chromatic Runs chromatic when label is applied label Sep 17, 2026

@connoromalleyatwork connoromalleyatwork left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!!

akash1810 and others added 3 commits September 17, 2026 11:19
This is a no-op change.

Co-authored-by: connoromalleyatwork <connoromalleyatwork@users.noreply.github.com>
We have performed a number of load tests against CODE and are happy with it's performance compared to EC2.

Co-authored-by: connoromalleyatwork <connoromalleyatwork@users.noreply.github.com>
After load testing, we're confident that ECS can serve production traffic.
10% equates to 70-100 requests/minute.

Co-authored-by: connoromalleyatwork <connoromalleyatwork@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

🚀 Image pushed to AWS ECR

Image digest: sha256:a560de7b6a831fc522dab401b18cc7431ef55c4e0ce3ae4bebfc66ca522c2cfd

🐛 Run the image locally

The following can be used to run the image locally:

# Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:a560de7b6a831fc522dab401b18cc7431ef55c4e0ce3ae4bebfc66ca522c2cfd"

# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-ecs-weighting"

# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-31292"

# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"

IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/artifacts --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"

# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY

# Pull the image
docker pull $IMAGE

# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE

From guardian/actions-publish-image.

@akash1810
akash1810 added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit 705ac02 Sep 17, 2026
33 checks passed
@akash1810
akash1810 deleted the ecs-weighting branch September 17, 2026 13:38
@gu-prout

gu-prout Bot commented Sep 17, 2026

Copy link
Copy Markdown

Seen on PROD (merged by @akash1810 9 minutes and 9 seconds ago) Please check your changes!

@akash1810

akash1810 commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

This is live as of https://riffraff.gutools.co.uk/deployment/view/7824d0b4-1a12-404f-9fb7-5287e6c72194.

image

Average latency is up very slightly, the availability SLI remains at 100% though:

image

@akash1810

akash1810 commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Metrics from ECS for the last 12 hours:

image

Looking pretty good! As a reminder, the EC2 infrastructure scales out at 400ms latency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Departmental tracking: work on a new feature run_chromatic Runs chromatic when label is applied Seen-on-PROD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants