Skip to content

ECS Scaling at 20% CPU usage. - #16693

Merged
connoromalleyatwork merged 1 commit into
mainfrom
dcr-1percent
Sep 16, 2026
Merged

connoromalleyatwork merged 1 commit into
mainfrom
dcr-1percent

Conversation

@connoromalleyatwork

@connoromalleyatwork connoromalleyatwork commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

What does this change?

Adds target scaling to ECS for tag page rendering in both CODE and PROD.

Target scaling was added to the cdk in GuLoadBalancedAppExperimental in pr ECS CPU Target Scaling - #2962

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

🚀 Image pushed to AWS ECR

Image digest: sha256:32c4d87d751a767c56cc9d7fe6dfb7843df38dee560fa70300842e6eb258273e

🐛 Run the image locally

The following can be used to run the image locally:

# Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:32c4d87d751a767c56cc9d7fe6dfb7843df38dee560fa70300842e6eb258273e"

# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-dcr-1percent"

# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-31167"

# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"

IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/artifacts --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"

# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY

# Pull the image
docker pull $IMAGE

# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE

From guardian/actions-publish-image.

@connoromalleyatwork connoromalleyatwork added the feature Departmental tracking: work on a new feature label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

@connoromalleyatwork
connoromalleyatwork marked this pull request as ready for review September 14, 2026 15:40
@connoromalleyatwork
connoromalleyatwork requested a review from a team as a code owner September 14, 2026 15:40

@akash1810 akash1810 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC we don't currently scale EC2 CODE - do we need to scale ECS CODE? Not a blocking comment, as we can update in a later PR.

Comment on lines +175 to +176
scaleInCooldown: Duration.seconds(60),
scaleOutCooldown: Duration.seconds(60),

@connoromalleyatwork connoromalleyatwork Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default is Duration.seconds(300), which I actually think I like more as it gives more time to prevent fluctuations in capacity.

We had the 60 seconds before this change when doing our load testing, however I think that may have been to speed up scaling events to reduce wait time to watch them happen. Not for any meaningful reason.

@connoromalleyatwork connoromalleyatwork Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akash1810 and I discussed this and we are going to stick with 60 seconds for each cooldown for now. We can tune this more later!

60 seconds seams reasonable for now, and it is hard to know how this will impact scaling and the services ability to maintain its SLI under increasing load without watching it in production. We did a load test in code and it coped fine, so we have no reason to believe 60 seconds is a bad choice.

@connoromalleyatwork
connoromalleyatwork force-pushed the dcr-1percent branch 2 times, most recently from 4ab965b to 72e6092 Compare September 15, 2026 14:17
@connoromalleyatwork connoromalleyatwork added the run_chromatic Runs chromatic when label is applied label Sep 16, 2026
@connoromalleyatwork
connoromalleyatwork added this pull request to the merge queue Sep 16, 2026
@github-actions github-actions Bot removed the run_chromatic Runs chromatic when label is applied label Sep 16, 2026
Merged via the queue into main with commit 2fa9dee Sep 16, 2026
37 checks passed
@connoromalleyatwork
connoromalleyatwork deleted the dcr-1percent branch September 16, 2026 12:32
@gu-prout

gu-prout Bot commented Sep 16, 2026

Copy link
Copy Markdown

Seen on PROD (merged by @connoromalleyatwork 9 minutes and 4 seconds ago) Please check your changes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Departmental tracking: work on a new feature Seen-on-PROD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants