Skip to content

DESIGN.md

Latest commit

 

History

History
87 lines (64 loc) · 5.39 KB

DESIGN.md

File metadata and controls

87 lines (64 loc) · 5.39 KB
Raw

Design Document

Overview

This CloudFormation template deploys a distributed, scalable architecture for migrating data from any S3-compatible storage provider to Amazon Simple Storage Service (Amazon S3) using rclone. The design addresses common challenges with single-instance migration approaches: lack of visibility, frequent failures, source-side throttling, and performance saturation.

Architecture Layers

1. Discovery Layer (Amazon ECS with AWS Fargate)

The lister runs as a Fargate task using the public python:3.13-slim image with an inline Python script. It connects to the source storage endpoint using credentials from AWS Secrets Manager, paginates through all objects, and batches them into groups of 20 keys per Amazon SQS message.

Design decisions:

  • Fargate over AWS Lambda: object enumeration can take hours for large buckets, exceeding Lambda's 15-minute timeout
  • Inline script over custom Docker image: eliminates ECR dependency, simplifies deployment
  • Batch size of 20: balances SQS message size limits (256 KB), fault isolation granularity, and processing efficiency

2. Queueing Layer (Amazon SQS)

Amazon SQS decouples discovery from execution. Each message is self-contained with source bucket, destination bucket, and a list of object keys.

Design decisions:

  • 12-hour visibility timeout: accommodates large file transfers that may take hours
  • maxReceiveCount of 2: messages move to DLQ after 2 failed attempts, preventing infinite retry loops
  • AWS KMS encryption: customer-managed AWS KMS key for encryption at rest

3. Execution Layer (Amazon EC2 Auto Scaling)

EC2 r5n.xlarge instances run 6 parallel rclone worker processes via systemd. The Auto Scaling Group scales from 0 to 5 instances based on queue depth.

Design decisions:

  • EC2 over Fargate for workers: r5n.xlarge provides up to 25 Gbps network throughput vs Fargate's ~10 Gbps per task
  • 6 workers per instance: maximizes the 25 Gbps network capacity (each rclone process achieves 500 Mbps to 6 Gbps depending on file size)
  • Scale-to-zero: no cost when the queue is empty
  • Scale-in protection: workers protect themselves from termination while actively processing a message

Security Considerations

Credential Management

Source storage credentials are stored in AWS Secrets Manager, encrypted at rest by default. Workers retrieve credentials at boot time via the setup script. The lister retrieves them at runtime via the boto3 SDK. No credentials are hardcoded in the template.

IAM Least Privilege

Three separate IAM roles scope permissions to each component:

  • ECS Task Execution Role: pull container images, write to Amazon CloudWatch Logs
  • ECS Task Role: send SQS messages, retrieve secrets from AWS Secrets Manager, use AWS KMS for SQS encryption
  • EC2 Instance Role: poll/delete/reset SQS messages, retrieve secrets from AWS Secrets Manager, read/write S3 objects, write Amazon CloudWatch Logs, manage scale-in protection, publish Amazon CloudWatch metrics

S3 permissions use Resource: '*' because both the source (external, S3-compatible) and destination (user-specified at runtime) bucket ARNs are not known at deploy time. This is documented as an accepted trade-off.

Network Security

  • VPC with public subnets and Internet Gateway (no NAT Gateway to avoid per-GB data processing charges)
  • Security group allows all outbound traffic (required for rclone to reach external endpoints) and no inbound traffic
  • IMDSv2 enforced on EC2 instances via MetadataOptions: HttpTokens: required

Encryption

  • SQS queues: customer-managed AWS KMS key with automatic annual key material rotation
  • CloudWatch Log Groups: customer-managed AWS KMS key with automatic annual key material rotation
  • AWS Secrets Manager: customer-managed AWS KMS key
  • Data in transit: rclone uses HTTPS by default for S3-compatible endpoints

Input Validation

  • Worker validates SQS message structure (required fields, correct types) before processing
  • Regex allowlists validate bucket names and object keys before passing to subprocess
  • Invalid messages are logged and deleted from the queue

Data Flow

1. User runs `aws ecs run-task` with SOURCE_BUCKET, DEST_BUCKET, optional PREFIX
2. Fargate lister retrieves credentials from Secrets Manager
3. Lister paginates source bucket, batches 20 keys per SQS message
4. EC2 workers poll SQS, validate message, run `rclone copyto` per key
5. On success: delete SQS message, publish FilesTransferred metric
6. On failure: reset message visibility to 0 for immediate retry, publish FailedTransfers metric
7. After 2 failures: SQS moves message to DLQ
8. Auto Scaling adjusts worker count based on queue depth

Trade-offs and Accepted Risks

Decision Trade-off Rationale
S3 Resource: '*' Broader than ideal IAM scope Source bucket is external (not AWS), destination is user-specified at runtime
Public subnets (no NAT) Instances have public IPs NAT Gateway charges $0.045/GB which is prohibitive for petabyte-scale migrations
rclone from official install script Supply chain risk Simplifies deployment for a sample; production deployments should pin version with checksum
Inline scripts in CloudFormation Harder to test independently Eliminates ECR/Docker build dependency; testable modules extracted for property-based testing
python:3.13-slim from Docker Hub External image dependency No custom image build required; standard public image