PRD-kubernetes-multi-account-aws.md

# Technical PRD — Kubernetes Multi-Account AWS Architecture

| Metadata | Value |
|----------|-------|
| **Project** | HelpDev'Ops Enterprise Infrastructure Stack |
| **Version** | 1.0.0 |
| **Date** | 2026-01-24 |
| **Status** | Draft |
| **Author** | Guilherme Biff Zarelli |

---

## Tecnical Summary

This document defines a Kubernetes platform architecture (EKS) on AWS designed for **enterprises requiring rapid scalability, robust security, and high resilience** as a greenfield initiative. The architecture establishes a multi-account AWS strategy with environment isolation (Staging/Production), GitOps-driven deployments using Argo CD, and a comprehensive observability stack (Grafana, Mimir, Loki, Tempo). Key platform capabilities include Service Mesh with Istio (mTLS, traffic management), centralized identity management via Keycloak, API Gateway with Kong, and a Developer Portal powered by Backstage. The design prioritizes scalability, security (Zero Trust), and operational excellence while preparing for multi-region and multi-Business Unit expansion.

### Initial Scope
- **Staging (HML):** Dedicated AWS account with EKS cluster
- **Production (Prod-Main):** Dedicated AWS account with EKS cluster and Central Grafana
- **Business Units (BU):** Architecture prepared for future expansion (documented, not implemented in v1)

### Business Objectives
1. Accelerate time-to-market for new services
2. Ensure isolation and governance between environments
3. Provide unified observability for operations
4. Establish scalable standards from the start

---

## Table of Contents

1. [Context and Problem](#1-context-and-problem)
2. [Architecture Principles](#2-architecture-principles)
3. [Solution Architecture](#3-solution-architecture)
4. [Technology Stack](#4-technology-stack)
   - [4.1 Infrastructure](#41-infrastructure)
   - [4.2 Networking](#42-networking)
   - [4.3 GitOps and CI/CD](#43-gitops-and-cicd)
   - [4.4 Observability](#44-observability-grafana-stack)
   - [4.5 Security and Identity](#45-security-and-identity)
   - [4.6 Storage](#46-storage)
   - [4.7 Developer Portal](#47-developer-portal)
5. [Repository Structure](#5-repository-structure)
   - [5.1 Overview](#51-overview)
   - [5.2 Repository Details](#52-repository-details)
   - [5.3 Naming Conventions](#53-naming-conventions)
   - [5.4 Standard Labels and Annotations](#54-standard-labels-and-annotations)
   - [5.5 Developer Portal (Backstage)](#55-developer-portal-backstage)
6. [Deploy Flow (GitOps)](#6-deploy-flow-gitops)
   - [6.1 Flow Overview](#61-developer-flow-overview)
   - [6.2 Staging Flow](#62-staging-flow-hml)
   - [6.3 Production Flow](#63-production-flow-prod)
   - [6.4 GitHub Actions Workflow](#64-complete-github-actions-workflow)
   - [6.5 Environment Promotion](#65-summary-environment-promotion)
   - [6.6 Rollback Strategies](#66-rollback-strategies)
7. [Observability](#7-observability)
8. [Service Mesh (Istio)](#8-service-mesh-istio)
9. [Identity and Access Management (Keycloak)](#9-identity-and-access-management-keycloak)
10. [API Management (Kong)](#10-api-management-kong)
    - [10.1 Overview](#101-overview)
    - [10.2 Repository Structure](#102-repository-structure)
    - [10.3 Kong Configuration](#103-kong-configuration)
    - [10.4 Pipeline Integration](#104-pipeline-integration)
    - [10.5 Backstage Integration](#105-backstage-integration)
11. [Security and Governance](#11-security-and-governance)
    - [11.1 IAM and IRSA](#111-iam-and-irsa)
    - [11.2 Network Policies](#112-network-policies-default-deny)
    - [11.3 Pod Security Standards](#113-pod-security-standards)
    - [11.4 GitOps Governance](#114-gitops-governance)
    - [11.5 Secrets Management](#115-secrets-management)
12. [Resilience and High Availability](#12-resilience-and-high-availability)
13. [Implementation Plan](#13-implementation-plan)
14. [Risks and Mitigations](#14-risks-and-mitigations)
15. [Success Metrics](#15-success-metrics)
16. [Architecture Decision Records (ADRs)](#16-architecture-decision-records-adrs)
17. [Glossary](#17-glossary)
18. [References](#18-references)
19. [Approvals](#19-approvals)
- [Annexes](#annexes)

---

## 1. Context and Problem

### 1.1 Current Situation
Greenfield project with no existing infrastructure. Need to establish a robust platform foundation with:
- Clear separation between environments (HML/Prod)
- Modern GitOps and IaC practices
- Native observability (metrics, logs, traces)
- Expansion path for multiple Business Units

### 1.2 Motivation
| Driver | Description |
|--------|-------------|
| **Isolation** | Controlled blast radius; billing and governance per account |
| **Traceability** | Git as source of truth for all cluster state |
| **Observability** | Unified view of all environments in a central point |
| **Scalability** | Architecture prepared for horizontal growth |

---

## 2. Architecture Principles

| # | Principle | Description |
|---|-----------|-------------|
| P1 | **Isolation by AWS Account** | Each environment in separate account (security, billing, blast radius) |
| P2 | **GitOps as Source of Truth** | All cluster state versioned in Git; PRs for code, Releases for production |
| P3 | **Observability per Environment** | Local telemetry with read federation in Central Grafana |
| P4 | **Standardization and Reuse** | Base Helm charts, pipeline templates, automated scaffolding |
| P5 | **Resilience by Design** | No single points of failure; HA in critical components |
| P6 | **Artifact Immutability** | Images referenced by digest; immutable tags in registry |
| P7 | **Zero Trust Network** | Mandatory mTLS between services; authentication at all layers |

---

## 3. Solution Architecture

### 3.1 Account, Region, and Cluster Topology

#### Multi-Region Principle

> **Regions are isolated by design.** Services in different regions DO NOT communicate directly. Each region is a complete and self-contained deployment. This enables true disaster recovery and regional isolation.

```
┌─────────────────────────────────────────────────────────────────────────────────────┐
│                              AWS Organization                                        │
│                                                                                      │
│  ┌─────────────────────────────────────────────────────────────────────────────┐    │
│  │                  helpdev-org-main (Shared Services Account)                  │    │
│  │                                                                               │    │
│  │   Route53 (DNS) ──── CloudFront (CDN) ──── ECR (us-east-1) ──── Grafana Central │
│  │         │                   │                   │                   │         │    │
│  │   ┌─────┴─────┐       ┌─────┴─────┐       ┌─────┴─────┐       ┌─────┴─────┐   │    │
│  │   │ ACM Certs │       │ WAF Rules │       │Cross-Acct │       │ Dashboards│   │    │
│  │   │ (wildcard)│       │ (shared)  │       │  Policies │       │  Alertas  │   │    │
│  │   └───────────┘       └───────────┘       └───────────┘       └───────────┘   │    │
│  └─────────────────────────────────────────────────────────────────────────────┘    │
│                                      │                                               │
│         ┌────────────────────────────┼────────────────────────────┐                 │
│         │        Cross-Account Access (IAM Roles + Resource Policies)               │
│         ▼                            ▼                            ▼                 │
│  ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────────────────────┐   │
│  │ helpdev-hml     │   │ helpdev-prod    │   │ helpdev-prod-bu-* (futuro)      │   │
│  │                 │   │                 │   │                                 │   │
│  │ us-east-1:      │   │ us-east-1:      │   │ Architecture prepared for       │   │
│  │ └─eks-hml-use1  │   │ └─eks-prod-use1 │   │ regional expansion per BU       │   │
│  │                 │   │                 │   │                                 │   │
│  │ sa-east-1:      │   │ sa-east-1:      │   │ Each BU consumes:               │   │
│  │ └─eks-hml-sae1  │   │ └─eks-prod-sae1 │   │ • Shared ECR                    │   │
│  │   (future)      │   │   (future)      │   │ • Shared Route53                │   │
│  │                 │   │                 │   │ • Central Grafana               │   │
│  └─────────────────┘   └─────────────────┘   └─────────────────────────────────┘   │
│                                                                                      │
│         ▲                            ▲                                               │
│         │     NO CROSS-REGION        │                                               │
│         │     COMMUNICATION          │                                               │
│         └────────────── ✕ ───────────┘                                               │
└─────────────────────────────────────────────────────────────────────────────────────┘
```

> **Note:** The `helpdev-org-main` account centralizes global resources following [AWS multi-account best practices](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/). See [ADR-012](adr/ADR-012-shared-services-account.md) for full justification.

#### AWS Accounts

| AWS Account | Purpose | Main Resources | v1 Status |
|-----------|-----------|---------------------|-----------|
| `helpdev-org-main` | Shared Services | Route53, CloudFront, ECR, Central Grafana, ACM | ✅ Implement |
| `helpdev-hml` | Staging | EKS clusters, Keycloak, Observability (Mimir/Loki/Tempo) | ✅ Implement |
| `helpdev-prod` | Production | EKS clusters, Keycloak, Observability (Mimir/Loki/Tempo) | ✅ Implement |
| `helpdev-prod-bu-*` | Business Units | Isolated EKS clusters per BU | 📋 Documented |

> **Note:** Global resources (DNS, CDN, Registry, Central Observability) reside in `helpdev-org-main` for isolation, clear billing, and scalability. See [ADR-012](adr/ADR-012-shared-services-account.md).

#### Clusters per Environment and Region

| AWS Account | Region | EKS Cluster | Purpose | v1 Status |
|-----------|--------|-------------|-----------|-----------|
| `helpdev-hml` | `us-east-1` | `eks-hml-use1` | Primary staging | ✅ Implement |
| `helpdev-hml` | `sa-east-1` | `eks-hml-sae1` | Secondary staging | 📋 Structure prepared |
| `helpdev-prod` | `us-east-1` | `eks-prod-use1` | Primary production | ✅ Implement |
| `helpdev-prod` | `sa-east-1` | `eks-prod-sae1` | Secondary production (Brazil) | 📋 Structure prepared |
| `helpdev-prod-bu-*` | `*` | `eks-prod-bu-*-{region}` | Isolated production per BU | 📋 Documented |

#### Region Codes

| AWS Region | Short Code | Location |
|------------|--------------|-------------|
| `us-east-1` | `use1` | N. Virginia (Primary) |
| `sa-east-1` | `sae1` | São Paulo |
| `eu-west-1` | `euw1` | Ireland (future) |
| `ap-southeast-1` | `apse1` | Singapore (future) |

#### v1 Strategy

- **Initial region:** `us-east-1` (structure prepared for multi-region)
- **Expansion:** Add regions by creating folders in `environments/{env}/{region}/`
- **New services:** Single-region default, multi-region option in Backstage

### 3.2 Components per Cluster

Each EKS cluster contains:

| Namespace | Components | Description |
|-----------|-------------|-----------|
| `argocd` | Argo CD (HA) | GitOps controller |
| `monitoring` | Grafana, Mimir, Loki, Tempo, OTel Collector | Complete observability |
| `istio-system` | Istiod, Ingress Gateway | Service Mesh control plane |
| `keycloak` | Keycloak (HA) + PostgreSQL | Identity and Access Management |
| `platform` | Backstage | Developer Portal and Software Catalog |
| `keda` | KEDA Operator | Event-driven autoscaling |
| `kube-system` | CoreDNS, kube-proxy, CNI, CSI | K8s core components |
| `apps-*` | Domain workloads | Microservices organized by domain (with Istio sidecar) |

### 3.3 Multi-Region Architecture Diagram

```mermaid
flowchart TB
    subgraph SHARED["Conta AWS: helpdev-org-main (Shared Services)"]
        R53["Route53<br/>(Latency-based routing)"]
        CF["CloudFront<br/>(CDN)"]
        ECR["ECR<br/>(us-east-1)"]
        GRAF_CENTRAL["Grafana Central 🎯"]
        ACM["ACM Certs<br/>(wildcard)"]
    end

    subgraph HML["Conta AWS: helpdev-hml"]
        subgraph HML_USE1["us-east-1"]
            EKS_HML_USE1["EKS: eks-hml-use1"]
            ARGO_HML_USE1["Argo CD"]
            KC_HML_USE1["Keycloak"]
            MON_HML_USE1["Mimir/Loki/Tempo"]
        end
        
        subgraph HML_SAE1["sa-east-1 (futuro)"]
            EKS_HML_SAE1["EKS: eks-hml-sae1"]
        end
    end

    subgraph PROD["Conta AWS: helpdev-prod"]
        subgraph PROD_USE1["us-east-1 (Primary)"]
            EKS_PROD_USE1["EKS: eks-prod-use1"]
            ARGO_PROD_USE1["Argo CD"]
            KC_PROD_USE1["Keycloak"]
            MON_PROD_USE1["Mimir/Loki/Tempo"]
        end
        
        subgraph PROD_SAE1["sa-east-1 (futuro)"]
            EKS_PROD_SAE1["EKS: eks-prod-sae1"]
            KC_PROD_SAE1["Keycloak"]
            MON_PROD_SAE1["Mimir/Loki/Tempo"]
        end
    end

    %% Routing (from shared account)
    R53 --> EKS_HML_USE1
    R53 --> EKS_PROD_USE1
    R53 -.-> EKS_PROD_SAE1
    
    %% ECR pull (cross-account)
    ECR --> EKS_HML_USE1
    ECR --> EKS_PROD_USE1
    ECR -.-> EKS_PROD_SAE1

    %% Observability Federation (cross-account queries)
    GRAF_CENTRAL --> MON_PROD_USE1
    GRAF_CENTRAL -.-> MON_PROD_SAE1
    GRAF_CENTRAL --> MON_HML_USE1
    
    %% Keycloak GitOps Sync (shared config)
    KC_HML_USE1 -.-|"GitOps Sync"| KC_PROD_USE1
    KC_PROD_USE1 -.-|"GitOps Sync"| KC_PROD_SAE1
    
    %% Isolation between regions
    EKS_PROD_USE1 ~~~ EKS_PROD_SAE1
    
    style GRAF_CENTRAL fill:#f96,stroke:#333
    style SHARED fill:#e1f5fe,stroke:#333
```

> **Note:** Regions marked as "future" have the directory structure prepared, but are not implemented in v1.

---

## 4. Technology Stack

### 4.1 Infrastructure

| Layer | Technology | Justification |
|--------|------------|---------------|
| **Cloud Provider** | AWS | Corporate standard |
| **Kubernetes** | Amazon EKS 1.29+ | Managed control plane; native AWS integration |
| **IaC** | Terraform | Declarative provisioning; state management |
| **Containers** | Docker / OCI | Industry standard |
| **Registry** | Amazon ECR | IAM integration; native scanning; tag immutability |
| **Autoscaling** | KEDA + HPA | Event-driven autoscaling; scale-to-zero |

#### Resources per Region

Each region has its own isolated infrastructure:

| Resource | Scope | Naming |
|---------|--------|--------------|
| **VPC** | Per region | `helpdev-{env}-{region}-vpc` |
| **EKS Cluster** | Per region | `eks-{env}-{region_short}` |
| **RDS (Keycloak)** | Per region | `helpdev-{env}-{region}-keycloak-db` |
| **S3 (Observability)** | Per region | `helpdev-{env}-{region}-{mimir\|loki\|tempo}` |
| **ECR** | Global (us-east-1) | `{account}.dkr.ecr.us-east-1.amazonaws.com` |

> **Note:** ECR is maintained only in `us-east-1` for simplicity. Images are pulled cross-region.

### 4.2 Networking

| Component | Technology | Justification |
|------------|------------|---------------|
| **VPC** | AWS VPC | Network isolation per account and region |
| **Subnets** | Public + Private (3 AZs) | HA and traffic segregation |
| **NAT** | AWS NAT Gateway | Internet egress from private subnets |
| **DNS** | Amazon Route 53 | Managed DNS; latency-based routing for multi-region |
| **Load Balancer** | AWS ALB (via AWS Load Balancer Controller) | Layer 7; Ingress integration |
| **Ingress Controller** | Istio Ingress Gateway | Unified ingress via Service Mesh |
| **Service Mesh** | Istio | mTLS, traffic management, observability |
| **Multi-Account Connectivity** | AWS Transit Gateway | Communication between VPCs from different accounts (same region) |

#### Regional Isolation

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         Isolated Regions                                 │
│                                                                          │
│  ┌─────────────────────────┐     ┌─────────────────────────┐            │
│  │     us-east-1           │     │     sa-east-1           │            │
│  │  ┌─────────────────┐    │     │  ┌─────────────────┐    │            │
│  │  │ VPC + EKS       │    │     │  │ VPC + EKS       │    │            │
│  │  │ Keycloak + RDS  │    │     │  │ Keycloak + RDS  │    │            │
│  │  │ Mimir/Loki/Tempo│    │     │  │ Mimir/Loki/Tempo│    │            │
│  │  └─────────────────┘    │     │  └─────────────────┘    │            │
│  └─────────────────────────┘     └─────────────────────────┘            │
│              ▲                              ▲                            │
│              │   NO DIRECT COMMUNICATION    │                            │
│              └────────────── ✕ ─────────────┘                            │
│                                                                          │
│  Communication between regions: Only via Route53 (users/clients)        │
└─────────────────────────────────────────────────────────────────────────┘
```

> **Principle:** Services in different regions DO NOT communicate directly. Each region is self-contained. This ensures true isolation for disaster recovery.

### 4.3 GitOps and CI/CD

| Component | Technology | Justification |
|------------|------------|---------------|
| **GitOps Controller** | Argo CD | Maturity; UI; Helm/Kustomize support |
| **CI Pipeline** | GitHub Actions | Corporate standard; extensive marketplace |
| **Git Hosting** | GitHub | Native integration with Actions |
| **Secrets** | AWS Secrets Manager + External Secrets Operator | Automatic rotation; no secrets in Git |
| **Image Scanning** | Trivy / ECR Scanning | Image vulnerabilities |
| **Artifact Storage** | Amazon S3 | Build artifacts, backups |

### 4.4 Observability (Grafana Stack)

| Signal | Technology | Suggested Retention |
|-------|------------|-------------------|
| **Metrics** | Mimir | HML: 7d / Prod: 90d |
| **Logs** | Loki | HML: 3d / Prod: 30d |
| **Traces** | Tempo | HML: 1d / Prod: 7d |
| **Collector** | OpenTelemetry Collector | Gateway mode + optional DaemonSet |
| **Visualization** | Grafana | Central in prod-main |
| **Alerting** | Alertmanager | Integrated with Grafana |
| **Dashboards** | Grafana Dashboards (as code) | Versioned in Git |

**Storage Backend (S3):**

| Component | Bucket Pattern | Retention |
|------------|----------------|----------|
| Mimir | `helpdev-{env}-mimir-blocks` | Lifecycle policy |
| Loki | `helpdev-{env}-loki-chunks` | Lifecycle policy |
| Tempo | `helpdev-{env}-tempo-traces` | Lifecycle policy |

### 4.5 Security and Identity

| Layer | Technology | Description |
|--------|------------|-----------|
| **IAM** | AWS IAM + IRSA | Workload identity; zero static credentials |
| **Encryption at Rest** | AWS KMS | Secrets, volumes (EBS), S3 |
| **Encryption in Transit** | Istio mTLS | Encrypted communication between services |
| **Network Isolation** | VPC + Security Groups + NetworkPolicies | Isolation at multiple layers |
| **Pod Security** | Pod Security Standards (Restricted) | Minimum security baseline |
| **Audit** | CloudTrail + K8s Audit Logs | Complete traceability |
| **Identity Provider** | Keycloak | SSO, OAuth2, OIDC for apps and tools |
| **API Gateway Auth** | Istio + Keycloak (JWT) | Token validation at mesh layer |

### 4.6 Storage

| Type | Technology | Usage |
|------|------------|-----|
| **Block Storage** | Amazon EBS (gp3) | Persistent volumes (PVCs) |
| **Object Storage** | Amazon S3 | Observability, backups, artifacts |
| **Database (Platform)** | Amazon RDS PostgreSQL | Keycloak, Backstage |
| **Database (Apps)** | RDS / DynamoDB / ElastiCache | Per service needs |

### 4.7 Developer Portal

| Component | Technology | Description |
|------------|------------|-----------|
| **Portal** | Backstage | Developer Portal and Software Catalog |
| **Templates** | Backstage Scaffolder | Automated service creation |
| **Docs** | TechDocs | Integrated documentation (docs-as-code) |
| **Database** | PostgreSQL (RDS) | Catalog persistence |
| **Plugins** | GitHub, Kubernetes, Argo CD, Keycloak | Native integrations |

### 4.8 Visual Stack Summary

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                              DEVELOPER EXPERIENCE                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │  Backstage  │  │   GitHub    │  │  Grafana    │  │  Argo CD    │         │
│  │  (Portal)   │  │  (Code/CI)  │  │  (Observe)  │  │  (Deploy)   │         │
│  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘         │
├─────────────────────────────────────────────────────────────────────────────┤
│                              PLATFORM SERVICES                               │
│  ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐      │
│  │ Keycloak  │ │   Istio   │ │   OTel    │ │   Mimir   │ │   KEDA    │      │
│  │ (Identity)│ │  (Mesh)   │ │(Collector)│ │Loki/Tempo │ │(Autoscale)│      │
│  └───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────┘      │
├─────────────────────────────────────────────────────────────────────────────┤
│                              KUBERNETES (EKS)                                │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  Namespaces: argocd | monitoring | istio-system | keycloak | keda   │    │
│  │              platform | apps-*                                       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
├─────────────────────────────────────────────────────────────────────────────┤
│                              AWS INFRASTRUCTURE                              │
│  ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐      │
│  │    VPC    │ │    EKS    │ │    RDS    │ │    S3     │ │    ECR    │      │
│  │   + TGW   │ │  + Nodes  │ │ PostgreSQL│ │  Storage  │ │  Registry │      │
│  └───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────┘      │
│  ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐      │
│  │    IAM    │ │    KMS    │ │ Route 53  │ │    ALB    │ │ Secrets   │      │
│  │  + IRSA   │ │Encryption │ │    DNS    │ │    LB     │ │  Manager  │      │
│  └───────────┘ └───────────┘ └───────────┘ └───────────┘ └───────────┘      │
└─────────────────────────────────────────────────────────────────────────────┘
```

### 4.9 Recommended Versions

| Component | Minimum Version | Notes |
|------------|---------------|-------|
| **EKS** | 1.29 | Pod Identity support |
| **Istio** | 1.20+ | Ambient mesh ready |
| **Argo CD** | 2.10+ | ApplicationSets GA |
| **KEDA** | 2.13+ | AWS Scalers (SQS, MSK, etc.) |
| **Grafana** | 10.x | Unified alerting |
| **Mimir** | 2.11+ | Ingest storage |
| **Loki** | 3.0+ | TSDB backend |
| **Tempo** | 2.4+ | vParquet3 |
| **OTel Collector** | 0.96+ | Connector support |
| **Keycloak** | 24+ | Quarkus-based |
| **Backstage** | 1.24+ | New frontend system |
| **Terraform** | 1.6+ | Testing framework |
| **Helm** | 3.14+ | OCI support |

---

## 5. Repository Structure

### 5.1 Overview

```
helpdev/
├── platform-terraform-aws-accounts/                 # Account and cluster IaC
├── platform-terraform-aws-modules/                  # Reusable and versioned Terraform modules
├── platform-terraform-aws-projects-resources/       # AWS resources per service (SQS, RDS, S3, etc.)
├── platform-argocd-config/      # App-of-Apps and Argo CD configuration
├── platform-helm-charts/        # Reusable charts
├── platform-observability/      # Observability stack
├── platform-keycloak/           # Keycloak configuration (realms, clients, roles)
├── platform-pipelines/          # GitHub Actions templates
├── platform-kong/               # Kong Gateway infrastructure (Helm, base config)
├── platform-apis/               # OpenAPI specs, routes, plugins configuration
│
├── service-xyz/                 # Service XYZ code
├── service-xyz-infra/           # Service XYZ GitOps infra
└── ...                          # 50+ services
```

### 5.2 Repository Details

#### `platform-terraform-aws-accounts`
```
platform-terraform-aws-accounts/
├── helpdev-org-main/                     # AWS Account: Shared Services
│   └── global/                           # Global resources
│       └── us-east-1/                    # Primary region for global resources
│           ├── main.tf                   # Route53, CloudFront, ECR, ACM
│           ├── ecr.tf                    # ECR repositories + cross-account policies
│           ├── route53.tf                # DNS zones + health checks
│           ├── cloudfront.tf             # CDN distributions
│           ├── grafana.tf                # Central Grafana (EKS or Grafana Cloud)
│           ├── variables.tf
│           ├── terraform.tfvars
│           └── backend.tf
├── helpdev-hml/                          # AWS Account: HML
│   └── hml/                              # Environment
│       └── us-east-1/                    # Primary HML region
│           ├── main.tf
│           ├── variables.tf
│           ├── terraform.tfvars
│           └── backend.tf                # S3 backend per region
├── helpdev-prod/                         # AWS Account: PROD
│   └── prod/                             # Environment
│       ├── us-east-1/                    # Primary PROD region
│       │   ├── main.tf
│       │   ├── variables.tf
│       │   ├── terraform.tfvars
│       │   └── backend.tf
│       └── sa-east-1/                    # Secondary region (structure prepared)
│           ├── main.tf
│           ├── variables.tf
│           ├── terraform.tfvars
│           └── backend.tf
└── README.md
```

> **Note:** The `helpdev-org-main` account manages shared resources. ECR cross-account access is configured via repository policies. See [ADR-012](adr/ADR-012-shared-services-account.md).

#### `platform-terraform-aws-modules`

Reusable and versioned Terraform modules for infrastructure and application resources:

```
platform-terraform-aws-modules/
├── modules/
│   ├── vpc/                               # VPC module (reusable)
│   ├── eks/                               # EKS module (reusable)
│   ├── iam/                               # IAM roles and policies
│   ├── kms/                               # KMS keys
│   ├── rds/                               # RDS module (databases)
│   ├── ecr/                               # ECR module (shared registry)
│   ├── route53/                           # Route53 module (DNS zones)
│   ├── sqs/                               # SQS queues + DLQ
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   │
│   ├── sns/                               # SNS Topics
│   │   └── ...
│   │
│   ├── s3/                                # S3 Buckets with policies
│   │   └── ...
│   │
│   ├── dynamodb/                          # DynamoDB Tables
│   │   └── ...
│   │
│   ├── elasticache/                       # Redis/Memcached
│   │   └── ...
│   │
│   ├── secrets-manager/                   # Application Secrets
│   │   └── ...
│   │
│   └── sqs-sns-fanout/                    # Composite: SNS → multiple SQS
│       └── ...
│
├── examples/
│   ├── sqs-basic/
│   ├── sqs-with-dlq/
│   ├── rds-postgresql/
│   └── s3-with-lifecycle/
│
├── tests/                                 # Terratest
│   └── ...
│
├── CHANGELOG.md
└── README.md
```

**Versioning:**
- Git tags: `v1.0.0`, `v1.1.0`, `v2.0.0`
- Reference by tag: `source = "git::https://github.com/helpdev/platform-terraform-modules.git//modules/sqs?ref=v1.2.0"`
- Breaking changes = major version bump
- Changelog maintained for each release

**Standard Module Interface:**

All modules follow a consistent interface:

```hcl
# Required variables (all modules)
variable "name" {
  description = "Resource name"
  type        = string
}

variable "environment" {
  description = "Environment (hml, prod)"
  type        = string
}

variable "region" {
  description = "AWS Region"
  type        = string
}

variable "owner_service" {
  description = "Service that owns the resource"
  type        = string
}

variable "owner_team" {
  description = "Team responsible for the resource"
  type        = string
}

# Standard outputs (all modules)
output "arn" {
  description = "Resource ARN"
}

output "id" {
  description = "Resource ID"
}

output "name" {
  description = "Resource name"
}
```

#### `platform-terraform-aws-projects-resources`

AWS resource definitions per service, provisioned via self-service in Backstage:

```
platform-terraform-aws-projects-resources/
├── helpdev-hml/                           # AWS Account: HML
│   └── hml/                               # Environment
│       └── us-east-1/                     # Region
│           ├── sqs/
│           │   ├── payment-api/
│           │   │   ├── order-events.tf    # Order events queue
│           │   │   └── terraform.tfvars
│           │   └── order-service/
│           │       └── ...
│           │
│           ├── sns/
│           │   └── notification-service/
│           │       └── ...
│           │
│           ├── rds/
│           │   └── user-service/
│           │       ├── main.tf
│           │       └── terraform.tfvars
│           │
│           ├── s3/
│           │   └── document-service/
│           │       └── ...
│           │
│           └── dynamodb/
│               └── session-service/
│                   └── ...
│
├── helpdev-prod/                          # AWS Account: PROD
│   └── prod/
│       ├── us-east-1/                     # Primary region
│       │   ├── sqs/
│       │   │   └── payment-api/
│       │   │       └── ...
│       │   └── rds/
│       │       └── user-service/
│       │           └── ...
│       │
│       └── sa-east-1/                     # Secondary region
│           ├── sqs/
│           │   └── payment-api/
│           │       └── ...
│           └── rds/
│               └── user-service/
│                   └── ...
│
├── .github/
│   └── workflows/
│       ├── terraform-plan.yaml            # On PR: plan and comment
│       ├── terraform-apply.yaml           # On merge: apply
│       └── terraform-drift.yaml           # Scheduled: detects drift
│
├── CODEOWNERS
└── README.md
```

**Path Pattern:** `{account}/{environment}/{region}/{resource-type}/{service-name}/`

**Resource Example (SQS):**

```hcl
# helpdev-prod/prod/us-east-1/sqs/payment-api/order-events.tf

module "order_events_queue" {
  source = "git::https://github.com/helpdev/platform-terraform-modules.git//modules/sqs?ref=v1.2.0"

  name        = "payment-api-order-events"
  environment = "prod"
  region      = "us-east-1"
  
  # Ownership
  owner_service = "payment-api"
  owner_team    = "payments-team"
  
  # Queue configuration
  visibility_timeout_seconds = 30
  message_retention_seconds  = 1209600  # 14 days
  
  # DLQ
  create_dlq        = true
  max_receive_count = 3
  
  # Access
  allowed_services = ["order-processor", "payment-api"]
  
  tags = {
    Service     = "payment-api"
    Environment = "prod"
    Region      = "us-east-1"
    ManagedBy   = "terraform"
    CreatedVia  = "backstage"
  }
}

output "queue_url" {
  value = module.order_events_queue.queue_url
}

output "queue_arn" {
  value = module.order_events_queue.queue_arn
}
```

**Provisioning Flow (Self-Service):**

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant BS as Backstage
    participant GH as GitHub
    participant CI as GitHub Actions
    participant AWS as AWS
    participant ESO as External Secrets

    Dev->>BS: 1. Requests resource (SQS, RDS, etc.)
    Note over BS: Fills form with config
    
    BS->>GH: 2. Creates PR in platform-project-resources
    Note over GH: Uses versioned module
    
    GH->>CI: 3. Trigger terraform plan
    CI->>GH: 4. Comments PR with plan output
    
    alt HML Environment
        GH->>GH: 5a. 1 approval from team
    else PROD Environment
        GH->>GH: 5b. 2 approvals (includes Platform Team)
    end
    
    GH->>CI: 6. Merge triggers terraform apply
    CI->>AWS: 7. Creates resource
    CI->>AWS: 8. Saves outputs in Secrets Manager
    
    ESO->>AWS: 9. Sync secrets to K8s
    Note over Dev: Service consumes via env vars
```

**Governance (CODEOWNERS):**

```
# platform-terraform-aws-projects-resources/CODEOWNERS

# PROD requires Platform Team approval
/helpdev-prod/              @helpdev/platform-team

# HML can be approved by service owners
/helpdev-hml/               @helpdev/developers

# RDS always requires DBA review
**/rds/**                   @helpdev/platform-team @helpdev/dba-team

# Sensitive resources require Security review
**/secrets-manager/**       @helpdev/security-team
```

| Environment | Approval Rules |
|----------|---------------------|
| HML | 1 approval from team member |
| PROD | 2 approvals, including Platform Team |
| RDS (any) | DBA Team approval required |
| Secrets | Security Team approval |

#### `platform-argocd-config`
```
platform-argocd-config/
├── clusters/
│   ├── hml/
│   │   └── us-east-1/
│   │       └── app-of-apps.yaml          # Apps for eks-hml-use1
│   └── prod/
│       ├── us-east-1/
│       │   └── app-of-apps.yaml          # Apps for eks-prod-use1
│       └── sa-east-1/
│           └── app-of-apps.yaml          # Apps for eks-prod-sae1 (future)
├── projects/
│   ├── platform.yaml
│   ├── apps-domain-a.yaml
│   └── apps-domain-b.yaml
├── applicationsets/
│   └── services.yaml
├── applications/
│   ├── service-checkout.yaml
│   ├── service-users.yaml
│   └── ...
└── README.md
```

**Responsibilities:**
- Argo CD Projects definition (isolation by domain)
- App-of-Apps per cluster
- ApplicationSets for automated deploy
- Individual Applications (created by Backstage)

**Multi-Region ApplicationSet:**

The ApplicationSet automatically generates Applications for each service × environment × region combination:

```yaml
# applicationsets/services.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: services
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          # Generator 1: Reads directory structure from infra repo
          - git:
              repoURL: https://github.com/helpdev/{{service}}-infra
              directories:
                - path: environments/*/*  # Captures: environments/{env}/{region}
          # Generator 2: Maps to correct cluster
          - list:
              elements:
                - cluster: eks-hml-use1
                  clusterUrl: https://eks-hml-use1.eks.amazonaws.com
                  env: hml
                  region: us-east-1
                - cluster: eks-hml-sae1
                  clusterUrl: https://eks-hml-sae1.eks.amazonaws.com
                  env: hml
                  region: sa-east-1
                - cluster: eks-prod-use1
                  clusterUrl: https://eks-prod-use1.eks.amazonaws.com
                  env: prod
                  region: us-east-1
                - cluster: eks-prod-sae1
                  clusterUrl: https://eks-prod-sae1.eks.amazonaws.com
                  env: prod
                  region: sa-east-1
  template:
    metadata:
      name: '{{service}}-{{env}}-{{region}}'
      labels:
        app.kubernetes.io/name: '{{service}}'
        helpdev.io/environment: '{{env}}'
        helpdev.io/region: '{{region}}'
    spec:
      project: '{{domain}}'
      source:
        repoURL: https://github.com/helpdev/{{service}}-infra
        targetRevision: HEAD
        path: environments/{{env}}/{{region}}
        helm:
          valueFiles:
            - ../../chart/values-common.yaml
            - values.yaml
      destination:
        server: '{{clusterUrl}}'
        namespace: '{{service}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
```

**How It Works:**
1. The `git` generator discovers directories in `environments/*/*`
2. The `list` generator defines the cluster → URL mapping
3. The matrix combines both to generate an Application per existing combination
4. Result: If `payment-api` has `environments/prod/us-east-1/`, an Application `payment-api-prod-us-east-1` is automatically created

#### `platform-helm-charts`
```
platform-helm-charts/
├── charts/
│   ├── service-base/                    # Base chart for microservices
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   ├── templates/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   ├── hpa.yaml
│   │   │   ├── pdb.yaml
│   │   │   ├── serviceaccount.yaml
│   │   │   ├── configmap.yaml
│   │   │   ├── secret.yaml
│   │   │   ├── ingress.yaml
│   │   │   ├── networkpolicy.yaml
│   │   │   ├── virtualservice.yaml      # Istio
│   │   │   ├── destinationrule.yaml     # Istio
│   │   │   ├── requestauthentication.yaml # Istio + Keycloak
│   │   │   ├── authorizationpolicy.yaml # Istio
│   │   │   └── _helpers.tpl
│   │   └── README.md
│   │
│   ├── cronjob-base/                    # Base chart for CronJobs
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   └── templates/
│   │       └── cronjob.yaml
│   │
│   ├── worker-base/                     # Base chart for workers/consumers
│   │   ├── Chart.yaml
│   │   ├── values.yaml
│   │   └── templates/
│   │       ├── deployment.yaml
│   │       └── keda-scaledobject.yaml   # KEDA ScaledObject
│   │
│   └── library/                         # Chart library (shared helpers)
│       ├── Chart.yaml
│       └── templates/
│           └── _helpers.tpl
│
├── docs/
│   ├── service-base.md
│   ├── cronjob-base.md
│   ├── worker-base.md
│   └── keda-scaling.md                  # KEDA scaling documentation
│
├── examples/
│   ├── java-api/
│   ├── nodejs-api/
│   └── python-worker/
│
└── README.md
```

**Responsibilities:**
- Reusable charts for all workload types
- Standardization of resources, probes, security contexts
- Integration with Istio (VirtualService, DestinationRule)
- Integration with Keycloak (RequestAuthentication)
- Integration with KEDA (ScaledObject for workers)
- Documentation and usage examples

**KEDA Configuration in Base Helm Chart:**

> 📄 **File:** [keda-values.yaml](samples/helm/values/keda-values.yaml)

**ScaledObject Template (worker-base):**

> 📄 **File:** [scaled-object.yaml](samples/helm/templates/scaled-object.yaml)

**Versioning:**
- Semantic versioning (e.g.: `1.2.3`)
- Published in OCI registry (ECR) or ChartMuseum
- Services reference specific version in `Chart.yaml`

#### `platform-observability`
```
platform-observability/
├── terraform/                           # AWS resources for observability
│   ├── modules/
│   │   ├── s3-mimir/                   # Buckets for Mimir
│   │   ├── s3-loki/                    # Buckets for Loki
│   │   ├── s3-tempo/                   # Buckets for Tempo
│   │   └── iam-observability/          # IRSA roles
│   ├── environments/
│   │   ├── hml/
│   │   │   └── us-east-1/              # S3/IAM resources per region
│   │   │       ├── main.tf
│   │   │       └── terraform.tfvars
│   │   └── prod/
│   │       ├── us-east-1/
│   │       │   ├── main.tf
│   │       │   └── terraform.tfvars
│   │       └── sa-east-1/              # Secondary region (prepared)
│   │           ├── main.tf
│   │           └── terraform.tfvars
│   └── README.md
│
├── helm/                                # Values for deploy via Argo CD
│   ├── mimir/
│   │   ├── values-common.yaml
│   │   └── environments/
│   │       ├── hml/
│   │       │   └── us-east-1/
│   │       │       └── values.yaml
│   │       └── prod/
│   │           ├── us-east-1/
│   │           │   └── values.yaml
│   │           └── sa-east-1/
│   │               └── values.yaml
│   │
│   ├── loki/
│   │   ├── values-common.yaml
│   │   └── environments/
│   │       └── ... (same structure)
│   │
│   ├── tempo/
│   │   ├── values-common.yaml
│   │   └── environments/
│   │       └── ... (same structure)
│   │
│   ├── grafana/
│   │   ├── values-common.yaml
│   │   ├── environments/
│   │   │   └── prod/
│   │   │       └── us-east-1/
│   │   │           └── values.yaml     # Central Grafana only in prod/us-east-1
│   │   ├── dashboards/                 # Dashboards as code (shared)
│   │   │   ├── platform-overview.json
│   │   │   ├── service-health.json
│   │   │   ├── istio-mesh.json
│   │   │   ├── keycloak.json
│   │   │   └── multi-region.json       # Multi-region dashboard
│   │   └── datasources/
│   │       └── datasources.yaml        # All remote datasources
│   │
│   ├── otel-collector/
│   │   ├── values-common.yaml
│   │   └── environments/
│   │       └── ... (same structure)
│   │
│   └── alertmanager/
│       ├── values-common.yaml
│       ├── alerting-rules/
│       └── notification-channels/
│
├── alerts/                              # Centralized alerts
│   ├── slo/
│   ├── infrastructure/
│   └── security/
│
└── README.md
```

**Responsibilities:**
- Terraform for AWS resources (S3, IAM, KMS) per region
- Helm values for Grafana stack (Mimir, Loki, Tempo, Grafana) per region
- Dashboards as code (JSON) - shared across regions
- Centralized alerting rules
- Datasource configuration (federation from all regions)

> **Central Grafana:** Deploy only in `prod/us-east-1`. Configures remote datasources for all other regions and environments.

#### `platform-keycloak`

> **Multi-Region Strategy:** Keycloak per region with **GitOps sync**. Realm configuration is separated by account (hml/prd) and synchronized across regions within the same account. Runtime data (sessions) is isolated per region.

> **Important:** GitOps sync happens only within the same account (cross-region), never cross-account. This ensures HML changes don't affect PRD and allows progressive rollout.

```
platform-keycloak/
├── terraform/                           # AWS Infra for Keycloak (per region)
│   ├── modules/
│   │   ├── rds/                         # RDS PostgreSQL for Keycloak
│   │   └── secrets/                     # Secrets Manager
│   ├── environments/
│   │   ├── hml/
│   │   │   └── us-east-1/
│   │   │       ├── main.tf
│   │   │       └── terraform.tfvars
│   │   └── prod/
│   │       ├── us-east-1/
│   │       │   ├── main.tf
│   │       │   └── terraform.tfvars
│   │       └── sa-east-1/
│   │           ├── main.tf
│   │           └── terraform.tfvars
│   └── variables.tf
│
├── helm/                                # Helm values for Keycloak
│   ├── values-common.yaml               # Shared configuration
│   └── environments/
│       ├── hml/
│       │   └── us-east-1/
│       │       └── values.yaml          # RDS endpoint, resources
│       └── prod/
│           ├── us-east-1/
│           │   └── values.yaml
│           └── sa-east-1/
│               └── values.yaml
│
├── realms/                              # Separated by account - GitOps sync within account only
│   ├── helpdev-hml/                     # HML account realms
│   │   ├── helpdev-platform/            # Realm for tools (Argo, Grafana)
│   │   │   ├── realm.json
│   │   │   ├── clients/
│   │   │   │   ├── argocd.json
│   │   │   │   ├── grafana.json
│   │   │   │   └── backstage.json
│   │   │   ├── roles/
│   │   │   │   ├── platform-admin.json
│   │   │   │   ├── platform-viewer.json
│   │   │   │   └── developer.json
│   │   │   └── identity-providers/
│   │   │       └── github.json          # SSO with GitHub
│   │   │
│   │   └── helpdev-services/            # Realm for services (APIs)
│   │       ├── realm.json
│   │       ├── clients/                 # Clients created via Backstage
│   │       │   └── .gitkeep
│   │       ├── roles/
│   │       │   ├── service-admin.json
│   │       │   └── service-reader.json
│   │       └── client-scopes/
│   │           ├── api-read.json
│   │           └── api-write.json
│   │
│   └── helpdev-prd/                     # PRD account realms
│       ├── helpdev-platform/            # Realm for tools (Argo, Grafana)
│       │   ├── realm.json
│       │   ├── clients/
│       │   │   ├── argocd.json
│       │   │   ├── grafana.json
│       │   │   └── backstage.json
│       │   ├── roles/
│       │   │   ├── platform-admin.json
│       │   │   ├── platform-viewer.json
│       │   │   └── developer.json
│       │   └── identity-providers/
│       │       └── github.json          # SSO with GitHub
│       │
│       └── helpdev-services/            # Realm for services (APIs)
│           ├── realm.json
│           ├── clients/                 # Clients created via Backstage
│           │   └── .gitkeep
│           ├── roles/
│           │   ├── service-admin.json
│           │   └── service-reader.json
│           └── client-scopes/
│               ├── api-read.json
│               └── api-write.json
│
├── scripts/
│   ├── import-realm.sh                  # Script to import realm
│   ├── export-realm.sh                  # Script to export realm
│   ├── create-client.sh                 # Used by Backstage
│   └── promote-to-prod.sh               # Script to promote HML → PRD configs
│
├── docs/
│   ├── sso-setup.md
│   ├── service-authentication.md
│   ├── multi-region-sync.md             # Documentation for sync between regions
│   ├── account-separation.md            # Documentation for account separation
│   └── troubleshooting.md
│
└── README.md
```

**Responsibilities:**
- Terraform for RDS PostgreSQL (HA) and Secrets Manager **per region**
- Helm values for Keycloak deploy in the cluster **per region**
- Realms versioned as code - **separated by account, synced across regions within account**
- Automation scripts for import/export and HML → PRD promotion
- SSO configuration and multi-region sync documentation

**Realms:**

| Account | Realm | Purpose | Clients | Synchronization |
|---------|-------|---------|---------|-----------------|
| `helpdev-hml` | `helpdev-platform` | Platform tools | Argo CD, Grafana, Backstage | GitOps (hml regions only) |
| `helpdev-hml` | `helpdev-services` | APIs and microservices | Created via Backstage | GitOps (hml regions only) |
| `helpdev-prd` | `helpdev-platform` | Platform tools | Argo CD, Grafana, Backstage | GitOps (prd regions only) |
| `helpdev-prd` | `helpdev-services` | APIs and microservices | Created via Backstage | GitOps (prd regions only) |

**Account-Separated Synchronization Flow:**

```mermaid
flowchart TB
    subgraph Git["Git Repository: platform-keycloak"]
        subgraph RealmsHML["realms/helpdev-hml/"]
            HML_PLATFORM["helpdev-platform/"]
            HML_SERVICES["helpdev-services/"]
        end
        subgraph RealmsPRD["realms/helpdev-prd/"]
            PRD_PLATFORM["helpdev-platform/"]
            PRD_SERVICES["helpdev-services/"]
        end
    end
    
    subgraph HMLAccount["AWS Account: helpdev-hml"]
        ARGO_HML["Argo CD hml/us-east-1"]
        KC_HML["Keycloak hml/us-east-1"]
    end
    
    subgraph PRDAccount["AWS Account: helpdev-prd"]
        ARGO_PRD_USE1["Argo CD prd/us-east-1"]
        ARGO_PRD_SAE1["Argo CD prd/sa-east-1"]
        KC_PRD_USE1["Keycloak prd/us-east-1"]
        KC_PRD_SAE1["Keycloak prd/sa-east-1"]
    end
    
    RealmsHML --> ARGO_HML --> KC_HML
    RealmsPRD --> ARGO_PRD_USE1 --> KC_PRD_USE1
    RealmsPRD --> ARGO_PRD_SAE1 --> KC_PRD_SAE1
    
    style RealmsHML fill:#ffd,stroke:#333
    style RealmsPRD fill:#dfd,stroke:#333
```

**GitOps Sync Rules:**

| ArgoCD Instance | Source Path | Syncs To |
|-----------------|-------------|----------|
| hml/us-east-1 | `realms/helpdev-hml/` | Keycloak hml/us-east-1 |
| prd/us-east-1 | `realms/helpdev-prd/` | Keycloak prd/us-east-1 |
| prd/sa-east-1 | `realms/helpdev-prd/` | Keycloak prd/sa-east-1 |

**HML → PRD Promotion Workflow:**

1. Developer commits realm change to `realms/helpdev-hml/`
2. Argo CD syncs to HML cluster
3. Testing and validation in HML environment
4. Create PR to copy changes from `realms/helpdev-hml/` → `realms/helpdev-prd/`
5. Code review by Platform Team + Security Team
6. Merge triggers Argo CD sync to PRD clusters (all regions)

> **Note:** User sessions and tokens remain isolated per region (no runtime data replication). Configuration changes must be validated in HML before promotion to PRD.

#### `platform-pipelines`
```
platform-pipelines/
├── .github/
│   └── workflows/
│       └── test-actions.yaml            # CI to test the actions
│
├── actions/                             # Reusable Composite Actions
│   ├── setup-java/
│   │   └── action.yaml
│   ├── setup-node/
│   │   └── action.yaml
│   ├── setup-go/
│   │   └── action.yaml
│   ├── setup-python/
│   │   └── action.yaml
│   │
│   ├── docker-build-push/
│   │   └── action.yaml                  # Build + Push to ECR (us-east-1)
│   │
│   ├── security-scan/
│   │   └── action.yaml                  # Trivy, Snyk, etc.
│   │
│   ├── update-gitops/
│   │   └── action.yaml                  # Updates digest in infra repo
│   │
│   ├── phased-rollout/
│   │   └── action.yaml                  # Multi-region deploy with order/delay
│   │
│   ├── notify-slack/
│   │   └── action.yaml
│   │
│   └── deploy-status/
│       └── action.yaml                  # Checks sync in Argo CD
│
├── workflows/                           # Workflow templates
│   ├── java-service.yaml
│   ├── kotlin-service.yaml
│   ├── nodejs-service.yaml
│   ├── go-service.yaml
│   ├── python-service.yaml
│   │
│   ├── multi-region-deploy.yaml         # Workflow for phased rollout
│   ├── helm-chart-release.yaml          # For platform-helm-charts
│   └── terraform-apply.yaml             # For platform-terraform
│
├── scripts/
│   ├── extract-digest.sh
│   ├── update-values.sh
│   ├── wait-argocd-sync.sh
│   └── phased-rollout.sh                # Multi-region rollout script
│
├── docs/
│   ├── java-pipeline.md
│   ├── security-scan.md
│   ├── gitops-update.md
│   └── multi-region-rollout.md          # Multi-region rollout documentation
│
└── README.md
```

**Responsibilities:**
- Reusable Composite Actions
- Workflow templates per language
- **Phased rollout action** for multi-region deploy
- Auxiliary scripts
- Usage documentation

**Phased Rollout (Multi-Region):**

The `phased-rollout` action reads the rollout configuration from the service's `catalog-info.yaml` and executes deploy region by region:

```yaml
# catalog-info.yaml of the service
spec:
  deployment:
    rollout:
      prod:
        regions:
          - name: us-east-1
            order: 1
          - name: sa-east-1
            order: 2
            delayAfterPrevious: 30m    # Waits 30 min after us-east-1
            healthCheckDuration: 5m    # Monitors health for 5 min
```

**Rollout Workflow:**

```mermaid
flowchart LR
    A[Build Image] --> B[Push ECR]
    B --> C[Read catalog-info.yaml]
    C --> D[Deploy Region 1]
    D --> E{Health OK?}
    E -->|Yes| F[Wait delay]
    E -->|No| G[Rollback]
    F --> H[Deploy Region 2]
    H --> I{Health OK?}
    I -->|Yes| J[✅ Complete]
    I -->|No| K[Rollback Region 2]
```

**Workflow Template Example (Java) - Used by Service:**

> 📄 **File:** [service-ci-cd.yaml](samples/github-actions/service-ci-cd.yaml)

**Reusable Workflow Template (Java):**

> 📄 **File:** [java-service-workflow.yaml](samples/github-actions/java-service-workflow.yaml)

#### `service-xyz` (Code Pattern per Service)
```
service-xyz/
├── src/
│   └── main/
│       ├── java/io/helpdev/xyz/
│       │   ├── Application.java
│       │   ├── config/
│       │   │   ├── SecurityConfig.java
│       │   │   └── OpenTelemetryConfig.java
│       │   ├── api/
│       │   │   ├── controllers/
│       │   │   └── dto/
│       │   ├── domain/
│       │   │   ├── entities/
│       │   │   ├── services/
│       │   │   └── repositories/
│       │   └── infrastructure/
│       │       ├── clients/
│       │       └── messaging/
│       └── resources/
│           ├── application.yaml
│           ├── application-hml.yaml
│           └── application-prod.yaml
│
├── src/test/
│   └── java/io/helpdev/xyz/
│       ├── unit/
│       ├── integration/
│       └── contract/
│
├── .github/
│   └── workflows/
│       └── ci-cd.yaml                   # Usa workflow template
│
├── Dockerfile
├── catalog-info.yaml                    # Backstage metadata
├── README.md
├── CONTRIBUTING.md
├── pom.xml                              # or build.gradle
└── .gitignore
```

**`catalog-info.yaml` Structure:**

> 📄 **File:** [catalog-info-example.yaml](samples/backstage/catalog-info-example.yaml)

**CI/CD Pipeline (`ci-cd.yaml`):**
```yaml
name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build-deploy:
    uses: helpdev/platform-pipelines/.github/workflows/java-service.yaml@main
    with:
      service-name: service-xyz
      java-version: '21'
    secrets: inherit
```

#### `service-xyz-infra` (Pattern per Service)
```
service-xyz-infra/
├── chart/
│   ├── Chart.yaml              # Base chart dependency
│   └── values-common.yaml      # Values shared between regions
├── environments/
│   ├── hml/
│   │   └── us-east-1/          # Primary HML region
│   │       ├── values.yaml     # Region-specific values
│   │       └── istio/          # Access policies
│   │           ├── request-authentication.yaml
│   │           └── authorization-policy.yaml
│   └── prod/
│       ├── us-east-1/          # Primary PROD region
│       │   ├── values.yaml
│       │   └── istio/
│       │       ├── request-authentication.yaml
│       │       └── authorization-policy.yaml
│       └── sa-east-1/          # Secondary region (when enabled)
│           ├── values.yaml
│           └── istio/
│               ├── request-authentication.yaml
│               └── authorization-policy.yaml
└── README.md
```

> **Note:** Each region has its own values and Istio policies. This allows different configurations per region (e.g., replicas, resources) and independent access control.

### 5.3 Naming Conventions

#### Region Codes

| AWS Region | Short Code | Usage |
|------------|------------|-------|
| `us-east-1` | `use1` | Primary region |
| `sa-east-1` | `sae1` | Brazil |
| `eu-west-1` | `euw1` | Europe (future) |
| `ap-southeast-1` | `apse1` | Asia (future) |

#### Infrastructure Resources

| Resource | Pattern | Example |
|---------|--------|---------|
| **AWS Account** | `helpdev-{env}` | `helpdev-hml`, `helpdev-prod` |
| **EKS Cluster** | `eks-{env}-{region_short}` | `eks-hml-use1`, `eks-prod-sae1` |
| **VPC** | `helpdev-{env}-{region}-vpc` | `helpdev-prod-us-east-1-vpc` |
| **S3 Bucket** | `helpdev-{purpose}-{env}-{region}` | `helpdev-mimir-prod-us-east-1` |
| **RDS** | `helpdev-{env}-{region}-{service}-db` | `helpdev-prod-us-east-1-keycloak-db` |
| **ECR** | `{account}.dkr.ecr.us-east-1.amazonaws.com/{service}` | ECR only in us-east-1 |

#### Kubernetes Resources

| Resource | Pattern | Example |
|---------|--------|---------|
| **Namespace** | `apps-{domain}` | `apps-payments`, `apps-users` |
| **Service Account** | `{service-name}` | `payment-api` |

#### Git Repositories

| Resource | Pattern | Example |
|---------|--------|---------|
| **Service Repo** | `service-{name}` | `service-checkout` |
| **Infra Repo** | `service-{name}-infra` | `service-checkout-infra` |

#### Folder Structure per Region

```
environments/
├── hml/
│   └── us-east-1/          # Primary region
│       ├── values.yaml
│       └── istio/
└── prod/
    ├── us-east-1/          # Primary region
    │   ├── values.yaml
    │   └── istio/
    └── sa-east-1/          # Secondary region (when enabled)
        ├── values.yaml
        └── istio/
```

#### Secrets (AWS Secrets Manager)

| Padrão | Exemplo |
|--------|---------|
| `helpdev/{env}/{region}/{domain}/{service}/{type}` | `helpdev/prod/us-east-1/payments/checkout/database` |

### 5.4 Standard Labels and Annotations

> 📄 **File:** [labels-annotations-example.yaml](samples/kubernetes/labels-annotations-example.yaml)

### 5.5 Developer Portal (Backstage)

**Backstage** is adopted as the Developer Portal and self-service platform for creating new services. It automates the entire scaffolding process, ensuring standardization and reducing onboarding time.

#### Overview

```mermaid
flowchart LR
    subgraph Backstage["Backstage Portal"]
        CATALOG["Software Catalog"]
        TEMPLATES["Software Templates"]
        TECHDOCS["TechDocs"]
    end
    
    subgraph Created["Automatically Created Resources"]
        REPO_CODE["service-xyz (code)"]
        REPO_INFRA["service-xyz-infra (GitOps)"]
        ARGO_APP["Argo CD Application"]
        KC_CLIENT["Keycloak Client"]
    end
    
    DEV["Developer"] --> TEMPLATES
    TEMPLATES --> REPO_CODE
    TEMPLATES --> REPO_INFRA
    TEMPLATES --> ARGO_APP
    TEMPLATES --> KC_CLIENT
    
    REPO_CODE --> CATALOG
    REPO_INFRA --> CATALOG
```

#### Backstage Components

| Component | Description |
|-----------|-------------|
| **Software Catalog** | Inventory of all services, owners, dependencies and documentation |
| **Software Templates** | Templates for automated creation of new services |
| **TechDocs** | Integrated technical documentation (docs-as-code) |
| **Plugins** | Integrations with GitHub, Argo CD, Kubernetes, Keycloak |

#### Namespace and Deploy

```yaml
# Backstage is deployed in the `platform` namespace
namespace: platform
components:
  - backstage (frontend + backend)
  - postgresql (or RDS)
```

#### Software Template: New Service

When a developer creates a new service via Backstage, the template automatically executes:

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant BS as Backstage
    participant GH as GitHub API
    participant Argo as platform-argocd-config
    participant KC as platform-keycloak (Git)
    
    Dev->>BS: Fills form (name, owner, domain, tier, regions)
    BS->>BS: Validates inputs and naming conventions
    
    par Parallel creation
        BS->>GH: Creates repo service-{name}
        Note over GH: Code + Dockerfile + CI pipeline
        
        BS->>GH: Creates repo service-{name}-infra
        Note over GH: environments/{env}/{region}/ for each selected region
        
        BS->>Argo: Commit in platform-argocd-config
        Note over Argo: ApplicationSet for multi-region
        
        BS->>KC: Commit client JSON in realms/helpdev-services/clients/
        Note over KC: GitOps sync propagates to all regions
    end
    
    BS-->>Dev: Returns URLs of created repos
    Dev->>GH: Clones and starts developing
```

#### Creation Form Fields

| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `name` | text | Service name | `payment-api` |
| `owner` | select | Responsible team | `payments-team` |
| `domain` | select | Business domain | `payments` |
| `tier` | select | Criticality | `tier-1`, `tier-2`, `tier-3` |
| `language` | select | Language/runtime | `java`, `kotlin`, `node`, `go` |
| `regions` | multiselect | Regions for deploy | `us-east-1`, `sa-east-1` |

> **Region Default:** If no region is selected, the service is created only for `us-east-1`. Additional regions can be enabled later by editing the `catalog-info.yaml`.

#### Structure Generated in Infra Repo

Based on the selected regions, the template generates:

```
service-xyz-infra/
├── chart/
│   └── values-common.yaml
├── environments/
│   ├── hml/
│   │   ├── us-east-1/              # ← Generated if us-east-1 selected
│   │   │   ├── values.yaml
│   │   │   └── istio/
│   │   └── sa-east-1/              # ← Generated if sa-east-1 selected
│   │       ├── values.yaml
│   │       └── istio/
│   └── prod/
│       ├── us-east-1/
│       │   ├── values.yaml
│       │   └── istio/
│       └── sa-east-1/
│           ├── values.yaml
│           └── istio/
└── catalog-info.yaml               # Includes rollout config
```

#### Generated catalog-info.yaml

```yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.name }}
  annotations:
    github.com/project-slug: helpdev/${{ values.name }}
    backstage.io/techdocs-ref: dir:.
    helpdev.io/regions: "${{ values.regions | join(',') }}"
spec:
  type: service
  lifecycle: experimental
  owner: ${{ values.owner }}
  system: ${{ values.domain }}
  
  deployment:
    rollout:
      hml:
        regions:
          - name: us-east-1
            order: 1
          - name: sa-east-1
            order: 2
            delayAfterPrevious: 0
      prod:
        regions:
          - name: us-east-1
            order: 1
            healthCheckDuration: 5m
          - name: sa-east-1
            order: 2
            delayAfterPrevious: 30m
            healthCheckDuration: 5m
```

#### Template Structure

> 📄 **File:** [new-microservice-template.yaml](samples/backstage/templates/new-microservice-template.yaml)

#### Code Repository Skeleton

```
skeleton/java/
├── src/
│   └── main/
│       ├── java/
│       │   └── io/helpdev/${{values.name | replace('-', '')}}/
│       │       ├── Application.java
│       │       ├── config/
│       │       └── api/
│       └── resources/
│           └── application.yaml
├── .github/
│   └── workflows/
│       └── build-deploy.yaml      # Pipeline already configured
├── Dockerfile
├── catalog-info.yaml              # Metadata for Backstage
├── README.md
└── pom.xml
```

#### Infra Repository Skeleton

```
skeleton/infra/
├── chart/
│   ├── Chart.yaml
│   └── values-common.yaml
├── environments/
│   ├── hml/
│   │   └── values.yaml
│   └── prod/
│       └── values.yaml
└── README.md
```

#### Automatically Generated Values

> 📄 **File:** [service-hml-values.yaml](samples/helm/values/service-hml-values.yaml)

#### Service Catalog

The automatically generated `catalog-info.yaml` file:

> 📄 **File:** [catalog-info-template.yaml](samples/backstage/catalog-info-template.yaml)

#### Recommended Plugins

| Plugin | Description |
|--------|-----------|
| `@backstage/plugin-github-actions` | View pipeline status |
| `@backstage/plugin-kubernetes` | View K8s pods, logs, events |
| `@roadiehq/backstage-plugin-argo-cd` | Argo CD sync status |
| `@backstage/plugin-techdocs` | Integrated documentation |
| Custom: `@helpdev/plugin-keycloak` | Keycloak client management |
| Custom: `@helpdev/plugin-service-permissions` | Service-to-service permissions management |
| Custom: `@helpdev/plugin-resources` | Self-service for AWS resources (SQS, RDS, S3) |

#### Software Template: New AWS Resource

Backstage allows creating AWS resources (SQS, RDS, S3, SNS, etc.) via self-service. The template generates a PR in the `platform-project-resources` repository.

```mermaid
flowchart LR
    subgraph Backstage["Backstage Portal"]
        FORM["Resource Request Form"]
    end
    
    subgraph GitHub["GitHub"]
        PR["Pull Request<br/>platform-project-resources"]
        TF["Terraform Code<br/>(uses versioned module)"]
    end
    
    subgraph AWS["AWS"]
        RES["Resource Created<br/>(SQS, RDS, S3...)"]
        SEC["Secrets Manager<br/>(outputs)"]
    end
    
    subgraph K8s["Kubernetes"]
        ESO["External Secrets<br/>Operator"]
        POD["Service Pod<br/>(consumes via env)"]
    end
    
    FORM --> PR
    PR --> TF
    TF -->|"terraform apply"| RES
    RES --> SEC
    SEC --> ESO
    ESO --> POD
```

##### Form Fields

| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `service` | select | Service that will use the resource | `payment-api` |
| `resourceType` | select | AWS resource type | `sqs`, `rds`, `s3`, `sns`, `dynamodb`, `elasticache` |
| `resourceName` | text | Resource name | `order-events` |
| `environment` | select | Target environment | `hml`, `prod` |
| `regions` | multiselect | Target regions | `us-east-1`, `sa-east-1` |
| `config` | dynamic | Type-specific configuration | (varies by type) |

##### Configurations per Resource Type

**SQS:**
| Field | Description | Default |
|-------|-------------|---------|
| `visibilityTimeout` | Visibility timeout (sec) | 30 |
| `messageRetention` | Message retention (days) | 14 |
| `createDlq` | Create Dead Letter Queue | true |
| `maxReceiveCount` | Attempts before DLQ | 3 |
| `allowedServices` | Services with access | [owner] |

**RDS:**
| Field | Description | Default |
|-------|-------------|---------|
| `engine` | Database engine | `postgres` |
| `engineVersion` | Version | `15.4` |
| `instanceClass` | Instance class | `db.t3.micro` (hml) / `db.r6g.large` (prod) |
| `allocatedStorage` | Storage in GB | 20 |
| `multiAz` | Multi-AZ (high availability) | false (hml) / true (prod) |

**S3:**
| Field | Description | Default |
|-------|-------------|---------|
| `versioning` | Enable versioning | true |
| `encryption` | Encryption | `aws:kms` |
| `lifecycleRules` | Lifecycle rules | [] |
| `allowedServices` | Services with access | [owner] |

##### Example: Creating an SQS Queue

1. Developer accesses Backstage → Templates → "New AWS Resource"
2. Fills form:
   - Service: `payment-api`
   - Type: `SQS`
   - Name: `order-events`
   - Environment: `prod`
   - Regions: `us-east-1`, `sa-east-1`
   - DLQ: enabled, max 3 retries
   - Allowed services: `order-processor`

3. Backstage creates PR in `platform-project-resources`:
   - `helpdev-prod/prod/us-east-1/sqs/payment-api/order-events.tf`
   - `helpdev-prod/prod/sa-east-1/sqs/payment-api/order-events.tf`

4. PR requires 2 approvals (PROD) including Platform Team

5. After merge, GitHub Actions executes `terraform apply`

6. Outputs are saved in Secrets Manager:
   - `helpdev/prod/us-east-1/payments/payment-api/sqs-order-events-url`
   - `helpdev/prod/us-east-1/payments/payment-api/sqs-order-events-arn`

7. External Secrets Operator syncs to K8s Secret

8. Service consumes via environment variables

##### Consuming Resources in the Service

After the resource is created, the service consumes it via External Secrets:

```yaml
# service-payment-api-infra/environments/prod/us-east-1/values.yaml

externalSecrets:
  enabled: true
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  data:
    - secretKey: SQS_ORDER_EVENTS_URL
      remoteRef:
        key: helpdev/prod/us-east-1/payments/payment-api/sqs-order-events-url
    - secretKey: SQS_ORDER_EVENTS_ARN
      remoteRef:
        key: helpdev/prod/us-east-1/payments/payment-api/sqs-order-events-arn

env:
  - name: SQS_ORDER_EVENTS_URL
    valueFrom:
      secretKeyRef:
        name: payment-api-resources
        key: SQS_ORDER_EVENTS_URL
```

#### Service Authentication and Authorization

Backstage centralizes identity management and access permissions between services, integrating with Keycloak for OIDC/OAuth2 and with Istio for enforcement via Service Mesh.

##### Service Identity Model

```mermaid
flowchart TB
    subgraph Backstage["Backstage Portal"]
        SC["Software Catalog"]
        SP["Service Permissions Plugin"]
    end
    
    subgraph Keycloak["Keycloak"]
        REALM["Realm: helpdev-services"]
        CLIENTS["Service Clients"]
        ROLES["Client Roles"]
        POLICIES["Authorization Policies"]
    end
    
    subgraph Istio["Service Mesh (Istio)"]
        RA["RequestAuthentication"]
        AP["AuthorizationPolicy"]
    end
    
    SC --> SP
    SP -->|"Creates/Manages"| CLIENTS
    SP -->|"Defines"| ROLES
    SP -->|"Configures"| POLICIES
    
    CLIENTS -->|"JWT Tokens"| RA
    POLICIES -->|"Sync"| AP
```

##### Service Identity Lifecycle

When a service is created via Backstage, automatically:

| Step | Resource Created | Location |
|------|------------------|----------|
| 1 | OIDC Client | Keycloak (`helpdev-services` realm) |
| 2 | Default Client Roles | Keycloak (read, write, admin) |
| 3 | Service Account | Keycloak (client credentials flow) |
| 4 | RequestAuthentication | Kubernetes/Istio |
| 5 | Base AuthorizationPolicy | Kubernetes/Istio |

##### Supported Authentication Types

| Type | Usage | OAuth2 Flow |
|------|-----|-------------|
| **User → Service** | End user accesses API | Authorization Code + PKCE |
| **Service → Service** | Communication between microservices | Client Credentials |
| **Service → External** | Service accesses external API | Client Credentials |

##### Permissions Management in Backstage

The `@helpdev/plugin-service-permissions` plugin allows developers to request access to other services and owners to approve via Pull Request.

> **Ownership Model:** The AuthorizationPolicy resides in the infra repository of the **destination service** (who exposes the API), as the enforcement is done by the destination's Envoy sidecar. The destination service owner has full control over who can access it.

```mermaid
sequenceDiagram
    participant Dev as Developer (order-api)
    participant BS as Backstage
    participant KC as Keycloak
    participant GH as GitHub
    participant Owner as Owner (payment-api)
    participant Argo as Argo CD
    participant Istio as Istio Sidecar
    
    Dev->>BS: Requests access: order-api → payment-api (env: hml, role: process-payment)
    BS->>BS: Validates services and available roles
    
    BS->>KC: Prepares role "process-payment" (staging)
    BS->>GH: Creates PR in payment-api-infra
    Note over GH: Changes environments/hml/istio/authorization-policy.yaml
    
    GH-->>Owner: PR notification
    Owner->>GH: Review + Approve + Merge
    
    GH->>KC: Webhook: Activates roles in order-api service account
    GH-->>Argo: Webhook: Sync triggered
    Argo->>Istio: Applies AuthorizationPolicy in HML cluster
    
    Note over Istio: payment-api sidecar (HML)<br/>allows requests from order-api
    
    BS-->>Dev: Notification: Access approved and active in HML
    
    Note over Dev,Istio: For PROD: repeat process<br/>with PR in environments/prod/istio/
```

##### Where Each Configuration Resides

| Resource | Location | Owner | Reason |
|----------|----------|-------|--------|
| **AuthorizationPolicy** | `payment-api-infra/environments/{env}/istio/` | Destination owner | Destination sidecar does enforcement |
| **RequestAuthentication** | `payment-api-infra/environments/{env}/istio/` | Destination owner | Validates JWT at destination |
| **Client Roles** | Keycloak (payment-api client) | Destination owner | Defines exposed permissions |
| **Service Account Roles** | Keycloak (order-api client) | Approved by destination | Roles that the origin possesses |

##### Structure in Infra Repository

```
payment-api-infra/
├── chart/
│   └── values-common.yaml
├── environments/
│   ├── hml/
│   │   ├── values.yaml
│   │   └── istio/
│   │       ├── request-authentication.yaml
│   │       └── authorization-policy.yaml
│   └── prod/
│       ├── values.yaml
│       └── istio/
│           ├── request-authentication.yaml
│           └── authorization-policy.yaml
└── README.md
```

> **Note:** Istio policies reside within each environment (`environments/{env}/istio/`) to allow:
> - Different permissions per environment (HML more permissive for testing)
> - Controlled promotion of policies (test in HML before PROD)
> - Specific CODEOWNERS for `/environments/prod/` requiring additional approvals

**authorization-policy.yaml** (managed via Backstage PRs):

```yaml
# environments/prod/istio/authorization-policy.yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payment-api-access
  namespace: apps-payments
spec:
  selector:
    matchLabels:
      app: payment-api
  action: ALLOW
  rules:
    # Rule added via PR approved by owner
    - from:
        - source:
            principals: ["cluster.local/ns/apps-orders/sa/order-api"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/payments/*"]
      when:
        - key: request.auth.claims[roles]
          values: ["process-payment"]
    
    # Health checks - always allowed
    - to:
        - operation:
            methods: ["GET"]
            paths: ["/health", "/ready", "/metrics"]
```

##### Differences between Environments

| Aspect | HML | PROD |
|--------|-----|------|
| **Permissions** | May include test/debug services | Only approved services |
| **PR Approval** | 1 reviewer | 2 reviewers (CODEOWNERS) |
| **Namespace** | `apps-*` in cluster `eks-hml` | `apps-*` in cluster `eks-prod-main` |
| **Promotion** | Test policies here first | Copy from HML after validation |

##### Service-to-Service Permissions Matrix

Example configuration managed via Backstage:

| Source Service | Destination Service | Allowed Roles | Endpoints |
|----------------|-----------------|------------------|-----------|
| `order-api` | `payment-api` | `process-payment` | `POST /payments/*` |
| `order-api` | `inventory-api` | `check-stock`, `reserve` | `GET /stock/*`, `POST /reservations` |
| `notification-api` | `user-api` | `read-contacts` | `GET /users/*/contacts` |
| `*` (any) | `health-check` | - | `GET /health`, `GET /ready` |

##### Keycloak Configuration (Automatically Generated)

```yaml
# Client generated for payment-api
client:
  clientId: payment-api
  protocol: openid-connect
  publicClient: false
  serviceAccountsEnabled: true
  authorizationServicesEnabled: true
  
  # Roles that this service exposes
  roles:
    - name: process-payment
      description: "Allows processing payments"
    - name: refund
      description: "Allows processing refunds"
    - name: read-transactions
      description: "Allows querying transactions"

  # Authorization policies
  authorizationSettings:
    policies:
      - name: "order-api-can-process"
        type: client
        logic: POSITIVE
        clients: ["order-api"]
    
    permissions:
      - name: "process-payment-permission"
        type: scope
        policies: ["order-api-can-process"]
        scopes: ["process-payment"]
```

##### Backstage Interface

The Service Owner visualizes and manages permissions through a unified interface:

```
┌─────────────────────────────────────────────────────────────────────────┐
│  payment-api > Permissions                                               │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  📥 INBOUND (Who can access this service)                            │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │ Service        │ Roles              │ Endpoints        │ Status   │ │
│  ├────────────────┼────────────────────┼──────────────────┼──────────┤ │
│  │ order-api      │ process-payment    │ POST /payments/* │ ✅ Active│ │
│  │ refund-api     │ refund             │ POST /refunds/*  │ ✅ Active│ │
│  │ admin-portal   │ read-transactions  │ GET /transactions│ ✅ Active│ │
│  └────────────────────────────────────────────────────────────────────┘ │
│  [+ Add Inbound Permission]                                              │
│                                                                          │
│  📤 OUTBOUND (Which services this can access)                         │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │ Service        │ Roles              │ Status           │ Request  │ │
│  ├────────────────┼────────────────────┼──────────────────┼──────────┤ │
│  │ fraud-api      │ check-fraud        │ ✅ Active        │ -        │ │
│  │ notification   │ send-notification  │ ⏳ Pending       │ View     │ │
│  └────────────────────────────────────────────────────────────────────┘ │
│  [+ Request Outbound Access]                                             │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

##### Access Request Flow

```mermaid
stateDiagram-v2
    [*] --> Requested: Dev requests access in Backstage
    Requested --> PRCreated: Backstage creates PR in destination repo
    PRCreated --> PendingReview: Destination owner notified
    PendingReview --> Approved: Owner approves and merges
    PendingReview --> Rejected: Owner rejects PR
    Approved --> KeycloakSync: Webhook activates roles in Keycloak
    KeycloakSync --> ArgoSync: Argo CD syncs policies
    ArgoSync --> Active: Sidecar applies AuthorizationPolicy
    Active --> [*]
    Rejected --> [*]
    
    Active --> PRRevoke: Owner or Admin creates revocation PR
    PRRevoke --> Revoked: Merge removes permission
    Revoked --> [*]
```

> **Note:** The entire permissions lifecycle goes through Pull Requests, ensuring complete auditing via Git history and explicit approval from the destination service owner.

##### Auditing and Compliance

All permission operations are audited:

| Event | Recorded Data | Destination |
|-------|---------------|-------------|
| Permission requested | Requester, services, roles, timestamp | Backstage DB + CloudWatch |
| PR created | PR URL, branch, diff | GitHub + Backstage |
| PR approved/rejected | Approver, review comments | GitHub history |
| Permission provisioned | Merge commit SHA, created resources | Git history + CloudWatch |
| Access performed | Source, destination, result | Istio access logs + Loki |
| Permission revoked | Revocation PR, reason | Git history + CloudWatch |

##### Tokens and Identity Propagation

```mermaid
sequenceDiagram
    participant User
    participant Gateway as Istio Gateway
    participant OrderAPI as order-api
    participant PaymentAPI as payment-api
    participant KC as Keycloak
    
    User->>KC: Login (Authorization Code)
    KC-->>User: Access Token (user context)
    
    User->>Gateway: POST /orders (Bearer: user-token)
    Gateway->>Gateway: Validates JWT
    Gateway->>OrderAPI: Forward + headers
    
    Note over OrderAPI: Needs to call payment-api
    
    OrderAPI->>KC: Client Credentials Grant
    KC-->>OrderAPI: Service Token (order-api context)
    
    OrderAPI->>PaymentAPI: POST /payments (Bearer: service-token)
    Note over PaymentAPI: Header: X-Original-User-Id preserved
    
    PaymentAPI->>PaymentAPI: Validates JWT + roles
    PaymentAPI-->>OrderAPI: 201 Created
    OrderAPI-->>User: 201 Order Created
```

##### Base Helm Chart Configuration

Services inherit authentication configuration from the base chart:

```yaml
# values.yaml - service-to-service auth section
serviceAuth:
  # This service's identity
  identity:
    clientId: "{{ .Values.service.name }}"
    clientSecretRef:
      name: "{{ .Values.service.name }}-oidc"
      key: "client-secret"
  
  # Services this can access (defined via Backstage)
  outbound: []
  # Example after configuration via Backstage:
  # - service: payment-api
  #   roles: ["process-payment"]
  
  # Original user context propagation
  propagateUserContext: true
  userContextHeader: "X-Original-User-Id"
```

#### Complete Flow: From Zero to Deploy

```mermaid
flowchart TB
    A[Developer accesses Backstage] --> B[Selects 'New Microservice' template]
    B --> C[Fills form]
    C --> D{Validation}
    D -->|Error| C
    D -->|OK| E[Backstage executes steps]
    
    E --> F[Creates service-xyz on GitHub]
    E --> G[Creates service-xyz-infra on GitHub]
    E --> H[Commit in platform-argocd-config]
    E --> I[Creates client in Keycloak]
    
    F --> J[CI Pipeline triggers on first push]
    J --> K[Build + Push image to ECR]
    K --> L[Adds deploy-hml label]
    L --> M[Updates environments/hml/]
    
    H --> N[Argo CD detects new Application]
    M --> O[Argo CD sync service-xyz-hml]
    N --> O
    
    O --> P[Pod running on EKS-HML]
    P --> Q[Metrics in Grafana]
    P --> R[Logs in Loki]
    P --> S[Traces in Tempo]
    
    style E fill:#f9f,stroke:#333
    style P fill:#9f9,stroke:#333
```

> **Note:** The first deploy is to HML. For production, the developer creates a Release after validating in HML.

#### Onboarding Time

| Step | Time | Responsible |
|------|------|-------------|
| Fill form in Backstage | ~2 min | Developer |
| Template execution | ~1 min | Automatic |
| First pipeline build | ~5 min | Automatic |
| Deploy in HML via Argo CD | ~2 min | Automatic |
| **Total: Service running in HML** | **~10 min** | - |

---

## 6. Deploy Flow (GitOps)

### 6.1 Developer Flow Overview

The deploy flow follows two distinct journeys based on the code lifecycle:

```mermaid
flowchart TB
    subgraph DEV["🧑‍💻 Development"]
        A[Opens PR] --> B[Build + Test]
        B --> C{Deploy to HML?}
        C -->|Yes| D[Adds label<br/>deploy-hml]
        C -->|No| A
    end

    subgraph HML["🧪 Homologation"]
        D --> E[Build image<br/>tag: SHA]
        E --> F[Push ECR]
        F --> G[Update<br/>environments/hml/]
        G --> H[Argo CD Sync]
        H --> I[Validate in HML]
    end

    subgraph APPROVAL["✅ Approval"]
        I --> J{Approved?}
        J -->|Yes| K[Merge PR]
        J -->|No| A
    end

    subgraph PROD["🚀 Production"]
        K --> L[Create Release<br/>tag: vX.Y.Z]
        L --> M[Build image<br/>tag: vX.Y.Z]
        M --> N[Push ECR]
        N --> O[Update<br/>environments/prod/]
        O --> P[Argo CD Sync]
        P --> Q[✓ In Production]
    end

    style DEV fill:#e1f5fe
    style HML fill:#fff3e0
    style APPROVAL fill:#f3e5f5
    style PROD fill:#e8f5e9
```

| Journey | Trigger | Image | Destination | Approval |
|---------|---------|-------|-------------|----------|
| **Homologation** | Label `deploy-hml` on PR | Commit SHA | `environments/hml/` | Automatic |
| **Production** | Merge to `main` + Release | Release tag (vX.Y.Z) | `environments/prod/` | Via Release |

### 6.2 Homologation (HML) Flow

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub (PR)
    participant CI as GitHub Actions
    participant ECR as Amazon ECR
    participant Infra as service-xyz-infra
    participant Argo as Argo CD (HML)
    participant EKS as EKS-HML

    Dev->>GH: 1. Opens Pull Request
    GH->>CI: 2. Trigger: PR opened/updated
    
    CI->>CI: 3. Lint & Test
    CI->>CI: 4. Build image
    CI->>CI: 5. Security scan (Trivy)
    CI->>ECR: 6. Push image (SHA: abc123)
    
    Note over Dev,GH: Dev adds label "deploy-hml"
    
    GH->>CI: 7. Trigger: label added
    CI->>Infra: 8. Update environments/hml/values.yaml
    Note over CI,Infra: image.tag = "abc123" (SHA)
    
    Argo->>Infra: 9. Detect change (webhook)
    Argo->>EKS: 10. Sync manifests
    EKS-->>Argo: 11. Health status
    
    Note over Dev,EKS: Validation in HML
```

**HML flow characteristics:**
- Image identified by **commit SHA** (immutable)
- **On-demand** deploy via `deploy-hml` label
- Allows multiple deploys from the same PR during development
- Doesn't require merge, enables validation before approving PR

### 6.3 Production (PROD) Flow

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub
    participant CI as GitHub Actions
    participant ECR as Amazon ECR
    participant Infra as service-xyz-infra
    participant Argo as Argo CD (PROD)
    participant EKS as EKS-PROD

    Dev->>GH: 1. Merge PR to main
    GH->>CI: 2. Trigger: push to main
    
    Note over Dev,GH: Dev creates Release (v1.2.3)
    
    GH->>CI: 3. Trigger: release published
    CI->>CI: 4. Build image (final)
    CI->>CI: 5. Security scan
    CI->>ECR: 6. Push image (tag: v1.2.3)
    
    CI->>Infra: 7. Update environments/prod/values.yaml
    Note over CI,Infra: image.tag = "v1.2.3" + digest
    
    CI->>Infra: 8. Commit + Push
    
    Argo->>Infra: 9. Detect change (webhook)
    Argo->>EKS: 10. Sync manifests
    EKS-->>Argo: 11. Health status
    
    Argo->>GH: 12. Update deployment status
```

**PROD flow characteristics:**
- Image identified by **semantic tag** (v1.2.3) + digest
- **Automatic** deploy after Release creation
- Release serves as approval gate and documentation
- Release history facilitates rollback

### 6.4 GitHub Actions Workflow (Complete)

> 📄 **File:** [ci-cd-pipeline.yaml](samples/github-actions/ci-cd-pipeline.yaml)

### 6.5 Summary: Promotion between Environments

| Stage | Trigger | Image Tag | Infra Folder | Approval |
|-------|---------|-----------|--------------|----------|
| **HML** | Label `deploy-hml` on PR | SHA (abc123) | `environments/hml/` | Automatic |
| **PROD** | Release (v1.2.3) | Semantic tag | `environments/prod/` | Via Release |

### 6.6 Rollback Strategies

| Environment | Strategy | Command/Action |
|-------------|----------|----------------|
| **HML** | Remove label and re-add with previous commit | GitHub UI |
| **PROD** | Revert in infra repo or new hotfix release | `git revert` + push or new release |
| **Emergency** | Rollback via Argo CD UI | Argo CD → History → Rollback |

**Rollback via CLI:**
```bash
# Rollback to previous revision in Argo CD
argocd app rollback service-xyz --revision=<previous-revision>

# Or via git revert in infra repo
cd service-xyz-infra
git revert HEAD
git push
```

### 6.7 Multi-Region Deploy (Phased Rollout)

In multi-region architecture, deploy follows a **phased rollout** strategy - gradual deployment region by region with health checks between each step.

#### Configuration in catalog-info.yaml

The rollout configuration is defined in the service's `catalog-info.yaml`:

```yaml
# catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-api
  annotations:
    helpdev.io/regions: "us-east-1,sa-east-1"  # Available regions
spec:
  type: service
  lifecycle: production
  owner: payments-team
  
  deployment:
    rollout:
      hml:
        regions:
          - name: us-east-1
            order: 1
          - name: sa-east-1
            order: 2
            delayAfterPrevious: 0  # No delay for HML
      prod:
        regions:
          - name: us-east-1
            order: 1
            healthCheckDuration: 5m
          - name: sa-east-1
            order: 2
            delayAfterPrevious: 30m  # Waits 30min after success in us-east-1
            healthCheckDuration: 5m
```

#### Configuration Options

| Field | Type | Description | Default |
|-------|------|-------------|---------|
| `regions[].name` | string | AWS region name | required |
| `regions[].order` | int | Deploy order (1 = first) | required |
| `regions[].delayAfterPrevious` | duration | Wait time after previous region | `0` |
| `regions[].healthCheckDuration` | duration | Time to monitor health before proceeding | `5m` |
| `regions[].rollbackOnFailure` | bool | Automatic rollback if deploy fails | `true` |

#### Phased Rollout Flow

```mermaid
sequenceDiagram
    participant CI as GitHub Actions
    participant Cat as catalog-info.yaml
    participant R1 as us-east-1
    participant R2 as sa-east-1
    participant Argo1 as Argo CD (use1)
    participant Argo2 as Argo CD (sae1)
    participant Obs as Observability
    
    CI->>Cat: 1. Reads rollout config for PROD
    Cat-->>CI: regions[], order, delays
    
    rect rgb(230, 245, 230)
        Note over CI,Argo1: Phase 1: us-east-1 (Primary)
        CI->>R1: 2. Update environments/prod/us-east-1/values.yaml
        R1->>Argo1: 3. Webhook trigger
        Argo1->>Argo1: 4. Sync manifests
        Argo1-->>CI: 5. Sync successful
        CI->>Obs: 6. Verifies metrics (5min)
        Obs-->>CI: 7. Health OK ✅
    end
    
    rect rgb(255, 250, 230)
        Note over CI: Wait Phase
        CI->>CI: 8. Waits delayAfterPrevious (30min)
        CI->>Obs: 9. Monitors region 1 during delay
    end
    
    rect rgb(230, 240, 250)
        Note over CI,Argo2: Phase 2: sa-east-1 (Secondary)
        CI->>R2: 10. Update environments/prod/sa-east-1/values.yaml
        R2->>Argo2: 11. Webhook trigger
        Argo2->>Argo2: 12. Sync manifests
        Argo2-->>CI: 13. Sync successful
        CI->>Obs: 14. Verifies metrics (5min)
        Obs-->>CI: 15. Health OK ✅
    end
    
    Note over CI: Rollout Complete! 🎉
```

#### Specific Region Deploy

For debugging or emergency, it's possible to deploy to a single region:

```bash
# Deploy only to us-east-1 (bypass phased rollout)
gh pr comment --body "/deploy-prod-us-east-1"

# Or via specific label
gh pr edit --add-label "deploy-prod-sa-east-1"
```

**Available labels:**
- `deploy-hml` → Deploy to all HML regions (phased)
- `deploy-hml-us-east-1` → Deploy only to HML/us-east-1
- `deploy-prod-us-east-1` → Deploy only to PROD/us-east-1 (requires approval)

#### Multi-Region Rollback

```mermaid
flowchart LR
    subgraph Detection
        A[Failure detected<br/>in sa-east-1] --> B{rollbackOnFailure?}
    end
    
    subgraph Rollback
        B -->|true| C[Automatic rollback<br/>sa-east-1]
        B -->|false| D[Alert to team]
        C --> E{Revert<br/>us-east-1?}
        E -->|Manual decision| F[Rollback us-east-1]
        E -->|Keep| G[us-east-1 with new version<br/>sa-east-1 with previous version]
    end
```

**Rollback strategy:**
1. **Automatic per region:** If `rollbackOnFailure: true`, rollback only in the failed region
2. **Optional cascade:** Rollback of previous regions is a manual decision
3. **Rollback via Git:** `git revert` on the specific region's commit

---

## 7. Observability

### 7.1 Multi-Region Telemetry Architecture

Each region has its own observability stack (Mimir, Loki, Tempo). The **Central Grafana** in `prod/us-east-1` aggregates data from all regions via remote datasources.

```mermaid
flowchart TB
    subgraph Global["Grafana Central (prod/us-east-1)"]
        GRAFANA["Grafana Central 🎯"]
    end
    
    subgraph PROD_USE1["prod/us-east-1"]
        MIMIR_PROD_USE1[(Mimir)]
        LOKI_PROD_USE1[(Loki)]
        TEMPO_PROD_USE1[(Tempo)]
    end
    
    subgraph PROD_SAE1["prod/sa-east-1 (futuro)"]
        MIMIR_PROD_SAE1[(Mimir)]
        LOKI_PROD_SAE1[(Loki)]
        TEMPO_PROD_SAE1[(Tempo)]
    end
    
    subgraph HML_USE1["hml/us-east-1"]
        MIMIR_HML_USE1[(Mimir)]
        LOKI_HML_USE1[(Loki)]
        TEMPO_HML_USE1[(Tempo)]
    end
    
    GRAFANA --> MIMIR_PROD_USE1
    GRAFANA --> LOKI_PROD_USE1
    GRAFANA --> TEMPO_PROD_USE1
    
    GRAFANA -.-> MIMIR_PROD_SAE1
    GRAFANA -.-> LOKI_PROD_SAE1
    GRAFANA -.-> TEMPO_PROD_SAE1
    
    GRAFANA --> MIMIR_HML_USE1
    GRAFANA --> LOKI_HML_USE1
    GRAFANA --> TEMPO_HML_USE1
    
    style GRAFANA fill:#f96,stroke:#333
```

### 7.2 Stack per Region

Each region has its own telemetry stack:

```mermaid
flowchart LR
    subgraph App["Application Pod"]
        APP[App Container]
        OTEL_SDK[OTel SDK]
        APP --> OTEL_SDK
    end

    subgraph Collector["OTel Collector (Gateway)"]
        RECV[Receivers]
        PROC[Processors]
        EXP[Exporters]
        RECV --> PROC --> EXP
    end

    subgraph Backends["Storage Backends (per region)"]
        MIMIR[(Mimir)]
        LOKI[(Loki)]
        TEMPO[(Tempo)]
    end

    OTEL_SDK -->|OTLP| RECV
    EXP -->|RemoteWrite| MIMIR
    EXP -->|Loki Protocol| LOKI
    EXP -->|OTLP| TEMPO
```

### 7.3 OTel Collector Configuration

> 📄 **File:** [otel-collector-config.yaml](samples/helm/values/otel-collector-config.yaml)

### 7.4 Datasource Federation (Central Grafana)

Grafana in `prod/us-east-1` aggregates data from all environments and regions:

> 📄 **File:** [grafana-datasources.yaml](samples/helm/values/grafana-datasources.yaml)

**Datasources per Region:**

| Environment | Region | Mimir URL | Loki URL | Tempo URL |
|----------|--------|-----------|----------|-----------|
| prod | us-east-1 | `http://mimir.monitoring:9009` (local) | `http://loki.monitoring:3100` | `http://tempo.monitoring:3200` |
| prod | sa-east-1 | `https://mimir.prod-sae1.internal` | `https://loki.prod-sae1.internal` | `https://tempo.prod-sae1.internal` |
| hml | us-east-1 | `https://mimir.hml-use1.internal` | `https://loki.hml-use1.internal` | `https://tempo.hml-use1.internal` |

### 7.5 Cross-Account Connectivity (for Federation)

| Option | Latency | Complexity | Cost | Recommendation |
|--------|---------|------------|------|----------------|
| **AWS Transit Gateway** | Low | Medium | Medium | ✅ Recommended |
| **VPC Peering** | Low | Low | Low | Alternative (peer limit) |
| **PrivateLink** | Low | High | High | For specific services |

> **Note:** Cross-region connectivity is used only for **observability federation** (Central Grafana reading from other regions). Application services do NOT communicate cross-region.

### 7.6 Standard Dashboards

| Dashboard | Scope | Main Metrics |
|-----------|--------|---------------------|
| **Platform Overview** | All clusters/regions | Nodes, pods, global resources |
| **Multi-Region Status** | All regions | Health per region, latency, traffic split |
| **Service Health** | Per service | Request rate, latency, errors (RED) |
| **Resource Usage** | Per namespace | CPU, memory, network |
| **SLO Dashboard** | Per critical service | SLI tracking, error budget |
| **Cost Dashboard** | Per account/namespace/region | Compute, storage, network costs |
| **Istio Mesh** | Per cluster | Traffic, mTLS status, latency mesh |
| **Keycloak** | Per environment/region | Logins, tokens, sessions |

---

## 8. Service Mesh (Istio)

### 8.1 Overview

The Service Mesh provides secure communication (mTLS), traffic observability and resilience policies between microservices. **Istio** is adopted as standard, with **mTLS STRICT** enabled by default in all application namespaces.

> **Multi-Region:** Each region has its own Istio control plane (Istiod). There is no mesh federation between regions (isolation by design).

### 8.2 Service Mesh Architecture

```mermaid
flowchart TB
    subgraph Cluster["EKS Cluster (per region)"]
        subgraph IstioSystem["istio-system namespace"]
            ISTIOD["Istiod (Control Plane)"]
            IGW["Istio Ingress Gateway"]
        end
        
        subgraph AppsNS["apps-* namespaces"]
            subgraph PodA["Pod: service-a"]
                APP_A["App Container"]
                ENVOY_A["Envoy Sidecar"]
            end
            subgraph PodB["Pod: service-b"]
                APP_B["App Container"]
                ENVOY_B["Envoy Sidecar"]
            end
        end
        
        ISTIOD -->|Config Push| ENVOY_A
        ISTIOD -->|Config Push| ENVOY_B
        IGW -->|mTLS| ENVOY_A
        ENVOY_A <-->|mTLS| ENVOY_B
    end
    
    Client["External Client"] -->|TLS| IGW
```

### 8.3 Base Configuration

#### PeerAuthentication (mTLS STRICT)

> 📄 **File:** [peer-authentication.yaml](samples/kubernetes/peer-authentication.yaml)

#### Namespace with Sidecar Injection

> 📄 **File:** [namespace-istio-example.yaml](samples/kubernetes/namespace-istio-example.yaml)

### 8.4 Configuration via Base Helm Chart

The base Helm chart for services includes configurable Service Mesh settings:

> 📄 **File:** [service-mesh-values.yaml](samples/helm/values/service-mesh-values.yaml)

### 8.5 Templates Generated by Helm Chart

#### VirtualService

> 📄 **File:** [virtualservice.yaml](samples/helm/templates/virtualservice.yaml)

#### DestinationRule

> 📄 **File:** [destinationrule.yaml](samples/helm/templates/destinationrule.yaml)

### 8.6 Usage Example per Service

```yaml
# service-checkout-infra/environments/prod/values.yaml
replicaCount: 3

image:
  repository: 123456789.dkr.ecr.us-east-1.amazonaws.com/service-checkout
  digest: sha256:abc123...

serviceMesh:
  enabled: true
  timeout: 15s
  retries:
    enabled: true
    attempts: 3
  circuitBreaker:
    enabled: true
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 60s
```

### 8.7 Observability Integration

Istio automatically exports metrics to the OTel Collector:

```yaml
# istio-meshconfig
defaultConfig:
  tracing:
    openCensusAgent:
      address: otel-collector.monitoring:55678
  proxyStatsMatcher:
    inclusionRegexps:
      - ".*"
```

Additional dashboards in Grafana:
- **Istio Mesh Dashboard**: Mesh overview
- **Istio Service Dashboard**: Metrics per service
- **Istio Workload Dashboard**: Metrics per workload

---

## 9. Identity and Access Management (Keycloak)

### 9.1 Overview

**Keycloak** is adopted as Identity Provider (IdP), providing:
- Authentication (OAuth2, OIDC, SAML)
- Single Sign-On (SSO)
- Identity federation
- User and group management
- Fine-grained authorization

### 9.2 Multi-Region Architecture

Each region has its own Keycloak instance with dedicated RDS. **Realm configuration is synchronized via GitOps**, ensuring consistency between regions while maintaining runtime isolation.

> **Principle:** Configuration (realms, clients, roles) is shared via Git. Runtime data (sessions, tokens) is isolated per region.

```mermaid
flowchart TB
    subgraph Git["Git: platform-keycloak"]
        REALMS["realms/<br/>clients, roles, policies"]
    end
    
    subgraph HML["AWS Account: HML"]
        subgraph HML_USE1["us-east-1"]
            KC_HML_USE1["Keycloak (HA)"]
            PG_HML_USE1["RDS us-east-1"]
            ARGO_HML_USE1["Argo CD"]
        end
        subgraph HML_SAE1["sa-east-1"]
            KC_HML_SAE1["Keycloak (HA)"]
            PG_HML_SAE1["RDS sa-east-1"]
            ARGO_HML_SAE1["Argo CD"]
        end
    end
    
    subgraph PROD["AWS Account: Prod"]
        subgraph PROD_USE1["us-east-1"]
            KC_PROD_USE1["Keycloak (HA)"]
            PG_PROD_USE1["RDS us-east-1"]
            ARGO_PROD_USE1["Argo CD"]
        end
        subgraph PROD_SAE1["sa-east-1"]
            KC_PROD_SAE1["Keycloak (HA)"]
            PG_PROD_SAE1["RDS sa-east-1"]
            ARGO_PROD_SAE1["Argo CD"]
        end
    end
    
    REALMS -->|sync| ARGO_HML_USE1 --> KC_HML_USE1 --> PG_HML_USE1
    REALMS -->|sync| ARGO_HML_SAE1 --> KC_HML_SAE1 --> PG_HML_SAE1
    REALMS -->|sync| ARGO_PROD_USE1 --> KC_PROD_USE1 --> PG_PROD_USE1
    REALMS -->|sync| ARGO_PROD_SAE1 --> KC_PROD_SAE1 --> PG_PROD_SAE1
    
    style REALMS fill:#f9f,stroke:#333
```

#### What is Synchronized vs Isolated

| Component | Synchronized (Git) | Isolated (Region) |
|-----------|-------------------|-------------------|
| Realm definitions | ✅ | |
| Clients (argocd, grafana, backstage) | ✅ | |
| Service clients (microservices) | ✅ | |
| Roles e permissions | ✅ | |
| Identity Providers (GitHub SSO) | ✅ | |
| Client scopes | ✅ | |
| RDS PostgreSQL | | ✅ |
| User sessions | | ✅ |
| Refresh tokens | | ✅ |
| Audit logs | | ✅ |

#### Sync Mechanism

Sync is done via **keycloak-config-cli**, executed as a Job after Keycloak deploy:

```yaml
# helm/values-common.yaml
keycloak:
  extraInitContainers:
    - name: keycloak-config-cli
      image: adorsys/keycloak-config-cli:latest
      env:
        - name: KEYCLOAK_URL
          value: "http://localhost:8080"
        - name: IMPORT_PATH
          value: "/config/realms"
      volumeMounts:
        - name: realm-config
          mountPath: /config/realms
```

### 9.3 Components per Region

| Component | Technology | Description |
|-----------|------------|-------------|
| **Keycloak** | Keycloak 24+ (Quarkus) | Identity Provider |
| **Database** | Amazon RDS PostgreSQL | Persistence (Multi-AZ in prod) |
| **Cache** | Infinispan (embedded) | Sessions and distributed cache intra-region |
| **Ingress** | Istio Ingress Gateway | External exposure with mTLS |

> **Note:** Infinispan is configured for clustering only within the same region. There is no cross-region replication of cache or sessions.

### 9.4 Deploy Configuration

> 📄 **File:** [keycloak-values.yaml](samples/helm/values/keycloak-values.yaml)

### 9.5 Realm Structure

| Realm | Purpose | Clients |
|-------|---------|---------|
| `master` | Keycloak administration | Admin Console |
| `helpdev-platform` | Internal collaborators and tools | Argo CD, Grafana, Backstage, Admin Tools |
| `helpdev-services` | Service-to-service auth | Microservices (client credentials) |
| `helpdev-customers` | End users (if applicable) | Public apps |

### 9.6 Platform Component Integration

#### Argo CD + Keycloak

> 📄 **File:** [argocd-oidc-config.yaml](samples/config/argocd-oidc-config.yaml)

#### Grafana + Keycloak

> 📄 **File:** [grafana-oauth-config.yaml](samples/config/grafana-oauth-config.yaml)

### 9.7 Base Helm Chart Configuration (Apps)

```yaml
# values.yaml of the base Helm chart
auth:
  enabled: true
  provider: keycloak
  
  # OIDC client configuration
  oidc:
    issuerUrl: "https://auth.{{ .Values.global.domain }}/realms/helpdev-services"
    clientId: ""  # Defined per service
    clientSecretRef:
      name: ""    # Secret with client_secret
      key: "client-secret"
    scopes: ["openid", "profile"]
  
  # JWT Validation (for APIs)
  jwt:
    enabled: true
    issuer: "https://auth.{{ .Values.global.domain }}/realms/helpdev-services"
    jwksUri: "https://auth.{{ .Values.global.domain }}/realms/helpdev-services/protocol/openid-connect/certs"
    audiences: []  # Defined per service
  
  # Service-to-service (client credentials)
  serviceAccount:
    enabled: false
    clientId: ""
    clientSecretRef:
      name: ""
      key: "client-secret"
```

### 9.8 Istio Integration (RequestAuthentication)

```yaml
{{- if and .Values.auth.enabled .Values.auth.jwt.enabled }}
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: {{ .Release.Name }}-jwt
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ .Release.Name }}
  jwtRules:
    - issuer: {{ .Values.auth.jwt.issuer }}
      jwksUri: {{ .Values.auth.jwt.jwksUri }}
      {{- if .Values.auth.jwt.audiences }}
      audiences:
        {{- toYaml .Values.auth.jwt.audiences | nindent 8 }}
      {{- end }}
      forwardOriginalToken: true
{{- end }}
```

### 9.9 AuthorizationPolicy com Istio

```yaml
{{- if .Values.auth.enabled }}
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: {{ .Release.Name }}-authz
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ .Release.Name }}
  action: ALLOW
  rules:
    # Allow health checks without auth
    - to:
        - operation:
            paths: ["/health", "/ready", "/metrics"]
    # Require valid JWT for other endpoints
    - from:
        - source:
            requestPrincipals: ["*"]
      to:
        - operation:
            paths: ["/*"]
{{- end }}
```

### 9.10 Domains per Environment

| Environment | Keycloak URL | Notes |
|-------------|--------------|-------|
| HML | `auth.hml.helpdev.io` | Test data |
| Prod | `auth.helpdev.io` | Production |
| BU (future) | `auth.bu-a.helpdev.io` | Per BU |

---

## 10. API Management (Kong)

> **ADR Reference:** [ADR-013: Kong API Gateway](adr/ADR-013-kong-api-gateway.md)

Kong Gateway provides API management capabilities for the HelpDev platform, enabling:
- External API exposure with authentication and rate limiting
- Internal API routing between service domains
- Centralized OpenAPI specifications for documentation
- GitOps-managed configuration (DB-less mode)

### 10.1 Overview

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         API MANAGEMENT STACK                                │
│                                                                             │
│  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐       │
│  │   Backstage      │    │   platform-apis  │    │  platform-kong   │       │
│  │                  │    │   (API Config)   │    │   (Infra Only)   │       │
│  │ • Create Kong    │    │ • OpenAPI specs  │    │ • Kong deploy    │       │
│  │ • Associate svc  │    │ • Routes config  │    │ • Helm values    │       │
│  │ • Update catalog │    │ • Plugins config │    │ • Base settings  │       │
│  └──────────────────┘    └──────────────────┘    └──────────────────┘       │
│                                   │                       │                 │
│                                   ▼                       ▼                 │
│                          ┌──────────────────────────────────────────┐       │
│                          │           Kong Runtime                   │       │
│                          │  (DB-less, config from platform-apis)    │       │
│                          └──────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────────────────────────┘
```

#### Kong Instance Types

| Type | Purpose | Network | Use Case |
|------|---------|---------|----------|
| **Internal** | Service-to-service | ClusterIP | APIs consumed within the cluster |
| **External** | Public APIs | LoadBalancer + WAF | APIs exposed to internet |

#### Kong Instance per Domain

Kong instances are created per business domain, not per service:

```
ecommerce-external     → checkout-api, cart-api, exchange-api
ecommerce-internal     → checkout-api, cart-api, exchange-api
payments-external      → payment-api, billing-api
payments-internal      → payment-api, billing-api
```

### 10.2 Repository Structure

#### Separation of Concerns

| Repository | Responsibility | Ownership |
|------------|----------------|-----------|
| `platform-kong` | Kong **infrastructure** (Helm, resources) | Platform Team |
| `platform-apis` | API **configuration** (specs, routes, plugins) | Service Teams |

#### `platform-apis`

Stores OpenAPI specs, routes, and plugins configuration:

```
platform-apis/
├── helpdev-hml/                           # AWS Account: HML
│   └── hml/                               # Environment
│       └── us-east-1/                     # Region
│           ├── ecommerce-internal/        # Kong instance name
│           │   ├── kong.yaml              # Aggregated config (auto-generated)
│           │   ├── checkout-api/
│           │   │   ├── openapi.yaml       # OpenAPI spec
│           │   │   ├── routes.yaml        # Route definitions
│           │   │   └── plugins.yaml       # Plugin configurations
│           │   └── cart-api/
│           │       └── ...
│           └── ecommerce-external/
│               └── ...
│
├── helpdev-prod/                          # AWS Account: PROD
│   └── prod/
│       ├── us-east-1/
│       │   └── ...
│       └── sa-east-1/
│           └── ...
│
├── templates/                             # Service config templates
│   ├── routes.yaml.tmpl
│   └── plugins/
│       ├── rate-limiting.yaml.tmpl
│       └── jwt.yaml.tmpl
│
├── .github/
│   └── workflows/
│       ├── validate-openapi.yaml
│       ├── validate-kong-config.yaml
│       └── aggregate-kong-config.yaml     # Builds kong.yaml per instance
│
├── CODEOWNERS
└── README.md
```

**Path Pattern:** `{account}/{environment}/{region}/{kong-instance}/{service-name}/`

#### `platform-kong`

Infrastructure-only repository:

```
platform-kong/
├── helpdev-hml/
│   └── hml/
│       └── us-east-1/
│           ├── ecommerce-internal/
│           │   ├── values.yaml            # Helm values
│           │   └── kustomization.yaml
│           └── ecommerce-external/
│               └── ...
│
├── helpdev-prod/
│   └── prod/
│       └── ...
│
├── base/                                  # Base Kong deployment
│   ├── Chart.yaml
│   ├── values-base.yaml
│   ├── values-internal.yaml
│   └── values-external.yaml
│
├── CODEOWNERS
└── README.md
```

### 10.3 Kong Configuration

Kong runs in **DB-less mode** with declarative YAML configuration.

#### Service Routes (`routes.yaml`)

> 📄 **File:** [routes.yaml](samples/kong/routes.yaml)

```yaml
service:
  name: checkout-api
  url: http://checkout-api.apps-ecommerce.svc.cluster.local:8080

routes:
  - name: checkout-api-v1
    paths:
      - /api/v1/checkout
    methods:
      - GET
      - POST
      - PUT
    strip_path: true
    protocols:
      - https
```

#### Service Plugins (`plugins.yaml`)

> 📄 **File:** [plugins.yaml](samples/kong/plugins.yaml)

```yaml
plugins:
  - name: rate-limiting
    config:
      minute: 100
      policy: local

  - name: jwt
    config:
      key_claim_name: kid
      claims_to_verify:
        - exp

  - name: correlation-id
    config:
      header_name: X-Request-ID
      generator: uuid
```

#### Aggregated Configuration

A GitHub Action aggregates service configs into `kong.yaml`:

> 📄 **File:** [kong-aggregated.yaml](samples/kong/kong-aggregated.yaml)

### 10.4 Pipeline Integration

Service pipelines automatically publish OpenAPI specs and routes:

```mermaid
flowchart LR
    subgraph Service["Service Pipeline"]
        A[Build & Test] --> B[Validate OpenAPI]
        B --> C[Build Image]
    end
    
    subgraph HML["HML Deploy"]
        C --> D[Deploy HML]
        D --> E[Publish API Config]
        E --> F[Kong Sync]
    end
    
    subgraph PROD["Prod Deploy (Phased)"]
        C --> G[Deploy Region 1]
        G --> H[Publish API Region 1]
        H --> I[Health Check]
        I --> J[Wait 30min]
        J --> K[Deploy Region 2]
        K --> L[Publish API Region 2]
    end
```

#### catalog-info.yaml Annotations

Services declare Kong integration via annotations:

```yaml
metadata:
  annotations:
    helpdev.io/kong-instance: ecommerce-external
    helpdev.io/kong-type: external
    helpdev.io/openapi-path: apis/openapi.yaml
    helpdev.io/openapi-auto-publish: "true"
```

#### Pipeline Actions

| Action | Purpose |
|--------|---------|
| `validate-openapi` | Validates OpenAPI syntax and checks for breaking changes |
| `publish-api-config` | Publishes OpenAPI, routes, metadata to platform-apis |
| `aggregate-kong-config` | Combines service configs into kong.yaml per instance |

> 📄 **Files:**
> - [validate-openapi.yaml](samples/github-actions/validate-openapi.yaml)
> - [publish-api-config.yaml](samples/github-actions/publish-api-config.yaml)
> - [aggregate-kong-config.yaml](samples/github-actions/aggregate-kong-config.yaml)

### 10.5 Backstage Integration

#### Kong Instance Template

Backstage provides a template to create new Kong instances:

> 📄 **File:** [kong-instance/template.yaml](samples/backstage/templates/kong-instance/template.yaml)

| Field | Description |
|-------|-------------|
| `name` | Instance name (e.g., `ecommerce-external`) |
| `type` | `internal` or `external` |
| `domain` | Business domain |
| `environments` | Target environments (hml, prod) |
| `regions` | Target regions |

#### Service Association

Services are associated with Kong instances via `catalog-info.yaml` annotations. When a service with Kong annotations is deployed, the pipeline automatically:

1. Validates the OpenAPI spec
2. Creates/updates routes in `platform-apis`
3. Triggers the aggregation workflow
4. ArgoCD syncs the new `kong.yaml` to Kong

#### Governance

| Repository | CODEOWNERS |
|------------|------------|
| `platform-kong` | `@helpdev/platform-team` |
| `platform-apis` (PROD) | `@helpdev/platform-team` |
| `platform-apis` (HML) | `@helpdev/developers` |
| `platform-apis` (per domain) | Domain team owners |

---

## 11. Security and Governance

### 11.1 IAM and IRSA

> 📄 **File:** [irsa-example.tf](samples/terraform/irsa-example.tf)

### 11.2 Network Policies (Default Deny)

> 📄 **File:** [network-policy-default-deny.yaml](samples/kubernetes/network-policy-default-deny.yaml)

### 11.3 Pod Security Standards

> 📄 **File:** [pod-security-namespace.yaml](samples/kubernetes/pod-security-namespace.yaml)

### 11.4 GitOps Governance

| Repository | Branch Protection | Reviewers | CODEOWNERS |
|-------------|-------------------|-----------|------------|
| `platform-terraform` | `main` protected; require 2 approvals | Platform Team | `/environments/prod/ @platform-leads` |
| `platform-argocd-config` | `main` protected; require 2 approvals | Platform Team | `/ @platform-team` |
| `platform-keycloak` | `main` protected; require 2 approvals | Platform Team | `/realms/ @security-team` |
| `platform-observability` | `main` protected; require 2 approvals | Platform Team | `/ @platform-team` |
| `service-*-infra` | `main` protected; require 1 approval | Service Team | `/environments/prod/ @team-lead` |
| `service-*` (code) | `main` protected; require 1 approval | Service Team | `/ @service-team` |

**Release Control (Production):**

| Rule | Description |
|------|-------------|
| **Release required** | Only releases (tags vX.Y.Z) trigger production deploy |
| **Release notes** | Recommended to describe changes and impacts |
| **Tag protection** | Release tags protected against deletion |
| **Auditing** | Release history trackable in GitHub |

### 11.5 Secrets Management

#### Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                           BACKSTAGE                                      │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │  Secrets Manager Plugin (UI)                                     │    │
│  │  - List secrets per service                                      │    │
│  │  - Create/Edit (based on role)                                   │    │
│  │  - Rotate secrets                                                │    │
│  │  - Audit log of accesses                                         │    │
│  └─────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼ (OIDC Auth)
┌─────────────────────────────────────────────────────────────────────────┐
│                           KEYCLOAK                                       │
│  Roles: secrets-admin | secrets-manager | developer | viewer            │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼ (RBAC)
┌─────────────────────────────────────────────────────────────────────────┐
│                     AWS SECRETS MANAGER                                  │
│  helpdev/{env}/{domain}/{service}/{secret-type}                         │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼ (Sync)
┌─────────────────────────────────────────────────────────────────────────┐
│                  EXTERNAL SECRETS OPERATOR                               │
│  ClusterSecretStore → ExternalSecret → Kubernetes Secret                │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         POD (Application)                                │
│  - Environment variables                                                 │
│  - Volume mounts                                                         │
└─────────────────────────────────────────────────────────────────────────┘
```

#### Components

| Component | Technology | Responsibility |
|-----------|------------|----------------|
| **UI** | Backstage Plugin | Management interface |
| **Authentication** | Keycloak OIDC | SSO for Backstage |
| **Authorization** | Keycloak Roles | RBAC per domain/service |
| **Storage** | AWS Secrets Manager | Secure storage |
| **K8s Sync** | External Secrets Operator | Syncs secrets to cluster |
| **Audit** | CloudWatch + Backstage | Access traceability |

#### Naming Convention

```
helpdev/{env}/{region}/{domain}/{service}/{secret-type}
```

| Part | Description | Example |
|------|-------------|---------|
| `env` | Environment | `hml`, `prod` |
| `region` | AWS Region | `us-east-1`, `sa-east-1` |
| `domain` | Business domain | `payments`, `users`, `orders` |
| `service` | Service name | `checkout`, `user-api` |
| `secret-type` | Secret type | `database`, `api-keys`, `certificates` |

**Examples:**
```
helpdev/prod/us-east-1/payments/checkout/database
helpdev/prod/sa-east-1/payments/checkout/database
helpdev/hml/us-east-1/users/user-api/database
helpdev/prod/us-east-1/orders/order-processor/message-broker
```

> **Note:** Secrets are regional. Each region has its own secrets in the AWS Secrets Manager of that region. This ensures isolation and allows different values per region (e.g., regional RDS endpoints).

#### Permissions Matrix (RBAC)

| Role | Scope | List | View Value | Create | Edit | Delete | Rotate |
|------|--------|--------|-----------|-------|--------|---------|------------|
| `secrets-admin` | Global | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| `secrets-manager` | Domain | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| `developer` | Service | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| `viewer` | Service | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |

**Keycloak Groups Mapping:**
```yaml
# platform-keycloak/realms/helpdev-prd/helpdev-platform/groups/
# (same structure in helpdev-hml for testing)
groups:
  - name: secrets-admin
    roles: [secrets-admin]
    members: [platform-team]
    
  - name: secrets-manager-payments
    roles: [secrets-manager]
    attributes:
      domain: payments
    members: [payments-team-leads]
    
  - name: developers-payments
    roles: [developer]
    attributes:
      domain: payments
    members: [payments-developers]
```

#### External Secrets Operator Configuration

> **Note:** We use `ClusterSecretStore` (cluster-wide) instead of `SecretStore` (namespace-scoped) to centralize authentication configuration with AWS Secrets Manager. This allows any namespace to create `ExternalSecrets` pointing to the same store, simplifying governance and reducing configuration duplication.

**ClusterSecretStore (per cluster):**

> 📄 **File:** [cluster-secret-store.yaml](samples/kubernetes/cluster-secret-store.yaml)

**IAM Policy for External Secrets:**

> 📄 **File:** [iam-policy-external-secrets.json](samples/config/iam-policy-external-secrets.json)

#### ExternalSecret Template in Base Helm Chart

**values.yaml (per service):**

> 📄 **File:** [secrets-values.yaml](samples/helm/values/secrets-values.yaml)

**Helm Template (external-secret.yaml):**

> 📄 **File:** [external-secret.yaml](samples/helm/templates/external-secret.yaml)

**Helm Template (deployment.yaml - secrets usage):**

> 📄 **File:** [deployment-secrets-snippet.yaml](samples/helm/templates/deployment-secrets-snippet.yaml)

#### Backstage Plugin: Secrets Manager

**Plugin Structure:**
```
backstage/
├── plugins/
│   ├── secrets-manager/                  # Frontend plugin
│   │   ├── src/
│   │   │   ├── components/
│   │   │   │   ├── SecretsTable.tsx
│   │   │   │   ├── SecretDetails.tsx
│   │   │   │   ├── CreateSecretDialog.tsx
│   │   │   │   └── RotateSecretDialog.tsx
│   │   │   ├── api/
│   │   │   │   └── SecretsManagerApi.ts
│   │   │   └── plugin.ts
│   │   └── package.json
│   │
│   └── secrets-manager-backend/          # Backend plugin
│       ├── src/
│       │   ├── service/
│       │   │   ├── router.ts
│       │   │   ├── permissions.ts
│       │   │   └── awsClient.ts
│       │   └── plugin.ts
│       └── package.json
```

**Permissions (permissions.ts):**

> 📄 **File:** [permissions.ts](samples/backstage/plugins/permissions.ts)

**Router with RBAC (router.ts):**

> 📄 **File:** [router.ts](samples/backstage/plugins/router.ts)

#### Backstage UI

**Listing Component:**

> 📄 **File:** [SecretsTable.tsx](samples/backstage/plugins/SecretsTable.tsx)

**Service Page (EntityPage):**
```tsx
// packages/app/src/components/catalog/EntityPage.tsx
import { SecretsCard } from '@internal/plugin-secrets-manager';

const serviceEntityPage = (
  <EntityLayout>
    {/* ... other tabs ... */}
    <EntityLayout.Route path="/secrets" title="Secrets">
      <SecretsCard />
    </EntityLayout.Route>
  </EntityLayout>
);
```

#### Secret Creation Flow via Backstage

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant BS as Backstage
    participant KC as Keycloak
    participant AWS as AWS Secrets Manager
    participant ESO as External Secrets Operator
    participant K8s as Kubernetes

    Dev->>BS: Accesses Secrets page of the service
    BS->>KC: Verifies user roles
    KC-->>BS: Roles: [secrets-manager, domain:payments]
    
    Dev->>BS: Clicks "Create New Secret"
    BS->>BS: Validates permission (secrets-manager for domain)
    BS->>BS: Shows form
    
    Dev->>BS: Fills: database-credentials
    Dev->>BS: Adds: username, password, host
    
    BS->>AWS: CreateSecret (helpdev/prod/payments/checkout/database)
    AWS-->>BS: Secret created
    
    BS->>BS: Records audit log
    BS-->>Dev: ✅ Secret created successfully
    
    Note over ESO,K8s: Next sync cycle (1h or trigger)
    
    ESO->>AWS: GetSecretValue
    AWS-->>ESO: Secret value
    ESO->>K8s: Creates/Updates Kubernetes Secret
    K8s-->>ESO: Secret synchronized
```

#### Auditing and Compliance

**CloudWatch Logs (via AWS SDK):**
```typescript
// All value read operations are logged
{
  "timestamp": "2026-01-25T01:55:00Z",
  "action": "SECRET_VALUE_READ",
  "user": "user:default/joao.silva",
  "userEmail": "joao.silva@helpdev.com.br",
  "secret": "helpdev/prod/payments/checkout/database",
  "sourceIP": "10.0.1.50",
  "userAgent": "Backstage/1.24.0"
}
```

**Grafana Dashboard for Auditing:**
- Secret accesses per user
- Most accessed secrets
- Denied access attempts
- Alerts for off-hours access

---

## 12. Resilience and High Availability

### 11.1 Multi-Region Strategy

Multi-region architecture provides the highest level of resilience, with completely isolated regions:

```mermaid
flowchart TB
    subgraph Global["Global Layer"]
        R53["Route53<br/>(Latency-based + Failover)"]
        CF["CloudFront<br/>(Global CDN)"]
    end
    
    subgraph USE1["us-east-1 (Primary)"]
        EKS1["EKS Cluster<br/>(Multi-AZ)"]
        RDS1["RDS<br/>(Multi-AZ)"]
        KC1["Keycloak<br/>(HA)"]
    end
    
    subgraph SAE1["sa-east-1 (Secondary)"]
        EKS2["EKS Cluster<br/>(Multi-AZ)"]
        RDS2["RDS<br/>(Multi-AZ)"]
        KC2["Keycloak<br/>(HA)"]
    end
    
    R53 --> USE1
    R53 --> SAE1
    CF --> R53
    
    style Global fill:#e8f5e9
    style USE1 fill:#e3f2fd
    style SAE1 fill:#fff3e0
```

#### Resilience Levels

| Level | Scope | Protection | Components |
|-------|-------|------------|------------|
| **Pod** | Application | Container failure | Liveness/Readiness probes, Replicas |
| **Node** | Cluster | EC2 instance failure | PodDisruptionBudget, TopologySpread |
| **AZ** | Region | Datacenter failure | Multi-AZ node groups, RDS Multi-AZ |
| **Region** | Global | Complete regional failure | Isolated multi-region, Route53 failover |

#### Region Failover

```mermaid
sequenceDiagram
    participant User as User
    participant R53 as Route53
    participant USE1 as us-east-1
    participant SAE1 as sa-east-1
    participant Alert as Alertmanager
    
    User->>R53: Request
    R53->>USE1: Routing (latency)
    
    rect rgb(255, 220, 220)
        Note over USE1: Failure detected
        USE1--xR53: Health check fails
        R53->>Alert: Trigger alarm
    end
    
    R53->>SAE1: Automatic failover
    SAE1-->>User: Response
    
    Note over User,SAE1: User continues operating<br/>in secondary region
```

**Route53 Configuration:**
```yaml
# Latency-based routing with failover
RecordSet:
  Type: A
  AliasTarget:
    DNSName: !GetAtt ALB.DNSName
  SetIdentifier: us-east-1
  Region: us-east-1
  HealthCheckId: !Ref HealthCheck
  Failover: PRIMARY

RecordSet:
  Type: A
  AliasTarget:
    DNSName: !GetAtt ALB.DNSName
  SetIdentifier: sa-east-1
  Region: sa-east-1
  HealthCheckId: !Ref HealthCheck
  Failover: SECONDARY
```

### 11.2 EKS Node Groups

> 📄 **File:** [node-groups.tf](samples/terraform/node-groups.tf)

### 11.3 PodDisruptionBudget

> 📄 **File:** [pdb-example.yaml](samples/kubernetes/pdb-example.yaml)

### 11.4 Topology Spread

> 📄 **File:** [topology-spread.yaml](samples/kubernetes/topology-spread.yaml)

### 11.5 Argo CD HA

> 📄 **File:** [argocd-ha-values.yaml](samples/helm/values/argocd-ha-values.yaml)

### 11.6 RTO/RPO per Scenario

| Scenario | RTO | RPO | Strategy |
|----------|-----|-----|----------|
| Pod crash | < 30s | 0 | Kubernetes auto-restart |
| Node failure | < 5min | 0 | Pod rescheduling |
| AZ failure | < 15min | 0 | Multi-AZ, PDB |
| Region failure | < 5min | 0* | Route53 failover |

> *RPO=0 for session data (isolated per region). For persistent data, depends on the chosen replication strategy.

---

## 13. Implementation Plan

### 12.1 Timeline

```mermaid
gantt
    title Implementation Roadmap - HelpDev'Ops Infrastructure Platform
    dateFormat  YYYY-MM-DD
    section Phase 1: Foundation
        Landing Zone AWS           :f1a, 2026-02-01, 2w
        VPC + Networking           :f1b, after f1a, 1w
        EKS Base (HML + Prod)      :f1c, after f1b, 1w
    section Phase 2: GitOps Core
        Argo CD Setup              :f2a, after f1c, 1w
        Repository Structure       :f2b, after f1c, 1w
        KEDA Operator              :f2d, after f1c, 1w
        Helm Charts Base           :f2c, after f2a, 2w
    section Phase 3: Service Mesh & IAM
        Istio Install + mTLS       :f3a, after f2c, 1w
        Keycloak + RDS             :f3b, after f2c, 2w
        SSO Integration            :f3c, after f3b, 1w
    section Phase 4: Observability
        OTel Collector             :f4a, after f3a, 1w
        Mimir + Loki + Tempo       :f4b, after f4a, 2w
        Grafana + Dashboards       :f4c, after f4b, 1w
    section Phase 5: Onboarding
        Backstage Setup            :f5a, after f3c, 2w
        Software Templates         :f5b, after f5a, 1w
        Secrets Manager Plugin     :f5d, after f5a, 2w
        10 First Services          :f5c, after f4c, 3w
    section Phase 6: Scale
        Remaining Services         :f6a, after f5c, 4w
        Refinements                :f6b, after f6a, 1w
    section Phase 7: Documentation
        BU-Ready Docs              :f7a, after f5c, 2w
        Runbooks                   :f7b, after f6b, 1w
```

### 12.2 Detailed Phases

> **Note:** The phases below may have parallel activities. The Gantt timeline reflects the actual dependencies. Total estimated time: **~24 weeks (6 months)**.

#### Phase 1: Foundation (Weeks 1-4)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F1.1 | AWS Organization + accounts (hml, prod) | Platform |
| F1.2 | IAM baseline (roles, policies, OIDC) | Platform |
| F1.3 | VPC per account (3 AZs, subnets, NAT) | Platform |
| F1.4 | KMS keys (EKS, secrets, volumes) | Platform |
| F1.5 | EKS clusters (hml + prod-main) | Platform |
| F1.6 | Transit Gateway between accounts | Platform |

**Acceptance Criteria:**
- [ ] AWS accounts created with security baseline
- [ ] VPCs provisioned with functional connectivity
- [ ] EKS accessible via kubectl with IRSA

#### Phase 2: GitOps Core (Weeks 5-8)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F2.1 | Argo CD installed (HA) in both clusters | Platform |
| F2.2 | `platform-argocd-config` repository | Platform |
| F2.3 | App-of-Apps configured | Platform |
| F2.4 | Base Helm chart for services (with Service Mesh, Auth and KEDA support) | Platform |
| F2.5 | External Secrets Operator | Platform |
| F2.6 | KEDA Operator installed | Platform |

**Acceptance Criteria:**
- [ ] Argo CD UI accessible with SSO
- [ ] Example app deploy via GitOps functional
- [ ] Secrets synchronized from AWS Secrets Manager
- [ ] KEDA operator functional with test ScaledObject

#### Phase 3: Service Mesh and IAM (Weeks 9-12)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F3.1 | Istio installed in both clusters | Platform |
| F3.2 | mTLS STRICT enabled by default | Platform |
| F3.3 | RDS PostgreSQL for Keycloak (HML + Prod) | Platform |
| F3.4 | Keycloak installed (HA in prod) | Platform |
| F3.5 | Realms and clients configured | Platform |
| F3.6 | Argo CD + Keycloak SSO integration | Platform |
| F3.7 | Grafana + Keycloak SSO integration | Platform |

**Acceptance Criteria:**
- [ ] mTLS functional between services
- [ ] SSO login in Argo CD and Grafana via Keycloak
- [ ] Service mesh metrics visible in Grafana

#### Phase 4: Observability (Weeks 13-16)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F4.1 | OTel Collector (gateway mode) | Platform |
| F4.2 | Mimir for metrics | Platform |
| F4.3 | Loki for logs | Platform |
| F4.4 | Tempo for traces | Platform |
| F4.5 | Central Grafana with datasources | Platform |
| F4.6 | Standard dashboards (including Istio and Keycloak) | Platform |
| F4.7 | Base alerts | Platform |

**Acceptance Criteria:**
- [ ] Metrics, logs and traces from example app visible in Grafana
- [ ] Remote datasources (hml) working in prod Grafana
- [ ] SLO alerts configured

#### Phase 5: Onboarding (Weeks 17-20)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F5.1 | **Backstage installed and configured** | Platform |
| F5.2 | **Software Templates created** (Java, Kotlin, Node, Go) | Platform |
| F5.3 | GitHub Actions templates integrated | Platform |
| F5.4 | **Secrets Manager Plugin in Backstage** | Platform |
| F5.5 | Onboarding documentation | Platform |
| F5.6 | First 10 services created via Backstage | Squads |

**Acceptance Criteria:**
- [ ] Backstage accessible with SSO via Keycloak
- [ ] New service created in **< 10 min** using Backstage
- [ ] Code and infra repos created automatically
- [ ] Application registered automatically in Argo CD
- [ ] Secrets Plugin functional with RBAC (list, read, create, rotate)
- [ ] 10 services in production via GitOps
- [ ] Zero downtime during deploy

#### Phase 6: Scale (Weeks 21-24)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F6.1 | Migration of remaining 40+ services | Squads |
| F6.2 | Resource tuning and autoscaling | Platform |
| F6.3 | Alert refinement | Platform |
| F6.4 | Basic chaos engineering | Platform |

**Acceptance Criteria:**
- [ ] All 50+ services in production
- [ ] SLOs defined and monitored
- [ ] Runbooks for common incidents

#### Phase 7: BU-Ready Documentation (Continuous)

| Deliverable | Description | Owner |
|-------------|-------------|-------|
| F7.1 | BU expansion documentation | Platform |
| F7.2 | Parameterized Terraform modules | Platform |
| F7.3 | New account/cluster playbook | Platform |

### 12.3 Multi-Region Expansion (Post MVP)

After consolidating the first region (`us-east-1`), expansion to new regions follows this playbook:

```mermaid
gantt
    title Multi-Region Expansion (per additional region)
    dateFormat  YYYY-MM-DD
    section Infrastructure
        VPC + Networking (new region)      :mr1, 2026-09-01, 1w
        EKS Cluster (new region)           :mr2, after mr1, 1w
    section Platform Services
        Keycloak (sync config)              :mr3, after mr2, 3d
        Observability Stack                 :mr4, after mr2, 1w
        Argo CD                             :mr5, after mr2, 3d
    section Validation
        Smoke tests                         :mr6, after mr5, 2d
        Service pilot (1-2 services)        :mr7, after mr6, 1w
    section Rollout
        Phased rollout enabled           :mr8, after mr7, 3d
        Documentation updated             :mr9, after mr8, 2d
```

#### Multi-Region Expansion Phases

| Phase | Deliverable | Est. Time | Owner |
|-------|-------------|-----------|-------|
| **MR1** | VPC + Networking in new region | 1 week | Platform |
| **MR2** | EKS Cluster configured | 1 week | Platform |
| **MR3** | Keycloak deploy (automatic config sync via Git) | 3 days | Platform |
| **MR4** | Mimir + Loki + Tempo in new region | 1 week | Platform |
| **MR5** | Argo CD configured with cluster | 3 days | Platform |
| **MR6** | Smoke tests (connectivity, auth, observability) | 2 days | Platform |
| **MR7** | Pilot: 1-2 services with phased rollout | 1 week | Platform + Squad |
| **MR8** | Enable phased rollout for all services | 3 days | Platform |
| **MR9** | Update runbooks and documentation | 2 days | Platform |

**Total time per additional region:** ~4-5 weeks

**Acceptance Criteria for New Region:**
- [ ] VPC with 3 AZs operational
- [ ] EKS cluster healthy with nodes in all AZs
- [ ] Keycloak sync working (clients visible)
- [ ] Observability data reaching Central Grafana
- [ ] Argo CD syncing apps correctly
- [ ] Phased rollout executed successfully on pilot service
- [ ] Route53 health checks configured
- [ ] Failover runbook documented and tested

#### Pre-Expansion Checklist

Before adding a new region, validate:

| Item | Validation |
|------|------------|
| Business demand | Does latency or compliance justify new region? |
| Cost approved | Budget for ~2x resources (VPC, EKS, RDS, etc)? |
| Team capacity | Does Platform team have bandwidth? |
| Compatible services | Do all services support stateless/isolated? |
| Regional data | Regional data strategy defined? |

### 12.4 Resources Platform (Application Resources)

Self-service platform implementation for AWS resources:

```mermaid
gantt
    title Roadmap - Application Resources Platform
    dateFormat  YYYY-MM-DD
    section Phase 1: Modules
        SQS Module                    :ar1, 2026-06-01, 1w
        SNS Module                    :ar2, after ar1, 1w
        S3 Module                     :ar3, after ar1, 1w
        RDS Module                    :ar4, after ar2, 2w
    section Phase 2: Infrastructure
        Repo platform-terraform-modules :ar5, 2026-06-01, 1w
        Repo platform-project-resources :ar6, after ar5, 1w
        GitHub Actions (plan/apply)   :ar7, after ar6, 1w
    section Phase 3: Backstage
        Resources Plugin              :ar8, after ar4, 2w
        Templates (SQS, S3)           :ar9, after ar8, 1w
        Templates (RDS, DynamoDB)     :ar10, after ar9, 1w
    section Phase 4: Rollout
        Pilot (2 services)            :ar11, after ar10, 2w
        Documentation                  :ar12, after ar11, 1w
        General rollout                :ar13, after ar12, 2w
```

#### Detailed Phases

| Phase | Deliverable | Est. Time | Owner |
|-------|-------------|-----------|-------|
| **AR1** | Terraform Modules (SQS, SNS, S3) | 2 weeks | Platform |
| **AR2** | RDS + DynamoDB + ElastiCache Module | 2 weeks | Platform |
| **AR3** | Repositories and GitHub Actions | 2 weeks | Platform |
| **AR4** | Backstage Plugin + Templates | 3 weeks | Platform |
| **AR5** | Pilot with 2 services | 2 weeks | Platform + Squads |
| **AR6** | Documentation and rollout | 2 weeks | Platform |

**Total time:** ~10-12 weeks

**Acceptance Criteria:**
- [ ] Modules tested (Terratest or similar)
- [ ] Versioning via Git tags working
- [ ] GitHub Actions executing plan/apply correctly
- [ ] Backstage creating PRs with correct code
- [ ] CODEOWNERS validating approvals
- [ ] Services consuming resources via External Secrets
- [ ] Complete documentation for developers

---

## 14. Risks and Mitigations

### 13.1 General Risks

| # | Risk | Probability | Impact | Mitigation |
|---|------|-------------|--------|------------|
| R1 | Repository sprawl (50+ services × 2) | High | Medium | Automated scaffolding; templates; naming conventions |
| R2 | Latency in federated observability | Medium | Medium | Caching; timeouts; fallback to local query |
| R3 | Multi-account complexity for teams | Medium | High | Clear documentation; abstractions via Argo CD |
| R4 | Observability cost | Medium | Medium | Sampling; differentiated retention; quotas per tenant |
| R5 | Drift between environments | Low | High | Strict GitOps; auto-sync; drift detection alerts |
| R6 | Transit Gateway dependency | Low | High | Fallback to VPC peering; DR documentation |
| R7 | Service Mesh complexity | Medium | Medium | Abstraction via Helm chart; training; runbooks |
| R8 | Keycloak as SPOF for authentication | Low | Medium | HA per region; RDS Multi-AZ; token caching |

### 13.2 Multi-Region Risks

| # | Risk | Probability | Impact | Mitigation |
|---|------|-------------|--------|------------|
| R9 | Desynchronized configuration between regions | Medium | High | GitOps sync; pre-merge validation; automated tests |
| R10 | Cost multiplied by region | High | Medium | Start with single region; add on demand; cost alerts |
| R11 | Cross-region troubleshooting complexity | Medium | Medium | Central Grafana; centralized logs; specific runbooks |
| R12 | Phased rollout failure (region 2 fails after region 1) | Low | High | Rigorous health checks; configurable delay; automatic rollback |
| R13 | Data inconsistency between regions (if applicable) | Medium | High | Isolated-by-design architecture; no shared data |
| R14 | Keycloak config sync latency | Low | Low | Webhook triggers; Argo CD auto-sync; drift monitoring |
| R15 | Slow DNS failover (Route53) | Low | Medium | Low TTL; aggressive health checks; documented manual fallback |

### 13.3 Multi-Region Mitigation Matrix

```mermaid
quadrantChart
    title Multi-Region Risks: Impact vs Probability
    x-axis Low Probability --> High Probability
    y-axis Low Impact --> High Impact
    quadrant-1 Monitor closely
    quadrant-2 Immediate action
    quadrant-3 Low priority
    quadrant-4 Contingency plan
    R9: [0.5, 0.8]
    R10: [0.7, 0.5]
    R11: [0.5, 0.5]
    R12: [0.3, 0.8]
    R13: [0.5, 0.8]
    R14: [0.3, 0.3]
    R15: [0.3, 0.5]
```

---

## 15. Success Metrics

### 14.1 Platform KPIs

| Metric | Baseline | Target (6 months) |
|---------|----------|------------------|
| **Deployment Frequency** | - | ≥ 10/day/service |
| **Lead Time for Changes** | - | < 1 hora |
| **Mean Time to Recovery (MTTR)** | - | < 30 min |
| **Change Failure Rate** | - | < 5% |
| **Platform Uptime** | - | 99.9% |

### 14.2 Adoption KPIs

| Metric | Target |
|--------|--------|
| Services onboarded (4 weeks) | 10 |
| Services onboarded (8 weeks) | 30 |
| Services onboarded (12 weeks) | 50+ |
| New service onboarding time (automated) | ~10 min |
| New service onboarding time (including preparation) | < 30 min |

### 14.3 Multi-Region KPIs

| Metric | Target |
|--------|--------|
| **Region failover time** | < 5 min |
| **Config consistency between regions** | 100% (after sync) |
| **Complete phased rollout time** | < 2 hours |
| **Rollback rate per region** | < 2% |
| **Keycloak sync latency** | < 5 min |

---

## 16. Architecture Decisions (ADRs)

Architecture decisions are documented in individual files in the `adr/` folder:

| ADR | Title | Status |
|-----|-------|--------|
| [ADR-001](adr/ADR-001-argocd-distributed.md) | Argo CD per Cluster vs Centralized | Accepted |
| [ADR-002](adr/ADR-002-observability-federated.md) | Federated vs Centralized Observability | Accepted |
| [ADR-003](adr/ADR-003-image-digest.md) | Image by Digest vs Tag | Accepted |
| [ADR-004](adr/ADR-004-istio-service-mesh.md) | Istio as Service Mesh | Accepted |
| [ADR-005](adr/ADR-005-keycloak-idp.md) | Keycloak as Identity Provider | Accepted |
| [ADR-006](adr/ADR-006-keycloak-per-environment.md) | Keycloak per Environment vs Centralized | Accepted |
| [ADR-007](adr/ADR-007-backstage-developer-portal.md) | Backstage as Developer Portal | Accepted |
| [ADR-008](adr/ADR-008-secrets-management.md) | AWS Secrets Manager + External Secrets + Backstage UI | Accepted |
| [ADR-009](adr/ADR-009-multi-region-architecture.md) | Multi-Region Architecture | Accepted |
| [ADR-010](adr/ADR-010-keycloak-gitops-sync.md) | Multi-Region Keycloak with GitOps Sync | Accepted |
| [ADR-011](adr/ADR-011-application-resources.md) | Application Resources Management via Self-Service | Accepted |
| [ADR-012](adr/ADR-012-shared-services-account.md) | Shared Services Account for Global Resources | Accepted |

---

## 17. Glossary

| Term | Definition |
|------|------------|
| **BU** | Business Unit - Business unit with operational autonomy |
| **GitOps** | Paradigm where Git is the source of truth for desired state |
| **IRSA** | IAM Roles for Service Accounts - workload identity in EKS |
| **OTel** | OpenTelemetry - open standard for instrumentation |
| **MIMIR** | Prometheus-compatible metrics backend (Grafana Labs) |
| **LOKI** | Logs backend (Grafana Labs) |
| **TEMPO** | Traces backend (Grafana Labs) |
| **Blast Radius** | Scope of failure impact |
| **mTLS** | Mutual TLS - mutual authentication between services |
| **Service Mesh** | Infrastructure layer for service-to-service communication |
| **Istio** | Open source Service Mesh for Kubernetes |
| **Keycloak** | Open source Identity and Access Management |
| **OIDC** | OpenID Connect - authentication protocol |
| **Backstage** | Open source Developer Portal by Spotify |
| **Software Template** | Backstage template for automated scaffolding |
| **Software Catalog** | Inventory of services and components in Backstage |
| **KEDA** | Kubernetes Event-Driven Autoscaling - event-based autoscaling |
| **ScaledObject** | KEDA CRD that defines autoscaling rules |
| **HPA** | Horizontal Pod Autoscaler - Kubernetes native autoscaling |
| **Multi-Region** | Architecture with deploy in multiple AWS regions |
| **Phased Rollout** | Gradual deploy region by region with delays |
| **Region Short Code** | Abbreviated region code (use1, sae1, euw1) |
| **External Secrets Operator (ESO)** | Operator that syncs secrets from external providers to Kubernetes |
| **AWS Secrets Manager** | AWS managed service for secure secret storage |
| **ClusterSecretStore** | ESO CRD that defines connection to secrets backend |
| **ExternalSecret** | ESO CRD that defines which secrets to sync |

---

## 18. References

- [EKS Best Practices Guide](https://aws.github.io/aws-eks-best-practices/)
- [Argo CD Documentation](https://argo-cd.readthedocs.io/)
- [OpenTelemetry Documentation](https://opentelemetry.io/docs/)
- [Grafana Mimir](https://grafana.com/oss/mimir/)
- [Grafana Loki](https://grafana.com/oss/loki/)
- [Grafana Tempo](https://grafana.com/oss/tempo/)
- [Istio Documentation](https://istio.io/latest/docs/)
- [Keycloak Documentation](https://www.keycloak.org/documentation)
- [Backstage Documentation](https://backstage.io/docs/)
- [Backstage Software Templates](https://backstage.io/docs/features/software-templates/)
- [KEDA Documentation](https://keda.sh/docs/)
- [KEDA Scalers](https://keda.sh/docs/scalers/)
- [External Secrets Operator](https://external-secrets.io/latest/)
- [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/)
- [Backstage Permissions](https://backstage.io/docs/permissions/overview)

---

## 19. Approvals

| Role | Name | Date | Status |
|------|------|------|--------|
| Platform Architect | | | Pending |
| Tech Lead | | | Pending |
| Engineering Manager | | | Pending |
| Security | | | Pending |

---

## Annexes

### A. Validation Checklist

- [ ] Each account has limits and guardrails (IAM/KMS/CloudTrail)
- [ ] EKS per environment with nodegroups per profile and HA in multiple AZs
- [ ] Argo CD per cluster with HA and RBAC
- [ ] GitOps: desired state always in Git; Release required for production
- [ ] Observability per cluster; Central Grafana reads remote datasources
- [ ] OTel with separate pipelines for metrics/logs/traces
- [ ] PDB + anti-affinity + probes in critical workloads
- [ ] Promotion and rollback strategy defined
- [ ] Istio installed with mTLS STRICT enabled
- [ ] Keycloak in HA with RDS Multi-AZ
- [ ] SSO configured for Argo CD and Grafana via Keycloak
- [ ] Base Helm chart with Service Mesh, Auth and KEDA support
- [ ] KEDA Operator installed with AWS scalers configured (SQS, Kafka, Prometheus)
- [ ] Backstage installed with SSO via Keycloak
- [ ] Software Templates created for all supported languages
- [ ] Service creation via Backstage generates repos and registers in Argo CD automatically
- [ ] External Secrets Operator installed and configured with AWS Secrets Manager
- [ ] Secrets Plugin in Backstage with RBAC via Keycloak
- [ ] Secrets naming convention documented and applied

### B. Next Steps (Post-MVP)

1. Implement Karpenter for more efficient autoscaling
2. Implement FinOps with Kubecost or similar
3. Expand to BUs when there is demand
4. Chaos Engineering with Litmus or similar
5. Gateway API as replacement for traditional Ingress
6. Advanced policies with OPA/Gatekeeper