Skip to content

README.md

Latest commit

 

History

History
389 lines (309 loc) · 18.8 KB

README.md

File metadata and controls

389 lines (309 loc) · 18.8 KB
Raw

AWS Config Tagging Compliance Module

Introduction

This Terraform module provisions an AWS Config custom rule that evaluates tagging compliance across your AWS resources. It integrates with a separate validation Lambda module that retrieves tagging rules from a DynamoDB table and automatically evaluates resources against those rules, marking them as compliant or non-compliant based on required tags and their values.

Key Features

  • Modular Architecture: Separates AWS Config rules from Lambda function management
  • Custom AWS Config Rules: Automatically evaluate resources for tagging compliance
  • Flexible Rule Configuration: Define rules via DynamoDB with support for:
    • Required vs. optional tags
    • Specific permitted tag values
    • Regex pattern matching for tag values
    • Account-specific or global rules
    • Enabled/disabled rule toggles
  • High Performance Rules Caching: Optional in-memory caching reduces DynamoDB reads by 80-90% and improves response times
  • Cross-Account Lambda Support: Allow any account in your organization to invoke the Lambda
  • Structured Logging: JSON-formatted logs for easy integration with CloudWatch Logs Insights
  • IAM Best Practices: Minimal permissions with least-privilege access
  • Encryption Support: Optional KMS encryption for CloudWatch Logs
  • Configurable Evaluation Frequency: Control how often resources are evaluated

Module Components

  • AWS Lambda Function: Python-based handler deployed via ../validation module that evaluates resource tags against rules stored in DynamoDB
  • AWS Config Custom Rule: Invokes the Lambda function to assess resource compliance
  • Lambda Permissions: Grants cross-account and organization-wide invocation rights
  • CloudWatch Logs: Structured JSON logging with optional encryption and retention policies

Architecture

The config module now uses a separated validation module:

┌──────────────────────────────┐
│   Config Module (main.tf)     │
│                               │
│  • Config custom rules        │
│  • Lambda permissions         │
│  • Cross-account access       │
└────────────┬──────────────────┘
             │
             │ Calls
             ▼
┌──────────────────────────────┐
│  Validation Module            │
│                               │
│  • Lambda function            │
│  • IAM role & policy          │
│  • CloudWatch logs            │
│  • DynamoDB integration       │
└──────────────────────────────┘

Usage

Basic Example (Same Account)

module "config_tagging_compliance" {
  source = "./modules/config"

  # DynamoDB table containing tagging rules
  dynamodb_table_arn = aws_dynamodb_table.tagging_rules.arn

  # Lambda configuration (passed to validation module)
  lambda_name        = "tagging-compliance-handler"
  lambda_description = "Evaluates tagging compliance for AWS resources"
  lambda_runtime     = "python3.12"
  lambda_timeout     = 30
  lambda_log_level   = "INFO"
  lambda_role_name   = "tagging-compliance-lambda-role"

  # Rules caching configuration (optional, improves performance)
  rules_cache_enabled     = true   # Enable in-memory caching of rules
  rules_cache_ttl_seconds = 300    # Cache rules for 5 minutes

  # AWS Config rule configuration
  config_name                    = "tagging-compliance"
  config_max_execution_frequency = "TwentyFour_Hours"
  config_resource_types          = ["AWS::EC2::Instance", "AWS::S3::Bucket", "AWS::RDS::DBInstance"]

  # CloudWatch Logs configuration
  cloudwatch_logs_retention_in_days = 7
  cloudwatch_logs_log_group_class   = "STANDARD"

  # Tags to apply to all resources
  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

Lambda and Config Separation

The config module now delegates Lambda function management to the validation module for better separation of concerns:

  • config module: Manages AWS Config rules and Lambda permissions
  • validation module: Manages Lambda function, IAM roles, and logging

This allows you to:

  • Share a single Lambda across multiple Config rules
  • Manage Lambda and Config separately
  • Reuse the validation module for other purposes

If you need to customize the Lambda function beyond what this module provides, you can directly use the validation module.

Rule Configuration in DynamoDB

The compliance rules are stored in a DynamoDB table with the following structure:

# Example: Require an Environment tag with specific values
{
  AccountIds  = ["*"]                    # Apply to all accounts
  Enabled     = true                     # Rule is active
  Required    = true                     # Tag must be present
  ResourceType = "AWS::EC2::*"           # Applies to all EC2 resources
  Tag         = "Environment"            # Tag key to check
  ValuePattern = ""                      # No regex pattern
  Values      = ["Production", "Staging", "Development"]  # Permitted values
}

# Example: Optional tag with email pattern matching
{
  AccountIds   = ["123456789012"]        # Specific account only
  Enabled      = true
  Required     = false                   # Tag is optional
  ResourceType = "AWS::EC2::Instance"    # Specific resource type
  Tag          = "Owner"
  ValuePattern = "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"  # Email regex
  Values       = []                      # No specific values
}

Rule Definition Reference

Field Type Required Default Description
AccountIds list(string) No ["*"] AWS account IDs to apply rule to. Use "*" for all accounts.
Enabled bool No true Whether the rule is active. Set to false to disable without deleting.
Required bool No true Whether the tag must be present. Set to false to only validate if present.
ResourceType string Yes — AWS resource type pattern (e.g., "AWS::EC2::*", "AWS::S3::Bucket"). Use "*" for all types.
Tag string Yes — Tag key to evaluate (e.g., "Environment", "Owner").
ValuePattern string No "" Optional regex pattern for validating tag values.
Values list(string) No [] List of permitted tag values (ignored if ValuePattern is set).

Examples

Example 1: Enforce Environment Tags

# Store rule in DynamoDB
resource "aws_dynamodb_table_item" "environment_tag_rule" {
  table_name = aws_dynamodb_table.tagging_rules.name
  item = jsonencode({
    ResourceType = { S = "AWS::EC2::*" }
    Tag          = { S = "Environment" }
    Enabled      = { BOOL = true }
    Required     = { BOOL = true }
    Values       = { SS = ["production", "staging", "development", "sandbox"] }
    AccountIds   = { SS = ["*"] }
  })
}

Example 2: Enforce Owner Email Tag (Pattern-based)

# Store rule in DynamoDB
resource "aws_dynamodb_table_item" "owner_email_rule" {
  table_name = aws_dynamodb_table.tagging_rules.name
  item = jsonencode({
    ResourceType  = { S = "AWS::*" }
    Tag           = { S = "Owner" }
    Enabled       = { BOOL = true }
    Required      = { BOOL = true }
    ValuePattern  = { S = "^[a-zA-Z0-9._%+-]+@company\\.com$" }
    AccountIds    = { SS = ["*"] }
  })
}

Example 3: Optional Cost Center Tag (Account-specific)

# Store rule in DynamoDB
resource "aws_dynamodb_table_item" "cost_center_rule" {
  table_name = aws_dynamodb_table.tagging_rules.name
  item = jsonencode({
    ResourceType = { S = "AWS::RDS::DBInstance" }
    Tag          = { S = "CostCenter" }
    Enabled      = { BOOL = true }
    Required     = { BOOL = false }  # Optional tag
    Values       = { SS = ["CC-001", "CC-002", "CC-003"] }
    AccountIds   = { SS = ["123456789012", "210987654321"] }  # Specific accounts
  })
}

Example 4: Full Module Configuration

# DynamoDB table for rules
resource "aws_dynamodb_table" "tagging_rules" {
  name           = "tagging-compliance-rules"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "ResourceType"
  
  attribute {
    name = "ResourceType"
    type = "S"
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# Deploy the config module
module "config_tagging_compliance" {
  source = "./modules/config"

  dynamodb_table_arn = aws_dynamodb_table.tagging_rules.arn

  lambda_name        = "tagging-compliance"
  lambda_description = "Evaluates resource tagging compliance via AWS Config"
  lambda_log_level   = "DEBUG"

  config_name                    = "resource-tagging-compliance"
  config_max_execution_frequency = "Six_Hours"
  config_resource_types          = ["AWS::EC2::Instance", "AWS::S3::Bucket", "AWS::RDS::DBInstance", "AWS::Lambda::Function"]

  cloudwatch_logs_retention_in_days = 14
  cloudwatch_logs_kms_key_id        = aws_kms_key.logs.id

  tags = {
    Environment = "production"
    Team        = "platform-engineering"
  }
}

# Output compliance status
output "config_rule_arn" {
  value = module.config_tagging_compliance.config_rule_arn
}

output "lambda_function_arn" {
  value = module.config_tagging_compliance.lambda_function_arn
}

Compliance Evaluation Logic

The Lambda function evaluates resources against rules using the following logic:

  1. Rule Matching: Find all enabled rules matching the resource type and account
  2. Tag Presence Check: For required tags, verify the tag exists on the resource
  3. Value Validation: If values are specified, verify the tag value is in the permitted list
  4. Pattern Matching: If a regex pattern is specified, verify the tag value matches the pattern
  5. Result: Mark resource as:
    • COMPLIANT: All required tags present with valid values
    • NON_COMPLIANT: Missing required tags or invalid values
    • NOT_APPLICABLE: No matching rules for the resource

Performance Optimization: Rules Caching

Overview

The Lambda function supports optional in-memory caching of compliance rules to significantly reduce DynamoDB read costs and improve evaluation performance. When enabled, rules are cached in Lambda's execution environment and reused across invocations within the same container.

Benefits

  • Cost Reduction: Reduces DynamoDB read capacity consumption by 80-90% in typical workloads
  • Performance: Faster evaluation times by eliminating DynamoDB API calls on cache hits
  • Server-Side Filtering: Only enabled rules are fetched from DynamoDB, reducing data transfer
  • Automatic Expiration: Configurable TTL ensures rules are periodically refreshed

Configuration

module "config_tagging_compliance" {
  source = "./modules/config"

  # ... other configuration ...

  # Enable rules caching (default: true)
  rules_cache_enabled = true

  # Cache TTL in seconds (default: 300)
  # Rules are refreshed after this period
  rules_cache_ttl_seconds = 300  # 5 minutes
}

How It Works

  1. First Invocation (Cold Start): Lambda fetches rules from DynamoDB and stores them in memory
  2. Subsequent Invocations (Warm Start): Lambda reuses cached rules if TTL hasn't expired
  3. Cache Expiration: After TTL expires, Lambda fetches fresh rules from DynamoDB
  4. Container Recycling: When AWS recycles the Lambda container, cache is reset

Trade-offs

Advantages:

  • Significant cost savings on DynamoDB reads
  • Improved Lambda execution time
  • Reduced DynamoDB throttling risk

Considerations:

  • Rule changes take up to TTL seconds to propagate to all Lambda instances
  • Multiple Lambda containers may have slightly stale rules during the TTL window
  • For most use cases, a 5-minute delay is acceptable given the cost savings

Recommendations

  • Production workloads: Enable caching with 300-600 second TTL
  • Testing/development: Disable caching or use shorter TTL (60 seconds) for faster iteration
  • High-frequency changes: Use shorter TTL (60-120 seconds) if rules change frequently
  • Cost optimization: Use longer TTL (600-3600 seconds) for stable rule sets

Monitoring and Troubleshooting

CloudWatch Logs

The Lambda function outputs structured JSON logs. Query them using CloudWatch Logs Insights:

fields @timestamp, @message, compliance_type, resource_id
| filter action = "lambda_handler"
| stats count() by compliance_type

AWS Config Console

View compliance status directly in the AWS Config console:

  • Navigate to Config Rules → tagging-compliance rule
  • See non-compliant resources and compliance timeline
  • Remediate resources or update rules as needed

Common Issues

  • No matching rules found: Verify the rule ResourceType pattern matches your resources
  • Lambda timeout: Increase lambda_timeout if evaluating many tags
  • Permission errors: Verify Lambda role has access to both DynamoDB and Config

Providers

Name Version
aws >= 6.0.0

Inputs

Name Description Type Default Required
compliance_rule_table_arn The ARN of the DynamoDB table to store tags for AWS resources. string n/a yes
config_resource_types List of AWS resource types to evaluate (https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html) list(string) n/a yes
cloudwatch_logs_kms_key_id The KMS key ID to encrypt CloudWatch Logs. If not provided, logs will not be encrypted. string null no
cloudwatch_logs_log_group_class The log group class for CloudWatch Logs. Valid values are STANDARD and INFREQUENT_ACCESS. string "STANDARD" no
cloudwatch_logs_retention_in_days The number of days to retain CloudWatch Logs. Valid values are 0 (retain indefinitely), 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, or 1827. number 7 no
config_name The name of the AWS Config rule string "tagging-compliance" no
configuration_item_change_enabled When true, the rule is invoked when AWS Config records a configuration change (including oversized configuration notifications) for resources in scope. Set to false and enable periodic_evaluation_enabled to run only on a schedule. bool true no
lambda_architectures The lambda architecture to use. Valid values are x86_64 and arm64. list(string) 
[
  "arm64"
]
 no
lambda_artifacts_dir The directory to store any generated artifacts for the lambda string "builds" no
lambda_create_role Indicates we should create the role bool true no
lambda_description The description of the Lambda function to handle AWS Organization account movements. string "Handles AWS Organization account movements for tagging compliance." no
lambda_log_level The log level for the Lambda function. Valid values are DEBUG, INFO, WARNING, ERROR, CRITICAL. string "INFO" no
lambda_memory_size The amount of memory in MB allocated to the Lambda function. number 128 no
lambda_name The name of the Lambda function to handle AWS Organization account movements. string "lz-tagging-compliance" no
lambda_role_name The name of the IAM role to be created for the Lambda function. string "lz-tagging-compliance" no
lambda_runtime The runtime environment for the Lambda function. string "python3.14" no
lambda_timeout The timeout for the Lambda function in seconds. number 30 no
maximum_execution_frequency Maximum interval between periodic evaluations when periodic_evaluation_enabled is true. Values: One_Hour, Three_Hours, Six_Hours, Twelve_Hours, TwentyFour_Hours. string "TwentyFour_Hours" no
organizations_table_arn The ARN of the DynamoDB table to store AWS Organizations account information. string null no
periodic_evaluation_enabled When true, AWS Config also invokes the rule on a recurring schedule; use maximum_execution_frequency to set how often. Combine with configuration_item_change_enabled for both change-driven and scheduled evaluations. bool false no
rules_cache_enabled Enable or disable caching of compliance rules in Lambda function memory. When enabled, rules are cached between invocations to reduce DynamoDB read costs and improve performance. bool true no
rules_cache_ttl_seconds Time-to-live (TTL) in seconds for cached compliance rules. After this period, the cache expires and rules are re-fetched from DynamoDB. Default is 300 seconds (5 minutes). number 300 no
tags A map of tags to apply to the Lambda function. map(string) {} no

Outputs

Name Description
config_arn The ARN of the AWS Config rule for tagging compliance.
lambda_arn The ARN of the Lambda function for tagging compliance.