Terraform in Production: State Isolation, Remote Locking, and Modular Architecture
Deep dive into Terraform production operations: configuring S3 with DynamoDB state locks, structuring monorepos vs multi-repo workspaces, and zero-downtime state migrations.
CloudPlatformLabs Engineering
Platform Architecture Team
1. Remote State Locking with AWS S3 and DynamoDB
Local state files are catastrophic for team collaboration, risking state corruption, accidental overwrite, and plaintext secret leakage in git repositories. A production backend requires high durability storage with concurrent write locking.
We use an Amazon S3 bucket with server-side encryption (SSE-KMS) and versioning enabled, combined with a DynamoDB table with a primary key named `LockID` to ensure atomic state locks during `terraform apply` operations.
terraform {
required_version = ">= 1.8.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.50"
}
}
backend "s3" {
bucket = "cpl-terraform-state-prod"
key = "networking/vpc/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "cpl-terraform-locks"
encrypt = true
}
}Enable Bucket Versioning
Always enable object versioning on your S3 state bucket. If a corrupted state file is uploaded or state is damaged, versioning allows instant point-in-time recovery.
2. The State Isolation Principle: Why Blast Radiuses Matter
Placing your entire cloud architecture (networking, databases, compute, DNS) into a single monolithic Terraform state file is a recipe for disaster. A failure or typo in an application DNS record can bring down primary RDS databases or VPC peerings.
Isolate your state by infrastructure lifecycle frequency and blast radius. Low-churn resources (VPCs, subnets, route tables) should live in dedicated state files completely separate from high-churn resources (Kubernetes deployments, ECS task definitions, lambda functions).
- Layer 1: Foundation (VPC, Subnets, NAT Gateways, Transit Gateways)
- Layer 2: Data Tier (PostgreSQL RDS, DynamoDB, Redis Clusters)
- Layer 3: Orchestration Platform (EKS Clusters, Node Groups, IAM Roles)
- Layer 4: Workloads & Routing (Ingress ALB, Route53, ACM Certificates)
3. Safe Resource Refactoring with 'moved' Blocks
Historically, renaming a resource or moving it into a child module caused Terraform to plan a destructive recreation: destroy the existing live resource and create a new one. In modern Terraform, declarative `moved` blocks instruct Terraform to update state pointers seamlessly without touching live infrastructure.
# Migrating standalone resource into a reusable module
moved {
from = aws_security_group.web_tier
to = module.security_groups.aws_security_group.web_tier
}
moved {
from = aws_s3_bucket.assets
to = aws_s3_bucket.static_assets
}Related Topics & Tags
Written by CloudPlatformLabs Engineering
Platform Architecture Team
CloudPlatformLabs publishes independent, peer-reviewed cloud and DevOps architecture blueprints, hands-on tutorials, and infrastructure case studies. Our mission is to make advanced infrastructure engineering transparent, resilient, and developer-friendly.