AWS CDK EC2 CodeDeploy Pipeline: ALB and Auto Scaling Without the 2am 502s

You wired up CodeDeploy, pushed a deploy, CodeDeploy reported success, and your users were getting 502s. Welcome to the club. If you are trying to get aws cdk codedeploy ec2 auto scaling working properly in production, not just the tutorial skeleton that falls apart on the first real load event, this one is for you.

---

Who Is This For

You are a backend or DevOps engineer already comfortable with CDK and AWS. You have shipped a few stacks. You know what an ALB is. You are here because you are trying to wire CodePipeline to CodeBuild to CodeDeploy to an ASG in a way that holds up under load, handles rollbacks, and does not wake you at 2 AM.

This is also for teams who have been burnt by container migrations. Maybe ECS is not the right fit yet, maybe the app has stateful assumptions, maybe the org just is not ready, and you need EC2-based deployments to be production-grade, not "good enough for now."

---

Problem and Why It Matters

Here is what nobody tells you about CodeDeploy plus Auto Scaling plus ALB: the compute layer is the easy part.

The hard part is the coordination layer. CodeDeploy has its own definition of healthy. The ALB has its own. The ASG has its own. All three have to agree, or you get a deploy that is technically successful while your users stare at 502s.

I have seen this exact failure twice on client projects. Both times the pipeline was green, CloudWatch showed the EC2 instances running, and the ALB was quietly draining targets because the health check path returned 404 during the ApplicationStart lifecycle phase, right when the new app version was still spinning up. CodeDeploy did not know. The ASG did not know. The ALB removed the instances from rotation and moved on.

Then there is the rollback problem. Everyone assumes rollback works. It does, until your S3 artifact bucket has a lifecycle policy aggressive enough to have deleted the previous revision. CodeDeploy reports that it is rolling back, then fails because there is nothing to roll back to. That is a discovery you do not want to make during an incident.

The operational glue - SSM parameters for runtime config, SNS for deploy notifications, EventBridge for pipeline state changes, CloudWatch alarms - is what separates "we have CodeDeploy" from "we have a production deployment system."

---

When This Approach Fits

EC2 plus CodeDeploy is still the right call in a few specific situations. Do not let anyone push you onto ECS if these apply:

• Your app has persistent local state, IPC between processes, or syscall requirements that make containers awkward
• You deploy fewer than about 10 times a day and the 5 to 15 minute CodeDeploy lifecycle overhead is acceptable
• The team owns the OS and runtime environment and wants to keep it that way
• You have compliance or licensing requirements tied to specific machine configurations
• You are migrating a legacy app and containerizing it is a quarter-long project, not a sprint

If you deploy 20 or more times a day and speed matters, containers are genuinely better. But for the workloads where EC2 fits, this architecture gives you zero-downtime rolling deploys without the ECS or EKS operational surface area.

---

Architecture or Implementation Overview

It is a layered model. Standard shape, but the details matter.

Network tier: Custom VPC with public and private subnets across 2 AZs. ALB lives in the public tier, internet-facing. EC2 instances live in the private tier with no direct internet exposure. The subnet design here follows the pattern I covered in the VPC subnet design for EC2 workloads guide. If you are unsure how many AZs or what your CIDR blocks should look like, start there.

Compute tier: Auto Scaling Group in the private subnets. Instances pull from SSM Parameter Store for runtime config, so no secrets baked into AMIs and no environment variables in user data. The CodeDeploy agent runs on each instance and polls for deployment instructions.

CI/CD spine: S3 as the artifact store. CodePipeline orchestrates source to build to deploy. CodeBuild handles the build stage. CodeDeploy runs rolling deployments onto the ASG using deployment groups tied to the ASG directly.

Observability: CloudWatch log groups with explicit retention (more on why that matters in the cost section) and alarms on the key signals. EventBridge routes pipeline state change events downstream. SNS handles notifications.

The non-obvious piece: CodeDeploy needs to know about the ALB. You configure the deployment group with the ALB target group, which tells CodeDeploy to deregister instances from the ALB before installing the new version and re-register after. That is how you get zero-downtime rolling deploys. Skip this wiring and CodeDeploy will happily deploy while the ALB is still routing traffic to instances being updated. You see errors. Not a lot. Just enough to be mysterious.

---

Step-by-Step Implementation

Here is a sanitized CDK snippet showing the ALB-to-ASG wiring, which is the part most examples gloss over:

// ALB → Target Group → ASG wiring
const targetGroup = new elbv2.ApplicationTargetGroup(this, 'AppTargetGroup', {
 vpc,
 port: 8080,
 protocol: elbv2.ApplicationProtocol.HTTP,
 targetType: elbv2.TargetType.INSTANCE,
 healthCheck: {
 path: '/health', // Make sure this exists and returns 200 BEFORE app starts
 healthyThresholdCount: 2,
 unhealthyThresholdCount: 3,
 interval: Duration.seconds(30),
 timeout: Duration.seconds(10),
 },
 deregistrationDelay: Duration.seconds(60), // Give in-flight requests time to drain
});

// ASG registers itself with the target group
asg.attachToApplicationTargetGroup(targetGroup);

// CodeDeploy deployment group knows about the ALB
const deploymentGroup = new codedeploy.ServerDeploymentGroup(this, 'DeploymentGroup', {
 application: codeDeployApp,
 autoScalingGroups: [asg],
 loadBalancer: codedeploy.LoadBalancer.application(targetGroup),
 deploymentConfig: codedeploy.ServerDeploymentConfig.ONE_AT_A_TIME,
 installAgent: true,
});

A few things worth calling out.

The health check path. /health needs to return HTTP 200 before the application is fully started. Some apps take 30 seconds to warm up caches and connection pools. If your health check fires every 10 seconds with a tight timeout during that window, you drain instances on every deploy. Set healthyThresholdCount: 2 and give yourself breathing room on the interval.

deregistrationDelay: 300 seconds by default. For most EC2 apps that is five whole minutes of waiting during every rolling deploy. Drop it to 30 to 60 seconds unless you have genuinely long-lived connections.

ONE_AT_A_TIME vs HALF_AT_A_TIME: For smaller fleets, under about 10 instances, ONE_AT_A_TIME is safer during rollouts. For larger fleets HALF_AT_A_TIME is faster but means half your fleet is unavailable during the install phase. Pick based on fleet size and traffic.

The IAM role construction, the appspec configuration, and the lifecycle hook setup for ASG suspension during deploys are where the real "here is why this breaks" detail lives, and they deserve their own walkthrough.

---

Trade-offs

Deploy speed vs operational simplicity. CodeDeploy lifecycle hooks (BeforeInstall, AfterInstall, ApplicationStart, ValidateService) add 5 to 15 minutes per rolling deploy. At 10 or more deploys a day that overhead compounds. Container swaps on ECS are measured in seconds. EC2 CodeDeploy is measured in minutes. That is not a bug, it is the cost of the model.

S3 artifacts vs container images. Zip-based artifacts are simple and cheap, but you lose layer caching and immutable image semantics. Every deploy re-transfers the full artifact. For small apps, irrelevant. For apps carrying 500MB of dependencies, it adds up.

SSM Parameter Store. Standard tier is free up to 10,000 parameters. Cross into advanced tier and it is $0.05 per parameter per month. The sneakier part: teams hit TooManyUpdates throttling during high-frequency deploys, when the CodeDeploy agent and the app both make SSM calls at once, and it surfaces as deploy failures that look like network issues.

---

Failure Modes

The one I see most: CodeDeploy reports success, ALB shows 502s. This happens when the ALB health check path does not exist or is not returning 200 during the ApplicationStart phase. CodeDeploy considers the instance healthy once the ApplicationStart hook exits successfully. The ALB has its own opinion based on the health check. The two are not synchronized. Build a ValidateService hook that waits until the ALB health check is actually passing before CodeDeploy marks the deploy done.

Rollback with nothing to roll back to. If your S3 artifact bucket lifecycle policy deletes old versions aggressively, a failed deploy has no previous revision to restore. Set a minimum 30-day retention on the artifact bucket. Do not let a lifecycle policy clean up the one thing CodeDeploy needs to recover.

Scale-in during deploy. The ASG decides to terminate an instance mid-deploy because a scale-in event fired. CodeDeploy reports partial success on a shrinking fleet. The fix is to suspend the Terminate scaling process during deployment using a lifecycle hook on the ASG. Most CDK examples skip this entirely, and it is a slow one to diagnose because nothing in the deploy logs points at the ASG.

Silent pipeline failures. Without a CloudWatch alarm on FAILED state transitions from your EventBridge rules, a stuck pipeline can go unnoticed for hours. Create the alarm. CloudWatch alarms for EC2 Auto Scaling fleets covers this pattern in detail.

---

Security and Operational Considerations

IAM is where this architecture is most likely to fail a security audit. CDK's default managed policies for CodeBuild and CodeDeploy are broad. AWSCodeBuildAdminAccess on a CodeBuild role is not least-privilege, it is "I will tighten this later," and later never comes. IAM least privilege for CodeBuild and CodeDeploy walks through the specific permission sets you actually need.

EC2 instance profiles need SSM access for Parameter Store reads, CloudWatch Logs access for the agent, and S3 access to pull deployment artifacts. That is it. Nothing else.

SNS email subscriptions require manual confirmation. Teams automate everything, forget to click the confirmation link, and discover their deploy notification channel is silent on the first real incident. Subscribe an SQS queue or Lambda instead if you want reliable delivery.

---

Cost Reality

Real numbers for a modest fleet (5 EC2 instances, t3.medium, us-east-1). Treat these as estimates, not quotes:

• EC2 (5x t3.medium On-Demand): roughly $150 to $170/month
• ALB: roughly $20/month base plus data processing charges
• NAT Gateway (private subnet egress): $0.045/hour, about $32.85/month per gateway at 730 hours, plus $0.045/GB processed (VPC pricing, checked June 2026)
• CodePipeline: about $1/month per active pipeline
• CodeBuild: first 100 build minutes/month free, then about $0.005/minute on general1.small
• S3 (artifact storage): under $5/month for most teams
• CloudWatch Logs (default never-expire retention): the surprise line, easily $50 to $200/month for a busy fleet

That CloudWatch Logs number is the one that catches people. Verbose app logs on 5 EC2 instances with no retention policy cost $0.50/GB to ingest and $0.03/GB/month to store, forever. Set explicit retention on every log group. 30 days is usually right, 90 days at most unless compliance requires more. Do not accept the CloudWatch default of forever.

For a more detailed breakdown of where CI/CD pipeline costs actually go, the CI/CD pipeline cost breakdown on AWS has the full analysis.

---

What I'd Do Differently

Three things.

1. VPC endpoints from day one. I have set up this pattern twice without VPC endpoints and paid NAT Gateway egress for SSM, S3, and CloudWatch Logs calls, all traffic that never needed to touch the internet. Gateway endpoints for S3 and DynamoDB are free; interface endpoints are about $7.30/month per AZ. At any real volume that beats NAT data processing.

2. Skip the email SNS subscription. Use an SQS queue to Lambda to Slack or PagerDuty instead. Email subscriptions need manual confirmation, deliver unreliably, and nobody reads the AWS notification inbox. Build the automation from day one.

3. Validate the ALB health check before the first real deploy. Sounds obvious. Almost nobody does it. Hit /health manually from an instance in the private subnet, confirm it returns 200 inside your timeout, then deploy. Do not discover the health check is broken mid-deploy.

---

Where to Start

The deploy pipeline that does not page you at 2 AM is mostly a config problem, not a compute problem. Before your next deploy, do three checks: confirm the ALB health check returns 200 during ApplicationStart, set a 30-day floor on the artifact bucket so rollback always has a target, and put a CloudWatch alarm on pipeline FAILED transitions so a stuck deploy cannot hide. Get the lifecycle hooks right, get the health checks aligned, set your log retention. The rest is wiring.