DevOps Transformation
SSH deploys + rollback terror → <10 min pipelines, <30s rollback, deploy-related incidents near zero.
- Role
- Lead Platform Engineer
- Context
- SaaS / Platform
- Duration
- 4 months
- Team
- 2 platform engineers (me + 1) supporting 25 developers
Tech stack
- GitHub Actions
- Terraform
- AWS ECS
- CodeDeploy
- Docker
- ECR
- S3
- DynamoDB
- CloudWatch
- Slack Webhooks
Business problem and production context.
Business problem
Engineering team of 25+ developers was deploying manually via SSH. Release cycles took 2-3 days with frequent rollbacks. No automated testing. Bugs were caught in production. Infrastructure was provisioned via ClickOps in the AWS console with no version control or reproducibility.
Architecture decision
Designed and implemented a complete CI/CD pipeline using GitHub Actions with reusable composite workflows. Migrated all infrastructure to Terraform with remote state in S3 + DynamoDB locking. Implemented blue-green deployments on ECS with automatic rollback on health check failures. Set up multi-stage environments (dev/staging/prod) with promotion gates. Added automated testing (unit, integration, E2E) with 95%+ coverage requirements enforced in CI.
The useful proof is the decision surface, not only the result number.
These are the parts of the work that show production judgment: ownership, constraints, rollback, cost, and observability.
Owned scope
- CI/CD, deployment safety, and rollback improvements for teams moving beyond manual releases
- Release health signals and ownership boundaries for production changes
- IaC and automation paths that reduced risky human coordination
Evidence artifacts
What would fail first?
Every project has constraints. The useful work is naming them early enough that rollback, cost, and ownership are designed before an incident.
Manual releases
Release confidence was limited by handoffs, manual checks, and unclear rollback.
Service ownership
Deployment automation needed owners who understood how to respond when a change failed.
Environment drift
CI/CD reliability depended on consistent environments and predictable configuration.
Trade-offs accepted on purpose.
Production risks and how they were controlled.
Small steps, visible changes, fewer surprises.
- 01
Documented existing deployment process: SSH into bastion → pull latest code → restart services → manually verify. Average deploy: 2-3 hours with 1 in 5 requiring rollback.
- 02
Migrated all 12 repositories to GitHub Actions with reusable composite workflows: test → build → push to ECR → deploy to ECS with CodeDeploy blue-green.
- 03
Converted all infrastructure from ClickOps to Terraform with remote state in S3 + DynamoDB locking. Ran `terraform import` on 180+ existing resources over 3 weeks.
- 04
Built multi-stage promotion pipeline: feature branch → dev (auto-deploy) → staging (auto-deploy + E2E tests) → prod (manual approval + blue-green + Slack notification).
- 05
Added test coverage enforcement in CI. PRs below 95% coverage automatically blocked. Wrote 400+ integration tests as bootstrap to get the team past the initial hurdle.
Before and after.
Rollback path
Rollback was treated as part of architecture: deployment strategy, state changes, owners, and verification.
Cost considerations
The main cost benefit was lower incident/release drag, plus cleaner non-production environment usage.
Observability notes
Signals included deploy frequency, failure rate, rollback time, environment health, and customer-impact alarms.
Related services
Have a similar production problem?
Send the current architecture, bill concern, deployment pain, or GenAI reliability issue. I will help you find the first real bottleneck.
Book a production review