Skip to content
Back to case studies
SanitizedEvent-Driven2021

Real-Time Processing

15-minute audit lag vs compliance → <5s pipeline, 1M+ req/min, ES search <100ms at billions of rows.

Public company context
Role
Full Stack Cloud Developer
Context
Media / Analytics
Duration
10 months
Team
6 engineers + 1 cloud developer (me)

Tech stack

  • ECS
  • EKS
  • Fargate
  • Python
  • Lambda
  • S3
  • ElasticSearch
  • Redis
  • Celery
Executive summary

Business problem and production context.

Business problem

Enterprise clients needed real-time activity tracking and audit logging across distributed microservices. Existing batch processing had 15-minute delays, failing compliance requirements for near-real-time visibility.

Architecture decision

Operated cloud-native microservices on ECS, EKS, and Fargate handling 1M+ requests/minute. Built audit logging with Lambda, S3, and ElasticSearch for real-time activity tracking. Implemented Celery task queues with Redis and SQS for background job processing.

awsAWS Account / streamingRequestResponseEVENT INGESTSTREAM PROCESSINGSEARCH + STORAGE12345678API GatewayHTTPS ingestKinesis1M+ req/minFirehosebuffer + batchLambdavalidatorKinesis Analyticswindowed aggLambdaenrichmentOpenSearch<100ms searchS3raw + parquetAthenaad-hoc auditCROSS-CUTTING: Shard scaling auto · DLQ + replay · PII redaction at ingest · Index lifecycle policy
What Rahul owned

The useful proof is the decision surface, not only the result number.

These are the parts of the work that show production judgment: ownership, constraints, rollback, cost, and observability.

Owned scope

  • Event-processing architecture for high-throughput, low-latency data paths
  • Backpressure, queue, and storage decisions around real-time query behavior
  • Operational signals for lag, failures, and customer-impacting delay

Evidence artifacts

Realtime pipelineSub-100ms query targetBackpressure planningReplay-safe operations
Constraints

What would fail first?

Every project has constraints. The useful work is naming them early enough that rollback, cost, and ownership are designed before an incident.

Latency

The system needed fast query responses while ingesting and processing continuous event streams.

Throughput

Burst traffic required explicit handling of queue depth, consumers, and storage limits.

Debuggability

Operators needed to know whether delay came from ingestion, processing, or query serving.

Decision table

Trade-offs accepted on purpose.

Option
Decision
Reason
ElasticSearch vs CloudWatch Logs Insights
ElasticSearch chosen for sub-second full-text search across billions of log entries
ElasticSearch vs CloudWatch Logs Insights: ElasticSearch chosen for sub-second full-text search across billions of log entries.
ECS vs EKS
Used both
ECS vs EKS: Used both. ECS for simpler services, EKS for teams needing Kubernetes-native tooling.
Redis vs ElastiCache
Self-managed Redis on ECS for finer control over eviction policies and cluster topology
Redis vs ElastiCache: Self-managed Redis on ECS for finer control over eviction policies and cluster topology.
Risk table

Production risks and how they were controlled.

Risk
Mitigation
Owner
Lag looked like data loss
Tracked consumer lag and separated delayed processing from failed processing.
Data platform owner
Hot partitions or query bottlenecks
Validated access patterns and capacity limits against traffic shape.
Backend/platform engineer
Replay created duplicate side effects
Designed idempotency and replay boundaries before incident recovery was needed.
Service owner
Implementation approach

Small steps, visible changes, fewer surprises.

  1. 01

    Instrumented existing batch pipeline to identify bottlenecks. We discovered 15-minute processing delays came from hourly S3-to-Redshift ETL jobs.

  2. 02

    Replaced batch ETL with streaming architecture: Lambda triggers on S3 events, pushing to ElasticSearch via Kinesis Data Firehose.

  3. 03

    Deployed audit microservices on ECS Fargate with auto-scaling based on SQS queue depth, scaling from 3 to 50 tasks during peak hours.

  4. 04

    Built real-time ElasticSearch dashboards for compliance teams: sub-second search across 2B+ audit log entries.

  5. 05

    Implemented Celery task queues with Redis for background analytics jobs, keeping the hot path (real-time logging) isolated from cold path (report generation).

Result

Before and after.

Log Processing Delay
15 minutes
<5 seconds
Search Latency
8-12s (Redshift)
<100ms (ElasticSearch)
Throughput
~10K req/min
1M+ req/min
Compliance Audit Time
2 weeks
2 hours

Rollback path

Processing changes needed replay-safe deployment and clear boundaries around idempotent writes.

Cost considerations

Cost control depended on matching retention, throughput, and query capacity to actual access patterns.

Observability notes

Useful signals were ingestion rate, processing lag, failed messages, query latency, and replay volume.

Related services

Forward Deployed AI EngineeringAWS Production Architecture ReviewGenAI / RAG Production Readiness

Have a similar production problem?

Send the current architecture, bill concern, deployment pain, or GenAI reliability issue. I will help you find the first real bottleneck.

Rahul LadumorRequest the reliability auditContact form