Backend reliability · AWS
Finding the bottleneck before adding servers.
Tracing peak-hour failures through Redis lookups, request-path I/O and misleading capacity signals.
- My contribution
- Platform architecture & performance lead
- Context
- Professional work · 2026
- My responsibility
- Led the investigation from production request behavior to infrastructure signals.
- The key decision
- Replace repeated cache scans and move expensive processing into background workers.
- What changed
- Cheaper request lookups, separated heavy processing and a repeatable capacity investigation process.
The problem & my responsibilities
A production application repeatedly slowed during peak demand. Adding capacity was an obvious response, but it would not explain why requests were doing expensive cache scans and file work in the first place.
I led the investigation from production request behavior to infrastructure signals, then focused the changes on the critical path: lookup shape, heavy I/O and capacity decisions. Representative user journeys were part of the validation approach.
Constraints that shaped the work
- A live service with recurring peak-hour incidents
- Cache work multiplied by concurrent requests
- File processing competing with interactive traffic
- Capacity decisions under cost pressure
How the system fits together
- 01ObserveRequest timings · errors · CloudWatch
- 02IsolateRedis scans · file I/O · hot instances
- 03ChangeCached lookups · asynchronous jobs
- 04ValidateAuthenticated journeys · concurrent load
The consequential decisions
| Decision | Choice | Alternative | Trade-off |
|---|---|---|---|
| Cache lookup | In-process lookup with a five-minute TTL | Scan Redis for each request | Avoid repeated network scans. Local copies introduce staleness and per-process consistency concerns; the TTL must fit the data. |
| Capacity signal | Requests per ALB target | Average CPU alone | Fleet averages can conceal hot instances. Request count is useful context, but request cost and latency still need observation. |
| Expensive I/O | RabbitMQ workers | Generate reports in the request path | Separate heavy work from interactive capacity. Queue delay, retries and job visibility become part of the user experience. |
| Validation | Authenticated user-flow load script | Isolated endpoint benchmark | Exercise the sequence the frontend actually uses. Synthetic load still needs comparison with production traffic shape. |
Where the work got difficult
Separate symptoms from causes
High CPU was a symptom shared by several failure modes. I correlated request behavior, Redis usage and file operations before choosing where to intervene.
Make recovery testable
A fast request in isolation is weak evidence. The validation approach followed authenticated workflows under concurrency and watched failure rates as well as response time.
Operating considerations
Reliability
Moving work to a queue needs retry limits and visibility into failed jobs.
Security
Load tests that follow authenticated journeys need test identities and controlled credentials. Use representative test data for repeatable load tests.
Cost
Right-sizing should follow representative demand and headroom requirements. Lower spend alone is not evidence of an adequate capacity margin.
Outcomes & lessons
The work replaced scan-heavy request behavior with cheaper lookups, separated heavy processing and established a repeatable path for capacity investigation.
Capacity planning starts with what each request asks the system to do. More instances can hide an expensive request shape without correcting it.