Data access · Asynchronous workloads
Keeping expensive work out of the request path.
Reducing unnecessary data work and separating background jobs from latency-sensitive APIs.
- My contribution
- Platform architecture lead
- Context
- Professional work · 2025–2026
- My responsibility
- Reviewed data access and execution patterns across platform workflows.
- The key decision
- Use lighter reads and bounded concurrency; make background processing an explicit boundary.
- What changed
- Reusable practices for data reads, asynchronous jobs and evidence-based cache sizing.
The problem & my responsibilities
As workflows evolved, routine reads accumulated framework overhead, independent operations ran serially, and report generation competed with user requests. Infrastructure growth alone could not fix that workload shape.
I reviewed data access and execution patterns across workflows, identified where batching or concurrency was safe, and made asynchronous processing a deliberate platform boundary.
Constraints that shaped the work
- Existing APIs and correctness expectations
- Independent versus dependent database operations
- Memory and database connection limits
- Background reports sharing application capacity
How the system fits together
- 01RequestValidate input · identify required data
- 02ReadProjection · lean results where suitable
- 03SeparateInteractive response / queued work
- 04OperateWorker capacity · failures · cache demand
The consequential decisions
| Decision | Choice | Alternative | Trade-off |
|---|---|---|---|
| Read results | Mongoose lean reads where suitable | Hydrated documents for every read | Avoid document machinery when getters, setters and document methods are unnecessary. Keep hydration when behavior depends on it. |
| Independent I/O | Batching or controlled concurrency | Serial round-trips | Reduce waiting between independent operations. Unbounded Promise.all can overload the connection pool; concurrency is a capacity decision. |
| Report generation | Asynchronous workers | Work inside API handlers | Keep expensive generation away from interactive requests. Job state and failure handling must remain visible to callers. |
| Cache capacity | Size from observed demand | Keep the inherited instance size | Use memory and traffic evidence over a representative window. Include burst behavior, eviction and recovery headroom. |
Where the work got difficult
Avoid universal performance rules
A lean read is useful only when the caller does not require document behavior. I treated query shape and correctness as part of the same review.
Expose hidden contention
Independent work can run concurrently, but it still competes for finite resources. Pool size, worker capacity and request demand belong in the decision.
Operating considerations
Reliability
Background workers need bounded retries and duplicate-safe side effects. Moving code into a queue does not by itself make the operation reliable.
Data & security
Read projections should select only what the caller needs. Lean results do not replace authorization checks or change which data may be returned.
Performance & cost
Measure CPU, pool waiting, queue delay and cache behavior under a representative workload. A faster isolated query does not establish whole-system improvement.
Outcomes & lessons
The work established reusable practices for lighter reads, intentional concurrency, background processing and evidence-based cache sizing.
Performance defaults should make the safe path easier without hiding the conditions under which that default stops being correct.