Voice AI · Stateful orchestration
A voice tutor with an explicit learning journey.
Connecting real-time speech with curriculum flow, ordered playback and recoverable learner state.
- My contribution
- Architecture & hands-on engineering
- Context
- Professional work · 2025–present
- My responsibility
- Designed the voice pipeline and the boundaries between teaching, assessment and conversation.
- The key decision
- Keep lesson progression in explicit learner state, with bounded agents for each learning activity.
- What changed
- Streaming speech, learning agents and recovery coordinated within one learning flow.
The problem & my responsibilities
An AI tutor needs to follow a lesson while responding to a child who may switch between Hindi, English and Hinglish, interrupt a video, or lose connectivity. A fluent answer alone does not make the learning experience coherent.
I designed the voice pipeline and the boundaries between teaching, assessment, doubt clearing and conversation. My focus was keeping the learning journey explicit while making speech, media and session state work together.
Constraints that shaped the work
- Noisy homes and variable microphones
- Network interruptions during lessons and media
- Ordered speech while model output is still streaming
- Curriculum progression that must survive agent handoffs
How the system fits together
- 01ListenLiveKit · noise handling · speech recognition
- 02RouteLearner state · teaching · assessment · doubts
- 03RespondModel stream · sentence chunks · FIFO audio
- 04RecoverRedis context · PostgreSQL progress · traces
The consequential decisions
| Decision | Choice | Alternative | Trade-off |
|---|---|---|---|
| Voice transport | LiveKit WebRTC | Custom WebSocket transport | Audio, media and data events share a session model. The trade-off is operating a real-time connection lifecycle, not just an HTTP request. |
| Speech delivery | Sentence chunking and an ordered queue | Wait for the complete answer | Start synthesis while generation continues. Queue ordering avoids interleaved speech; interruption still needs explicit cancellation. |
| Teaching flow | Bounded agents and explicit learner state | One open-ended prompt | Separate teaching, assessment and doubt handling. More transitions must be tested, but lesson progression is easier to inspect. |
| Session persistence | Redis context with periodic database persistence | Persist every update synchronously | Reduce writes on the conversational path. Periodic persistence leaves a recovery window; it cannot guarantee that every recent update survives a crash. |
| Operating evidence | Langfuse and OpenTelemetry | Application logs alone | Connect model calls with the voice pipeline. Traces are useful only when timing boundaries, retention and access are defined. |
Where the work got difficult
Keep the lesson in control
Teaching, assessment and doubt clearing have distinct responsibilities. I made learner progression explicit so a conversational handoff did not become the source of truth for what should happen next.
Protect the conversational turn
Noise filtering and voice activity detection sit before transcription. Their timing changes when a turn is considered complete; model and synthesis latency must be interpreted separately from that delay.
Coordinate interruptions
A question during a video affects audio, player state and the lesson cursor. Recovery requires those states to agree before the experience can continue.
Operating considerations
Latency & reliability
Measure turn detection, transcription, generation, synthesis and client playback separately. A first-audio measurement after turn finalization is not end-to-end conversational latency.
Retrieval & privacy
Curriculum context should be bounded by the active lesson. Voice transcripts and learner progress require deliberate access and retention rules.
Cost & observability
Token usage and synthesis work need session-level attribution. Trace collection also has a storage and privacy cost; capture what is needed to investigate a turn.
Outcomes & lessons
The resulting architecture brings streaming speech, specialized learning agents, session context and operating traces into one learning flow. The central engineering contribution is explicit coordination between components that otherwise fail independently.
A useful voice product needs both responsive speech and a dependable state model. Improving one does not automatically improve the other.