Speech assessment
Turning a spoken attempt into useful feedback.
Connecting audio preparation, transcription and word alignment so reading feedback stays attached to the learner’s attempt.
- My contribution
- Backend & speech-pipeline engineering
- Context
- Speech assessment
- My responsibility
- Worked on the backend flow from recorded speech to stored learner feedback.
- The key decision
- Align the transcript with the expected passage before calculating word-level feedback.
- What changed
- A structured assessment record with expected words, matched transcript words and per-word feedback.
The problem & my responsibilities
Learners skip words, insert words and restart phrases while reading aloud. Comparing a transcript with the expected passage by position can shift every subsequent match. The backend also has to handle different recording formats and preserve the result against the correct learner and activity.
My work connects audio ingestion, transcription and assessment output in a learning platform. The flow prepares recordings for speech recognition, aligns the resulting words with the expected passage, applies scoring rules and stores the analysis with learner context.
Constraints that shaped the work
- Different recording formats and microphone quality
- Missing or extra words in spoken attempts
- Multiple assessment paths with different response formats
- Transcription failures and variable processing time
How the system fits together
- 01PrepareValidate the recording and convert it to a consistent audio format
- 02TranscribeRun speech recognition and capture the resulting transcript
- 03AlignMatch expected and spoken words with gaps for omissions
- 04ReportCalculate word feedback and store the learner’s analysis
The consequential decisions
| Decision | Choice | Alternative | Trade-off |
|---|---|---|---|
| Audio preparation | Normalise the recording before recognition | Pass every source format through unchanged | A consistent PCM audio input makes the speech-recognition boundary explicit. Conversion is a separate processing stage with its own runtime and failure modes. |
| Word alignment | Sequence alignment with insertion and omission handling | Compare words at matching positions | Skipping one word should not shift the rest of the passage’s feedback. Alignment retains expected-word positions while accounting for missing and extra transcript words. |
| Assessment output | Map results into a shared word-feedback structure | Expose each engine’s raw response to the client | Expected words, transcribed words, scores and match information can be handled consistently by storage and presentation code. |
Where the work got difficult
Separate recognition from scoring
A transcription mismatch may come from the recording, recognition or the spoken attempt. Keeping alignment and scoring as separate steps makes their behaviour easier to inspect and adjust.
Keep a failed attempt understandable
The custom path checks transcription readiness and returns an explicit failure when recognition cannot complete. Analysis needs learner and activity context so support can identify the affected attempt.
Operating considerations
Latency
Upload, audio conversion, recognition, alignment and persistence contribute different delays. Timing those stages separately is more useful than treating the entire endpoint as a model call.
Feedback quality
Word similarity and phonetic matching support reading feedback. Assessment quality also depends on the expected passage, recording conditions and the behaviour of the selected recognition engine.
Data handling
Recordings, transcripts and learner identifiers need consistent access and retention rules across analysis storage and downstream reporting.
Outcomes & lessons
The assessment flow connects recorded speech with expected-word alignment, structured feedback and stored learner analysis. Its stages provide concrete places to investigate recognition errors, score behaviour and processing delays.
Useful speech feedback depends on preserving the relationship between what was expected, what was heard and how the result was calculated.