AnnotationPrep

Audio, Speech & Video Temporal Grounding practice questions

20 questions from the Multimodal vision, audio & video eval practice test.

Start practising

Audio, Speech & Video Temporal Grounding practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Sample scenarios

PROMPT

Transcribe the attached 12-second insurance-call audio clip verbatim.

OBSERVATION

At 00:07, the policyholder clearly says, "I can't authorize the deductible change until Friday," with a brief glottal stop on can't.

MODEL A

I can authorize the deductible change until Friday.

MODEL B

I can't authorize the deductible change until Friday.

Why does Model A receive a major ASR severity deduction despite having only a single-word error?

  1. Low WER tolerance: verbatim ASR tasks automatically fail any transcript whenever overall word error rate exceeds 2%.
  2. Dropped negation: turning can't into can reverses the speaker's legal authorization and meaning.
  3. Contraction expansion rule: Model A violated style guidelines by failing to expand spoken contractions into cannot.
  4. Timestamp omission error: Model A did not bracket the glottal stop at 00:07 with an inline acoustic hesitation tag.
Show the answer

B. Dropped negation: turning can't into can reverses the speaker's legal authorization and meaning.

In ASR RLHF evaluation, not all single-word errors carry equal weight. Dropping a negation (can't → can) flips the polarity of a binding authorization statement, making it a critical semantic failure even though the raw word-error count is only one word.

Principle: ASR Evaluation: Negation Drops & Polarity-Flipping Errors

PROMPT

Transcribe the attached radiology dictation clip accurately.

OBSERVATION

The physician dictates: "Small polyp noted in the terminal ileum; no fracture of the right ilium." Both words sound nearly identical phonetically (/ˈɪliəm/), but ileum is the small intestine and ilium is the pelvic bone.

AI RESPONSE

Small polyp noted in the terminal ilium; no fracture of the right ileum.

What ASR failure must an evaluator flag here?

  1. Diarization mismatch: the model merged the radiologist's clinical impression with a background nurse's note.
  2. Punctuation normalization error: clinical dictation guidelines forbid semicolons between two anatomical clauses.
  3. Acoustic clipping artifact: rapid dictation speed truncated the final syllable of both Latin medical terms.
  4. Homophone swap: it inverted ileum (intestine) and ilium (pelvic bone), corrupting clinical sense.
Show the answer

D. Homophone swap: it inverted ileum (intestine) and ilium (pelvic bone), corrupting clinical sense.

Homophones sound identical acoustically, so accurate ASR depends on domain and syntactic context. Swapping ileum (where a polyp occurs) and ilium (the pelvic bone that can fracture) produces a major clinical accuracy error that human evaluators must catch and penalize.

Principle: ASR Evaluation: Context-Dependent Homophones & Domain Terms

PROMPT

Produce a speaker-diarized transcript and summarize who requested the refund in this two-person support call.

OBSERVATION

Speaker 1 (Customer) asks at 00:18, "Please cancel my annual plan and refund the charge." Speaker 2 (Support Agent) replies at 00:24, "I've initiated the cancellation, and the refund will post in five business days."

AI RESPONSE

[Speaker 2]: Please cancel my annual plan and refund the charge. [Speaker 1]: I've initiated the cancellation... Summary: The support agent requested a refund from the customer.

What exact audio evaluation defect occurred?

  1. Speaker diarization swap: inverted speaker labels reversed who requested versus processed the refund.
  2. Acoustic silence hallucination: the model inserted phantom words during the 6-second pause between 00:18 and 00:24.
  3. Prosody misattribution: the model mistook polite agent intonation for an urgent customer escalation request.
  4. Verbatim filler omission: the model stripped vocal pauses from Speaker 1, shifting the turn boundary by six seconds.
Show the answer

A. Speaker diarization swap: inverted speaker labels reversed who requested versus processed the refund.

Even when the transcribed words are 100% accurate, swapping speaker identities (Speaker 1 vs Speaker 2) corrupts who said what and propagates directly into downstream summarization errors. Diarization attribution accuracy is a core axis in multi-speaker audio RLHF.

Principle: ASR Evaluation: Speaker Diarization & Role Attribution

Other topics