Multimodal vision, audio & video eval: practice drill
10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Timed mock assessment
20 scenarios in 24 minutes, drawn from every topic. No answers shown until the end, then a full score breakdown, your weakest topics and a review of what you missed.
Prices are in US dollars, plus tax where it applies. Stripe, our payment processor and the seller of record, adds tax at checkout.
All sales are final. If something is really wrong, such as a duplicate charge or a purchase that does not unlock what you bought, write to support@annotationprep.com and we will put it right.
One-time payment. Nothing renews.
You have an assessment ticket for this practice test. Start the mock assessment to use it.
Practice tests that prepare you for this assessment
Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.
Sample scenarios
PROMPT
Transcribe the attached 12-second insurance-call audio clip verbatim.
OBSERVATION
At 00:07, the policyholder clearly says, "I can't authorize the deductible change until Friday," with a brief glottal stop on can't.
MODEL A
I can authorize the deductible change until Friday.
MODEL B
I can't authorize the deductible change until Friday.
Why does Model A receive a major ASR severity deduction despite having only a single-word error?
Low WER tolerance: verbatim ASR tasks automatically fail any transcript whenever overall word error rate exceeds 2%.
Dropped negation: turning can't into can reverses the speaker's legal authorization and meaning.
Contraction expansion rule: Model A violated style guidelines by failing to expand spoken contractions into cannot.
Timestamp omission error: Model A did not bracket the glottal stop at 00:07 with an inline acoustic hesitation tag.
Show the answer
B. Dropped negation: turning can't into can reverses the speaker's legal authorization and meaning.
In ASR RLHF evaluation, not all single-word errors carry equal weight. Dropping a negation (can't → can) flips the polarity of a binding authorization statement, making it a critical semantic failure even though the raw word-error count is only one word.
Principle: ASR Evaluation: Negation Drops & Polarity-Flipping Errors
PROMPT
Transcribe the attached radiology dictation clip accurately.
OBSERVATION
The physician dictates: "Small polyp noted in the terminal ileum; no fracture of the right ilium." Both words sound nearly identical phonetically (/ˈɪliəm/), but ileum is the small intestine and ilium is the pelvic bone.
AI RESPONSE
Small polyp noted in the terminal ilium; no fracture of the right ileum.
What ASR failure must an evaluator flag here?
Diarization mismatch: the model merged the radiologist's clinical impression with a background nurse's note.
Punctuation normalization error: clinical dictation guidelines forbid semicolons between two anatomical clauses.
Acoustic clipping artifact: rapid dictation speed truncated the final syllable of both Latin medical terms.
Homophone swap: it inverted ileum (intestine) and ilium (pelvic bone), corrupting clinical sense.
Show the answer
D. Homophone swap: it inverted ileum (intestine) and ilium (pelvic bone), corrupting clinical sense.
Homophones sound identical acoustically, so accurate ASR depends on domain and syntactic context. Swapping ileum (where a polyp occurs) and ilium (the pelvic bone that can fracture) produces a major clinical accuracy error that human evaluators must catch and penalize.
Principle: ASR Evaluation: Context-Dependent Homophones & Domain Terms
PROMPT
Produce a speaker-diarized transcript and summarize who requested the refund in this two-person support call.
OBSERVATION
Speaker 1 (Customer) asks at 00:18, "Please cancel my annual plan and refund the charge." Speaker 2 (Support Agent) replies at 00:24, "I've initiated the cancellation, and the refund will post in five business days."
AI RESPONSE
[Speaker 2]: Please cancel my annual plan and refund the charge. [Speaker 1]: I've initiated the cancellation... Summary: The support agent requested a refund from the customer.
What exact audio evaluation defect occurred?
Speaker diarization swap: inverted speaker labels reversed who requested versus processed the refund.
Acoustic silence hallucination: the model inserted phantom words during the 6-second pause between 00:18 and 00:24.
Prosody misattribution: the model mistook polite agent intonation for an urgent customer escalation request.
Verbatim filler omission: the model stripped vocal pauses from Speaker 1, shifting the turn boundary by six seconds.
Show the answer
A. Speaker diarization swap: inverted speaker labels reversed who requested versus processed the refund.
Even when the transcribed words are 100% accurate, swapping speaker identities (Speaker 1 vs Speaker 2) corrupts who said what and propagates directly into downstream summarization errors. Diarization attribution accuracy is a core axis in multi-speaker audio RLHF.
Principle: ASR Evaluation: Speaker Diarization & Role Attribution
Taking this assessment soon?
Tell us what skills or question formats you want more drills on.