AnnotationPrep

Outlier AI Factuality, Groundedness & Citation Rubric Test practice test

11 practice questions for this assessment, each with the grading rationale and underlying rule.

Start practising

Binary rubric design & groundedness: practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Practice tests that prepare you for this assessment

Practise by topic

About this assessment

Platform
Outlier AI
Availability
Remote / Global

Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.

Sample scenarios

In Mercor Rubric Academy / auto-grader rubric design, what makes a rubric criterion 'atomic'?

  1. It checks 3 to 5 related facts in a single sentence to save space.
  2. It tests exactly ONE verifiable condition or detail so an automated judge can answer cleanly Yes or No without partial credit.
  3. It uses subjective adjectives like 'well-written' or 'insightful'.
  4. It is written in fewer than 3 words.
Show the answer

B. It tests exactly ONE verifiable condition or detail so an automated judge can answer cleanly Yes or No without partial credit.

Non-atomic ('bundled') criteria like 'Mentions X, Y, and uses a friendly tone' break binary grading when a model nails X and Y but misses the tone. Split every requirement into its own single-check criterion.

Principle: Rubric Design: Atomicity Principle

In prompt-writing tasks where your goal is to 'stump the model' (induce a realistic failure), which prompt is strongest?

  1. 'Tell me about World War II.'
  2. A realistic multi-constraint prompt combining a specific role, a negative constraint, an exact output schema, and a subtle false premise or source-only rule.
  3. 'Write a 10,000-word essay using no vowels.'
  4. 'What is 2 + 2?'
Show the answer

B. A realistic multi-constraint prompt combining a specific role, a negative constraint, an exact output schema, and a subtle false premise or source-only rule.

Silly impossible constraints ('no vowels') are rejected by QA reviewers as unnatural. Strong adversarial prompts look like real user tasks that layer multiple realistic constraints or tricky edge cases.

Principle: Adversarial Prompting: Realistic Failure Modes

Which rubric criterion is properly written for a binary (Yes/No) LLM judge?

  1. 'Does the response give a good, helpful overview of Python decorators?'
  2. 'Does the response explicitly state that a decorator is a callable that takes a function and returns a modified/wrapped function?'
  3. 'Is the code clean and Pythonic?'
  4. 'Does the response explain decorators and also include 2 examples without bugs?'
Show the answer

B. 'Does the response explicitly state that a decorator is a callable that takes a function and returns a modified/wrapped function?'

Option B is specific, objective, and tests a single atomic fact. Options A and C are vague/subjective, and Option D bundles two separate checks (explanation + 2 bug-free examples).

Principle: Rubric Design: Objective vs. Subjective Criteria

Taking this assessment soon?

Tell us what skills or question formats you want more drills on.

Other Rubric Design & Prompting assessments