10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Sample scenarios
In Mercor Rubric Academy / auto-grader rubric design, what makes a rubric criterion 'atomic'?
It checks 3 to 5 related facts in a single sentence to save space.
It tests exactly ONE verifiable condition or detail so an automated judge can answer cleanly Yes or No without partial credit.
It uses subjective adjectives like 'well-written' or 'insightful'.
It is written in fewer than 3 words.
Show the answer
B. It tests exactly ONE verifiable condition or detail so an automated judge can answer cleanly Yes or No without partial credit.
Non-atomic ('bundled') criteria like 'Mentions X, Y, and uses a friendly tone' break binary grading when a model nails X and Y but misses the tone. Split every requirement into its own single-check criterion.
Principle: Rubric Design: Atomicity Principle
Which rubric criterion is properly written for a binary (Yes/No) LLM judge?
'Does the response give a good, helpful overview of Python decorators?'
'Does the response explicitly state that a decorator is a callable that takes a function and returns a modified/wrapped function?'
'Is the code clean and Pythonic?'
'Does the response explain decorators and also include 2 examples without bugs?'
Show the answer
B. 'Does the response explicitly state that a decorator is a callable that takes a function and returns a modified/wrapped function?'
Option B is specific, objective, and tests a single atomic fact. Options A and C are vague/subjective, and Option D bundles two separate checks (explanation + 2 bug-free examples).
Principle: Rubric Design: Objective vs. Subjective Criteria
Why must every rubric item be 'self-contained' (never referencing 'the ideal response' or 'Criterion 1')?
Because automated judges evaluate each criterion independently against the prompt and candidate response alone.
Because JSON does not allow numbers.
Because ideal responses are hidden from humans.
To make the rubric longer.
Show the answer
A. Because automated judges evaluate each criterion independently against the prompt and candidate response alone.
In evaluation pipelines, each criterion is often scored in isolation. Writing 'Same as Criterion 2' or 'Matches the reference answer' fails when the judge only sees that single criterion and the model's output.