In Mercor Rubric Academy / auto-grader rubric design, what makes a rubric criterion 'atomic'?
It checks 3 to 5 related facts in a single sentence to save space.
It tests exactly ONE verifiable condition or detail so an automated judge can answer cleanly Yes or No without partial credit.
It uses subjective adjectives like 'well-written' or 'insightful'.
It is written in fewer than 3 words.
Show the answer
B. It tests exactly ONE verifiable condition or detail so an automated judge can answer cleanly Yes or No without partial credit.
Non-atomic ('bundled') criteria like 'Mentions X, Y, and uses a friendly tone' break binary grading when a model nails X and Y but misses the tone. Split every requirement into its own single-check criterion.
Principle: Rubric Design: Atomicity Principle
In prompt-writing tasks where your goal is to 'stump the model' (induce a realistic failure), which prompt is strongest?
'Tell me about World War II.'
A realistic multi-constraint prompt combining a specific role, a negative constraint, an exact output schema, and a subtle false premise or source-only rule.
'Write a 10,000-word essay using no vowels.'
'What is 2 + 2?'
Show the answer
B. A realistic multi-constraint prompt combining a specific role, a negative constraint, an exact output schema, and a subtle false premise or source-only rule.
Silly impossible constraints ('no vowels') are rejected by QA reviewers as unnatural. Strong adversarial prompts look like real user tasks that layer multiple realistic constraints or tricky edge cases.
Which rubric criterion is properly written for a binary (Yes/No) LLM judge?
'Does the response give a good, helpful overview of Python decorators?'
'Does the response explicitly state that a decorator is a callable that takes a function and returns a modified/wrapped function?'
'Is the code clean and Pythonic?'
'Does the response explain decorators and also include 2 examples without bugs?'
Show the answer
B. 'Does the response explicitly state that a decorator is a callable that takes a function and returns a modified/wrapped function?'
Option B is specific, objective, and tests a single atomic fact. Options A and C are vague/subjective, and Option D bundles two separate checks (explanation + 2 bug-free examples).
Principle: Rubric Design: Objective vs. Subjective Criteria
A prompt says: 'Using ONLY the attached lease excerpt, state the late fee.' The attached excerpt says nothing about a late fee. Model A says '$50 (standard)'. Model B says 'The provided excerpt does not mention a late fee.' Which model passes?
Model A, because $50 is common.
Model B, because strict groundedness requires acknowledging missing information rather than guessing from outside knowledge.
Both fail.
Model A, because it is more decisive.
Show the answer
B. Model B, because strict groundedness requires acknowledging missing information rather than guessing from outside knowledge.
When a prompt restricts the model to provided context, inventing an unstated fact—even a plausible one—is a groundedness failure.
Principle: Groundedness: Abstaining on Missing Context
Why must every rubric item be 'self-contained' (never referencing 'the ideal response' or 'Criterion 1')?
Because automated judges evaluate each criterion independently against the prompt and candidate response alone.
Because JSON does not allow numbers.
Because ideal responses are hidden from humans.
To make the rubric longer.
Show the answer
A. Because automated judges evaluate each criterion independently against the prompt and candidate response alone.
In evaluation pipelines, each criterion is often scored in isolation. Writing 'Same as Criterion 2' or 'Matches the reference answer' fails when the judge only sees that single criterion and the model's output.