Exams in nodes is a feature for generation and quorum nodes that lets you filter out low-performing users before they start working on production tasks. Exam submissions can be evaluated not only against exact ground-truth values, but also against instructions or production-style quality criteria.
Why we need it
Exams provide a self-service way to check expert quality before production work starts.
Many tasks cannot be evaluated by simple exact-match checks, especially open-answer, bounding box, and other complex-output projects.
A unified Training → Exam onboarding flow makes it easier to prepare experts and move them into production only after they pass the required quality bar.
How it works
In the relevant node section, you can enable exams for generation or quorum nodes.
Upload an exam dataset, or ask an agent to create one.
Select which output entities should be evaluated in the exam and choose how they should be checked.
Configure thresholds: how many exam tasks are required and how many must be accepted.
Experts can move more seamlessly between onboarding steps: after completing Training, they immediately see a button to continue to the next step.
Exam evaluation options
Exact ground-truth matching
Use this option when expert submissions can be compared directly with reference values from the exam dataset.
Most relevant for classification labels and other structured outputs.
The uploaded dataset must contain the expected ground-truth values.
Instruction-based ground-truth comparison
Use this option when reference answers exist, but cannot be compared to submissions directly.
Most relevant for bounding box tasks and other outputs where exact comparison is not enough.
The LLM QA agent uses a specific instruction to compare expert answers with the ground truth.
This allows the exam to check whether the submitted answer is acceptable according to the task-specific evaluation logic.
Production quality criteria
Use this option when the exam should be checked against the same quality criteria that are used in production.
Most relevant for open-answer tasks and other projects with complex outputs.
You can select which criteria should be used in the exam.
Experts will see a quality criteria panel with the chosen criteria.
Limitations
Agent-generated exam datasets should be reviewed and evaluated by a human.