Verita AISoftware Engineering

Coding & QC Expert

Code reviewRubric evaluationQuality controlAI benchmarksSoftware engineeringData annotationEvidence-based feedback

About the role

Verita AI is seeking Coding & Rubric QC Experts for a paid pilot evaluating the quality of AI benchmark tasks. Each task includes a question, a proposed gold answer and a rubric of claims describing what a correct answer should contain.

Critically audit existing work rather than create new content. Assess whether rubric claims hold up and whether the reference answer is actually correct.

This is evaluative, judgment-driven work suited to reviewers who enjoy code review, quality assurance and auditing other people's work. Strong performers may be considered for continued or expanded work after the approximately one-week pilot.

Scope of Work

  • Review every rubric claim for clarity, specificity and verifiability, identifying vague, redundant or unusable criteria.
  • Mark claims as pass, fail or flagged for follow-up, with a short written reason.
  • Identify requirements that do not connect to the question or proposed answer.
  • Check the correctness of the proposed gold answer before treating it as ground truth.
  • Give an overall accept or reject decision on each task, supported by specific findings.

What you’ll bring

  • Be based in the United States or Canada under the combined role-specific North America requirement and Verita AI partner policy.
  • Completed bachelor's degree in Computer Science, or an equivalent completed degree.
  • At least 3 months of human-data, annotation or AI-training experience of any kind.
  • At least 1 year of software engineering experience outside human-data work. Any language or stack is acceptable; no specific stack is required.

Benefits

  • Paid pilot
  • Flexible scheduling
  • Remote work within eligible countries

Schedule and availability

Approximately 10+ hours minimum during a paid pilot of about one week, with flexible scheduling. Average task time is expected to be around 20 minutes.

Contract & Payment Terms

  • Independent contractor engagement paid per completed, accepted task, not hourly.
  • Payment is per completed, accepted task, not for total hours worked.
  • Paid pilot of approximately one week, with potential for continued or expanded work for strong performers.