MercorOperations

AI Evaluation Rubric Reviewer

About the role

We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:

Scope of Work

  • Was the information available?
  • Does the rubric accurately grade what the prompt asked for?
  • Would a valid answer have been marked wrong?
  • Is the rubric structure correct per the guidelines?
  • Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.

What you’ll bring

  • Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
  • Can read a long agent transcript and find where it went wrong.
  • Precise writing, and you check the data instead of trusting the reference answer.

Benefits

Not provided in the source listing.

Schedule and availability

Hourly contract; fully remote and completed on your own schedule (independent contractor).

Contract & Payment Terms

  • We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.
  • You will be engaged as an independent contractor.
  • This is a fully remote role that can be completed on your own schedule.
  • Projects can be extended, shortened, or concluded early depending on needs and performance.
  • Your work at Mercor will not involve access to confidential or proprietary information from any employer, client, or institution.
  • Payments are weekly on Stripe or Wise based on services rendered.
  • Please note: We are unable to support H1-B or STEM OPT candidates at this time.