Skip to main content

Task & Evaluation Specialist

L4

verita aiNew York, NY8 days ago
What this role does Design realistic, difficult shopping-related tasks (e.g. meal planning, restocking, gifting, travel prep) that stress-test AI models, then build the rubrics used to grade how well a model performs them. Who we're looking for Subject-matter expertise in retail, e-commerce, consumer shopping behavior, or a related domain (nutrition, logistics, home/office management, etc.) Strong analytical and writing skills — able to define precise, low-variance grading criteria Comfortable working with AI outputs and evaluating them critically and objectively Prior experience in eval design, QA, tutoring, or curriculum/rubric design is a plus
Core responsibilities: Author realistic shopping tasks with a clear, concrete end goal Build a grading rubric/verifier for each task that two independent reviewers would score consistently Review an AI agent's attempt at the task and grade it against the rubric Document where the AI succeeded or failed, with clear reasoning Work style High attention to detail and consistency Comfortable with ambiguity — tasks are self-directed with light guidelines
Apply now (opens in a new tab)
Apply now (opens in a new tab)

Level

MidL4

Location

New York, NY

Occupation

Market Research Analysts and Marketing Specialists

Industry

Other Scientific and Technical Consulting Services

Posted

8 days ago

To get sharper similar jobs, create your profile using the link below.

Create profile