Task & Evaluation Specialist
verita aiNew York, NY
Task & Evaluation Specialist
L4
verita aiNew York, NY8 days ago
Occupations
Market Research Analysts and Marketing SpecialistsSoftware Quality Assurance Analysts and TestersComputer and Information Research ScientistsIndustries
Other Scientific and Technical Consulting ServicesSupermarkets and Other Grocery RetailersCustom Computer Programming ServicesWhat this role does Design realistic, difficult shopping-related tasks (e.g. meal planning, restocking, gifting, travel prep) that stress-test AI models, then build the rubrics used to grade how well a model performs them.
Who we're looking for
Subject-matter expertise in retail, e-commerce, consumer shopping behavior, or a related domain (nutrition, logistics, home/office management, etc.)
Strong analytical and writing skills — able to define precise, low-variance grading criteria
Comfortable working with AI outputs and evaluating them critically and objectively
Prior experience in eval design, QA, tutoring, or curriculum/rubric design is a plus
Core responsibilities:
Author realistic shopping tasks with a clear, concrete end goal
Build a grading rubric/verifier for each task that two independent reviewers would score consistently
Review an AI agent's attempt at the task and grade it against the rubric
Document where the AI succeeded or failed, with clear reasoning
Work style
High attention to detail and consistency
Comfortable with ambiguity — tasks are self-directed with light guidelines
Apply now (opens in a new tab)
Level
MidL4
Location
New York, NY
Occupation
Market Research Analysts and Marketing Specialists
Industry
Other Scientific and Technical Consulting Services
Posted
8 days ago
To get sharper similar jobs, create your profile using the link below.