AI Research Engineer
virtual vocationsDenver, CO
AI Research Engineer
L6
virtual vocationsDenver, CO4 days ago
Occupations
Computer and Information Research ScientistsComputer Systems Engineers/ArchitectsSoftware DevelopersIndustries
Research and Development in the Physical, Engineering, and Life SciencesAll Other Professional, Scientific, and Technical ServicesComputer Systems Design ServicesTo support a growing AI research initiative, the full-time AI Research Engineer will focus on designing and validating evaluation rubrics, developing long-term benchmarks with real user data, and publishing research findings, all while working remotely.
Key responsibilities:
Design and test evaluation rubrics for validity and reliability, calibrating AI judges against human judgment
Create and maintain long-horizon benchmarks involving real users and tasks, ensuring ongoing accuracy and relevance
Conduct research on human feedback's impact on AI models and develop methods for practical application in live programs
Required qualifications:
Published research or strong preprints in areas such as evaluation metrics, human-AI interaction, or measurement
A PhD, a PhD in progress, or an MS with relevant research internships, or equivalent experience in machine learning research
Strong applied statistics skills, including inter-rater reliability and confidence intervals
Experience with LLM-as-judge and rubric-based evaluation methodologies
Proficiency in Python for building research prototypes and evaluation pipelines
Apply now (opens in a new tab)
Level
LeadL6
Location
Denver, CO
Occupation
Computer and Information Research Scientists
Industry
Research and Development in the Physical, Engineering, and Life Sciences
Posted
4 days ago
To get sharper similar jobs, create your profile using the link below.