Applied Scientist, AI Evaluation Platform
Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or Apple Store experience we deliver is the result of us making each other's ideas stronger. That happens because every one of us shares a belief that we can make something wonderful and share it with the world, changing lives for the better. It's the diversity of our people and their thinking that inspires the innovation that runs through everything we do. When we bring everybody in, we can do the best work of our lives. Here, you'll do more than join something - you'll add something.
Description
Our team, part of Apple Services Engineering, is looking for an Applied Scientist to lead the design and continuous development of automated benchmarking methodologies for AI-powered code assistant tools. In this role, you will investigate how coding-focused LLM agents behave, create rigorous evaluation frameworks, and establish scientific standards for assessing their quality and reliability. This role is crafted to enable the development of scalable evaluation frameworks that ensure our engineers have the right tools to create products that surprise and delight our customers.
Want more jobs like this?
Get jobs in Seattle, WA delivered to your inbox every week.
The successful candidate will have a proactive approach with the ability to work independently and collaboratively on a wide range of projects. In this role, you will work alongside a small but impactful team, collaborating with ML and data scientists, software developers, project managers and other teams at Apple to understand requirements and translate them into scalable, reliable, and efficient evaluation frameworks. ","responsibilities":"Design scientifically grounded benchmarking methodologies for code assistants, covering multiple dimensions of quality (e.g. correctness, performance) across several use cases.
Developing automated evaluation pipelines that collect, automatically judge, and analyze model outputs at scale.
Create and curate datasets, tasks, and coding scenarios that represent realistic developer workflows across multiple languages and domains.
Define and validate new metrics for complex phenomena such as tool reliability, reasoning quality, or multi-turn developer interaction patterns.
Apply statistical rigor and reproducibility to above mentioned metrics.
Work closely with engineering and research teams to translate experimental findings into actionable model improvements.
Publish internal reports and external papers.
Monitor evolving industry practices and academic work to ensure benchmarks remain relevant.
Preferred Qualifications
Publications in ML evaluation or related fields.
Experience with automated testing frameworks.
Experience constructing human-in-the-loop or multi-turn evaluation setups.
Prior work on agentic developer tools.
Minimum Qualifications
Advanced degree (MS or PhD) in Computer Science, Software Engineering, or equivalent research/work experience.
Strong research background in empirical evaluation, experimental design, or benchmarking.
Strong proficiency in Python. Intermediate proficiency in Swift.
Deep familiarity with software engineering workflows and developer tools.
Experience working with or evaluating AI/ML models, preferably LLMs or program synthesis systems.
Strong analytical and communication skills, including the ability to write clear reports.
Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant .
Perks and Benefits
Health and Wellness
Parental Benefits
Work Flexibility
Office Life and Perks
Vacation and Time Off
Financial and Retirement
Professional Development
Diversity and Inclusion
Company Videos
Hear directly from employees about what it is like to work at Apple.