Research Scientist / Engineer, Foundation Model Evaluation
We are looking for a Research Scientist or Engineer to join our Foundation Model Evaluation
team. In this role, you will design and build evaluation methodology that measures what matters
- how well our models perform at the frontier of key capabilities, and how well they serve real
users across Apple products on billions of active devices. You will turn evaluation insights into
signals that make models better.
Description
This is a hands-on role focused on the models that power Apple products used daily by over a
billion people. You will design evaluation systems where the outcome is not just a score, but an
actionable signal - one that drives model improvement and predicts real user experience.
Working alongside model training and product teams, you will close the loop between evaluation
and improvement.
Our work spans three areas:
• Frontier capability assessment: benchmarking against the state of the art in reasoning,
code, knowledge, and agentic workflows
• Product-aligned evaluation: measuring model quality in ways that reflect real user
experience
• Evaluation-to-training integration: feeding actionable insights back into the model
development cycle
You may focus on one area or work across multiple, depending on your background and
interests.","responsibilities":"Benchmark Design & Development: Design and implement evaluation benchmarks,
metrics, and test suites that rigorously measure model capabilities across reasoning,
knowledge, code, and agentic workflows.
Product-Aligned Evaluation: Develop evaluation methods that capture how models
behave in real product settings, and validate that evaluation metrics predict user-
perceived quality and product outcomes.
Evaluation Methodology Research & Tooling: Research and apply state-of-the-art
evaluation techniques - including scoring frameworks, model-based judging, and
contamination-resistant benchmark design. Build reusable tools, scorer libraries, and
analysis frameworks that scale across the team's benchmark portfolio.
Experimental Analysis: Design and execute rigorous experiments comparing model
capabilities, engage with third-party vendors on benchmarking, and perform detailed gap
analysis to guide model development priorities.
Cross-Team Collaboration: Work closely with model training, training data, and product
teams to ensure evaluation insights inform training strategies, data decisions, and product
quality improvements.
Preferred Qualifications
PhD in Computer Science, Machine Learning, NLP, or a related field
Want more jobs like this?
Get jobs in Cupertino, CA delivered to your inbox every week.
Direct experience evaluating large language models, e.g. benchmark design, model-based
judging
Track record of collaborating with model training and data teams to turn evaluation
findings into training improvements
Experience building reusable evaluation tooling or analysis frameworks adopted across
teams
Familiarity with human evaluation methodology and experience partnering with
annotation teams or vendors to assess model quality
Minimum Qualifications
3+ years of experience in AI model evaluation, NLP, or a related area (e.g., natural
language generation, information retrieval, or conversational AI)
Strong fundamentals in machine learning, natural language processing, and statistical
analysis
Proficiency in Python and experience with ML frameworks (PyTorch, JAX, or
equivalent)
Demonstrated ability to translate research insights into practical implementations
Strong experimental design skills: ability to design rigorous comparisons and draw valid
conclusions from results
Clear technical communication: ability to distill evaluation results into actionable
recommendations for cross-functional partners
MS or PhD in Computer Science, Machine Learning, Natural Language Processing or a related
technical field. Equivalent practical experience will be considered.
Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant .
Pay & Benefits
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $181,100 and $318,400, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits.
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
Perks and Benefits
Health and Wellness
Parental Benefits
Work Flexibility
Office Life and Perks
Vacation and Time Off
Financial and Retirement
Professional Development
Diversity and Inclusion
Company Videos
Hear directly from employees about what it is like to work at Apple.