Software Engineer, AIML Evaluation

At Apple, great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring passion and dedication to your job and there's no telling what you could accomplish. Do you want to make Siri and Apple Intelligence features smarter for our users? Contribute to the next revolution in AI and Machine Learning. We're a dynamic team seeking engineers who are enthusiastic about influencing the future of Apple Intelligence.

Description

As a Software Engineer in AIML Evaluation, you will:

Work closely with product managers, backend engineers, data scientists and other stakeholders to improve the overall quality and user experience of numerous Apple products.

Design and develop evaluation system and framework to enhance the quality of Apple Intelligence, ensuring our AI/ML models meet the highest standards.

Tuning evaluation infrastructure, tightening operational controls, improving auditability and debug trails, and scaling the workflows .

You'll play a pivotal role in developing Apple Intelligence, driving the next generation of groundbreaking products across all Apple platforms.

Responsibilities:

Benchmark Design & Development: Design and implement evaluation benchmarks, metrics, and test suites that rigorously measure model capabilities across reasoning, knowledge, code, and agentic workflows.

Product-Aligned Evaluation: Develop evaluation methods that capture how models behave in real product settings, and validate that evaluation metrics predict user-perceived quality and product outcomes.

Evaluation Methodology Research & Tooling: Research and apply state-of-the-art evaluation techniques - including scoring frameworks, model-based judging, and contamination-resistant benchmark design. Build reusable tools, scorer libraries, and analysis frameworks that scale across the team's benchmark portfolio.

Experimental Analysis: Design and execute rigorous experiments comparing model capabilities, engage with third-party vendors on benchmarking, and perform detailed gap analysis to guide model development priorities.

Cross-Team Collaboration: Work closely with model training, training data, and product teams to ensure evaluation insights inform training strategies, data decisions, and product quality improvements.

Preferred Qualifications

Eager to explore and apply cutting-edge tech solutions.

Experience working on large, cross-functional, and multi-team projects.

Proven ability to operate independently on ambiguous, open-ended problems where the right answer is not obvious.

Striving to elevate every aspect of the user experience.

Minimum Qualifications

3+ years of professional software engineering experience building and operating production systems

Ability to tackle complex challenges, think critically, and deliver innovative solutions.

Fluent use of AI-assisted development tools (coding agents, code review assistants, etc.) to work effectively at scale

Experience designing and scaling distributed systems (load balancing, active-active topologies, capacity planning, throughput-bound services)