AI Experience Researcher, Product Evaluation, Vision Products Group

Description

The central challenge of this role is figuring out what "good" means for an AI experience, and then designing rigorous evaluations that measure those qualities reliably and at scale. This requires both deep theoretical grounding in human experience and a solid analytical mindset to operationalize that understanding into scalable evaluation frameworks.

Leaning on research in human sciences, you will decompose complex AI interactions into their constituent parts, reason about how those parts interact, and build evaluation frameworks that hold up under the scrutiny of non-deterministic nature of AI experiences and the pressures of iterative product development. You will derive experimental designs, create golden data sets, write tests, and turn them into prompts for LLM judges or instructions for human raters. You will run automated evaluations, analyze results, and present findings to diverse stakeholders.

Candidates who bring both quantitative rigor and a qualitative sensibility - to recognize patterns in model behaviors and outputs, and to develop an interpretive understanding of what the data is and isn't capturing from a human perspective - will thrive in this role.What matters most is the ability to hold both orientations at once - to think carefully about what makes an experience work, and to measure complex human dimensions with precision. We are also looking for someone who is excited to co-create what this discipline looks like going forward - bringing intellectual curiosity and a point of view about where human-centered AI evaluation should be headed.","responsibilities":"Develop scalable automated evaluation methodologies by operationalizing complex multi-modal multi-turn AI experiences into observable and measurable metrics that work across diverse use cases, features, or product area

Produce comprehensive evaluation plans detailing evaluation scope, validation and data strategy, tooling requirements, resource allocation, and timelines

Derive experimental designs and write test instructions for LLM judges or for human raters

Define requirements for, or curate datasets that represent realistic usage; support data generation and annotation workflows to ensure coverage, quality, and alignment with product goals

Implement and analyze automated evaluations, maintaining rigor around reproducibility, identifying key insights, and areas for improvement across both qualitative and quantitative patterns

Prepare and present clear, concise, and impactful evaluation findings to diverse stakeholders, translating results into actionable recommendations for model training, ranking, and product decisions

Partner with engineers, QA, data scientists, designers, and product managers throughout the product development lifecycle to integrate evaluation insights and drive continuous improvement

Contribute to evolving human-centered AI evaluation methodologies and help to define best practices for AI experience evaluation as the field matures

Preferred Qualifications

Familiarity with methods for capturing experiential quality beyond task success - such as cognitive interviews, think-aloud protocols, interaction analysis, or discourse and conversation analysis

Experience designing and implementing automated evaluation pipelines, including writing prompts for LLM judges and constructing human-in-the-loop or multi-turn evaluation setups

Experience working with multimodal or agentic systems, AI/ML models, preferably Large Language Models

Familiarity with automated testing frameworks and tooling

Experience with data generation and annotation workflows, including curating datasets, scenarios, and tasks that represent realistic usage

Portfolio demonstrating previous evaluation frameworks, research findings, or measurable contributions to product improvement

Background in learning sciences or instructional design, with experience reasoning about what makes a complex human experience effective is a plus

Minimum Qualifications

Advanced degree in Cognitive Psychology, Human-Computer Interaction (HCI), User Experience (UX) Research, Learning Sciences, Learning Analytics, Psychometrics, Applied Behavioral Science, or a related field with a focus on human cognition, behavior, and empirical evaluation

A strong data-driven mindset with experience designing and conducting rigorous empirical research or evaluation - including experimental design, data analysis, and interpretation of various qualitative and quantitative data - particularly in the context of complex human-system interactions

Ability to reason from theoretical grounding about what makes an experience good in a given context, and to translate that reasoning into evaluation frameworks and measurement designs

Demonstrated ability to operationalize research literature, qualitative user feedback, and quantitative behavioral data into actionable evaluation criteria, observable metrics, and product insights

Proficiency in data analysis and interpretation, with a strong understanding of statistical validity in evaluation contexts

Exceptional collaboration skills with a track record of working effectively in cross-functional teams that include engineering, ML, design, QA, leadership, and subject matter experts of diverse domains

Strong communication skills, with the ability to translate complex research findings and evaluation results into clear, actionable recommendations for both technical and non-technical audiences

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant .

Pay & Benefits

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $134,800 and $245,800, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits.

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Client-provided location(s): Boulder, CO

Job ID: apple-200655171-2605_rxr-662

Employment Type: OTHER

Posted: 2026-04-05T19:43:35