Manual Evaluations Program Leader

About the Role

The Manual Evaluations Program Leader will own the end-to-end strategy, design and execution of human evaluations for Uber's GenAI-powered products, including conversational AI, voice AI, agent workflows and auto-evaluation systems. This role sits within the Global Digital Experience team, the operational arm of Uber's customer support tech organisation, and is a critical driver of quality, safety, and performance across Uber's next-generation AI solutions.

This leader will build and scale Uber's Manual Evaluation framework: defining methodologies, creating evaluation rubrics, ensuring annotation quality, and generating the insights that shape model tuning, product improvements, and release decisions. They will partner closely with Product, Engineering, Data Science and Product Ops to translate evaluation outcomes into clear technical and operational actions.

The role includes both strategic leadership and operational execution. The Program Leader will directly manage a team of three and indirectly oversee a distributed network of evaluators across global business sites. They will be responsible for setting the quality bar for evaluations, ensuring consistent delivery at scale, and driving continuous improvement of the evaluation pipeline.

The ideal candidate brings strong technical literacy in GenAI systems, exceptional program design and operational skills, and the ability to lead high-impact cross-functional initiatives. They are comfortable navigating ambiguity, building strong partnerships across Uber and influencing product direction through rigorous evaluation insights. This is a rare opportunity to play a leading role in one of Uber's most transformative technology programs and help shape the future of Uber's AI-driven experiences.

1. Own the end-to-end strategy, design, and execution of Manual Evaluations for Uber's GenAI-powered products (chatbots, voice AI, automated workflows, autoeval systems)
2. Develop and continuously improve evaluation methodologies, including rubrics, taxonomies, annotation guidelines, quality standards and success metrics
3. Partner with Product, Engineering, Data Science and Product Operations to ensure human evaluations directly inform model tuning, safety improvements, product design changes, and release decisions, as well as scaled operations teams to delivery on time, at short notice and to a high quality standard
4. Lead evaluation projects across multiple AI products simultaneously, ensuring timelines, quality and delivery expectations are met
5. Package insights into clear, actionable narratives and present them to cross-functional leaders, influencing product and operational strategy
6. Oversee a global manual evaluations operation, including direct management of a core team, indirect leadership of evaluators at multiple business sites and ongoing assessment of internal vs external resources to deliver the best evaluation outcomes
7. Establish processes and tools that scale, including workflow optimization, evaluator training, QA systems and feedback loops.
8. Serve as Uber's subject-matter expert in human evaluation for GenAI, staying current with best practices in safety testing, multimodal evaluation and human-in-the-loop systems.

1. Bachelors degree in engineering or similar
2. 5+ years of experience in program management, product operations, quality operations, research operations, or technical program leadership, ideally in a technology or AI-related environment.
3. Experience with GenAI systems, LLM evaluation, model safety, failure pattern analysis, prompt evaluation, or AI product quality.
4. Experience designing or running structured evaluation or quality frameworks, such as human labeling, annotation, audit workflows or manual review processes.
5. Familiarity with evaluation methodologies (rubric design, taxonomies, annotation guidelines, reliability scoring, inter-rater agreement, etc.).
6. Proven track record of managing teams, including coaching, performance management and resource planning.
7. Strong project management abilities, with experience running multiple complex programs simultaneously.
8. Proven experience managing outsourced teams to execute high-quality manual evaluation processes

1. Demonstrated ability to work cross-functionally with Product, Engineering, Data Science, and Operations teams.
2. Knowledge of automated evaluation systems, LLM-as-judge frameworks, or hybrid human+machine evaluation pipelines.
3. Background in service design, conversational AI, voice UX, or agent workflows.
4. Strong analytical and problem-solving skills, with experience turning ambiguous data into clear insights.
5. Excellent written and verbal communication skills, capable of translating technical evaluation outputs into business-relevant insights.
6. Experience in global operations, including scaling teams, training processes, and quality management across regions.