Manager Digital Product - AI Quality Assurance & Testing

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

This AI Quality Assurance & Testing is the first of its kind and critical role for accelerating the AI agenda across UHC Consumer Operations. This position will report to the Director of AI Quality and Assurance testing.. This leader's mission is to drive 3 things a) AI Program Pre-Production Testing b) AI Program Postproduction Testing/Monitoring c) AI Operational readiness and go to market strategies
This leader's mission is to lead a team of approximately 5-7 individuals) to rigorously test seamless AI enabled consumer experiences, maximize AI agentic delivery and ensure UHC wide ambitious cost savings due to consumer self-service adoption.

Primary Responsibilities:

• Establish and oversee AI/ML testing environments and frameworks for functionality, performance, fairness, bias detection and robustness
• Implement pre-launch validation and sign off processes for AI models and features
• Build QA playbook for AI products/features covering regression, stress testing, explainability and edge case validation
• Develop post launch QA playbook covering live shadow testing against ground truth datasets, A/B and multivariant testing to drive adoption of AI risk management practices
• Oversee integration of RHLF (reinforcement learning from human in the loop) pipelines - structured human oversight in model evaluation, labeling, QA to ensure that models align with ethical and business standards
• Build RLAIF frameworks (Reinforcement learning AI feedback loop) leveraging AI-assisted evaluation and reinforcement learning for continuous feedback loops at scale
• Define and run operational readiness reviews (ORR) before AI product launches
• Drive release readiness assessments across product, engineering, operations, technology
• Champion responsible AI practices ensuring fairness, transparency and accountability in AI Models
• Hire, mentor and scale a high performing AI delivery and business integration organization
• Define program KPI's and regularly report process to executive leadership
• Drive a culture of blameless postmortems to strength resilience and accountability
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regard to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

• 10 + years in product and/or operations with 2+ years of AI/ML focused testing leadership roles
• Solid understanding of machine learning, generative AI, data platforms and AI ethics
• Experience in establishing data integrity and governance frameworks at scale
• Track record of delivering AI products to market with measurable impact
• Demonstrated intellectual capacity and sound judgment to lead in changing and ambiguous environments
• Proven track record of:
  • Working collaboratively in a highly matrixed organization
  • Building relationships with people from diverse areas of an organization
  • Bringing fresh ideas and new thinking to the team (with views that change opportunistically)
  • Being energized and passionate with the ability to persuasively sell ideas and concepts
  • Operating effectively at executive levels
  • Thinking broadly about overall business issues with both an "inside-out" and "outside-in" perspective
  • Managing, motivating and developing a team and attracting and retaining outstanding talent
  • Empowering team members and holding them accountable
  • Rapidly assessing situations and absorbing & applying new information to decision-making and direction setting
  • Demonstrated confidence and perseverance in addressing challenges and overcoming obstacles
  • History of addressing difficult issues with candor, speaking the truth, expressing a point of view and taking a stand

• Past experience partnering with AI forward companies like Amazon, Google, Microsoft and the likes
• Familiarity with global data protection and AI regulatory frameworks
• Well-rounded in AI literacy and upskilling programs