AI QA Engineer - GenAI

Job Description Summary

GE Vernova is seeking a highly skilled AI QA Engineer to lead the development of benchmark frameworks for AI systems & use cases deployed. This role is critical to ensuring the accuracy, reliability, safety, and compliance of AI models driving innovation at GE Vernova. The ideal candidate will design and implement rigorous testing frameworks tailored for AI use cases unique to GE Vernova.

Job Description

• Design, develop, and execute comprehensive benchmark frameworks to evaluate AI use cases, prompt performance, robustness, across diverse array of use cases.
• Develop automated testing suites to validate AI functionalities such as prediction accuracy, response consistency, case handling, bias detection, and model degradation over time.
• Collaborate closely with Prompt Engineers, AI Agent Engineers and product owners to define quality standards, acceptance criteria, and key performance indicators (KPIs) for AI deployments.
• Conduct quantitative and qualitative evaluations of AI outputs ensuring alignment with business objectives, regulatory standards, and ethical AI principles.
• Monitor AI use cases performance in production, systematically identify issues, and recommend improvements to maintain high standards of AI quality and safety.
• Document testing methodologies, framework designs, quality metrics, and ensure thorough reporting to stakeholders.
• Support continuous improvement initiatives by incorporating feedback loops and new benchmarking techniques as AI technologies evolve.
• Provide training and mentorship on AI testing best practices within GE Vernova's Prompt engineering and AI Agent Engineering team.
• Stay current with emerging AI technologies, contribute to platform and tooling improvements, and share knowledge within the team.

• Master's degree in Computer Science, AI Engineering, Data Science, or related field.
• 3+ years of experience in software quality assurance with at least 2 years focused on AI/ML systems.
• Proven track record designing and implementing AI benchmark frameworks or quality assurance strategies.
• Strong programming skills in Python and experience with automated testing frameworks (e.g., pytest, Selenium).
• Familiarity with RAG systems, vector databases, and GenAI architectures.
• Deep understanding of machine learning models, LLMs, and AI validation methodologies.
• Experience with AI fairness, bias detection, and responsible AI practices.
• Knowledge of cloud environments (AWS, Azure, or GCP), containerization, and deployment pipelines.
• Ability to analyze complex datasets and performance metrics quantitatively and qualitatively.
• Strong communication skills with ability to document and present technical information clearly.
• Hands-on experience with LLMs, prompt engineering, and natural language processing (NLP).
• Knowledge of agent orchestration platforms and multi-agent systems (e.g., AutogenAI, LangGraph, MCP protocol).