Senior Agentic AI Test and Evaluation Engineer

Description

At Leidos, you'll contribute to AI solutions that serve critical national and global missions-ranging from defense and intelligence to healthcare, energy, and space exploration. Our work emphasizes Trusted Mission AI: systems that are transparent, ethical, resilient, and accountable. You'll collaborate with multidisciplinary teams to transition AI research into operational environments where accuracy, security, and reliability are non-negotiable. Joining Leidos means applying your expertise to solve some of the most complex and meaningful challenges of our time.

We are looking for a motivated Agentic AI Test and Evaluation Engineer who wants to work on challenging problems in a variety of domains - including enterprise IT, health, defense, intelligence, and energy - to get results that apply and go beyond the state of the art for measurably better outcomes. We apply our knowledge, capabilities, and experience to develop and deploy Trusted Mission AI - AI that deserves to be trusted by system owners, end users, and the public - to be accurate, ethical, reliable, and adaptable.

You will work with a team of agentic AI scientists, agentic AI scientists, data scientists and data engineers to operationalize new approaches for test and evaluation of Agentic AI models that produce measurable advances over state of the art solutions.

Primary Responsibilities

• Develops AI Models Test and Evaluation CONOPS
• Creates scalable Test and Model Evaluation plans for Agentic AI systems including process, techniques and tools.
• Works with AI scientists, agentic AI scientists, data scientists and data engineers to understand the AI system under test to develop test procedures
• both positive and negative testing and evaluation
• Collect performance metrics as part of evalulation results documentation
• Works with MLOps engineers to integrate testing tools and procedures with the CI/CD pipeline
• Analyzes existing processes and resultant metrics to recommend potential improvements
• Collaborates with AI Governance team to maintain visibility and explainability through testing
• Implements testing process in the AI system design, development and deployment life cycle
• Identifies the risk in testing of projects, particularly for assessing the limitations of planned tests on complex AI systems
• Works within teams of AI/ML researchers and engineers using Agile development processes

• Bachelor's degree in Computer Science, Data Science or related field and over 8 years of relevant experience, Masters with 6 years experience. Additional experience may be considered in lieu of degree.
• Strong Python programming fundamentals
• Experience with system and subsystem level test process and automation
• Experience with creating user acceptance test scenarios
• Experience with SecDevOps tooling and MLOps pipeline development
• Experience with software test automation techniques
• Experience with AI Performance and vulnerability assessment
• AI model assurance evaluation
• Experience applying and automating AI interpretability & explainability tools and methods
• Experience with developing CONOPS and presentations
• Good understanding of machine learning algorithms, tools and platforms
• Self-starter with high intellectual curiosity
• Great communication skills, able to explain model and test results to a non-technical audience
• Proficient in data exploration techniques and tools

• Experience with data visualization libraries such as Plotly, Streamlit, and matplotlib.
• Experience with AI/ML tools, such as common Python packages (e.g., scikit-learn, NumPy, Pandas) and Jupyter notebooks
• Experience with database administration and data repositories
• Experience in data exploration techniques and tools
• Experience with building LLM and other Generative AI applications.
• Willing to learn new skills and platforms to support data analytics.
• Ability and willingness to obtain a Top Secret security clearance