Staff QA Engineer (AI)

About the role

We're Matillion, the company behind Maia. Maia is the first AI Data Automation platform that rethinks manual data work by autonomously creating, managing, and evolving data products for humans and AI agents at scale. Quality isn't an afterthought here; it's embedded into everything we ship. As a Staff Quality Engineer, you'll take a senior position within one of our small, focused engineering teams - bringing advanced technical guidance, deep hands-on expertise and a relentless drive to embed quality across the entire development lifecycle.

This is primarily a hands-on technical role, with a secondary focus on technical mentorship and upskilling the team in AI quality practices. You'll set the quality standard for the team, champion automation-first and shift-left testing methodologies, and play a pivotal role in shaping the overall quality journey across Matillion's Engineering department. If you believe quality is everyone's responsibility and you have the experience to prove it - we want to hear from you.

This role sits within the AI team - the group building Maia's core capabilities. Quality here means something sharper than it does elsewhere: our systems are built on large language models, which are non-deterministic by nature, so testing them takes more than traditional pass/fail assertions. You'll help define what "quality" even means for a system like Maia, and build the evals, harnesses and tooling that let us prove it.

What you will be doing

- Building and evolving the eval suites that quantify Maia's performance - including LLM-as-judge test harnesses - so the team can change models, prompts and pipelines with confidence instead of guesswork

- Curating and maintaining the golden datasets our evals run against, and teaching developers across the team how to contribute to and maintain both the evals and the data behind them - quality here is shared, not something that lives solely with QE

- Owning and evolving how we test Maia's agentic workflows - multi-step tool use, reasoning chains and error recovery - a discipline this team has built from the ground up and keeps pushing forward

- Turning eval and test results into visualisations and trend reporting that show us, release over release, whether Maia's capabilities are genuinely moving forward

- Continuously auditing test coverage against real customer usage, so our eval suites stay representative of how Maia is actually used rather than how we assume it's used

- Extending quality practices beyond pre-release testing into production - monitoring live outputs alongside the rest of the team to catch drift and regressions that CI alone won't surface

- Partnering daily with Matillion's Data Science team on the statistical rigour behind our evals - designing sound experiments and correctly reading whether a change is a genuine improvement or just noise

- Stepping back from individual test suites to spot gaps and friction in our broader testing frameworks and methodology - and taking the initiative to revisit, rework or introduce entirely new approaches when the way we test needs to evolve

- Acting as a source of gen AI testing expertise for the wider Engineering organisation - helping quality engineers in other teams stand up evals and learn how to test generative AI features as they build their own AI functionality

- Staying ahead of the curve on generative AI testing - researching new techniques, frameworks and tools on your own initiative, without needing to be asked

- Leading and overseeing the testing of Matillion products - hands-on in the work while mentoring and guiding junior engineers to raise the bar across the team

- Designing and driving robust quality frameworks that span automation, security, accessibility and performance testing - with an automation-first mindset at the core

- Championing shift-left testing across the development lifecycle, embedding quality from design through to deployment via CI/CD pipelines

- Taking a leadership role in cross-functional collaboration - building a culture of quality excellence and elevating standards across the wider Engineering organisation

- Identifying areas for improvement in the quality journey, driving continuous iteration and playing an active role in talent acquisition and team development

What we are looking for

- An understanding of large language models and why their non-deterministic behaviour makes traditional, assertion-based testing insufficient on its own

- Experience building or working with evals and LLM-as-judge techniques to score and quantify the quality of generative AI systems

- A working grasp of statistical thinking - you know that non-deterministic outputs make single-run results noise, and how to design and read evals across samples rather than as simple pass/fail checks

- Deep hands-on experience building and maintaining test automation frameworks (Cypress, Pact or equivalent), with strong proficiency in Java or JavaScript and a solid understanding of CI/CD pipelines

- Experience with at least one major cloud platform (AWS, GCP or Azure) and advanced knowledge of databases and SQL

- A genuine QA mindset - you challenge norms, push back when quality is being compromised and continuously improve how the team works

- Autonomy and inquisitiveness in equal measure - you dig into problems independently, understand causes not just effects, and bring a growth mindset to everything you do

Client-provided location(s): Manchester, United Kingdom

Job ID: d08101cc-3465-4694-b0b5-1602338a9978

Employment Type: OTHER

Posted: 2026-04-14T13:40:22