284 jobs for "Test and Evaluation Engineer"

Machine Learning Platform Engineer, Apple Services Engineering

At Apple - Seattle

Posted on Oct 17

View job Apply on company site

Intern, R&D Undergraduate Summer - Advanced Concepts Engineering, Onsite

At Sandia National Laboratories

Posted on Nov 5

View job Apply on company site

Intern, R&D Graduate Summer - Advanced Concepts Engineering, Onsite

At Sandia National Laboratories

Posted on Nov 5

View job Apply on company site

Enterprise Analytics and Modeling - Quantitative Engineering - Lead Associate

At Fannie Mae - Washington · Reston

Posted on Nov 3

View job Apply on company site

Apple

Machine Learning Platform Engineer, Apple Services Engineering

Seattle, WA$175,000 – $263,300Posted 23 days ago

Apply on company site View full job

We're building the evaluation platform that will serve all of Apple's generative AI and agent systems. Evaluating non-deterministic AI systems is one of the hardest unsolved problems in production ML - and one Apple has to get right at scale. We're building the platform that makes it tractable for every team here.

This is a hands-on engineering role with a lot of autonomy. You'll write a lot of Python and own meaningful pieces of the platform end-to-end. You'll be partnering closely with research engineers, model and serving teams, product and feature teams, and the infra and data platform groups this work integrates with.

Description

Build and ship: Take ownership of features and services within the evaluation platform: APIs, SDKs, orchestration components, evaluation runners. You'll have the room to make calls on your own work and the support to deliver it well.

Productionize ML research: Partner with research engineers to take their prototype code and turn it into reliable services. You'll learn their world quickly and translate research patterns into clean Python that holds up under real load.

Move fast, responsibly: You'll get scoped problems with room to figure out the how. We trust you to balance speed with care, to know when something needs a quick prototype and when it needs a design doc, tests, and a careful rollout.

Improve as you go: Notice the rough edges and pick them up. The flaky test, the slow build, the confusing API, the runbook that's out of date. We want someone who leaves the codebase a little better every week.

Developer experience: Help build the SDKs and abstractions that other Apple teams use to evaluate their models and agents. You'll feel the friction of bad ergonomics directly, which puts you in a great position to fix it.

Operational ownership: Your code runs in production. You write the tests, set up the CI, add the metrics, and stay close when something breaks. You don't need to be an SRE, but you take care of what you ship.","responsibilities":"Build and ship: Take ownership of features and services within the evaluation platform: APIs, SDKs, orchestration components, evaluation runners. You'll have the room to make calls on your own work and the support to deliver it well.

Productionize ML research: Partner with research engineers to take their prototype code and turn it into reliable services. You'll learn their world quickly and translate research patterns into clean Python that holds up under real load.

Move fast, responsibly: You'll get scoped problems with room to figure out the how. We trust you to balance speed with care, to know when something needs a quick prototype and when it needs a design doc, tests, and a careful rollout.

Improve as you go: Notice the rough edges and pick them up. The flaky test, the slow build, the confusing API, the runbook that's out of date. We want someone who leaves the codebase a little better every week.

Developer experience: Help build the SDKs and abstractions that other Apple teams use to evaluate their models and agents. You'll feel the friction of bad ergonomics directly, which puts you in a great position to fix it.

Operational ownership: Your code runs in production. You write the tests, set up the CI, add the metrics, and stay close when something breaks. You don't need to be an SRE, but you take care of what you ship.

Preferred Qualifications

Experience working on developer platforms, internal tools, or SDKs

Production experience with LLM/agent systems - building, evaluating, or operating them

Familiarity with job orchestration frameworks (Temporal.io, Airflow, or similar)

Distributed compute experience (Ray, Dask, or Kubernetes-based job systems)

Experience with experiment tracking or ML lifecycle tooling (Weights & Biases, MLflow, etc.)

Startup or early-stage experience where you wore multiple hats and shipped under constraint

Minimum Qualifications

4-8 years of software engineering experience building and shipping production services.

Strong Python. You're fluent with FastAPI, Pydantic, and the modern Python ecosystem. You write code that's clean, tested, and easy for the next person to pick up.

Builder's mindset. You enjoy shipping. You're comfortable iterating quickly on scoped problems and knowing when to slow down for the parts that need it.

Fluency with AI coding tools. You actively use tools like Claude Code (or equivalents) in your day-to-day workflow, including features like skills, slash commands, and agent-style workflows. You have a good intuition for when to lean on them, when to steer them, and how to get high-quality output.

Familiarity with the agentic LLM landscape. You stay current on how modern LLM systems work in production - tool use, MCP servers, agent frameworks, context management, multi-step reasoning. You can hold a real conversation about the tradeoffs.

Hands-on evaluation experience. You've built evaluations for your own agents or LLM systems, or you've worked with evaluation orchestration frameworks like Inspect, Braintrust, LangSmith, Promptfoo, or equivalents (including internal tooling). You understand what makes an evaluation trustworthy vs. theatrical.

Real working knowledge of LLMs in production. You're comfortable with prompt iteration, dataset curation, judge models, and statistical reasoning about non-deterministic outputs. You understand the lifecycle around models even if you haven't trained them yourself.

Solid engineering fundamentals. You understand testing, CI/CD, containerization (Docker), and basic observability. You've shipped services that others depend on and stayed close when they broke.

Clear communicator. You write clear PRs, ask sharp questions, and flag blockers early. You're comfortable disagreeing thoughtfully and changing your mind when the argument is good.

Ownership. When something is broken or unclear, you tend to pick it up rather than wait. You either move it forward or surface it clearly.

Pay & Benefits

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $175,000 and $263,300, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

About Apple

Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.

Size: 10,000+ employees
Industry: Technology

View company profile