5 jobs for "Evaluation Science Lead"

Responsibilities

About the team
The Automation Team works closely with Product, Engineering, Algorithms, Operations, and other departments to strategically drive moderation automation and AI initiatives and maximize impact and ensure goal attainment.

About the role
You will be working on initiatives such as:
- AI agents that assist human moderators with decision-making
- LLM-powered operational workflows
- Prompt engineering and evaluation for moderation use cases
- Automation solutions that reduce manual review effort while maintaining quality
- AI quality monitoring and continuous model improvement
- Human-in-the-loop systems combining AI and operational expertise

Responsibilities:
- Lead end-to-end AI automation programs that improve the efficiency, quality, and scalability of Trust & Safety operations.
- Develop prompts, workflows, and evaluation strategies for LLM-powered and AI agent solutions to improve operational outcomes.
- Analyze AI model performance, including precision, recall, leakage, overkill, and human review quality, to identify improvement opportunities.
- Use data-driven insights to optimize prompts, business rules, and automation logic to continuously improve AI performance.
- Build monitoring frameworks and dashboards to measure AI adoption, model quality, operational efficiency, and business impact.
- Drive experimentation through pilots and A/B testing to validate new AI capabilities before large-scale deployment.
- Translate operational pain points into technical requirements and partner with Product and Engineering teams to influence product roadmap prioritization.
- Coordinate multiple cross-functional initiatives across Product, Engineering, ML, Policy, and Operations teams while managing risks, dependencies, and delivery timelines.
- Communicate program progress, technical findings, and strategic recommendations to leadership stakeholders.

Qualifications

Minimum Qualification(s)
-Bachelor's or above in Computer Science, Information Systems or a related field
-Strong analytical skills with experience interpreting operational or product data to drive process improvements.
-Experience working cross-functionally with Product, Engineering, and Operations teams in a technology environment.
-Ability to manage multiple complex initiatives simultaneously while influencing stakeholders without direct authority.
-Excellent communication and stakeholder management skills.

Preferred Qualification(s)
-Hands-on experience working with Large Language Models (LLMs), Generative AI, AI agents, or Machine Learning products.
-Experience designing prompts, prompt evaluation methodologies, or AI workflow optimization.
-Familiarity with coding or scripting (such as Python, SQL, JavaScript, or similar languages) for automation, data analysis, or prototyping.
-Experience using APIs, AI platforms, or workflow automation tools to build operational solutions.
-Experience evaluating AI model performance using metrics such as precision, recall, leakage, overkill, hallucination, or similar quality measurements.
-Experience working in Trust & Safety, content moderation, fraud, risk, compliance, or large-scale operations is highly preferred.
-Familiarity with agentic AI frameworks, Retrieval-Augmented Generation (RAG), prompt engineering, or AI evaluation frameworks is a plus.

Responsibilities

Qualifications

Qualifications

Minimum Qualification(s)
-Bachelor's or above in Computer Science, Information Systems or a related field
-Strong analytical skills with experience interpreting operational or product data to drive process improvements.
-Experience working cross-functionally with Product, Engineering, and Operations teams in a technology environment.
-Ability to manage multiple complex initiatives simultaneously while influencing stakeholders without direct authority.
-Excellent communication and stakeholder management skills.

Preferred Qualification(s)
-Hands-on experience working with Large Language Models (LLMs), Generative AI, AI agents, or Machine Learning products.
-Experience designing prompts, prompt evaluation methodologies, or AI workflow optimization.
-Familiarity with coding or scripting (such as Python, SQL, JavaScript, or similar languages) for automation, data analysis, or prototyping.
-Experience using APIs, AI platforms, or workflow automation tools to build operational solutions.
-Experience evaluating AI model performance using metrics such as precision, recall, leakage, overkill, hallucination, or similar quality measurements.
-Experience working in Trust & Safety, content moderation, fraud, risk, compliance, or large-scale operations is highly preferred.
-Familiarity with agentic AI frameworks, Retrieval-Augmented Generation (RAG), prompt engineering, or AI evaluation frameworks is a plus.

About TikTok

TikTok is the leading destination for short-form mobile video and our mission is to inspire creativity and bring joy.

Size: 5,001 - 10,000 employees
Industry: Entertainment & Gaming

View company profile