AI Model Evaluation Project Lead - AI Data Service and Operations (Eco Governance)
Responsibilities
About the Team
The AI Data Service and Operations (ADSO) team provides safety and non-safety data annotation services, search operation services, and customer services for ByteDance's international products, helping them to build their own data ecological security. In order to optimize user experience while upholding negative content governance on our platforms, the Eco Governance team within ADSO focuses on data labeling work to support online content strategies and AI/LLM model development.
As the AI Model Evaluation Project Lead in the Eco Governance team, you will lead end-to-end delivery of AI data annotation projects and play a critical role in evaluating AI/LLM model performance. You will directly manage a team of AI Project Managers while driving rigorous model evaluation, analyzing results, identifying gaps in model behavior, and delivering clear, actionable recommendations to improve model quality. You will bridge annotation operations with model performance insights to ensure high-quality training/evaluation data translates into measurable improvements in AI capabilities.
Responsibilities
- Lead data annotation and model evaluation projects: Manage end-to-end execution of multiple projects, ensuring both annotation quality targets/SLAs and model performance benchmarks are met.
- Design and execute AI model evaluations: Develop or refine evaluation frameworks, create test cases/datasets (including adversarial/safety-focused ones), run evaluations on LLM outputs, and assess metrics such as accuracy, safety, relevance, bias, and robustness in content governance scenarios.
Want more jobs like this?
Get Data and Analytics jobs in Singapore delivered to your inbox every week.
- Analyze model performance and provide recommendations: Deep-dive into evaluation results, perform root cause analysis on model failures or quality issues, identify patterns in errors, and translate findings into concrete recommendations for annotation guideline improvements, data collection strategies, model fine-tuning, or process changes.
- Serve as the primary stakeholder interface: Translate product, safety, and business needs into clear annotation + evaluation requirements; align on targets and success metrics; and present evaluation insights and recommendations to algorithm, product, and leadership teams.
- Drive delivery governance and cross-functional collaboration: Establish operating rhythms, conduct evaluation reviews, and build escalation frameworks across QA, vendors, annotation teams, and business stakeholders.
- Leverage data for performance management: Monitor dashboards for both annotation and model metrics, detect anomalies, conduct in-depth data and root cause analysis, and drive continuous improvements in quality and efficiency.
- Lead continuous improvement and optimization: Identify gaps between annotation quality and model performance; design workflow enhancements, hybrid (machine + human) labeling strategies, and automation opportunities; partner with tooling and algorithm teams to scale evaluation capabilities.
- Risk and change management: Proactively identify risks related to data quality, model safety, or delivery timelines; propose mitigation plans; and lead operational transitions.
- Deliver strategic reporting: Synthesize annotation and model evaluation data into clear insights, performance summaries, and forward-looking recommendations for leadership and cross-functional partners.
Qualifications
Minimum Qualification(s)
- Bachelor's degree or above.
- At least 5 years of project/program management experience in AI data annotation, large-scale data operations, content moderation, or AI model evaluation environments.
- At least 3 years of hands-on experience in AI/LLM model evaluation, including designing evaluation methodologies, analyzing model outputs, conducting root cause analysis on performance issues, and providing actionable recommendations to improve models.
- At least 3 years of regional team management experience, with proven ability to lead a team of project managers or similar roles.
- Strong proficiency in both spoken and written English, with excellent stakeholder management and communication skills - especially in presenting complex evaluation findings and recommendations to regional/global teams in a fast-paced environment.
- Strong data skills (Excel, SQL, dashboard development) and proven experience using data-driven approaches for performance analysis, root cause investigation, and driving operational/process improvements.
Preferred Qualification(s)
- Majors in Data Science, Statistics, Mathematics, Computer Science, or related fields are preferred.
- Direct experience with LLM training, fine-tuning, or evaluation (e.g., safety alignment, red-teaming, preference/reward modeling, or RLHF workflows).
- Hands-on experience implementing or optimizing machine labeling, hybrid (human + machine) annotation, or automated evaluation pipelines.
- Familiarity with evaluation metrics and benchmarks relevant to content safety and generative AI.
- Project Management certifications (PMP, Agile, Lean Six Sigma) or equivalent.
Perks and Benefits
Health and Wellness
- Health Insurance
- Dental Insurance
- Vision Insurance
- HSA
- Life Insurance
- Fitness Subsidies
- Short-Term Disability
- Long-Term Disability
- On-Site Gym
- Mental Health Benefits
- Virtual Fitness Classes
Parental Benefits
- Fertility Benefits
- Adoption Assistance Program
- Family Support Resources
Work Flexibility
- Flexible Work Hours
- Hybrid Work Opportunities
Office Life and Perks
- Casual Dress
- Snacks
- Pet-friendly Office
- Happy Hours
- Some Meals Provided
- Company Outings
- On-Site Cafeteria
- Holiday Events
Vacation and Time Off
- Paid Vacation
- Paid Holidays
- Personal/Sick Days
- Leave of Absence
Financial and Retirement
- 401(K) With Company Matching
- Performance Bonus
- Company Equity
Professional Development
- Promote From Within
- Access to Online Courses
- Leadership Training Program
- Associate or Rotational Training Program
- Mentor Program
Diversity and Inclusion
- Diversity, Equity, and Inclusion Program
- Employee Resource Groups (ERG)
Company Videos
Hear directly from employees about what it is like to work at TikTok.