TikTok LIVE - AI/ ML Data Operations Manager

- Align Models with Policies. Convert policy provisions and expert judgment into executable prompts and Agent skills, establishing: Prompt-writing standards, Writer principles, Admission criteria, Standard operating procedures, Quality and consistency requirements , Explainability requirements , Chain-of-thought or title and sub-title-level decision structures
- Partner with Labeling Teams. Drive the delivery of high-quality training and evaluation data through: Labeling specifications , Quality-control SOPs , Ground-truth alignment , Edge-case mining, Throughput and quality SLAs, Error analysis, Dataset refresh, Model and strategy iteration
- Define Tooling Requirements. Work with Product and Engineering to define requirements for: Case retrieval, Batch testing, Prompt debugging, Online-result backfill, Guidance and policy tools, Self-service RCA tools, Analysis assistants, Data-iteration workflows
- Own Evaluation, Iteration, and Rollout. Diagnose false positives, false negatives, and metric movements across indicators such as: Accuracy , Risk, False Rejection, Coverage
- Prioritize remediation actions, prepare Launch Review evidence, and support: Canary testing, A/B testing, Rollout decisions, Cross-queue launches
- Build and Scale the PE Operating System. Standardize the full operating workflow: Intake → Sampling → Labeling → QC → Evaluation → Release
- Build program-tracking and multi-queue delivery mechanisms so that new prompts and maintenance cycles are delivered predictably and at scale.
- Drive Continuous Improvement from Production Signals. Convert online bad cases, RCA findings, and hard-policy constraints into: Prompt updates, Skill updates, Data updates, Reusable playbooks, Clear separation between confirmed conclusions and hypotheses

Qualifications

Minimum Qualification(s):
- Bachelor's degree or above.
- 3+ years of experience in AI/ML data operations, prompt engineering, content-moderation strategy, annotation operations, technical program management, strategy, or related fields.
- Hands-on experience designing or operating prompts and LLM workflows for production systems, including training, evaluation, or online inference.
- Proven experience building evaluation frameworks, including golden sets, benchmarks, labeling SOPs, quality control, offline and online metrics, and experiment-ready evaluation packages; strong skills with a proactive approach to identifying and driving process improvements.
- Strong experience working with labeling vendors or internal annotation teams.
- Experience in taxonomy and specification design.
- Experience in quality control and delivery management.

Preferred Qualification(s):
- Experience in content governance, Trust & Safety, or other high-stakes decision systems.
- Exposure to multimodal model evaluation across video, image, text, URL, or webpage content.
- Exposure to Agent systems, including planners, skills, tool use, and chain-of-thought or structured decision-making.
- Familiarity with A/B experiments, metric decomposition, numerator and denominator analysis, mix-shift analysis, and case-level RCA.
- Experience productizing internal tools for prompt debugging, case replay, and data-iteration platforms.
- Experience owning zero-to-one operating systems, including SOPs, workflow tools such as Meego or Jira, dashboards, and scalable delivery processes.
- Basic SQL and data-analysis capability.
- Python scripting experience for evaluation pipelines is a plus.
- PMP or equivalent program-management experience is a plus.

Client-provided location(s): Singapore

Job ID: TikTok-7673405628338145541

Employment Type: OTHER

Posted: 2026-08-13T22:25:59