About the job
Do you want to decide whether a generative imaging feature is good enough to ship? Our team evaluates generative image and video features. These features reach a very large number of people on Apple devices. You will build the evaluation systems, human-annotation programs, and automated metrics that turn "this looks better" into measurable, trustworthy quality signals. Those signals guide model development and release decisions. You'll own quality evaluation end to end — from evaluation set design through annotation, scoring, analysis, and the recommendations that shape what engineering and product build next. Your findings determine which models ship.
What you'll do
Build the measurement foundation. Design and maintain golden sets and evaluation sets, analyzing data distributions so results stay representative and known defects stay covered.
Run human evaluation at quality. Write and iterate annotation guidelines and label schemas, build or adapt lightweight labeling tools for image and video review, monitor annotator quality, and resolve disagreements and guideline ambiguity.
Automate the repeatable parts. Develop automated metrics and LLM-as-judge systems, calibrate them against human labels, and track where they diverge.
Make the results decision-ready. Report model-vs-model comparisons with confidence intervals, significance testing, and breakdowns by hotspot, content type, and failure mode — then turn them into root-cause findings and prioritized recommendations.
Set the bar with your partners. Work with ML, engineering, and design to define quality standards, and communicate results clearly to technical and non-technical audiences alike.
What you'll do
Build the measurement foundation. Design and maintain golden sets and evaluation sets, analyzing data distributions so results stay representative and known defects stay covered.
Run human evaluation at quality. Write and iterate annotation guidelines and label schemas, build or adapt lightweight labeling tools for image and video review, monitor annotator quality, and resolve disagreements and guideline ambiguity.
Automate the repeatable parts. Develop automated metrics and LLM-as-judge systems, calibrate them against human labels, and track where they diverge.
Make the results decision-ready. Report model-vs-model comparisons with confidence intervals, significance testing, and breakdowns by hotspot, content type, and failure mode — then turn them into root-cause findings and prioritized recommendations.
Set the bar with your partners. Work with ML, engineering, and design to define quality standards, and communicate results clearly to technical and non-technical audiences alike.
