As part of the work on next generation Siri, we are developing novel measurements of its quality. To ensure that the evaluation systems we are building are reliable, we plan benchmarking their accuracy on a wide range of features, locales, and platforms using humans in the loop.

Minimum Qualifications

Agentic Coding proficiency to achieve data-science, data collection and visualisation tasks
Good understanding of metrics, crowd science, data collection, annotation analysis, statistics
Ability to work independently and cross-functionally to integrate in partner team reporting systems and pipelines
Excellent communication skills and the ability to thrive in a highly collaborative work environment

Preferred Qualifications

Attunement to computational linguistics, language quality, human in the loop evaluation
Good engineering practices to create sustainable and easy to use data management pipelines
Python experience and other tools for data collection and visualisation