Human Data Quality Analyst, AI Business
Human Data Quality Analyst, AI Business
Prolific
Prolific isn’t just enabling AI innovation - we’re redefining it. While foundational AI technologies are becoming commoditized, Prolific’s human data infrastructure provides the high-quality, diverse data required to train the next generation of AI models. Through our platform, we empower researchers and companies to access a global, ethically curated participant base, ensuring cutting-edge AI research and training grounded in inclusivity and precision.
The Role
Prolific provides the human data that powers the next generation of AI models, working with frontier labs to capture the complex human judgments researchers need to train, evaluate and improve them. As a Human Data Quality Analyst, you'll be on the front line of making sure that data captures the right signal and is genuinely good enough to do its job.
This isn’t traditional, back-office QA. You’ll be doing real analytical work: digging into datasets, identifying patterns and failure modes, investigating why quality has shifted, and turning complex findings into clear insights that help us improve how data is collected, reviewed and delivered. You'll spend real time reading annotations closely, but that is how you gather evidence, not what you produce. What you produce is analysis, practical recommendations and better quality controls.
You'll get hands-on exposure to human data, annotation, machine learning pipelines and AI evaluation, working alongside Quality, Engineering, Operations and Delivery on new and evolving problems. There won't always be an established playbook. You'll be guided by our quality engineers, but you'll also need to run your own analysis, test your assumptions and recognise when you need input.
It is a role with a steep learning curve from day one and a strong opportunity for someone early in their career to build deep, practical experience in a fast-moving area of AI.
What You’ll Be Doing
- Run day-to-day quality measurement across live human-data programmes, combining statistical analysis with regular hands-on review to determine whether the data is fit for its intended use.
- Find the signal behind quality issues by investigating changes, quantifying their impact and distinguishing individual errors from wider problems in guidance, task design, training or tooling.
- Help build the data-quality pipeline alongside our quality engineers, turning repeated analysis into reliable validations and automated systems, including rule-based, LLM, and ML assisted approaches.
- Turn complex analysis into a clear story through reporting and visualisations that help stakeholders understand what is happening, why it matters, and what action the evidence supports.
- Support new programmes from design through to launch by helping develop the rubrics, guidelines, gold sets, calibration exercises, and production gates to measure quality from the start.
- Use what we learn from live programmes to improve annotation design, guidance, training and quality processes, then measure whether those changes work.
What You’ll Bring to the Role
- You have 1 to 2 years of practical experience analysing real data in an analytical, quality, research or data-focused role.
- You can use SQL and Python to write your own queries and analysis, check whether the results make sense, and explain how you reached them.
- You can apply statistical concepts such as sampling, distributions and variability, and judge whether a result reflects a meaningful pattern or limited evidence.
- You can turn evidence into a clear account of what is happening, supported by concrete examples and practical recommendations that non-technical colleagues can act on.
- You can assess detailed work carefully and consistently while looking for patterns across the wider dataset, process or programme.
- When the right method is unclear, you can test a sensible approach, document your reasoning and recognise when you need input.
Even Better if you have
- Experience working with data produced or judged by people, such as annotation, data labelling, research coding, content review or survey research, including measures such as agreement, gold-set accuracy or defect rates.
- Knowledge of psychology, behavioural science, linguistics or a related field, particularly how people interpret meaning and make judgements.
- An understanding of how human data supports model training and evaluation, including supervised fine-tuning, reinforcement learning from human feedback (RLHF), preference data and evaluation benchmarks.
- Experience creating clear data visualisations or dashboards that help people understand and act on findings.
Why Prolific is a great place to work
We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems.
We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation - one that reflects the breath and the best of humanity.
Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research.
Join us to enjoy a competitive salary, benefits, and hybrid working within our impactful, mission-driven culture. At Prolific, our compensation packages for eligible roles include base salary, equity, and benefits. Many roles also include the opportunity to earn a cash variable element, such as a bonus or commission. Your recruiter will also be happy to share the specific salary range for your preferred location during the hiring process.
Links to More Information
Privacy Statement
By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.