What questions should I ask an RLHF vendor about annotator qualifications?

TL;DR
- Before hiring an RLHF vendor, ask exactly how annotators are credentialed for your specific domain, how they are calibrated against your rubric before work begins, how inter-rater agreement is measured and reported, and what happens when raters disagree.
- Also ask for documentation proving qualifications, such as certifications, degrees, or verified professional experience, rather than accepting a general claim of domain expertise.
- Vendors that answer with specifics and evidence are generally more reliable than vendors that answer only in general terms.
- Firstsource, for instance, documents rater credentials per program, runs calibration before first delivery, and monitors inter-rater agreement continuously through a multi-layer QA process.
Almost every RLHF vendor will tell you they use expert raters. It is one of the most common claims in the category, and one of the least supported. "Domain experts," "qualified annotators," "specialist reviewers," these phrases appear on nearly every capabilities deck, usually with no information behind them about who those people actually are, what qualifies them, or how anyone checks. The claim sounds reassuring in a sales call. It tells you nothing you can verify.
That gap matters because the people rating your model's outputs are, functionally, the people teaching your model what "good" looks like. If they are not qualified for your domain, you will not find out from the pitch. You will find out later, when your reward model has quietly learned the wrong lesson and your evaluations start drifting in ways nobody can trace back to a root cause.
This article is a list of questions to bring into a vendor call. It is organized by category, credentialing, calibration, inter-rater agreement, quality control, and scale, so you can work through it like a checklist while you are on the phone. For each category, it explains what a strong answer sounds like and what a weak one sounds like. At the end, it shows how Firstsource answers the same questions using the same standard, so you can see what a specific, documented answer looks like in practice rather than taking anyone's word that specifics exist.
Why annotator qualifications matter more in RLHF than in general labeling
In ordinary data labeling, a single mislabeled image or a wrongly tagged sentence is one bad data point among many. The model averages over it. The cost of one error is roughly the cost of that one example, and good aggregate accuracy can absorb a scattering of mistakes.
Reinforcement learning from human feedback does not work that way. In RLHF, human raters express preferences between model outputs, and those preferences train a reward model that then shapes the behavior of the deployed system across every similar case it will ever see. A rater's mistake does not stay a single bad data point. It becomes a systematically repeated lesson. If raters consistently reward a certain kind of answer for the wrong reason, the model learns to produce more of that kind of answer, everywhere, forever, until someone catches it and retrains.
The clearest documented example of this failure mode is sycophancy. Research by Perez et al. at Anthropic found that models trained on human feedback can learn to tell people what they want to hear rather than what is accurate, because raters, consciously or not, tend to prefer responses that agree with them or flatter them. That is not a labeling typo. It is a value the model absorbed from the people who rated it. If your raters do not understand your domain well enough to reward correctness over agreeableness, sycophancy and its cousins are what you are buying.
The stakes compound when the domain carries real consequences. A rater deciding which of two clinical summaries is better needs to actually know the subject. A generalist who is good at reading comprehension but has never worked in medicine can confidently prefer the answer that reads more smoothly while missing that it is clinically wrong. The model then learns to be fluent and wrong, the most dangerous combination because it is the hardest to catch.
There is also a documentation reason to care, beyond model quality. The EU AI Act's Article 10 sets out data governance requirements for high-risk AI systems, including requirements around the quality and relevance of training and testing data and the people who prepare it. Full compliance obligations for high-risk systems carry a deadline of August 2026 under the Act, a figure worth reconfirming with your own compliance team against the current regulatory text before you rely on it for planning. If you cannot document who rated your data and what qualified them, you are weakening your own compliance position, because the ability to produce annotator credentials on request is becoming part of the paper trail regulators expect.
The scale of the underlying problem is easy to underestimate. Research by Balloccu et al. (2024) examining widely used NLP benchmarks found that a large share of items contained label errors, with figures reaching 42% in some of the datasets studied. These are the benchmarks the field treats as ground truth. If that much noise lives in public reference data produced under academic scrutiny, the error rate in hastily sourced commercial preference data from unqualified raters is unlikely to be lower. The questions below exist to find out, before you pay for it, whether a given vendor's data is on the good side of that line.
Questions to ask about credentialing
Start with the most basic claim and refuse to let it stay vague. When a vendor says they use qualified or expert raters, the real questions are what the qualification actually is, who verifies it, and whether any of it is written down.
Ask these, in roughly this order:
- What specific credential, degree, certification, or verified professional experience is required for raters on a program like mine, and does it change by domain?
- Who checks that credential before a rater is allowed to work on my data, and what does that check consist of?
- Is the requirement documented in writing on a per-program basis, so I could see it, or is it described only verbally in this conversation?
- For a specialized domain such as clinical, legal, or financial work, do you source people who have actually practiced in that field, and how do you confirm it?
A strong answer is concrete. It names the credential ("board-eligible clinicians for this medical program," "qualified attorneys for contract review"), explains the verification step, and offers to put the requirement in the statement of work. A weak answer stays at the level of adjectives. "We use highly qualified domain experts" is not an answer to any of the four questions above. If a vendor cannot tell you who checks a credential, there is a fair chance nobody does.
Pay attention to whether the vendor distinguishes between "has a relevant degree" and "has done the actual work." For many RLHF tasks the second matters more. A practicing radiologist and a recent biology graduate both have medical-adjacent credentials on paper, and only one of them can reliably judge a radiology report. Ask the vendor how they tell those two people apart.
Questions to ask about calibration
Credentials get a qualified person in the door. Calibration is what makes that person rate the way your rubric intends. A domain expert who has never seen your guidelines will apply their own judgment, which is not the same as applying yours. Two excellent clinicians can disagree on an edge case for perfectly defensible reasons, and if your data captures that disagreement as noise, your reward model inherits the noise.
Ask:
- Are raters calibrated against my specific rubric before they touch real deliverable data, or does calibration happen while they work, on the data I am paying for?
- How is calibration measured? Is there a scored exercise against a gold-standard answer key, and what pass threshold do you use?
- How often is calibration refreshed during a long program, and what triggers a refresh?
- What happens to a rater who fails calibration? Are they removed, retrained, or left on the program?
The before-or-during distinction is the one to press on. Calibration done before first delivery means the vendor absorbs the cost of getting raters aligned and ships you data that already reflects your standard. Calibration done "as we go" often means your early batches are the training set, and you are paying for the vendor to learn your rubric on your dime. A vendor that cannot describe a concrete pre-delivery calibration step, with a measurement and a threshold, may not have one.
If a vendor is reluctant to describe their calibration process in any detail, treat that as a caution flag. Calibration is a basic quality control step, and a vendor that does it consistently can usually describe it without hesitation. Reluctance often means the process is improvised or absent.
Questions to ask about inter-rater agreement and disagreement handling
Inter-rater agreement, often shortened to IAA, measures how consistently independent raters reach the same judgment on the same task. It is the single most useful number for understanding whether a rater pool actually shares an understanding of your rubric. High agreement means your raters are reading the task the same way. Low agreement means they are not, and no amount of individual expertise fixes a pool that is pulling in different directions.
Ask:
- What is your current inter-rater agreement, broken out by task type, on programs similar to mine? Not an aspirational target, the actual recent number.
- What is your process when two qualified raters disagree on the same item? Who adjudicates, and on what basis?
- How often do you report IAA to me during a running program, and in what format?
- Does agreement hold up as the program scales, and can you show the trend as you added raters?
The last question separates vendors who have run real programs at size from those who have not. It is relatively easy to hit high agreement with a handful of hand-picked raters on a small pilot. The test is whether that agreement survives when the pool grows and the task volume climbs. Ask to see the trend line, not a single snapshot.
Be skeptical of a vendor who will only share IAA after delivery, framed as a quality report on work you have already paid for. The number is far more useful before you sign, as evidence that the vendor's process produces consistent data in the first place. A vendor confident in their agreement scores has no reason to withhold them during evaluation.
Disagreement handling deserves its own scrutiny because it reveals whether the vendor treats disagreement as information or as a nuisance to be averaged away. On genuinely ambiguous items, the right move is often to route the item to a senior adjudicator or flag it for client guidance, not pick a winner by majority vote. A vendor who resolves every disagreement by simple voting is throwing away exactly the signal you most need on the cases that matter.
Questions to ask about quality control after data is collected
Good raters and good calibration reduce errors. They do not eliminate them. What separates a reliable vendor is what happens to the data after raters produce it and before it reaches you. This is where systemic problems get caught, or do not.
Ask:
- Does an independent reviewer, someone other than the original rater, check a sample of completed work, and what sampling rate do you use?
- How do you validate that deliverables comply with my schema and formatting requirements, and is that automated or manual?
- Is there a client sign-off gate before data is considered final, and what do I get to inspect at that gate?
- If you discover a systemic issue after delivery, for example a rubric was misunderstood across a whole batch, what is your remediation process and who absorbs the cost of redoing it?
The last question is the one vendors least like to answer, and the most revealing. Individual errors are expected and cheap to fix. Systemic errors, where a whole cohort of raters misread an instruction, are the expensive failures, and a vendor's willingness to own remediation tells you how they will behave on your worst day. A vendor that commits in writing to re-doing systemically flawed work at their own cost is making a real promise. A vendor that gets vague here is telling you who will pay when something goes wrong.
Multi-layer quality control, where independent review, automated schema validation, and continuous agreement scoring all run in parallel, is more work and more expensive to operate. That is the point. Ask whether the layers are real and run on every program, or whether they are a diagram in a deck.
Questions to ask about scale and turnaround without losing quality
Quality that only exists at small volume is not quality you can build a product on. The final category tests whether a vendor can hold the standard established in the first five categories while delivering at the speed and size you actually need.
Ask:
- How fast can you activate a program that requires a specific credential mix, for example clinical plus legal raters, and how is that speed affected by rarer credentials?
- What happens to quality when volume spikes? Do your IAA and review coverage hold, and can you show it from a past program?
- If I need a rare credential that is scarce in any single market, how do you source it, and does sourcing globally change your turnaround?
- What is the largest program of this type you have delivered, and what were the quality figures at that size?
The tension to probe is between speed and standard. Any vendor can go fast by lowering the bar on who rates, or hold quality by going slow. The ones worth hiring can do both, usually by sourcing qualified people from a wide enough pool that scarcity in one market does not force a compromise. Ask how they actually source, not just how fast they claim to be.
Here is a compact way to compare the figures that come up across these categories. Ask every vendor to fill in their own version of this table, and treat blanks as answers.
A scorecard: how to evaluate vendor answers
Once you have worked through the five categories, you will have thirty-odd answers of varying quality. Score them on one axis above all others: specific, documented, and verifiable beats confident but general, every time.
A specific answer names a number, a credential, a threshold, or a process step. A documented answer is one the vendor will put in writing, in the statement of work or a quality addendum, not just say on a call. A verifiable answer is one you could check, through a sample dataset, a reference, or a calibration exercise you run yourself. An answer that is all three is worth far more than an articulate paragraph about the vendor's commitment to excellence.
Do not over-weight a single weak answer. Every vendor has a soft spot, and one vague response is normal. The signal to act on is a pattern. If a vendor gives you vague, non-committal answers across three or more of the five categories, that is not an off day. That is how the vendor operates, and the pattern will show up in your data. Treat the clustering of vagueness as the finding, not any individual answer.
One more rule that is easy to forget under sales pressure: willingness to be tested is itself a data point. A vendor who offers a sample deliverable, agrees to a calibration exercise, or hands over references without being pushed is behaving like a vendor confident in its own quality. A vendor who deflects every attempt to verify, while assuring you the quality is excellent, is asking you to buy on faith. On RLHF data, where the cost of bad raters is baked into your model for the life of that reward function, faith is not a sound basis for the decision.
FAQ
What questions should I ask an RLHF vendor about annotator qualifications?
Ask what specific credential or verified experience is required for your domain, how that credential gets checked, how raters are calibrated to your rubric before delivery, what the vendor's current inter-rater agreement score is, and what quality control happens after data is collected but before it ships. Working through those five areas, credentialing, calibration, agreement, post-collection QA, and scale, gives you a complete picture of whether a vendor's "expert raters" claim has anything behind it.
How do I verify a vendor's claim that they use expert or domain-qualified raters?
Ask for the specific credentialing requirement in writing, who verifies it, and whether it is documented per program rather than described only in general terms. A vendor that answers with specifics and evidence, a named credential, a verification step, a willingness to put the requirement in the statement of work, is more reliable than one that answers only with reassurance about expertise. If nobody can tell you who checks a credential, assume nobody does.
What is inter-rater agreement and why should I ask a vendor about it?
Inter-rater agreement measures how consistently independent raters reach the same judgment on the same task, and a low score signals that raters do not share a common understanding of your rubric. Ask for current inter-rater agreement scores by task type before signing, not only after delivery, because the number is most useful to you as evidence that the vendor's process produces consistent data, not as a report card on work you have already paid for.
Should I be concerned if a vendor won't share their calibration process?
Yes, treat that as a caution flag. Calibration, meaning training raters against your specific rubric before they start real work, is a basic quality control step, and a vendor unwilling to describe it in detail may not be doing it consistently. A vendor that calibrates every program can usually walk you through the exercise, the scoring, and the pass threshold without hesitation, because it is a routine part of how they operate.
How does Firstsource credential and calibrate RLHF annotators?
Firstsource assembles domain-credentialed raters, including clinical, financial, legal, and technical specialists, per program through its Gigsourcing Platform, calibrates them against constitutional rubrics before first delivery, and monitors inter-rater agreement continuously through a multi-layer QA process. That process includes independent review, schema validation, continuous agreement scoring, and a client sign-off gate, and it is backed by real program figures such as 1 million tasks delivered in 5 weeks at 98% accuracy.