What is inter-annotator agreement (IAA) scoring in AI evaluation?

Inter-annotator agreement (IAA) scoring explained: what it measures, how it's calculated, and why it matters for trustworthy AI model evaluation programs.
November 18, 2026
The Firstsource team
What is inter-annotator agreement (IAA) scoring in AI evaluation?

TL;DR

  • Inter-annotator agreement (IAA) scoring measures how consistently two or more independent evaluators reach the same judgment when grading the same piece of AI model output.
  • It is typically calculated with statistical agreement measures such as Cohen's kappa (for two raters), Fleiss' kappa (for more than two raters), or Krippendorff's alpha, all of which adjust for the agreement you would expect by chance alone.
  • A high IAA score means a rubric is clear and evaluators are applying it consistently; a low score usually signals an ambiguous rubric, insufficient evaluator training, or a task that is more subjective than the scoring method assumes.
  • Serious AI evaluation programs report IAA broken out by category and experiment, and use it alongside other calibration methods, such as Elo-style head-to-head comparison, to keep evaluation quality auditable over time.

Asking five qualified people to grade the same AI response raises an immediate question: how often do they actually agree, and what happens when they don't? Hand the same model output to five annotators with the same rubric, and you will rarely get five identical scores. Two might call it excellent, two might call it adequate, and one might flag it as unacceptable. Those gaps are not noise to be ignored. They are information about whether your evaluation is measuring anything stable at all.

That question is why inter-annotator agreement scoring exists, and why it shows up in nearly every serious AI evaluation methodology. If you are grading model outputs to decide which version of a system ships, to build a preference dataset, or to prove to a regulator that your evaluation was rigorous, you need to know the grades mean the same thing no matter who assigned them. IAA is the number that tells you. This guide walks through what IAA measures, how it is calculated, what counts as a good score, why it gets harder in multimodal work, what drives it down, and how to use it as a buyer.

What inter-annotator agreement measures

Inter-annotator agreement is a consistency check, not a correctness check. It answers one question: when independent evaluators grade the same item against the same rubric, how often do they land on the same judgment? It says nothing about whether that judgment is right.

This distinction matters more than it first appears. A group of evaluators can reach near-perfect agreement on a rubric that is quietly wrong. Imagine a rubric that tells raters to reward long, confident answers and penalize short, hedged ones. Five raters could apply that rule with total consistency and produce a beautiful agreement score, while systematically rewarding verbose answers that happen to be factually worse. The IAA number would look excellent. The evaluation would still be steering the model in the wrong direction.

So IAA and accuracy answer different questions. Accuracy compares a judgment to a known correct answer, a gold label. IAA compares judgments to each other. You need both, because they fail in different ways. High accuracy with low agreement usually means your raters are guessing and occasionally guessing right. High agreement with low accuracy means your raters are consistently applying a flawed standard. The healthy state is high agreement on a rubric you have separately validated as correct, which is why IAA is almost always used alongside other quality checks rather than on its own.

The practical value of IAA is that it makes the invisible visible. Disagreement among evaluators happens whether or not you measure it. Scoring it turns a vague worry into a number you can track across experiments, compare across categories, and act on when it drops.

How IAA is calculated

The simplest way to express agreement is percent agreement: out of every item two people graded, what fraction did they score identically? If two raters agree on 85 of 100 items, that is 85 percent agreement. It is easy to compute and easy to explain, and it has a blind spot serious enough that no careful program relies on it alone.

The blind spot is chance. Suppose a task has only two possible labels, "pass" and "fail," and both raters hand out "pass" about 90 percent of the time out of habit. Two raters guessing independently at those base rates would still agree a large share of the time purely by luck, with no shared understanding of the rubric at all. Raw percent agreement counts that lucky overlap as if it were real agreement, so it flatters tasks with few categories or lopsided label distributions.

The family of statistics built to correct for this is what serious programs report. Each one subtracts out the agreement you would expect by chance and rescales what is left.

  • Cohen's kappa handles the two-rater case. It takes the observed agreement, subtracts the agreement expected by chance given each rater's label frequencies, and divides by the room that was left above chance. A kappa of 0 means the raters did no better than chance; a kappa of 1 means perfect agreement beyond chance.
  • Fleiss' kappa generalizes the same idea to more than two raters, which matters when a panel of evaluators grades each item rather than a fixed pair.
  • Krippendorff's alpha is the most flexible of the three. It handles missing ratings (not every rater has to grade every item), works across different measurement scales (nominal categories, ordered ratings, continuous scores), and accommodates any number of raters, which makes it the common choice for messy real-world datasets.

A short worked example shows why the chance correction changes the story. Take two raters grading 100 items on a pass/fail task. They agree on 80 of them, so observed agreement is 0.80. Now look at how often each one says "pass." Rater A marks "pass" 70 times, rater B marks "pass" 75 times. By chance alone, they would both say "pass" on about 0.70 times 0.75, or 0.525, of items, and both say "fail" on about 0.30 times 0.25, or 0.075, of items. Expected chance agreement is 0.525 plus 0.075, which is 0.60. Cohen's kappa is the observed agreement minus the chance agreement, divided by one minus the chance agreement: (0.80 minus 0.60) divided by (1 minus 0.60), which is 0.20 divided by 0.40, or 0.50. So 80 percent raw agreement collapses to a kappa of 0.50 once you account for how easy it was to agree by luck. That gap between 0.80 and 0.50 is exactly what the chance correction is for, and it is why a program that reports only percent agreement is telling you less than it seems.

What counts as a good IAA score

There is no single universal threshold, and anyone who gives you one without asking what the task is should be treated with suspicion. The right target depends on how objective the task is.

Objective classification tasks can reasonably expect high agreement. If evaluators are labeling whether an image contains a stop sign, or whether a response contains personally identifiable information, the answer is close to a matter of fact, and a well-trained panel should land in near-perfect territory, often above 0.8. Subjective judgment tasks are different by nature. Ask evaluators which of two chatbot responses is more helpful, or whether a summary captures the "right" tone, and honest, well-trained people will disagree more, because the task genuinely contains judgment. A lower kappa there is not automatically a failure. It can be an accurate reflection of how much legitimate interpretation the task involves.

Field convention offers rough bands to orient yourself, as long as you treat them as a starting point and not a verdict. As a general methodological convention, scores above 0.8 are often read as near-perfect agreement, 0.6 to 0.8 as substantial, and anything below 0.4 as poor. These bands are useful shorthand, but applying them mechanically is a mistake. A kappa of 0.65 might be excellent for a nuanced preference task and mediocre for a binary safety classification. The number only means something once you know what was being graded.

Kappa band Common interpretation What it usually tells you
Above 0.8 Near-perfect agreement Reasonable for objective, well-specified classification tasks
0.6 to 0.8 Substantial agreement Common healthy range for many trained-panel tasks
0.4 to 0.6 Moderate agreement Worth investigating; may be fine for subjective tasks
Below 0.4 Poor agreement Usually a rubric, training, or task-design problem

Bands reflect general methodological convention for interpreting chance-corrected agreement, not a fixed standard; read them against the task being graded.

The more useful habit is to compare a program's IAA against its own history for the same task type, and to expect transparency about which tasks run hot and which run cool. A vendor that reports one blended agreement number across everything is hiding exactly the variation that tells you where the evaluation is weak.

Why IAA matters more in multimodal evaluation

Text classification is, relatively speaking, the easy case for agreement. The item sits on the page, the categories are usually discrete, and two careful readers can often reconcile a disagreement by rereading. Multimodal evaluation, grading voice, video, and image outputs, is where agreement gets harder and where measuring it carefully pays off most.

The reason is that multimodal tasks carry more subjective judgment per item. Rating the naturalness of a synthesized voice, assessing whether a video model reasoned correctly about where objects are in space, or judging how much useful detail an image caption captures all ask evaluators to make calls that resist crisp category boundaries. Two reasonable people can hear the same text-to-speech sample and split on whether a slightly flat intonation counts as "unnatural" or just "neutral." That is a real disagreement about a genuinely fuzzy quality, not a mistake.

Because each modality has its own sources of subjectivity, a single blended IAA number across a mixed workload actively hides problems. Suppose a program runs image labeling, which agrees at 0.85, alongside audio naturalness rating, which agrees at 0.45, and reports only the average, around 0.65. That blended figure looks "substantial" and tells you nothing true. It masks a serious consistency problem in the audio work behind a healthy image score. Tracking IAA separately per modality is what catches the modality-specific failure. Watching the audio-naturalness agreement sit at 0.45 is the signal that the naturalness rubric needs sharper anchors, more example clips defining each point on the scale, or a different scoring method better suited to a subjective quality. You only see that signal if you refuse to average it away.

What causes low IAA and how programs fix it

When agreement drops, it is almost always traceable to one of a few root causes, and each has a matching fix.

The most common culprit is an ambiguous rubric. If the scoring guide leaves real room for interpretation, raters will interpret, and they will interpret differently. The fix is to revise the rubric with sharper definitions and concrete anchor examples for each score, then recalibrate the panel against the new version. A close relative is insufficient training on edge cases. Raters may agree easily on clear-cut items and scatter on the hard ones precisely because no one showed them how to handle those cases. Focused edge-case training, walking the panel through the exact items that split them and resolving each one explicitly, tightens agreement where it actually breaks.

Evaluator fatigue is a quieter cause. Agreement that is healthy at the start of a long batch and decays by the end is a sign that raters are tiring, not that the task changed. Capping batch length and rotating evaluators keeps judgment fresh and stops late-batch drift from contaminating the data.

The last cause is different in kind: some tasks are simply more subjective than the scoring method assumes. When a task genuinely involves preference, no amount of rubric tightening will force a high kappa, because the disagreement is real and legitimate. The fix there is not to chase agreement but to add a complementary method suited to subjective judgment. Elo-style head-to-head comparison is the usual choice. Instead of asking raters to assign an absolute score, you ask them which of two outputs is better. People are far more consistent at relative comparisons than at absolute ratings, and aggregating many pairwise judgments into an Elo ranking produces a stable ordering even when single-item scoring would scatter. Using IAA to decide which tasks need this treatment, and then measuring whether the Elo approach restores consistency, is how a mature program handles subjectivity instead of pretending it away.

Why this matters for choosing an evaluation partner

Here is the practical payoff for a buyer. Asking a prospective evaluation vendor for a sample IAA report, broken out by category, is itself a qualification test, and it is one of the fastest ways to separate a real evaluation program from a generic annotation service.

The logic is simple. If a vendor cannot produce agreement numbers broken out by task category, it has no way to show that its raters are actually consistent. It can tell you the raters are experienced, that they passed a screening, that there is a QA layer. None of that is evidence that two of them would grade the same output the same way. The IAA report is that evidence. A vendor that tracks it by category can show you where agreement is strong, where it is weak, and what they do about the weak spots. A vendor that cannot produce the report is telling you, without meaning to, that it does not measure its own consistency. A reader who understands IAA well enough to ask for that report by category is already most of the way to being a qualified buyer.

This is where Firstsource's GenAI Data Services approach is built to answer the question directly rather than with general assurance. Inside the Agentic AI Studio, IAA is tracked automatically as part of the Monitor layer, which surfaces more than 35 KPIs per evaluation program, including inter-rater reliability. Outlier judges, raters whose scores diverge systematically from the panel, are flagged through those dashboards so they can be retrained or recalibrated before their judgments skew a dataset. And the rubrics those scores are built on are co-designed with client research teams, so the standard being measured is the client's standard, not a generic template. That combination, automatic per-program tracking, outlier detection, and client-co-designed rubrics, is what lets the program produce the category-level IAA report that a careful buyer should be asking for.

IAA in practice

Two Firstsource programs show how IAA scoring works when it is wired into a live evaluation pipeline rather than reported after the fact.

The first is an audio question-and-answer pilot run for an e-commerce AI lab. The program began deliberately small, 5,000 tasks spread across 8 content genres, and the point of starting small was to validate agreement before committing to volume. IAA was checked on that initial batch to confirm that evaluators were applying the rubric consistently across the different genres. Only once agreement held up did the program scale, growing to 50,000 pairs. The sequence matters: validating consistency on a 5,000-task sample before expanding to 50,000 is what keeps a volume program from quietly producing 50,000 tasks' worth of inconsistent grades.

The second is a large voice AI preference program. It involves more than 300,000 preference evaluations across 9 or more languages, which works out to roughly 30,000 evaluations per language. Running at that scale across many languages is exactly the situation where a single blended agreement number would be misleading, because consistency can hold in one language and slip in another for reasons that have nothing to do with the others. Tracking IAA per language, rather than as one global figure, is what makes it possible to catch a language-specific drop and address it in that language alone. The voice AI program treats per-language agreement as the unit of measurement for precisely that reason.

Both programs illustrate the same principle. IAA is most useful when it is broken out, by genre, by language, by modality, and used as a gate on whether to scale, not as a single headline number reported once at the end.

IAA and regulatory documentation

There is a compliance reason to keep rigorous IAA records, and it is becoming concrete rather than theoretical. The EU AI Act sets documentation requirements for the data used to evaluate high-risk AI systems. Under the Act, these evaluation-data documentation obligations tied to Article 10 are set to apply from August 2026, with penalties that can reach up to 3 percent of global annual revenue for noncompliance (source: EU AI Act, Article 10). These figures and the exact scope of the obligation should be reconfirmed against the final regulatory text and with your own compliance team before you rely on them, since timelines and thresholds in this area are still settling.

What IAA offers in that context is auditable evidence. A regulator asking you to document that your evaluation process was rigorous and repeatable is asking a question that category-level agreement scores answer well. An IAA report shows that your evaluation was not one person's opinion but a process that independent, trained evaluators applied consistently, with the disagreements measured and the outliers caught. That is the difference between claiming your evaluation was careful and being able to show it. As the Article 10 documentation expectations take effect, the programs that already track IAA per category and per experiment will have that evidence on hand, and the ones that report a single blended number, or none at all, will be reconstructing it under deadline.

‍

FAQ

What is inter-annotator agreement (IAA) scoring in AI evaluation?

IAA scoring measures how consistently independent evaluators reach the same judgment when grading the same AI model output, using statistical measures like Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha that correct for chance agreement. It tells you whether a rubric is being applied consistently, not whether the judgment itself is correct.

What is a good IAA score?

There is no single universal threshold. Objective classification tasks can reasonably expect high agreement, while subjective judgment tasks, like comparing which response is more helpful, naturally produce lower agreement even among well-trained evaluators. What matters more is that a program reports IAA transparently across both types of tasks.

How is IAA different from accuracy?

Accuracy measures whether an evaluator's judgment matches a known correct answer, while IAA only measures whether evaluators agree with each other, regardless of whether that shared judgment is correct. A group of evaluators can have high IAA while still being consistently wrong if the rubric itself is flawed.

Why does low IAA happen, and how do teams fix it?

Low IAA usually points to an ambiguous rubric, insufficient evaluator training, evaluator fatigue, or a task that is more subjective than the scoring method assumes. Teams typically fix it by revising the rubric, running additional calibration sessions, and adding a complementary method like Elo-style head-to-head comparison for inherently subjective tasks.

Why does IAA matter when choosing an AI evaluation vendor?

If a vendor cannot produce a sample IAA report broken out by category, it has no way to show its raters are actually consistent, only that they were hired. Asking for this report is one of the fastest ways to separate a real evaluation program from a generic annotation service.

Recent Blogs

What is named entity recognition (NER) annotation used for in LLM training?

What is named entity recognition (NER) annotation used for in LLM training?

Technology
November 18, 2026
What are the main types of data annotation used in computer vision vs. NLP?

What are the main types of data annotation used in computer vision vs. NLP?

Technology
November 18, 2026
What are the best providers of pre-built, licensed AI training datasets?

What are the best providers of pre-built, licensed AI training datasets?

Technology
November 18, 2026
Technology