Skip to content
FAS Localize

Glossary

What is error span annotation in MT evaluation?

Dao HuyBy , Senior Project Manager4 min readFacts checked Oct 6, 2026
A reviewer does error span annotation in red on a printed French text headed "Des machines vont s'occuper des PR", next to a laptop
Photo by cottonbro studio on Pexels

Error span annotation (ESA) is a human method for scoring machine translation. A reviewer marks every error, labels each one major or minor, then rates the segment from 0 to 100. It is faster than MQM. WMT, the main MT benchmark, now runs its human evaluation on a variant of it.

Definition

Error span annotation (ESA): A human evaluation protocol in which annotators mark the problematic parts of a translation (a character, word, phrase or whole sentence), give each span a major or minor severity, and then score the segment from 0 to 100.

Marking error spans is also how FAS Localize reports quality to clients. Every MTPE delivery from faslocalize.com is scored against MQM and ships with an MQM error report, so you see each marked error and its severity, not just a number.

How does error span annotation scoring work?

  1. Mark the errors

    The annotator highlights everything that is wrong in the translated text, from a single character to a whole sentence.

  2. Grade each span

    Each span is major (the meaning changed, for example) or minor (grammar or style, for example). There are only two levels.

  3. Score the segment

    With the errors in front of them, the annotator gives the segment one score from 0 to 100.

The order is deliberate. According to the ESA authors, marking errors first primes annotators to score more accurately. They judge the segment with the evidence already on the page.

Error span annotation vs MQM: which one when?

ESAMQM
What the annotator recordsError spans (major or minor) plus a 0-100 scoreAn error type and a severity for every issue
Needs MQM-trained annotatorsNoYes
Median time per paragraph29 s38 s
Average of each annotator's median34 s (32% faster)49 s
Error-type breakdownNoYes
Timing figures from the 2024 ESA study.

ESA is quicker and more consistent, but it does not diagnose. In the 2024 ESA study, median time per paragraph fell from 38 seconds with MQM to 29 seconds with ESA, a 23% cut. Agreement between annotators was also higher. However, ESA gives no breakdown of error types, because its goal is evaluation. In our view the split is simple. Use ESA to compare engines, prompts or vendors quickly. Use MQM when you need to know *why* quality dropped.

What does ESA say about translation jobs?

Human evaluation of MT now looks like review work. The WMT25 general task evaluated 60 systems, mostly with ESA and professional annotators. For WMT26, human evaluation runs in the Pearmut tool with contrastive ESA. Its reference translations also encourage post-editing of machine output.

So even at the top benchmark, human work has shifted from writing translations to reviewing machine output. Reviewers find what is wrong and judge how serious it is. Beyond that, Pearmut offers an AI mode that pre-fills error spans for the annotator to confirm or correct.

A reader checks a research paper on a laptop, the screen-based review work behind error span annotation
Photo by Mikhail Nilov on Pexels

That is the job our linguists do at FAS in machine translation post-editing. An MT engine drafts, and a native linguist corrects and refines the text. Every delivery is scored against MQM and ships with an error report. To see how we score segments without a reference, read our guide to reference-free vs reference-based QE.

What to ask an MTPE vendor

  • Do your reviewers mark errors before they score, or do they give one overall impression?
  • Do you report major and minor errors separately?
  • Can I see a sample error report, not just a quality percentage?
  • When a score drops, can you tell me which error types caused it?

Want to see an error report on your own text? Send a short sample for a free test translation through our MTPE service. A native linguist post-edits the AI draft, and you get the marked errors back with the file. Rates start at $0.03 per word, see pricing.

Frequently asked questions

Is ESA the same as MQM?

No, they differ in detail. MQM records an error type and a severity for every issue. ESA records only where the error is and whether it is major or minor. It then adds one overall score, which is why it is faster.

Do ESA annotators need special training?

No MQM training is needed. ESA does not require annotators trained in MQM error categories. Annotators still need to read both languages well enough to spot a changed meaning.

Can AI pre-mark the errors?

Yes, in some tools. The Pearmut tool has an ESA AI mode where a model pre-fills the error spans. The annotator then reviews, edits or deletes them before scoring.

Sources

  1. Error Span Annotation (full text, v2), arXiv. Accessed Oct 6, 2026.
  2. Findings of the WMT25 General Machine Translation Shared Task, ACL Anthology. Accessed Oct 6, 2026.
  3. WMT26 General Machine Translation Task, WMT. Accessed Oct 6, 2026.
  4. zouharvi/pearmut README, GitHub. Accessed Oct 6, 2026.
  5. MTPE Services: Machine Translation Post-Editing, FAS Localize. Accessed Oct 6, 2026.
  6. MQM Framework, MQM Council. Accessed Oct 6, 2026.

See what post-editing does to your own content.

Get a free sample edit

Dao Huy

Senior Project Manager · EN · ZH · FR to Vietnamese

Your point of contact from brief to delivery: he scopes the job, sets up the AI and native-review workflow, and keeps quality, deadlines and budget on track.