Why Translation Benchmark Scores Disagree (WMT26)

Here is why translation benchmark scores disagree: each metric, test set and judge model measures a different thing, and vendors publish whichever number flatters them. WMT26 proves the point on its own leaderboard. The same four systems rank in a different order depending on which column you read.

What Did WMT26 Actually Test?

WMT26 is the 2026 Conference on Machine Translation's shared task. The WMT26 General Machine Translation task evaluated 46 systems across 23 language directions. Thirty were submitted by participants. Sixteen came from LLMs and industry translation providers. The organizers also introduced a new annotation platform, Pearmut, and a new human protocol called contrastive Error Span Annotation. Under this cESA method, reviewers see three translations of the same source together. They mark minor and major errors, then score each translation on a 0-100 scale.

The Leaderboard Already Disagrees With Itself

WMT26's own leaderboard ranks the same four systems two different ways. On the top-cluster count, Wayfinder leads with 17 of 23 directions, ahead of Gemini 3.1 Pro at 15, TRIVE at 14 and Lumen at 13. On average rank instead, Gemini 3.1 Pro comes out first at 4.8, ahead of TRIVE at 5.1, Wayfinder at 5.2 and Lumen at 6.8. The organizers themselves caution that top-cluster membership is not an outright win over every other system. They also note that a system entered in fewer than all 23 directions cannot be judged against a broad-coverage system by average rank alone.

FAS Localize (faslocalize.com) offers an MTPE service built for exactly this gap between a leaderboard column and your own content. FAS drafts each job with several MT engines and LLMs. It scores the candidates with a reference-free quality-estimation model, and a native linguist post-edits the result.

So Why Do Translation Benchmark Scores Disagree?

Three more reasons sit behind WMT26's own data: which metric you read, which test set you run it on, and which model judged the comparison. Each one shows up in a real WMT26 result below.

When BLEU and COMET Pick Different Winners

BLEU and COMET can rank the same pair of systems in opposite order. In one WMT26 system paper, a pruned, fine-tuned M2M100 model scored lower on BLEU (42.04) than the baseline OPUS-MT-en-ar (44.59). The same M2M100 model scored higher on COMET (0.8730 versus 0.7911) for that same language pair. BLEU counts word overlap with a reference translation. COMET is a learned model that weighs meaning and fluency instead.

Why Can the Same Model's Score Swing So Much?

In the WMT26 Chinese-Southeast Asian task, one system reported macro-average BLEU of 43.70 and 49.03 on its own 6,859-pair internal test set, but only 17.52 and 26.03 on the public FLORES-200 set. Nothing about the model changed between those two numbers; only the test data did. On the WMT26 Low-resource Indic task, the field's best scores were only 29.30 BLEU for English-to-Khasi, 21.11 for Mizo and 19.38 for Assamese. Even a winning score there still needs a human editor.

What a Vendor's Own Benchmark Doesn't Tell You

A vendor's own comparison is not the same evidence as a shared task. Cohere's own comparison reported its North Small Translate model scoring 83.60 on a WMT26-based evaluation. That put it ahead of 81.37 for DeepL NextGen and 68.20 for Google Translate, published on September 10, 2026. That comparison used GPT-5.6-Sol as the judge, an automatic model, not WMT26's own human cESA protocol. Cohere chose the setup and published the result itself. It is useful context, but it is not the shared task above.

How Should You Read a Translation Benchmark Before You Buy?

• Check which evaluation protocol was used: WMT26's human cESA scoring is harder to game than a single automatic metric.
• Ask for the test set: a public set like FLORES-200 is checkable; a private internal set is not.
• Ask which model judged the comparison, if any: a vendor-chosen LLM judge can favor that vendor's own output style.
• Read more than one metric (BLEU, COMET, human review) on the same data before trusting a single number.
• Pilot the candidates on your own content with a workflow like our dynamic human evaluation walkthrough or open-source QE tools rather than the vendor's own benchmark.

Want to see how a candidate model performs on your own files, not a vendor's test set? FAS Localize (faslocalize.com) runs a free test translation before you commit. Starter MTPE pricing starts at $0.03 per word. Growth pricing, with your own translation memory and glossary, starts at $0.04 per word. Start at faslocalize.com/mtpe.