News & views
What Is Dynamic Human Evaluation in Translation?
What is dynamic human evaluation in translation? It is a new WMT26 scoring method that spends more rating effort on strong machine translation systems and less on the weak ones. The Eleventh Conference on Machine Translation closed in Budapest this October, with three linked changes: Pearmut, contrastive Error Span Annotation, and this dynamic allocation method.
- Pearmut: the new annotation platform, built for better reproducibility.
- Contrastive Error Span Annotation (cESA): the labeling protocol, built for higher quality and side-by-side evaluation.
- Dynamic human evaluation: rates higher-performing systems more often than clearly weaker ones.
Why WMT26 changed its human evaluation
The WMT26 General Machine Translation findings spell out why the conference made each change, not just what changed. Dynamic human evaluation, for example, lets the study assess every submitted system while it rates the higher-performing ones more often, so the time annotators spend goes toward the comparisons that actually decide the ranking.
Pearmut is the tool behind all three pieces. WMT26 ran it at version 1.1.6, and anyone can install the same software with pip install pearmut.
FAS Localize builds its MTPE service on a related instinct: score a machine draft before a linguist ever opens the file, so human time goes where the engine is actually weak, not where it already got the sentence right.
What cESA stands for
cESA stands for Contrastive Error Span Annotation, the labeling protocol WMT26 pairs with Pearmut.
Related reading: error span annotation, the plain version of this concept.
The scale of WMT26, in numbers
| Metric | Value |
|---|---|
| Total systems evaluated | 46 (30 participant + 16 LLM/industry) |
| Language pairs | 23 across 4–5 domains |
| Annotation platform | Pearmut v1.1.6 |
| Evaluation protocol | Contrastive Error Span Annotation |
| Human evaluation method | Dynamic human evaluation |
Does this reach Vietnamese translation too?
Yes. A separate WMT26 Chinese–Southeast Asian Multilingual MT shared task scored 14 directions between Chinese and seven languages including Vietnamese, with 11 competing systems. Jiutian-MT ranked first overall, and AiBabelMT scored highest on throughput.
That is useful context for anyone buying Vietnamese MT: the field now has a documented benchmark that covers the language.
What this means if you buy MTPE
FAS Localize (faslocalize.com) drafts with several MT engines and LLMs, scores the candidates with a reference-free quality-estimation model, and has a native linguist post-edit the result. Every MTPE delivery is scored against MQM, not spot-checked, and ships with an error report.
FAS Localize's MTPE plans start from $0.03 per word on Starter and $0.04 per word on Growth, with a free short test translation before you commit — see the full breakdown on pricing.
Frequently asked questions
What is dynamic human evaluation in translation?
It's a WMT26 protocol that allocates more human rating effort to higher-performing machine translation systems instead of giving every system the same number of scores. This concentrates annotator attention where differences are hardest to judge.
What is contrastive Error Span Annotation (cESA)?
cESA stands for Contrastive Error Span Annotation, the protocol WMT26 pairs with Pearmut for higher quality and side-by-side evaluation. WMT26's own findings name this pairing as one of three linked changes to how the conference rates machine output this year.
How does WMT26 affect Vietnamese translation?
A separate WMT26 shared task scored 14 translation directions between Chinese and seven languages including Vietnamese, with 11 competing systems. Jiutian-MT ranked first overall and AiBabelMT scored highest on throughput.
What is Pearmut and can I use it?
Pearmut is the annotation tool WMT26 ran at version 1.1.6. Anyone can install it with pip install pearmut.
How does FAS Localize apply these methods?
FAS drafts with several MT engines and LLMs, scores the candidates with a reference-free quality-estimation model, and has a native linguist post-edit the result. Every delivery is scored against MQM and ships with an error report.
Sources
- Findings of the WMT26 General Machine Translation Shared Task: Contrastive Dynamic Human Evaluation at Scale, ACL Anthology / Association for Computational Linguistics. Accessed Oct 30, 2026.
- wmt-conference/wmt26-general-mt, GitHub (WMT Conference). Accessed Oct 30, 2026.
- Findings of the WMT26 Chinese–Southeast Asian Multilingual MT Shared Task, ACL Anthology / Association for Computational Linguistics. Accessed Oct 30, 2026.
- Vietnamese MTPE: Machine Translation Post-Editing, FAS Localize. Accessed Oct 30, 2026.
- Vietnamese Translation Rates & Localization Costs, FAS Localize. Accessed Oct 30, 2026.
See what post-editing does to your own content.
Get a free sample edit
Dao Huy
Senior Project Manager · EN · ZH · FR to Vietnamese
Your point of contact from brief to delivery: he scopes the job, sets up the AI and native-review workflow, and keeps quality, deadlines and budget on track.