Skip to content
FAS Localize

Comparison

Reference-free vs reference-based QE: what each measures

Dao HuyBy , Senior Project Manager4 min readFacts checked Oct 7, 2026
Two people comparing printed pages next to an open laptop, the kind of side-by-side check a reference-based quality score performs
Photo by Alexander Suhorucov on Pexels

Reference-free vs reference-based QE is a choice between two scoring jobs. Reference-based metrics compare a machine translation with a human reference. QE without references judges the output alone, using only the source text. The first is more precise per sentence, while the second works when no reference exists.

What does reference-based scoring check?

Reference-based scoring measures how close an output is to a human translation. For example, COMET's default model, wmt22-comet-da, is an XLM-R regression model trained on WMT17 to WMT20 direct assessments, according to the Unbabel/COMET repository. It returns scores from 0 to 1, where 1 means a perfect translation. However, the method needs a trusted reference for every new output. Without one, it simply cannot run.

What does QE without references check instead?

Quality estimation judges a translation with no human reference at all. The same COMET repository shows a reference-free command that drops the reference file and uses a model such as wmt22-cometkiwi-da. Meanwhile, newer systems add structure to this job.

FACET is a reference-free submission to the WMT26 Automated Translation Quality Evaluation Task. It splits evaluation into three passes:

  • Fluency: does the output read naturally?
  • Accuracy: does it keep the source meaning?
  • Consistency: does it stay consistent across segments?

A single fixed model is prompted three times, with no trained components. In FACET's rankings, post-edited human translation comes first. Also, the Consistency pass changes about a tenth of segment scores, yet the system ranking stays nearly unchanged.

Reference-based vs reference-free: the trade-offs

Each approach wins in a different setting. The table sums up the main differences.

DimensionReference-basedReference-free (QE)
Needs a human referenceYesNo
Segment-level results (WMT25)Baselines still leadLLMs trail
System-level results (WMT25)StrongStrong for large LLMs
Typical useOffline tests, model selectionLive monitoring, routing
Example modelsCOMET wmt22-comet-da, MetricX-24COMET wmt22-cometkiwi-da, MetricX-24, FACET
Example score scaleCOMET: 0 to 1, higher is betterMetricX-23: 0 to 25, lower is better
Core trade-offs between reference-based and reference-free scoring

The WMT25 shared task findings back this split. Large LLMs performed strongly at the system level. Even so, reference-based baseline metrics still outperformed LLMs at the segment level. The paper's title says it plainly: "Linguistic Diversity is Challenging and References Still Help."

Can one model run both modes?

Yes, some models now do both. The google-research/metricx repo says the MetricX-24 models are all hybrid. So the same model can score with a reference or without one. By contrast, MetricX-23 scores on a 0 to 25 scale where lower is better, because it predicts an error score.

How does FAS use reference-free QE?

In short, we use QE where no reference exists yet. FAS drafts with several MT engines and LLMs, a reference-free quality-estimation model scores the candidates, and a native linguist post-edits the result. Every MTPE delivery is scored against the MQM framework and ships with an MQM error report. For how error spans are labelled, see our guide to error span annotation.

If you are building this yourself, you will need to pick a QE model, set thresholds, route low scores to reviewers and keep a reference set for regression tests. If you would rather plug into ours, faslocalize.com connects the same QE gate, native linguist verification and LQA with MQM scoring to your repository, CMS or API, with a fixed quote before work starts. Scope a pipeline with us or send us your setup.

Frequently asked questions

When should I use reference-based scoring?

Use it when you already have trusted human translations. That makes it a good fit for regression tests, model selection and other offline checks. In those cases, segment-level precision matters, and WMT25 found reference-based baselines still lead there.

When is QE without references the right choice?

Choose it when no human reference exists yet. For example, live production output or real-time routing cannot wait for a reference translation. Instead, a QE model scores the source and the output together and returns a judgment right away.

Can one model do both kinds of scoring?

Yes, if you pick a hybrid model. Google says its MetricX-24 models are all hybrid, so the same model can score with a reference or without one. As a result, one deployment can serve both offline tests and live monitoring.

Has reference-free scoring caught up yet?

Only partly. The WMT25 findings report strong results for large LLMs at the system level. However, reference-based baseline metrics still outperform LLMs at the segment level, which is where single-sentence decisions are made.

Sources

  1. FACET at WMT 2026 Automated Translation Quality Evaluation Task, arXiv. Accessed Oct 7, 2026.
  2. Findings of the WMT25 Shared Task on Automated Translation Evaluation Systems, ACL Anthology. Accessed Oct 7, 2026.
  3. Unbabel/COMET: A Neural Framework for MT Evaluation, GitHub (Unbabel). Accessed Oct 7, 2026.
  4. google-research/metricx, GitHub (Google Research). Accessed Oct 7, 2026.

Want localization to run on autopilot?

Talk to us about your pipeline

Dao Huy

Senior Project Manager · EN · ZH · FR to Vietnamese

Your point of contact from brief to delivery: he scopes the job, sets up the AI and native-review workflow, and keeps quality, deadlines and budget on track.