Skip to content
FAS Localize

How-to

How to Evaluate a QE System Before Deploying It

Dao HuyBy , Senior Project Manager5 min readFacts checked Nov 19, 2026
A person at a desk pointing a pen at a printed Gantt-style bar chart next to an open laptop, the kind of result chart used to compare a QE system's scores against a gold standard.
Photo by Karolina Kaboompics on Pexels

How to evaluate a QE system before deploying it: run a controlled pilot on your own data using the recipe AMTA's Quality Estimation Users Working Group published in August 2026. Collect a representative sample and a gold standard, score it with the candidate QE, read the confusion matrix, set a threshold that matches your error costs, then validate on a hold-out set.

What does it mean to evaluate a QE system before deploying it?

Evaluation is not a benchmark leaderboard. It is a pass-fail test for your specific workflow. You need to know whether the QE system will route the right segments to post-editing, skip the clean ones, and flag the risky ones without drowning your linguists in false alarms. The AMTA framework published in August 2026 gives a repeatable recipe for that decision.

Agencies that sell AI translation need this same kind of QE step. FAS Localize (faslocalize.com) scores each AI draft with a reference-free quality-estimation model and keeps the best candidate. A native linguist then post-edits it, starting at $0.03 per word.

Step 1-2: Collect representative data and a gold standard

AMTA's working group recommends roughly 500 to 1,000 segments per combination of evaluation aspects as a starting point. Include the full quality spread: raw MT that is publish-ready, MT that needs light touch-up, and MT that must be retranslated. Have two senior linguists independently label each segment with a binary decision: clean (publish-ready) or needs review. Resolve disagreements. That resolved set is your gold standard. Without it, every subsequent number is a guess.

Step 3-4: Score each system as a confusion matrix

Score the same segments with every QE candidate you are considering. Do not stop at a single correlation metric. Build a 2×2 confusion matrix for each system at its default threshold:

Gold: Needs reviewGold: Clean
QE: Needs reviewTrue PositiveFalse Positive (unnecessary review)
QE: CleanFalse Negative (error slips through)True Negative
Confusion matrix for a binary QE decision

AMTA's working group defines a false negative as an error-containing segment incorrectly cleared from review, which creates quality risk. A false positive sends an error-free segment for unnecessary review instead, which raises cost. The appropriate threshold depends on both numbers together, not on system performance alone. It also depends on your own quality requirements and risk tolerance. Count both error types before you compare systems on a single accuracy number.

Step 5-6: Interpret limits and decide the threshold

Plot precision-recall or cost curves by sweeping the QE score threshold. Choose the point where the extra cost of a false positive matches the savings from a true positive. Base that choice on your own quality requirements and risk tolerance, not a vendor default. Then run the locked configuration on a fresh hold-out sample to confirm the error rates hold before production.

Why an LLM-as-judge QE system needs an extra check

AMTA's working group found that some LLM-as-judge QE systems show a 'people-pleaser effect'. The model flags issues even in content that has none, inflating the error rate on clean segments. The group also found that identical inputs and prompts can produce different results across runs. Setting the model to its lowest-randomness option reduces but does not remove this variation. Before you trust any LLM-as-judge QE, add known-good, error-free segments to your gold standard specifically to test for the people-pleaser effect, and re-run the same batch twice to see how much the score moves.

Trained QE models remain a strong alternative to LLM-as-judge systems. A 2026 AMTA research-track paper describes a compact, dedicated QE-and-post-editing model. It reached a QE correlation of r=0.907 and 88.4% post-editing decision accuracy, with less over-editing than large commercial LLMs on the same data. Put it on your shortlist next to any LLM-as-judge candidate.

What FAS does differently

If you want to see this kind of QE step on your own content before committing to an engine, FAS Localize offers a free test translation at faslocalize.com/mtpe: a reference-free quality-estimation model scores the AI draft, and a native linguist post-edits the result. For a shortlist of QE tools worth testing first, see our rundown of open-source quality estimation tools.

  1. Collect a representative sample

    500+ segments per language pair and domain, covering the full quality spectrum from publish-ready to retranslate.

  2. Build a gold standard

    Two senior linguists label each segment independently as clean or needs-review; resolve disagreements to create one binary reference.

  3. Score with candidate QE systems

    Run every QE engine you are evaluating on the exact same sample.

  4. Read the confusion matrix

    Count true positives, false positives, false negatives, and true negatives at the default threshold.

  5. Set the operating threshold

    Sweep the threshold to match your cost model; lock the point where marginal cost equals marginal savings.

  6. Validate on a hold-out set

    Run the locked threshold on a fresh batch of segments it has not seen; reject the system if error rates no longer hold.

Frequently asked questions

How many segments do I need for a reliable evaluation?

AMTA's working group recommends roughly 500 to 1,000 segments, or 5,000 to 10,000 words, per combination of evaluation aspects as a starting point. Use that minimum so the confusion matrix carries enough statistical weight to trust.

What is the 'people-pleaser effect' in LLM-as-judge QE?

It is a tendency some LLM-based QE systems show when asked to find translation errors: they flag issues even in content that has none. AMTA recommends adding known error-free segments to your evaluation data specifically to catch this before you trust the system's scores on real jobs.

Can I skip the gold-standard step if I trust the vendor's benchmark?

No. A vendor benchmark uses the vendor's own test set and judge model, which rarely matches your language pair, domain, terminology or error tolerance. AMTA's central principle is that QE performance is use-case and context specific, so you still need your own gold standard before deployment.

When should I choose reference-free over reference-based QE?

Choose reference-free QE when you lack reliable human reference translations at the scale you need to score, which is the common case in live production. See our comparison of reference-free vs reference-based QE for the trade-offs between the two approaches in practice.

Sources

  1. AMTA Publishes Framework for Evaluating Translation QE Systems, Slator. Accessed Nov 19, 2026.
  2. FAS Localize MTPE: Machine Translation Post-Editing, FAS Localize. Accessed Nov 19, 2026.
  3. FAS Localize Pricing, FAS Localize. Accessed Nov 19, 2026.
  4. Predict and Fix: A Unified Model for Translation Quality Estimation and Post-Editing, ACL Anthology / Association for Machine Translation in the Americas. Accessed Nov 19, 2026.

See what post-editing does to your own content.

Get a free sample edit

Dao Huy

Senior Project Manager · EN · ZH · FR to Vietnamese

Your point of contact from brief to delivery: he scopes the job, sets up the AI and native-review workflow, and keeps quality, deadlines and budget on track.