Academic Journals Database
Disseminating quality controlled scientific knowledge

For an external evaluation of MT systems by task-based methods Pour l'évaluation externe des systèmes de TA par des méthodes fondées sur la tâche

Author(s): Hervé Blanchon | Christian Boitet

Journal: Traitement Automatique des Langues
ISSN 1248-9433

Volume: 48;
Issue: 1;
Start page: 33;
Date: 2008;
VIEW PDF   PDF DOWNLOAD PDF   Download PDF Original page

Keywords: evaluation | machine translation | usability evaluation

External methods for evaluating MT systems define various measures based on MT results and their usage. While operational systems are mostly evaluated since long by task-based methods, evaluation campaigns of the last years use (parsimoniously) quite expensive subjective methods based on unreliable human judgments, and (for the most part) methods based on reference translations, that are impossible to use during the real usage of a system, less correlated with human judgments when quality increases, and totally unrealistic in that they force to measure progress on fixed corpora, endlessly retranslated, and not on new texts to be translated for real needs. There are also numerous biases introduced by the desire to diminish costs, in particular the usage of parallel corpora in the direction opposed to that of their production, and of monolingual rather than bilingual judges. We prove the above by an analysis of the history of MT evaluation, of the « mainstream » evaluation methods, and of certain recent evaluation campaigns. We propose to abandon the reference-based methods in external evaluations, and to replace them by strictly task-based methods, while reserving them for internal evaluations.
Save time & money - Smart Internet Solutions     

Tango Rapperswil
Tango Rapperswil