What do the metrics mean? A critical analysis of the use of Automated Evaluation Metrics in Interpreting

Abstract: With the growth of interpreting technologies, from remote interpreting and Computer-Aided Interpreting to automated speech translation and interpreting avatars, there is now a high demand for ways to quickly and efficiently measure the quality of any interpreting delivered. A range of approaches to fulfil the need for quick and efficient quality measurement have been proposed, each involving some measure of automation. This article examines these recently-proposed quality measurement methods and will discuss their suitability for measuring the quality of authentic interpreting practice, whether delivered by humans or machines, concluding that automatic metrics as currently proposed cannot take into account the communicative context and thus are not viable measures of the quality of any interpreting provision when used on their own. Across all attempts to measure or even categorise quality in Interpreting Studies, the contexts in which interpreting takes place have become fundamental to the final analysis.

• The paper demonstrates that automated metrics transplanted from MT, like BLEU, show limited accuracy (~62.96%) when applied to interpreting.
• The paper critiques decontextualized quality assessments, highlighting the mismatch between numeric scores and the contextual nature of interpreting.
• The paper advocates for hybrid evaluation models that combine automated metrics with context-sensitive, human judgment to better assess quality.

Critical Analysis of Automated Evaluation Metrics in Interpreting

Introduction

The analyzed paper provides a comprehensive critique of Automated Evaluation Metrics (AEMs) as applied to interpreting, with an emphasis on both foundational and recent developments in the field. As interpreting technologies evolve, there is acute demand for rapid and efficient quality assessment tools. The paper methodically dissects the theoretical and empirical underpinnings of interpreting AEMs, situating them in relation to their machine translation (MT) counterparts, and interrogates their conceptual suitability and operational limitations. This work offers rigorous insights into the tension between decontextualized, replicable metrics and the situated, inherently variable nature of authentic interpreting.

Taxonomy and Historical Context of Interpreting AEMs

The paper establishes a bifurcated typology, distinguishing between non-MT-based and MT-derived metrics. Non-MT approaches include both partial evaluations—such as semantic frame proximity and fluency estimation via acoustic features—and full evaluations relying on supervised machine learning to predict aggregate scores. Although these methods attempt to tailor evaluation to interpreting-specific characteristics, empirical results reveal limited predictive accuracy (e.g., best algorithms achieving 62.96% correlation with human scoring), underscoring an inherent complexity in human assessment that resists straightforward modeling.

Conversely, the transplantation of MT AEMs (most notably BLEU and its string-based derivatives) into interpreting is rooted in historical expediencies from system development contexts, where speed and repeatability are prioritized over nuanced quality determination. The paper echoes widely documented critiques in MT scholarship, such as low correlation with human judgments and inadvertent overfitting of system outputs to perform well on the metrics rather than delivering genuine communicative adequacy.

Theoretical Considerations: Latent Quality and Contextual Blindness

A central section of the paper exposes the latent, context-free definition of "quality" that is universalized across interpreting AEMs. Every operationalization boils down to a mathematical assessment of source-target relationships, relegating pragmatic concerns, communicative efficacy, and the social-ecological embedding of interpreting events to irrelevance. This reveals a deep epistemological alignment with a product-oriented, backward-looking conception of quality, which inherently prioritizes measurable alignment over situated usefulness.

The paper highlights a crucial contradiction: as interpreting studies theory increasingly foregrounds context, agency, and embodiment as inextricable aspects of quality, AEMs systematically erase these variables in pursuit of objectivity and replicability. Empirical evidence is presented showing that human and automated evaluations concur only when human rubrics themselves abandon context, thus revealing not objective convergence but methodological circularity.

Survey and Product-Based Research on Interpreting Quality

Synthesizing decades of interpreting studies research, the paper demonstrates that stakeholder (e.g., client) expectations of quality are contingent and often contradictory, with local event context shaping perceptions and requirements. The so-called "Eraslan Effect"—the disjuncture between abstract notions of quality and context-driven expectations—further complicates the project of defining and measuring quality generically. Survey-based studies, especially in dialogic interpreting, reveal that expectations for interpreter roles are fluid and contextually negotiated, rendering decontextualized metrics epistemically insufficient.

Product-based approaches, including classic error typologies and error counts, are likewise shown to flounder when faced with the necessity of ascribing meaning and pragmatic function to observed deviations. Studies demonstrate that what constitutes an "error" is often a strategic adaptation to communicative affordances, and context-insensitive classifications can generate misleading diagnostics. Recent holistic studies on AI-based medical interpreting echo these findings: while automated measures capture semantic fidelity, they fail to assess contextual appropriateness, fitness-for-purpose, or adaptation to dialogic requirements—dimensions repeatedly identified as critical in actual practice.

Situatedness and the Future of Interpreting Evaluation

The most robust conclusion drawn by the paper is the necessity for a situatedness orientation in interpreting quality assessment. Generic, decontextualized metrics (whether automated or human) are empirically and theoretically incapable of serving as sufficient measures of service adequacy, especially in high-stakes or interactive environments. Instead, quality must be treated as emergent from local context, purpose, and stakeholder needs. The paper advocates for the use of AEMs as ancillary tools—for initial baseline comparisons, or as hypothesis-testing instruments in comparative datasets—rather than as standalone arbiters of quality.

This theoretical repositioning aligns closely with broader moves in translation studies toward functionalist and user-centered paradigms. The situatedness approach privileges contextual relevance over metric convenience, and places primary evaluative authority with stakeholders enmeshed in the actual communicative event.

Implications and Research Trajectory

The implications are profound for both system development and the assessment of human and machine interpreting. Practically, no fully automated metric can be deployed as a reliable stand-in for context-sensitive human judgment. Theorists and practitioners must combine AEMs with nuanced, purposive, and context-aware evaluative methodologies, making explicit that any use of AEMs must be accompanied by clearly articulated caveats about their limitations.

There is unrealized potential for AEMs in large-scale comparative research, particularly for exploring cross-contextual features (prosody, terminological accuracy, etc.) in a way that can inform theory while not dictating practical judgments about quality. As interpreting technology advances, reflexive integration of quantitative and qualitative paradigms will be required to avoid epistemic drift and ensure that metrics serve, rather than distort, the field’s core evaluative commitments.

Conclusion

AEMs can expedite comparative assessment and hypothesis exploration in interpreting, but fundamentally lack the capacity to measure context-bound, situated interpreting quality on their own. The dominance of context in quality determination is now axiomatic in the field, and any assessment regime predicated solely on automated, decontextualized metrics is theoretically unsound and practically inadequate. Future research should direct efforts toward hybrid, context-sensitive evaluation models, with strong methodological caveats on the use and reporting of AEM-based findings.