Dialogues of the Deaf: Communication and AI

• Dialogues of the Deaf is a term that defines both miscommunication between Deaf and hearing participants and critiques of AI solutions that perpetuate interpreter-centric biases.
• It highlights limitations in current sign language AI methods, including pipeline translation errors, dataset biases, and the undervaluation of native Deaf signing practices.
• It calls for Deaf-led, co-designed research and evaluation frameworks that integrate multimodal approaches and authentic community metrics to improve accessibility.

Dialogues of the Deaf refer both to a sociolinguistic phenomenon—persistent miscommunication or lack of mutual understanding between deaf and hearing participants in dialogic contexts—and, in the past decade, to the framing and critique of technological approaches to bridging the deaf–hearing communication gap. The term encompasses not only face-to-face and mediated interactional breakdowns, but also describes how mainstream AI and computational linguistics research has at times enacted its own “dialogue of the deaf”—pursuing technical solutions that, while nominally inclusive, encode biases and neglect Deaf-centered perspectives, sustai­ning asymmetrical power relations in communicative access, design, and governance.

1. Historical and Theoretical Foundations

Historically, sign language interpreting has been privileged as the dominant model for providing “access” to Deaf communities, especially since its professionalization in the late twentieth century in Europe, North America, Australia, and New Zealand. The assumption that the presence of interpreters equates to access led to institutional targets based on interpreter numbers and availability, conflating quantitative provision with genuine inclusion. This orientation laid the groundwork for an “interpreter-centric” ideology, now embedded in contemporary AI design. AI solutions—ranging from ASR captioning to SL recognition—are consistently benchmarked against interpreter-mediated outcomes and training data, often sourced from interpreter-facilitated environments rather than native Deaf-to-Deaf signing. As a result, design and system evaluation have reflected a hearing-centered logic, further amplifying existing hierarchies and linguistic subordination for Deaf participants (Meulder, 5 May 2025).

Within the academic field, the term “dialogue of the deaf” has gained further resonance, describing not only social but epistemic misalignment: researchers, dominantly hearing and non-signing, often set the research agenda, data protocols, and annotation practices without meaningful, ongoing Deaf leadership or co-design, resulting in collective talking past one another (Desai et al., 5 Mar 2024).

2. AI-Driven Architectures for Bridging Communication

Sign language AI research has experimented with several architectures to enable fluid dialogues across modalities. Paradigmatic pipeline designs include:

• End-to-End English–Sign Translation: The AI4KSL architecture addresses Kenyan Sign Language (KSL) by building a parallel corpus of 14,000 English sentences with KSL glosses, 20,000 signed videos, and 4,000 words phonetically transcribed via HamNoSys. The process involves (1) ASR or text input, (2) English-to-gloss seq2seq translation, (3) conversion of glosses to HamNoSys phonetic vectors, and (4) retrieval or synthesis of video segments or avatar animation. This architecture enables instant rendering of hearing participants' utterances in sign, and, inverting the pipeline, real-time glossing and speech output for Deaf signers (Wanzare et al., 23 Oct 2024).
• Editing Program Models: Li et al. present a neural agent for spoken-language-to-gloss “glossification.” The model synthesizes and executes minimal “editing programs” that transform spoken sentences into sign-language gloss sequences via a DSL with ADD, COPY, DEL, and FOR actions. The agent combines imitation learning on minimal edit paths with policy optimization for BLEU-based surface form match, supporting grammatical, contextually faithful glossification for downstream sign animation (Li et al., 2021).
• Multilingual Visual Mediation: The MUGCAT system circumvents spoken/written language by mapping recognized sign-language gestures to expressive images, reconstructing sentences from partial gesture-derived keywords via semantic matching (Sentence-BERT cosine similarity) and image captioning. The visual output, decoupled from language specificity, functions as a universal bridge, allowing both deaf and hearing interlocutors to ground dialogue in shared visual context (Huynh et al., 2022).
• Wearable and Embedded Communication Assistants: Low-cost EMG-based devices, such as the Myo Armband+Smartphone DMEAS, map muscle signals directly to predefined words or phrases, offering instant, local, gesture-to-speech or text translation. However, these systems are constrained by limited gesture vocabularies and an inability to parse continuous signing (Hasan, 19 Mar 2025).
• Mixed-Reality and Avatars: Participatory design with Deaf and DHH users points to “AI interpreters” built into MR headsets (e.g., Apple Vision Pro). Such systems integrate real-time SL recognition, speech recognition, bidirectional translation, and hyper-customizable avatars supporting control over visual overlay, signing/voice style, emotion filtering, and situational awareness. Synchronization between sign and speech is prioritized for semantic and affective fidelity (Chen et al., 2 Oct 2024).

3. Systemic Failure Modes, Biases, and Critiques

A growing body of meta-research exposes deeply entrenched biases underlying current “dialogue of the deaf” efforts in AI:

• Interpreter-Mediation Bias: Training and benchmarking datasets—such as BOBSL, RWTH-Phoenix—are dominated by interpreter-mediated signing, which fails to capture the phonological, syntactic, and sociolinguistic range of native Deaf signing, and may perpetuate hearing-centric or “standardized” forms, marginalizing community variants (e.g., Black ASL, queer signing) (Meulder, 5 May 2025, Desai et al., 5 Mar 2024).
• Overemphasis on Communication Barrier Framing: Approximately 63% of surveyed sign-AI papers foreground “overcoming communication barriers” as a central motivation, often reducing sign languages to simple accessibility intermediates and neglecting their full status or diverse usage in Deaf communities (Desai et al., 5 Mar 2024).
• Annotation and Modeling Deficiency: Widespread reliance on gloss annotations without linguistic rigor—"tyranny of glossing"—along with use of pose estimators and vision models not tailored for sign language, results in both semantic flattening and technical misalignment (missed non-manuals, failed dialectal coverage) (Desai et al., 5 Mar 2024).
• Technological Technoableism and Modal Chauvinism: Mainstream voice assistants and gestural interfaces often privilege normative speech/accent models, reinforcing the marginal status of Deaf languaging and introducing “one-way” accessibility (sign-to-voice but not vice versa) (Meulder, 5 May 2025).

4. Evaluation Frameworks and User-Centric Metrics

Recent research proposes evaluation frameworks attentive to both the technical and socio-political stakes of “dialogues of the deaf.” Relevant metrics include:

• Standard Technical Scores: BLEU for text-to-gloss accuracy, WER for ASR, and Translation Error Rate for gloss synthesis (Wanzare et al., 23 Oct 2024).
• Semantic Similarity: Cosine similarity in Sentence-BERT space quantifies meaning preservation through visual or reconstructed outputs (Huynh et al., 2022).
• Usability and Satisfaction: System Usability Scale (SUS), task completion time, and NASA-TLX for cognitive load in smart assistant trials (Maria et al., 30 Nov 2024).
• Inclusivity and Trustworthiness: Quantitative indices such as Inclusivity Index (the proportion of community-approved linguistic varieties), Trustworthiness Score (rate of AI hallucinations), and Access Equity Function

$AE = \min_i \left(\frac{Access_i}{Demand_i}\right)$

where $i$ indexes community subgroups (Meulder, 5 May 2025).

• Contextual Risk Assessment: An “access matrix” (impact, interaction, duration, planning) guiding the use of pure-AI or hybrid human-in-command solutions (Meulder, 5 May 2025).

Participatory design methods further surface qualitative success criteria: controllability of avatar features, transparency in translation/rendering steps, synchronization of emotional content, and the degree to which privacy and boundary preferences are respected (Chen et al., 2 Oct 2024).

5. Participatory, Deaf-Led, and Community-Governed Approaches

Both systemic reviews and design-led studies converge on the necessity of shifting away from hearing-dominated research pipelines to Deaf-led, co-designed, and community-regulated models. Key recommendations include:

• Leadership by the Most Impacted: Research agendas, dataset design, and evaluation criteria must be set by Deaf researchers and community advisory groups (Desai et al., 5 Mar 2024, Meulder, 5 May 2025).
• Data Sovereignty and Representativeness: Prioritize datasets capturing the diversity of signing repertoires (age, race, disability), not solely “standardized” forms or interpreter speech (Meulder, 5 May 2025).
• Multi-Tiered Annotation: Combine glossing with phonological and spatial-linguistic features to reflect genuine sign structure (Wanzare et al., 23 Oct 2024).
• Transparency and Reflexivity: Full disclosure of dataset composition, annotator backgrounds, and model limitations; routine reevaluation by Deaf stakeholders (Desai et al., 5 Mar 2024).
• Human-in-Command Safeguards: For high-impact domains, maintain escalation pathways to human interpreters, with AI outputs equipped with transparent confidence measures (Meulder, 5 May 2025).
• Negotiated Consent and Personalization: Implementation of user-controlled overlays, avatar appearances, and boundary protocols in MR systems (Chen et al., 2 Oct 2024).

6. Future Directions and Open Research Problems

The field continues to evolve from text-centric and static gesture mapping paradigms toward:

• Robust, Multimodal Bi-directional Dialogue Systems: Multimodal SHAs and MR avatars combining sign, gesture, and environmental sound recognition, with real-time visual AR feedback, promise more equitable household and public-sphere communication (Maria et al., 30 Nov 2024, Chen et al., 2 Oct 2024).
• First-Person, Ambient, and Egocentric Sensing: Improved SL recognition that does not require fixed camera view, expanding naturalistic use cases (Huynh et al., 2022).
• Joint Training Across the Pipeline: Integrated optimization for end-to-end semantic accuracy from recognition through captioning or avatar synthesis (Huynh et al., 2022).
• Granular Metrics and Real-World Deployment: Larger scale, field-validated studies using both technical and user-centered metrics to continuously audit performance across community demographics (Wanzare et al., 23 Oct 2024, Maria et al., 30 Nov 2024).
• Ethical Governance and Policy Coupling: Binding policy safeguards (e.g., the EU AI Act) with community co-regulation to prevent the coercive substitution of AI for irreplaceable human interaction (Meulder, 5 May 2025).

The future of “dialogues of the deaf”—whether as an AI-mitigated communication barrier or a persistent product of asymmetric power and epistemic structures—depends on the extent to which the field centers Deaf autonomy, collective access, and linguistic rights in both design and implementation.