Visual Scene Displays in AAC

• Visual Scene Displays (VSDs) are AAC interfaces that use contextual images with embedded hotspots to convey language concepts effectively.
• The study employs comparative methods, including part-of-speech analysis and expert evaluations, to assess AI-generated COs against human-created options.
• Challenges persist in personalization and developmental appropriateness, highlighting the need for clinician oversight and context-driven customization.

Searching arXiv for Visual Scene Displays in AAC and automated just-in-time programming. Visual scene displays (VSDs) are a form of augmentative and alternative communication (AAC) that use photographs or other contextual images with interactive “hotspots” embedded in the image to represent language concepts and/or communication options (COs). In practice, a user taps a region in the image to activate a word, phrase, or message. Within AAC, VSDs are presented as especially useful for beginning communicators, including children with autism or other developmental disabilities, because they use personally relevant imagery, preserve the relationship between people and objects, combine multiple elements of an activity within a single visual context, reduce cognitive demands by aligning with natural visual processing, and support early communicative functions such as engagement, requesting, commenting, and social interaction (Zastudil et al., 2024).

1. Definition and place within AAC

VSDs are used to embed communication options within contextualized images rather than presenting vocabulary as a decontextualized grid of symbols or text. The image itself functions as the organizing substrate for interaction: communicative content can be embedded directly in the image as clickable hotspots, or represented as separate buttons on the display. The central design premise is that language access is coupled to a depicted activity, scene, or event, so that the communicative interface preserves contextual structure rather than abstracting away from it (Zastudil et al., 2024).

The reported rationale for VSDs in AAC is strongly scene-based. They preserve the relationship between people and objects, and they combine multiple elements of an activity within a single visual context. This suggests that VSDs are intended not merely as display surfaces but as representations of situated experience. A plausible implication is that their effectiveness depends on the degree to which the depicted scene aligns with the user’s current activity, prior experience, and communicative goals.

The population emphasized in the available study consists of beginning communicators, including children with autism or other developmental disabilities. In that framing, VSDs are not presented as generic multimodal interfaces; they are specialized AAC instruments designed to support early communication in ecologically grounded settings (Zastudil et al., 2024).

2. Contextual structure and communicative use

The described VSD use cases are activity-centered. Three recurring contexts are identified: playing, shared storybook reading, and retelling a past activity. These are not incidental examples; they specify the kinds of interactional settings in which VSDs are expected to support communication (Zastudil et al., 2024).

The communication options embedded in a VSD are tied to the communication stage of the child. The study distinguishes two stages. The first is pre-linguistic, described as “building engagement in interactions and the emergence of words.” The second is multiword, described as “beginning to combine words.” In the generative-AI pipeline, these stage descriptors are explicitly included in the prompt, indicating that communication-stage conditioning is treated as a necessary input to scene-specific vocabulary generation (Zastudil et al., 2024).

These activity and stage distinctions are consequential because the same image can support different communicative functions depending on the intended developmental target. A sandbox scene, for example, may support engagement-oriented and socially directed options in a pre-linguistic setting, but more combinatorial language in a multiword setting. The data do not formalize this as a control policy, but they do show that VSD programming is inherently conditional on both scene context and communicative stage.

3. Manual authoring and the just-in-time programming problem

A major limitation of existing VSD practice is that default imagery and communication options are often only relevant in specific settings. As a result, effective deployment typically requires manual programming of the communication options by a partner such as a clinician, teacher, or caregiver. The study characterizes this process as time-consuming, dependent on partner presence, and difficult to sustain when vocabulary must be reconfigured “on the fly” to remain contextually relevant and engaging (Zastudil et al., 2024).

This burden motivates just-in-time (JIT) programming, defined as the practice of creating or updating AAC vocabulary in real time to match the scene or activity. JIT programming is presented as a partial solution, but still one that depends on a human expert being available. The underlying problem is therefore not only vocabulary selection but the latency and labor associated with making VSD content scene-specific at the moment of use (Zastudil et al., 2024).

The study’s framing makes clear that VSD quality is not separable from contextual relevance. A VSD populated with generic or stale communication options may still be technically functional, yet miss the immediate interactional affordances of the scene. This suggests that VSD authoring is fundamentally a context-adaptation problem rather than merely a display-design problem.

4. Generative AI for automated VSD programming

The reported automation approach evaluates whether a large multimodal model (LMM), specifically OpenAI GPT-4V / gpt-vision-preview, can automatically generate contextually appropriate COs from images for use in VSDs. The input is a photograph intended for a VSD. The prompt specifies that the child is on the autism spectrum, identifies the communication stage as pre-linguistic or multiword, states that the image is for a VSD, and requests contextually relevant communication options (Zastudil et al., 2024).

The pipeline uses prompt chaining. Initial prompts yielded lists of about 20 options on average, and a second prompt was then used to narrow the list: “Using the communication options you generated, please identify the five most relevant communication options.” The appendix states that responses were collected in April 2024 and that the model used was GPT-4V, specifically gpt-vision-preview (Zastudil et al., 2024).

The evaluation design includes two studies. In Study 1, survey participants were N = 13 speech-language pathologists (SLPs) and AAC researchers, with experience ranging from 1 to 25 years and mean experience $\bar{x} = 9.3$; 7/13 were frequent users or researchers of VSDs. Expert VSD evaluators were N = 5, drawn from that group, selected because they had more than 5 years of experience and extensive experience using/configuring VSDs. They did not rate the COs they personally created, and they were not told which COs were LMM-generated. Study 2 consisted of semi-structured interviews with N = 5 expert VSD researchers or clinicians with experience ranging from 5 to 15 years and mean experience $\bar{x} = 9$ (Zastudil et al., 2024).

Study 1 compares human- and LMM-generated COs using part-of-speech (POS) analysis, deductive coding based on Light’s Four Functions of Communication, and expert ratings. Coding agreement was 0.65, interpreted as substantial agreement. This evaluation structure indicates that the automation question is treated not only as a generation-quality problem but also as a communicative-function and expert-acceptability problem (Zastudil et al., 2024).

5. Empirical characteristics of generated communication options

The corpus-level comparison reports that human participants produced 306 COs, whereas GPT-4V produced 379 COs. POS distributions were broadly similar: nouns accounted for 39.2% of human COs and 33.8% of LMM COs; verbs 25.8% and 31.7%; adjectives 7.6% and 10.1%; adverbs 5.0% and 8.9%; pronouns 6.7% and 6.9%. The study interprets this as similar lexical structure, with nouns and verbs dominant in both sources (Zastudil et al., 2024).

The functional coding uses the categories expressing wants and needs, information transfer, social closeness, social etiquette, and other. The distributions are reported as follows: expressing wants and needs, human 16.7% and LMM 16.9%; information transfer, human 69.9% and LMM 79.4%; social closeness, human 6.2% and LMM 0.8%; social etiquette, human 2.0% and LMM 1.8%; other, human 5.2% and LMM 2.3%. Both humans and the LMM mainly focused on information transfer and wants/needs, but the LMM generated very few social-closeness options. Humans generated more “other” options, often sound effects such as “vroom” and “bawk bawk” (Zastudil et al., 2024).

Example contrasts further clarify the difference in output style. In the sandbox example, human-only COs included “vroom,” “girl’s name,” “boy’s name,” “my turn,” and “watch me.” LMM-only COs included “finished,” “yellow,” “want truck,” “friend,” and “I play.” Shared COs included “sand,” “truck,” “play,” “shovel sand,” and “more sand.” The study uses these examples to illustrate that humans provided more socially oriented and identity-specific language, whereas the LMM provided more generic contextual or action language (Zastudil et al., 2024).

Expert ratings indicated that LMM-generated COs were often similar to human-generated ones, and in some cases preferred. Context-specific preference varied: for playing, human-generated COs were generally preferred; for storybook reading, LMM-generated COs were generally preferred; for retelling a past activity, LMM-generated COs were generally preferred. The overall conclusion was that the two sources were roughly comparable in quality, and that the LMM-generated COs were contextually relevant and often resembled those created by humans (Zastudil et al., 2024).

6. Personalization, developmental appropriateness, and open problems

The reported results are explicitly cautious. Although the communication options generated by the LMM were contextually relevant and often resembled those created by humans, vital questions remain before LMMs can be confidently implemented in AAC devices. The main concerns raised in the interviews were lack of personalization and developmentally inappropriate output (Zastudil et al., 2024).

Personalization is treated as a major gap because SLPs rely on deep knowledge of the child’s interests, family context, culture, and developmental level. AI-generated COs may therefore be contextually plausible while still missing family names, preferred activities, culturally specific language, or other highly individualized content. This suggests that scene relevance alone is not an adequate criterion for VSD quality; person relevance is also required (Zastudil et al., 2024).

Developmental appropriateness is a second concern. Participants worried that AI might generate COs that do not match the user’s communicative stage, and there was concern that inappropriate vocabulary could hinder language development. The study also notes uncertainty about harmful stereotypes, biases, and other inappropriate outputs in AAC contexts. While such bias was not observed in the reported evaluation, prior LMM research was cited as evidence that harmful, stereotyped output remains a realistic deployment risk (Zastudil et al., 2024).

The resulting controversy is not whether automated JIT programming is technically feasible, but whether it can be made clinically acceptable. The study identifies several conditions for future progress: personalized user models, mitigation strategies for bias and harmful content, and mechanisms to ensure that AI-generated COs are developmentally appropriate. A plausible implication is that future VSD systems will require clinician-informed safeguards and personalization mechanisms rather than direct substitution of automated generation for expert configuration (Zastudil et al., 2024).

7. Research significance and current scope

Within the available evidence, VSDs are presented as AAC interfaces whose distinctive property is the embedding of language options in contextualized images. Their practical value lies in supporting communication in naturalistic activities such as play, storybook reading, and retelling past events, particularly for beginning communicators. Their practical limitation lies in the effort required to configure communication options that remain relevant to a specific scene and a specific child (Zastudil et al., 2024).

The recent generative-AI work positions VSD research at the intersection of AAC interface design, vocabulary selection, and multimodal scene-conditioned generation. The central contribution is not a redefinition of VSDs themselves, but evidence that automated just-in-time programming may be feasible: GPT-4V-generated communication options were generally comparable to human-generated COs in content, focus, and quality, and often looked human-like. At the same time, the study argues that deployment cannot be justified on contextual relevance alone. Personalization, developmental appropriateness, bias mitigation, and clinician oversight remain unresolved prerequisites for implementation in AAC devices (Zastudil et al., 2024).

In that sense, the contemporary research trajectory does not displace the original AAC logic of VSDs. It extends it. The scene remains the organizing unit, the communication option remains the operative element, and contextual relevance remains essential; the open question is whether these properties can be operationalized at scale without losing the individualized judgments on which effective AAC practice depends.