V(is)owel: Cross-Domain Vowel Visibility

Updated 5 July 2026

V(is)owel is a cross-domain framework that treats vowels as observable, information-rich probes bridging acoustic, articulatory, and orthographic modalities.
It integrates methods from lipreading viseme classification, neural TTS accent adaptation via vowel space tracking, and tongue MRI for articulated tongue postures.
The approach reveals a low-dimensional vowel space that enables actionable feedback, robust model adaptation, and deep typological insights.

V(is)owel denotes a line of inquiry in which vowels are treated as explicitly visible objects: as visemes in lipreading, as trajectories in acoustic vowel space, as articulatory tongue configurations, and as orthographic entities that may be omitted and restored. The literal title is used for an interactive vowel chart for second-language pronunciation training (Kiesel et al., 8 Jul 2025), but the underlying research program spans visual speech language modeling (Bear, 2018), vowel-based Korean visemes (Won et al., 2014), vowel-space tracking in neural TTS adaptation (Abeysinghe et al., 2022), multimodal inference over tongue MRI (Sakajo et al., 29 Jan 2025), probabilistic typology of vowel inventories (Cotterell et al., 2017), mathematical mappings between vocal-tract parameters and $(f_1,f_2)$ space (Berthommier, 2021), and omission-tolerant restoration of Arabic vowel diacritics (Neme et al., 2019). A useful synthesis is that vowels function here as compact, information-rich probes of speech structure: they expose co-articulation, accent adaptation, articulatory control, inventory organization, and orthographic underspecification.

1. Scope and representational regimes

Across these literatures, a vowel is not represented in a single way. In acoustic phonetics and TTS, the canonical representation is the vowel space defined by the first two formants, where $F_1$ correlates with tongue height or jaw opening and $F_2$ with tongue frontness or backness (Abeysinghe et al., 2022). In articulatory work, especially MRI-based probing, the operative representation is tongue height and tongue backness for a five-vowel system $/a, i, u, e, o/$ , with /i/ high-front, /u/ high-back, /e/ mid-front, /o/ mid-back, and /a/ low-back (Sakajo et al., 29 Jan 2025). In visual speech, vowels are constrained by lip rounding versus spreading, jaw opening, and lip protrusion, and these gestures overlap under co-articulation, so the visible signal is not a simple one-to-one sequence of phonemic vowel frames (Bear, 2018). In Korean lip-reading, the viseme inventory is explicitly vowel-centered because vowels dominate the visible information in the mouth region (Won et al., 2014).

Regime	Primary representation	Representative claim
Visual speech	Visemes, phonemes, words	Vowel co-articulation makes visual sequences context-dependent (Bear, 2018)
Acoustic visualization	$F_1$ - $F_2$ vowel space	TTS fine-tuning can be tracked by vowel-space drift (Abeysinghe et al., 2022)
Articulatory vision	Tongue height/backness in MRI	VLMs improve when given reference examples (Sakajo et al., 29 Jan 2025)
Typology and modeling	Inventory subsets or low-dimensional manifolds	Natural inventories reflect dispersion and focalization (Cotterell et al., 2017)

This multiplicity of representations is central rather than incidental. The same vowel may be a discrete phoneme, a viseme class, a point or trajectory in $F_1$ - $F_2$ space, a tongue posture, or an omitted diacritic. This suggests that V(is)owel is best understood as a cross-domain framework for making vowel structure observable.

2. Visible vowels in lipreading and viseme theory

In HMM-based lipreading, decoding follows the noisy-channel objective

$\hat{W} = \arg\max_W P(X \mid W)\,P(W),$

where $P(X \mid W)$ is the visual model and $F_1$ 0 is the LLM. The key issue is the unit on which $F_1$ 1 is defined: visemes, phonemes, or words (Bear, 2018). A viseme is a unit of visible speech, typically a class of phonemes that share a visually indistinguishable mouth pattern. For vowels, this collapse is especially strong because multiple phonemic vowels share similar lip shapes and are further blurred by visual co-articulation. The 2018 comparison on the RMAV audio-visual speech dataset reports that a viseme-based LM performs extremely poorly, with mean word correctness $F_1$ 2, whereas a phoneme LM yields $F_1$ 3 and a phoneme classifier paired with a word LM reaches $F_1$ 4 (Bear, 2018). The paper’s explicit conclusion is that viseme classifiers can be competitive, but viseme-based LLMs are not; for vowels, the collapse should occur in the visual model rather than in the LM.

The Korean lip-reading study pushes the visible-vowel idea further by defining 10 visemes “with Korean vowels as its axis” (Won et al., 2014). The inventory comprises vowel-based classes $F_1$ 5 plus a bilabial class $F_1$ 6 for /m, b, p/ and implicitly /pp/. Six of these are single visemes with stable mouth shapes, while $F_1$ 7 are double visemes whose identity depends on the dynamic transition between constituent shapes. The feature representation combines static and dynamic information: an 8-dimensional region feature, a 6-dimensional contour feature, and a 12-dimensional dynamic contour feature derived from smoothed temporal differences of lip landmarks, yielding the proposed 20-dimensional visual feature vector (Won et al., 2014). Recognition is performed with tri-viseme HMMs in HTK. For isolated-word recognition, top-3 accuracy is 89.7% on training words and 78.9% on non-training words when manual viseme information is provided, versus 65.4% and 56.5% without it. For speaker robustness, manual-viseme accuracy is 92.3% on the training speaker and 83.8% and 82.1% on two non-training speakers (Won et al., 2014).

Taken together, these results establish two complementary points. First, visible-vowel groupings are operationally useful at the classifier level because they reflect what the visual modality can discriminate. Second, once vowels are collapsed too early—especially at the language-model level—the resulting homophemy destroys discriminability. This is the core visual-speech version of V(is)owel.

3. Vowel space as a visualization of learning and feedback

In neural TTS, vowel space is used as an “intermediary evaluation” for accent adaptation (Abeysinghe et al., 2022). A Tacotron2-based system is pre-trained on LJSpeech, a General American English single-speaker corpus of 13,100 short clips and about 24 hours of speech, for 120,000 steps, then fine-tuned on Mansfield, a New Zealand English single-speaker corpus of 1,095 sentences and about 3 hours of speech, for 28,000 further steps. During fine-tuning, the system synthesizes /hVd/ words such as “hid,” “head,” “had,” “hud,” “hod,” “hood,” “who’d,” “hard,” “heard,” and “heed.” Phone boundaries are obtained with WebMAUS using its NZE option, and vowel formants are extracted with Praat’s formant_burg() using default settings. The sampled checkpoints are $F_1$ 8 and $F_1$ 9 (Abeysinghe et al., 2022).

The resulting plots show the synthetic vowel space moving from a GAE-like trapezoid toward an NZE-like triangle. The largest changes occur in the first $F_2$ 0k fine-tuning steps, especially for TRAP, DRESS, STRUT, LOT, and FOOT, and the trajectories are explicitly described as “far from linear” (Abeysinghe et al., 2022). A perception study with 23 listeners corroborates the visualization. At step 0, 34% of responses rate the voice as “mainly/completely GAE,” falling to 4% by 28k steps; NZE ratings rise from 20% to 33%, with the largest perceptual changes also concentrated in the first 10k steps (Abeysinghe et al., 2022). The methodological significance is that vowel space functions as a linguistically interpretable training curve.

The 2025 CAPT system named V(is)owel makes the same representational logic interactive for human learners (Kiesel et al., 8 Jul 2025). It presents a trapezoidal vowel chart labeled front–mid–back and high–mid–low, overlays a native Spanish model speaker’s vowel trajectories with the learner’s own productions, and supports playback by clicking plotted lines. Calibration uses four English corner vowels through the words “beet,” “boot,” “bat,” and “bought.” These define a learner-specific quadrilateral in raw $F_2$ 1- $F_2$ 2 space, which is then mapped by a projective transformation into a standardized trapezoid with height $F_2$ 3, top width $F_2$ 4, and bottom width $F_2$ 5 (Kiesel et al., 8 Jul 2025). The plotted objects are trajectories rather than static points, and only the last 10 attempts are shown, with older attempts fading in opacity.

The within-subject study involved $F_2$ 6 participants, all L1 American English speakers with limited Spanish exposure (Kiesel et al., 8 Jul 2025). Relative to an audio-only control with the same recording and playback UI but no chart, V(is)owel elicited more recordings per word: mean $F_2$ 7 versus $F_2$ 8, with a Friedman-test result of $F_2$ 9. There was no significant usability difference by SUS ( $/a, i, u, e, o/$ 0), and NASA-TLX showed no significant difference between the two pronunciation conditions when the baseline was excluded ( $/a, i, u, e, o/$ 1) (Kiesel et al., 8 Jul 2025). Qualitatively, all eight participants used the visual target as a goal; the paper reports that visual feedback motivated more practice and that explicit anatomical feedback directly mapping onto physical movement is beneficial for phonetically untrained learners (Kiesel et al., 8 Jul 2025).

These two strands share a strong design principle. Vowel space is not merely a descriptive artifact; it is a control surface. In TTS it reveals what the model has learned; in CAPT it provides actionable feedback tied to articulatory interpretation.

4. Articulatory visibility and multimodal model probing

Tonguescape asks whether multimodal LMs can infer vowels from actual tongue configurations visible in real-time MRI (Sakajo et al., 29 Jan 2025). The source is Maekawa’s rtMRIDB: lateral real-time MRI of 22 Japanese speakers, with synchronized audio at 14 FPS or 27 FPS. From this, the study builds three datasets. VowelVideo contains 120 silent MRI videos of isolated vowels plus 1,653 consonant+vowel videos, for a total of 1,773 videos; the split is 1,658 train, 5 dev, and 110 test. VowelImage extracts one representative frame from each of the 120 isolated-vowel videos, giving 5 train, 5 dev, and 110 test images. VowelImageWithGuide adds ellipses marking the palate and tongue regions to the same 120 images (Sakajo et al., 29 Jan 2025).

The task is five-way classification over $/a, i, u, e, o/$ 2, framed via prompting that asks the model first to read tongue height and backness and then predict the vowel. Human performance is 71.55% on VowelVideo, 61.82% on VowelImage, and 62.73% on VowelImageWithGuide (Sakajo et al., 29 Jan 2025). Zero-shot VLM performance is generally near chance. On VowelImage, GPT-4o scores 20.91%, Gemini 16.36%, Qwen2-VL-7B 20.00%, and Qwen2-VL-72B 13.63%; on VowelVideo, GPT-4o scores 24.55% and VideoLLaMA2 16.36% (Sakajo et al., 29 Jan 2025). Performance improves under few-shot prompting. In the five-shot condition, GPT-4o reaches 37.27% on VowelImage and 40.00% on VowelImageWithGuide; Qwen2-VL-72B reaches 35.45% on VowelImage and 40.00% on VowelImageWithGuide (Sakajo et al., 29 Jan 2025).

The feature-level analysis is equally revealing. In the five-shot guided setting, GPT-4o achieves 43.64% on direct height prediction and 66.36% on direct backness prediction; via vowel prediction, height+backness reaches 40.00%. Qwen2-VL-72B reaches 46.36% on direct height, 65.45% on backness via vowel, and 40.00% on joint height+backness via vowel (Sakajo et al., 29 Jan 2025). The paper’s interpretation is cautious: LMs exhibit potential for understanding vowels and tongue positions when reference examples are provided, but they have difficulty without them (Sakajo et al., 29 Jan 2025).

This suggests that the articulatory version of V(is)owel is currently only partially realized in foundation models. They can leverage relative anchors and simple guides, but zero-shot grounding from raw MRI to vowel identity remains weak. The visible vowel is therefore informative for model probing precisely because it exposes the gap between textual phonetic knowledge and visually grounded articulatory understanding.

5. Vowel space as a probabilistic and mathematical object

A different branch of the literature treats the vowel system itself as an object to be modeled. In probabilistic typology, a vowel inventory is a random subset $/a, i, u, e, o/$ 3 drawn from a distribution over 53 IPA vowel types observed in 223 languages (Cotterell et al., 2017). The models are point processes over sets: a Bernoulli point process with unary focalization terms, a Markov point process with pairwise repulsion

$/a, i, u, e, o/$ 4

and a determinantal point process with

$/a, i, u, e, o/$ 5

The deep parameterization maps acoustic $/a, i, u, e, o/$ 6 inputs into embeddings $/a, i, u, e, o/$ 7, allowing the model to learn a perceptual geometry that operationalizes dispersion and focalization (Cotterell et al., 2017). On the reported experiments, DPP $/a, i, u, e, o/$ 8 MPP $/a, i, u, e, o/$ 9 BPP across embedding schemes. For prototype-based models, cross-entropy improves from 12.83 for pBPP to 10.95 for pMPP and 10.29 for pDPP; cloze-012 accuracy improves from 40.56% to 45.01% and 45.46%, respectively (Cotterell et al., 2017). The typological lesson is that vowels are best modeled not as isolated points but as mutually constrained sets.

A complementary mathematical treatment constructs an approximate bijection between a low-dimensional articulatory parameterization and the acoustic vowel space $F_1$ 0 (Berthommier, 2021). The generic area-function model is built from the first two odd cosine components: $F_1$ 1 with an added radius $F_1$ 2 to fill a 2D domain (Berthommier, 2021). A coordination function,

$F_1$ 3

is then shown to act similarly with Fant’s model and with the 4-Tube DRM derived from the generic model. For the generic model and reduced DRM, the mapping between the first two odd cosine coefficients and formant deviations is approximated by

$F_1$ 4

which helps explain why the vowel space takes on a triangular shape (Berthommier, 2021).

These approaches converge on a common claim: vowel spaces are low-dimensional but structured. One formalism captures that structure probabilistically across languages; the other derives it from coordinated articulatory-acoustic mappings. Both treat vowels as system-level entities rather than mere segment labels.

6. Orthographic visibility, omission, and restoration

In Arabic written technology, the visible-vowel problem appears in orthography rather than speech. The system described in Arabic-Unitex distinguishes 34 bare letters and 9 optionally written diacritics, plus initial hamza diacritics (Neme et al., 2019). The relevant vowel-related marks include the short vowels /a/, /i/, /u/, sukūn, the three nunation marks, shadda, superscript alif, and initial hamza above or below alif. In ordinary running text, vowels are usually omitted: the paper cites 1.6% of words with at least one diacritic in the Penn Arabic Treebank part 3, about 3% in a newspaper sample, typically 2–3% in newspapers, up to 12–15% in carefully edited articles, and about 35% in one highly vowelized al-Adab corpus (Neme et al., 2019). The generalization stated in the abstract is that more than 97 percent of written words do not explicitly show any of the vowels they contain (Neme et al., 2019).

Arabic-Unitex addresses this with omission-tolerant dictionary lookup. The resource contains about 76,000 fully vowelized lemmas and about 6 million fully vowelized inflected forms, expanded through finite-state transducers and agglutination grammars (Neme et al., 2019). It recognizes at least about 500 million valid fully vowelized agglutinated forms and several trillions of valid partially vowelized forms once omission rules are considered. The typographical rule set formalizes fatha, damma, kasra, sukūn, nunation, shadda+vowel omissions, superscript alif omission, initial hamza equivalences, and variants such as accusative $F_1$ 5 order. The resulting program analyzes 5,000 words per second, or about 20 pages per second, and provides lexical coverage of more than 99 percent of the words used in popular newspapers (Neme et al., 2019).

Here the vowel is “visible” only intermittently, yet its latent structure remains essential. This extends V(is)owel beyond phonetics and visualization in the narrow sense. Vowels can be rendered visible by plots, trajectories, visemes, or MRI frames, but they can also be made visible retrospectively through lexicon-driven restoration.

7. Synthesis and research significance

The surveyed literature supports several stable conclusions. First, vowels are disproportionately informative. They dominate visible articulation in Korean lip-reading, structure the most consequential ambiguities in English visual speech, anchor accent transfer in neural TTS, and provide compact probes of multimodal grounding in tongue MRI [(Won et al., 2014); (Bear, 2018); (Abeysinghe et al., 2022); (Sakajo et al., 29 Jan 2025)]. Second, making vowels visible is useful only when the representation preserves the right level of abstraction. Viseme collapse helps classifier design but damages language modeling; vowel charts become actionable when mapped to anatomy rather than raw Hertz; MRI guides help only when coupled with reference examples; orthographic restoration works when omission is formalized against a fully vowelized lexicon (Bear, 2018, Kiesel et al., 8 Jul 2025, Sakajo et al., 29 Jan 2025, Neme et al., 2019).

Third, vowel systems exhibit a recurring low-dimensional geometry. This is explicit in $F_1$ 6- $F_1$ 7 plots for TTS and CAPT, in height/backness reasoning for MRI-based probing, in probabilistic point-process models of inventories, and in cosine-based articulatory-acoustic mappings (Abeysinghe et al., 2022, Kiesel et al., 8 Jul 2025, Sakajo et al., 29 Jan 2025, Cotterell et al., 2017, Berthommier, 2021). A plausible implication is that V(is)owel is not merely a theme of “visualizing vowels,” but a broader epistemic strategy: vowels are the place where high-dimensional speech systems become interpretable.

Within that strategy, different disciplines optimize for different observables. Lipreading studies seek robust decoding under co-articulation; pronunciation-training systems seek actionable feedback for learners; TTS work seeks an intermediary evaluation of model adaptation; multimodal LM probing seeks grounded articulatory understanding; typology seeks distributions over possible inventories; mathematical phonetics seeks bijections between articulatory and acoustic coordinates; Arabic NLP seeks restoration of underspecified orthography. The shared object is the vowel, but the shared method is visibility.