TELEVAL: Dual Evaluation Frameworks

• TELEVAL is a term for two distinct evaluation frameworks, one focused on validating pedestrian-flow models via VR and another on benchmarking spoken language models in realistic Chinese settings.
• In its telepresence usage, TELEVAL utilizes motion compression, real-human navigation data, and haptic feedback to calibrate and validate evacuation training simulations.
• The spoken-language-model TELEVAL benchmark assesses explicit semantics, paralinguistic cues, and system robustness under challenging acoustic conditions.

TELEVAL denotes distinct evaluation constructs in the technical literature. In one usage, it refers to a telepresence-based validation framework built around Extended Range Telepresence for evacuation training and pedestrian-flow model evaluation, in which a real walking user is embedded inside a virtual building populated with simulation agents (Arias et al., 2010). In a later and unrelated usage, TELEVAL denotes “TELephonic EVALuation,” a dynamic benchmark for end-to-end spoken LLMs (SLMs) in realistic Chinese interactive scenarios, designed to assess explicit semantics, paralinguistic and implicit semantics, and system abilities (Li et al., 24 Jul 2025). A common source of confusion is that these systems share a name while targeting different problem domains, methodologies, and output modalities.

1. Terminology and scope

A plausible first distinction is terminological. The earlier TELEVAL is a rigorous evaluation tool for pedestrian-flow models implemented through Extended Range Telepresence, with an explicit training function for emergency evacuation scenarios (Arias et al., 2010). The later TELEVAL is a dynamic, user-centered benchmark for spoken dialogue agents, intended to evaluate SLMs as conversational systems rather than as isolated speech recognizers or question-answering engines (Li et al., 24 Jul 2025).

Usage of “TELEVAL”	Domain	Core purpose
Extended Range Telepresence TELEVAL	Pedestrian simulation and evacuation training	Validation and calibration of pedestrian-flow models
“TELephonic EVALuation”	Spoken LLMs in Chinese interactive scenarios	Benchmarking conversational, paralinguistic, and robustness abilities

The term should also not be conflated with Tele-VQA, which is a separate telepresence video-quality assessment framework for multimodal live-streaming telepresence content rather than a TELEVAL system (Ying et al., 2022).

2. TELEVAL in Extended Range Telepresence

In the pedestrian-simulation literature, Extended Range Telepresence is a virtual-reality telepresence framework specifically tailored to both train people for evacuation scenarios and serve as a rigorous evaluation tool for pedestrian-flow models (Arias et al., 2010). Its core principle is Motion Compression. The user wears a head-mounted display and walks freely within a small physical “user environment,” for example $4 \times 7$ m, while perceiving a much larger virtual “target environment” (Arias et al., 2010).

The Motion Compression pipeline has three stages. Path prediction guesses the user’s intended target-environment trajectory from head-orientation and map knowledge, or purely from view-direction if no prior map is loaded. Path transformation nonlinearly warps that target path into the user environment such that path length and cumulative heading changes are identical in both spaces, while curvature is altered. User guidance then imposes slight avatar-heading offsets, unnoticeable to the user, in order to steer real footsteps along the compressed user path (Arias et al., 2010).

The hardware stack comprises a head-mounted display with dual $1280 \times 1024$ panels and $60^\circ$ field of view per eye, an acoustic tracking system updating user position and orientation at $50$ Hz with sub-decimeter accuracy, and a hand-held haptic device capable of rendering multi-axis force vectors (Arias et al., 2010). On the software side, the system includes an MC server for Motion Compression, CORBA network clients and servers for exchanging user posture, avatar pose, and haptic data, and a VISSIM pedestrian simulator module that produces live crowds and computes per-agent social-force interactions (Arias et al., 2010).

3. Pedestrian-model validation, calibration, and experimental findings

The validation methodology associates the telepresent user’s trajectory $P=\{p_{\text{user}}(t_i)\}$ with the corresponding simulated agent path $Q=\{q_{\text{sim}}(t_i)\}$. At experiment start, the user’s instantaneous state—position $p_{\text{user}}(t)$, orientation $\theta_{\text{user}}(t)$, and velocity $v_{\text{user}}(t)$—initializes a parallel VISSIM simulation, and both real-user data and simulated-agent data are logged synchronously at each time step $t_i$ (Arias et al., 2010).

Trajectory discrepancy is quantified using distance metrics including the Hausdorff distance, RMSE over time-aligned samples, and mean absolute deviation. The RMSE is defined as

$1280 \times 1024$0

The framework also notes that more advanced studies may employ the continuous Fréchet distance or Dynamic Time Warping for non-uniform time alignment (Arias et al., 2010). Calibration proceeds by minimizing one of these metrics with respect to global simulation parameters such as gap-acceptance thresholds, route-choice utility weights, or social-force coefficients.

Haptic integration is based on the Social Force Model. When a virtual pedestrian collides with another agent or an obstacle, two component forces are calculated—normal body compression and tangential friction—and the transmitted net social force is

$1280 \times 1024$1

but only when actual overlap occurs; otherwise $1280 \times 1024$2 (Arias et al., 2010). The resulting 3D force vector is compressed by the same nonlinear mapping used for Motion Compression and rendered on the hand-held haptic interface in real time, so that the user feels crowd push and may slow down, sidestep, or choose another path (Arias et al., 2010).

The reported validation experiment used a simple route-choice scenario in a virtual hall with three exits. In simulation, $1280 \times 1024$3 agents circulated continuously, while $1280 \times 1024$4 test users walked physically through the telepresence setup; each run lasted approximately $1280 \times 1024$5 s, and recorded metrics included chosen gate, path length, completion time, and full-trajectory logs (Arias et al., 2010). The key findings were that the simulation sent approximately $1280 \times 1024$6 of agents through Gate 1, whereas only $1280 \times 1024$7 of test users did so, with the remainder favoring Gate 2; trajectory overlays showed that users exploited the full hall width in the final segment, whereas simulated agents followed narrow lanes; RMSE between user and best-matching simulated path averaged approximately $1280 \times 1024$8 m over all runs; and iterative dynamic assignment with weighting $1280 \times 1024$9, for example $60^\circ$0 for the latest iteration, converged simulated distributions to real-user distributions within $60^\circ$1–$60^\circ$2 calibration loops (Arias et al., 2010).

These results support the intended TELEVAL role: embedding a real human in the simulation loop yields ground-truth trajectories that are otherwise costly or unsafe to obtain in real evacuation drills, especially under panic or high-density conditions (Arias et al., 2010).

4. TELEVAL as a spoken-language-model benchmark

In spoken-language modeling, TELEVAL is a dynamic benchmark specifically designed to evaluate SLMs’ effectiveness as conversational agents in realistic Chinese interactive settings (Li et al., 24 Jul 2025). Its stated motivation is to bridge the gap between benchmarks that focus on complex tasks comparable to those tackled by LLMs and the way users naturally interact in real-world conversational scenarios (Li et al., 24 Jul 2025).

TELEVAL defines three evaluation dimensions. Explicit Semantics targets understanding of literal content and production of factually correct, coherent replies without prompting. Paralinguistic and Implicit Semantics targets perception of vocal cues such as emotion, age, dialect, non-speech sounds, and ambient scenes, and the integration of those cues into appropriate responses without additional instructions. System Abilities targets robustness and responsiveness under real-world acoustic conditions (Li et al., 24 Jul 2025).

The benchmark adopts a dialogue format consistent with real-world usage and evaluates text and audio outputs separately (Li et al., 24 Jul 2025). Inputs are pre-recorded user audio, real for paralinguistic tasks and TTS-synthesized otherwise. Outputs include both textual and speech responses from the SLM. Two data formats are used: Factoid Audio QA (FAQA), in which the user asks a question and the model replies with a concise answer, and Open-Ended Audio Conversation (OEAC), in which the model produces a free-form reply (Li et al., 24 Jul 2025).

The first release contains more than $60^\circ$3 samples across $60^\circ$4 test sets (Li et al., 24 Jul 2025). Multi-turn evaluation includes $60^\circ$5 dialogues for each of $60^\circ$6, $60^\circ$7, and $60^\circ$8 turns, with only the final user turn being a FAQA question and earlier turns supplying context and distractors (Li et al., 24 Jul 2025).

5. Task structure, annotation, and metrics in the SLM benchmark

Within Explicit Semantics, TELEVAL includes Basic Knowledge, Chinese Dialect Comprehension, Context-Memory, Domain Knowledge, Safety and Values, and Chit-Chat Human-Likeness (Li et al., 24 Jul 2025). The benchmark statistics list, for example, LlamaQA-en $60^\circ$9, TriviaQA-en $50$0, WebQ-en $50$1, ChineseSimpleQA-zh $50$2, ChineseQuiz-zh $50$3, dialect comprehension subsets for Cantonese $50$4, Henan $50$5, Northeastern Mandarin $50$6, Shanghainese $50$7, and Sichuanese $50$8, Multiturn_memory-zh $50$9 dialogues), LivelihoodPolicy-zh $P=\{p_{\text{user}}(t_i)\}$0, HumanAccept-zh $P=\{p_{\text{user}}(t_i)\}$1, and HumanChitchat-zh $P=\{p_{\text{user}}(t_i)\}$2 (Li et al., 24 Jul 2025).

Within Paralinguistic and Implicit Semantics, the benchmark includes Audio Event Detection with AED-zh $P=\{p_{\text{user}}(t_i)\}$3 clips), emotion perception and response with ESD-zh $P=\{p_{\text{user}}(t_i)\}$4 samples, $P=\{p_{\text{user}}(t_i)\}$5 emotions $P=\{p_{\text{user}}(t_i)\}$6 each), Chitchat-dialect subsets for five dialects, Non-Speech Vocalization response with Para_mix300-zh $P=\{p_{\text{user}}(t_i)\}$7, and Age-zh $P=\{p_{\text{user}}(t_i)\}$8 for child and elder styles (Li et al., 24 Jul 2025). GOAT-SLM describes the same benchmark as a multi-dimensional benchmark suite measuring both semantic intelligence and paralinguistic and speaker-aware interaction, and enumerates sub-tasks including Common Audio Question Answering, Dialectal AQA, Multi-Turn Dialogue (Memory), Dialect Following, Emotion-Conditioned Response, Non-Speech Vocal Signal Response, Age-Aware Interaction, quantitative speech metrics, and subjective dialectal speech generation (Chen et al., 24 Jul 2025).

Annotation and scoring are heterogeneous by task. For FAQA, TELEVAL uses a structured set of reference spans and logical composition rules for string-matching (Li et al., 24 Jul 2025). For OEAC, it uses LLM-based $P=\{p_{\text{user}}(t_i)\}$9–$Q=\{q_{\text{sim}}(t_i)\}$0 scoring via task-specific prompts, with three independent judgments and power scaling $Q=\{q_{\text{sim}}(t_i)\}$1 (Li et al., 24 Jul 2025). Audio outputs are evaluated with DNSMOS for speech quality, CER/WER between TTS transcript and ASR of response for text-audio consistency, emotion alignment via emotion2vec softmax scoring with neutral filtering, and binary dialectal audio accuracy via a dialect classifier (Li et al., 24 Jul 2025). GOAT-SLM further summarizes benchmark-level criteria as accuracy for QA and dialogue, appropriateness rate for paralinguistic tasks, dialect-mirroring rate averaged across five dialects, CER for spoken replies, DNSMOS, and emotion appropriateness judged by human raters (Chen et al., 24 Jul 2025).

A notable methodological feature is that TELEVAL does not yield a single scalar “TELEVAL score” in the GOAT-SLM description; instead, each sub-task returns its own metric, and researchers are encouraged to report per-dimension results or compute an application-dependent aggregate if needed (Chen et al., 24 Jul 2025).

6. Empirical results, interpretive significance, and limitations

The first TELEVAL release evaluates nine SLMs under common settings: GLM-4-Voice, MiniCPM-o-2.6, Baichuan-Omni-1.5, LLaMA-Omni, SpeechGPT-2.0-preview, Freeze-Omni, Qwen2.5-Omni, Kimi-Audio, and GPT4o-Audio (Li et al., 24 Jul 2025). In Basic Knowledge, GPT4o-Audio reaches an average of $Q=\{q_{\text{sim}}(t_i)\}$2, while MiniCPM-o-2.6 reaches $Q=\{q_{\text{sim}}(t_i)\}$3 and GLM-4-Voice $Q=\{q_{\text{sim}}(t_i)\}$4 (Li et al., 24 Jul 2025). In Dialect Comprehension, Qwen2.5-Omni attains an average of $Q=\{q_{\text{sim}}(t_i)\}$5, exceeding GPT4o-Audio’s $Q=\{q_{\text{sim}}(t_i)\}$6 (Li et al., 24 Jul 2025). In Context Memory, Qwen2.5-Omni reports $Q=\{q_{\text{sim}}(t_i)\}$7, MiniCPM-o-2.6 $Q=\{q_{\text{sim}}(t_i)\}$8, and GLM-4-Voice $Q=\{q_{\text{sim}}(t_i)\}$9 (Li et al., 24 Jul 2025).

The reported interpretation is that current SLMs handle audio understanding reasonably well but struggle to generate natural, context- and user-aware speech; paralinguistic and implicit cues are recognized but seldom integrated into reply generation; and all models degrade sharply under heavy noise, with low SNR $p_{\text{user}}(t)$0 dB) reducing accuracy by up to $p_{\text{user}}(t)$1 for some models (Li et al., 24 Jul 2025). The benchmark authors also note that no statistical significance tests or confidence intervals are reported in the first release (Li et al., 24 Jul 2025).

GOAT-SLM provides a second empirical view of TELEVAL. Its example results are reported as follows: Common AQA overall percent on eight datasets in the range $p_{\text{user}}(t)$2–$p_{\text{user}}(t)$3, average approximately $p_{\text{user}}(t)$4; Dialectal AQA average over five dialects $p_{\text{user}}(t)$5; Multi-turn memory $p_{\text{user}}(t)$6; Dialect Following average $p_{\text{user}}(t)$7; Emotion $p_{\text{user}}(t)$8; NSV $p_{\text{user}}(t)$9; Age $\theta_{\text{user}}(t)$0; spoken-response CER $\theta_{\text{user}}(t)$1; DNSMOS $\theta_{\text{user}}(t)$2; and emotion appropriateness $\theta_{\text{user}}(t)$3 (Chen et al., 24 Jul 2025). Its subjective dialectal speech generation scores are reported as Cantonese $\theta_{\text{user}}(t)$4, Henan $\theta_{\text{user}}(t)$5, Northeastern Mandarin $\theta_{\text{user}}(t)$6, Shanghainese $\theta_{\text{user}}(t)$7, and Sichuanese $\theta_{\text{user}}(t)$8 (Chen et al., 24 Jul 2025).

A common misconception is that strong semantic QA performance implies strong conversational competence. TELEVAL’s published results argue against that equivalence: models that perform competitively on explicit semantics still show low scores on emotion, dialect following, age-aware response, NSV response, or robustness under distortion (Li et al., 24 Jul 2025).

7. Conceptual relation between the two TELEVAL usages

The two TELEVAL usages are historically and technically unrelated, but a plausible commonality is methodological rather than domain-specific. Both are designed to move evaluation closer to realistic interaction loops. In the pedestrian-simulation setting, realism is achieved by placing a real human inside a virtual crowd and logging position, orientation, velocity, and haptic-force data at high frequency (Arias et al., 2010). In the SLM setting, realism is pursued by using dialogue-shaped tasks, real or TTS user audio, separate text and audio scoring, and explicit treatment of paralinguistic and robustness conditions (Li et al., 24 Jul 2025).

This suggests that “TELEVAL” has functioned less as a single established framework than as a recurring label for interaction-centered evaluation. In one case, the objective is validation and calibration of pedestrian-flow simulators under conditions that are costly or unsafe to stage physically (Arias et al., 2010). In the other, the objective is diagnosis of whether spoken agents can respond appropriately to literal content, implicit vocal cues, and acoustic degradation in realistic Chinese conversational settings (Li et al., 24 Jul 2025).

The principal encyclopedic caution is therefore disambiguation. References to TELEVAL in evacuation training, crowd simulation, Motion Compression, VISSIM, and haptic feedback belong to the Extended Range Telepresence framework (Arias et al., 2010). References to explicit semantics, paralinguistic and implicit semantics, system abilities, FAQA, OEAC, DNSMOS, CER/WER consistency, and Chinese dialect interaction belong to the spoken-language-model benchmark introduced in 2025 (Li et al., 24 Jul 2025).