DSTC12 Track 1 Overview

• The paper introduces a dual-task challenge that evaluates dialogue quality and safety by integrating multi-dimensional metrics with multilingual and multicultural detection.
• It employs a two-subtask structure where Task 1 uses a Llama-3-8B baseline (Spearman's correlation 0.1681) and Task 2 benchmarks multilingual safety (ROC-AUC up to 0.9648) versus cultural safety.
• The track highlights the need for robust evaluation methods as high performance in one safety domain does not guarantee success in culturally specific contexts.

Searching arXiv for the specified DSTC^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^ paper and nearby related records. arxiv_search(^^^^^^^^9query9 DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^^^^^", ^^^^^^^^9max_results9 ^^^^^^^^9sort_by9 to=arxiv_search code 福利彩票天天彩json {"^^^^^^^^9query9 DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^^^^^"," RTLRassistant to=arxiv_search code 彩神争霸官方json {"^^^^^^^^9query9 of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC ^^^^9submittedDate9query9^^^^\""," DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^,"^^^^^^^^9sort_by9 {"^^^^^^^^9query9 ^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^ dialog system evaluation dimensionality language culture safety","^^^^^^^^9max_results9 DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^,"^^^^^^^^9sort_by9 to=arxiv_search code 娱乐开号ിയassistant to=arxiv_search code 天天中彩票是json {"^^^^^^^^9query9 Track ^^^^9submittedDate9^^^^ multilingual multicultural safety detection dialogue-level multi-dimensional automatic evaluation metrics","^^^^^^^^9max_results9 DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^,"^^^^^^^^9sort_by9 Reviewing the search results for the paper metadata and any related task papers. Fetching broader arXiv records around DSTC^^^^9submittedDate9query9^^^^ and dialog evaluation to ground the article. arxiv_search(^^^^^^^^9query9 ^^^^9submittedDate9query9^^^^ dialog system technology challenge ^^^^9query9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9query9sort_by9^^^^ arXiv", ^^^^^^^^9max_results9 DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^, ^^^^^^^^9sort_by9 to=arxiv_search code 【อ่านข้อความเต็มjson {"^^^^^^^^9query9 ^^^^9submittedDate9query9^^^^ dialog system technology challenge ^^^^9query9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9query9sort_by9^^^^ arXiv","^^^^^^^^9max_results9 DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^,"^^^^^^^^9sort_by9 DSTC^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^, titled "Dialog System Evaluation: Dimensionality, Language, Culture and Safety," is a Dialog System Technology Challenge track devoted to dialogue system evaluation under contemporary large-language-model conditions. Published in ^^^^9query9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9query9sort_by9^^^^ as "Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC ^^^^9submittedDate9query9^^^^" (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&), it addresses the claim that the rapid advancement of LLMs has intensified the need for robust dialogue system evaluation, while traditional metrics often prove insufficient and safety considerations are frequently narrowly defined or culturally biased. The track comprised two subtasks: Dialogue-level, Multi-dimensional Automatic Evaluation Metrics, and Multilingual and Multicultural Safety Detection. Its reported results are notable for two reasons: in Task ^^^^9submittedDate9^^^^, a Llama-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9ti:\9^^^^B baseline achieved the highest average Spearman's correlation of ^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9submittedDate9submittedDate9ti:\9submittedDate9^^^^; in Task ^^^^9query9^^^^, participating teams significantly outperformed a Llama-Guard-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9submittedDate9^^^^B baseline on the multilingual safety subset, while the baseline proved superior on the cultural subset (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&).

^^^^9submittedDate9^^^^. Scope and problem formulation

DSTC^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^ is framed around a broad problem in dialogue system evaluation: comprehensive assessment remains challenging even as LLM capabilities improve (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). The track situates itself against two limitations identified in the abstract. First, traditional metrics often prove insufficient for evaluating dialogue systems. Second, safety considerations are frequently narrowly defined or culturally biased. The track is explicitly presented as part of an ongoing effort to address these critical gaps.

This formulation places evaluation and safety within the same technical program rather than treating them as independent concerns. This suggests that DSTC^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^ regarded dialogue quality, multilinguality, culture, and safety as coupled evaluation axes rather than separable post hoc checks. A plausible implication is that the track was designed to expose failure modes that remain hidden when systems are assessed with single-score metrics or monocultural safety assumptions.

^^^^9query9^^^^. Task structure

The track comprised two subtasks (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). Their names and immediate emphases are as follows.

Subtask	Focus stated in the record	Headline outcome
Dialogue-level, Multi-dimensional Automatic Evaluation Metrics	^^^^9submittedDate9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^ dialogue dimensions	Llama-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9ti:\9^^^^B baseline achieved the highest average Spearman's correlation, ^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9submittedDate9submittedDate9ti:\9submittedDate9^^^^
Multilingual and Multicultural Safety Detection	Multilingual safety subset and cultural subset	Teams outperformed Llama-Guard-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9submittedDate9^^^^B on multilingual safety; baseline was superior on the cultural subset

The first subtask targets dialogue-level, multi-dimensional automatic evaluation metrics. The second targets multilingual and multicultural safety detection. The available record does not enumerate the ^^^^9submittedDate9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^ dialogue dimensions, the dataset composition, or the internal submission format, but it does state that the paper describes the datasets and baselines provided to participants, as well as submission evaluation results for each of the two proposed subtasks (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&).

From an evaluation-design perspective, the two-task structure separates automatic quality estimation from safety detection while keeping both inside one track. This suggests a deliberate attempt to test whether advances in one area transfer to the other; the reported results indicate that such transfer cannot be assumed.

^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^. Dialogue-level, multi-dimensional automatic evaluation

Task ^^^^9submittedDate9^^^^ focused on ^^^^9submittedDate9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^ dialogue dimensions (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). The reported summary statistic is average Spearman's correlation, and the highest reported value in the abstract is ^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9submittedDate9submittedDate9ti:\9submittedDate9^^^^, obtained by a Llama-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9ti:\9^^^^B baseline. The abstract explicitly notes that this indicates substantial room for improvement.

Several technical implications follow directly from this phrasing. Because the subtask is dialogue-level, the target of prediction is at the dialogue granularity rather than a narrower unit explicitly identified in the record. Because it is multi-dimensional, the evaluation objective is not collapsed into a single undifferentiated quality label; instead, the subtask is organized around ^^^^9submittedDate9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^ dimensions. Because average Spearman's correlation is reported, the task outcome is tied to rank-order agreement rather than an exclusively classification-style endpoint.

The most striking fact is the magnitude of the best reported average Spearman's correlation. A score of ^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9submittedDate9submittedDate9ti:\9submittedDate9^^^^, even when described as the highest average correlation achieved by the baseline, is presented as evidence that comprehensive automatic dialogue evaluation remains unresolved (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). This suggests that multi-dimensional dialogue assessment remains difficult even when a modern LLM baseline is applied. A plausible implication is that the relevant dimensions are either weakly captured by current automatic metrics, difficult to align with annotation targets, or both; however, the available record does not specify which explanation dominates.

^^^^9max_results9^^^^. Multilingual and multicultural safety detection

Task ^^^^9query9^^^^ is named Multilingual and Multicultural Safety Detection (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). The abstract reports results separately for a multilingual safety subset and a cultural subset, indicating that the task did not treat safety as a single homogeneous category.

On the multilingual safety subset, participating teams significantly outperformed a Llama-Guard-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9submittedDate9^^^^B baseline, with the top ROC-AUC reported as ^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.9^^^^9submittedDate9max_results9ti:\9^^^^ (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). On the cultural subset, however, the Llama-Guard-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9submittedDate9^^^^B baseline proved superior, with ^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9sort_by9submittedDate9query9submittedDate9^^^^ ROC-AUC. The abstract further states that this highlights critical needs in culturally-aware safety.

These results support a narrow but important conclusion: high performance on multilingual safety detection does not automatically imply strong performance on culturally conditioned safety detection. This suggests that language coverage and cultural awareness are distinct properties. A plausible implication is that systems can generalize across languages while still failing to model culturally specific safety boundaries, norms, or harms. The paper’s wording does not reduce this gap to data scarcity, model architecture, or annotation practice, so no stronger causal claim is warranted from the available record.

^^^^9sort_by9^^^^. Baselines, metrics, and interpretation of results

The baselines explicitly named in the abstract are Llama-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9ti:\9^^^^B for Task ^^^^9submittedDate9^^^^ and Llama-Guard-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9submittedDate9^^^^B for Task ^^^^9query9^^^^ (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). The evaluation metrics explicitly named are average Spearman's correlation for the dialogue-evaluation subtask and ROC-AUC for the safety-detection subtask. These choices establish two different measurement regimes: correlation-based ranking agreement for multi-dimensional dialogue evaluation, and threshold-independent discrimination performance for safety detection.

The reported outcomes can be summarized compactly:

Component	Reported metric	Reported value
Task ^^^^9submittedDate9^^^^, Llama-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9ti:\9^^^^B baseline	Highest average Spearman's correlation	^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9submittedDate9submittedDate9ti:\9submittedDate9^^^^
Task ^^^^9query9^^^^, multilingual safety subset	Top ROC-AUC	^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.9^^^^9submittedDate9max_results9ti:\9^^^^
Task ^^^^9query9^^^^, cultural subset, Llama-Guard-^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^-^^^^9submittedDate9^^^^B baseline	ROC-AUC	^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^.^^^^9sort_by9submittedDate9query9submittedDate9^^^^

Two interpretive points are especially important. First, the strongest Task ^^^^9submittedDate9^^^^ result reported in the abstract is still characterized as leaving substantial room for improvement, underscoring the difficulty of dialogue-level, multi-dimensional automatic evaluation. Second, Task ^^^^9query9^^^^ exhibits an asymmetry: systems were strong on the multilingual safety subset relative to the named baseline, but not on the cultural subset, where the baseline was superior. This asymmetry is the clearest evidence in the abstract that multilingual safety detection and culturally-aware safety should not be conflated.

A common misconception would be to read the multilingual result as evidence that safety evaluation has largely been solved once language coverage is adequate. The reported cultural-subset result argues against that interpretation (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&).

^^^^9submittedDate9^^^^. Documentation status and place within DSTC evaluation work

The publication record identifies the paper as an overview paper describing the datasets and baselines provided to participants, as well as submission evaluation results for each of the two proposed subtasks (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&). At the same time, the associated note in the available record states that a supplied document was actually a LaTeX template with example citations and did not contain the DSTC-^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^ material such as objectives, datasets, metrics, baselines, or results. That note explains why the available high-level summary is concentrated in the abstract rather than in recoverable full-text details.

This limitation matters for scholarly use. It means that the track can be characterized reliably at the level of its formal scope, task decomposition, named baselines, and headline metrics, but not at the level of dataset statistics, the identities of participating teams, the ten evaluation dimensions, or implementation specifics. For researchers, the practical consequence is that DSTC^^^^9submittedDate9query9^^^^ Track ^^^^9submittedDate9^^^^ is presently best understood as a benchmark-setting effort whose abstract already establishes three durable points: dialogue evaluation is multi-dimensional, safety evaluation must account for both language and culture, and current LLM-era baselines leave unresolved technical gaps in both areas (&&&^^^^9(Mendonça et al., 16 Sep 2025) DSTC12 Track 1 Overview of Dialog System Evaluation Track Dimensionality Language Culture and Safety9^^^^&&&).