Weights Task Dataset (WTD) for CPS Analysis
- Weights Task Dataset (WTD) is a structured, goal-directed corpus featuring ten triads engaged in a balance scale task, exemplifying collaborative problem-solving.
- The dataset integrates gold-standard and automated transcriptions with VAD-based segmentation, enabling comparative analysis of CPS detection pipelines.
- It supports multi-label classification of CPS facets, revealing trade-offs between transcription quality and segmentation granularity on predictive performance.
Searching arXiv for the Weights Task Dataset and prior WTD-based CPS work. Searching arXiv for Bradford et al. WTD CPS detection work. The Weights Task Dataset (WTD) is a small-group, face-to-face, goal-directed collaborative learning corpus centered on a structured problem-solving activity in which ten triads determine the weights of a set of blocks using a balance scale. In recent work, it has served as a benchmark for Collaborative Problem-Solving (CPS) detection because it combines group audio/speech interactions, gold-standard (“Oracle”) segmentation and transcripts, Google ASR transcripts with segmentation based on Google’s Voice Activity Detector, and CPS annotations grounded in the framework of Sun et al. (2020) (Venkatesha et al., 6 Jul 2025). Within that usage, WTD is not an open-ended discussion dataset; it is a situated collaborative corpus designed around a shared physical task and utterance-level CPS analysis.
1. Corpus scope and collaborative setting
WTD, as described in the recent automation study, consists of ten triads, each engaged in a structured collaborative problem-solving task that requires participants to infer the weights of blocks by using a balance scale. This makes the corpus a dataset of triadic collaboration rather than dyadic tutoring, monologic explanation, or free conversation. Its interactional content is therefore shaped by coordination around a concrete external artifact and a common goal, which is central to the type of CPS behaviors annotated in the corpus (Venkatesha et al., 6 Jul 2025).
The same study treats WTD as its sole experimental testbed, which underscores the dataset’s role as a reference corpus for CPS detection rather than as one example among many. The dataset includes group audio/speech interactions, manually annotated Oracle segmentation and transcripts, automatically generated Google ASR transcripts, and CPS annotations. At the same time, several corpus-level details are not reported there: the paper does not provide participant demographics, age, educational level, session durations, total audio duration, or a native train/validation/test partition for the dataset.
2. Annotation framework and label structure
WTD’s annotation scheme is grounded in the CPS framework of Sun et al. (2020) and contains 19 distinct markers of collaborative behavior. In the cited study, those marker-level annotations are collapsed into three higher-level CPS facets: Constructing Shared Knowledge (Const), Negotiation and Coordination (Neg), and Maintaining Team Function (Maintain) (Venkatesha et al., 6 Jul 2025).
The prediction problem is therefore formulated not at the original 19-marker granularity but as a three-label multi-label classification problem. Each utterance or segment may express more than one facet simultaneously. The paper gives a concrete example: if an utterance contains subcategories such as Suggests appropriate ideas and Confirms understanding, then that utterance is labeled Const = 1, Neg = 0, and Maintain = 0, because those subcategories belong to the Constructing Shared Knowledge facet.
This aggregation has methodological consequences. It simplifies the output space and permits facet-level AUROC reporting, but it also removes some of the original annotation granularity. A plausible implication is that WTD supports both fine-grained discourse annotation and coarser CPS-facet prediction, but the cited study evaluates only the latter.
3. Segmentation regimes and units of analysis
The unit of analysis in WTD is the utterance/segment. Under Oracle conditions, this is a manually segmented utterance. Under automated conditions, it is an automatically segmented speech segment produced by Google’s VAD-based pipeline. This distinction is central because the paper’s main question is whether changes in transcription and segmentation alter downstream CPS detection or only the form in which the same interaction is represented (Venkatesha et al., 6 Jul 2025).
| Segmentation regime | Utterances | Const / Neg / Maintain |
|---|---|---|
| Oracle segmentation | 2482 | 906 / 866 / 391 |
| Google segmentation | 1824 | 664 / 642 / 328 |
The shift from 2482 Oracle utterances to 1824 Google-segmented utterances corresponds to a reduction of
or 26.5%. The paper attributes this coarsening largely to segment merging. It reports 518 instances in which multiple Oracle segments were merged into one Google segment, including 18 cases in which more than two Oracle segments were merged.
A crucial methodological step is the transfer of CPS labels onto automated segments. The Google segments inherit the labels of all of the Oracle segments that fall under the time span of the Google segment. This means that a single automatically segmented unit may accumulate several CPS labels from distinct underlying utterances. The result is greater multi-label density, reduced turn-level precision, and less direct access to “who said what and when.”
4. Feature extraction, models, and evaluation protocol
The study uses multimodal utterance-level features from WTD. Linguistic features are extracted with BERT-base-small and prosodic/acoustic features with openSMILE; the two feature types are then concatenated into a single multimodal representation (Venkatesha et al., 6 Jul 2025). The paper does not specify the exact BERT embedding extraction strategy, the exact openSMILE feature set used in practice, whether features are normalized, or whether speaker identity or diarization features are included.
CPS detection is evaluated with traditional supervised classifiers rather than deep sequence models. The models considered are Random Forest and AdaBoost, and deep learning models are explicitly not used because the dataset was considered too small to support them without overfitting. Hyperparameters are tuned with Hyperopt for 500 search iterations, using average AUROC as the optimization target. Tuning is performed only on the Oracle segmentation + Oracle transcription condition, and the selected model is then reused across the remaining conditions.
The best model is a Random Forest with criterion = entropy, max features = None, and number of estimators = 148. Evaluation uses leave-one-group-out cross-validation, which is appropriate because WTD contains ten triads and each fold can leave out one group. Reported metrics are AUROC, plus average precision and average recall across the three CPS facets.
5. Reported performance under manual and automatic pipelines
The central empirical comparison is across four pipeline conditions: Oracle segmentation with Oracle transcription, Oracle segmentation with Google transcription, Google segmentation with Oracle transcription, and Google segmentation with Google transcription (Venkatesha et al., 6 Jul 2025).
| Segmentation | Transcription | Average AUROC | Avg. Precision | Avg. Recall |
|---|---|---|---|---|
| Oracle | Oracle | 0.744 | 0.704 | 0.297 |
| Oracle | 0.698 | 0.528 | 0.246 | |
| Oracle | 0.740 | 0.658 | 0.339 | |
| 0.679 | 0.601 | 0.342 |
The best overall condition is Oracle segmentation + Oracle transcription, with average AUROC 0.744. The closest second is Google segmentation + Oracle transcription, at 0.740. The worst overall condition is the fully automated pipeline, Google segmentation + Google transcription, with average AUROC 0.679.
A central finding is that automated transcription hurts performance more clearly than automated segmentation. With Oracle segmentation fixed, replacing Oracle transcripts by Google transcripts lowers average AUROC from 0.744 to 0.698, precision from 0.704 to 0.528, and recall from 0.297 to 0.246. By contrast, with Oracle transcripts fixed, replacing Oracle segmentation by Google segmentation changes AUROC only slightly, from 0.744 to 0.740, while precision drops from 0.704 to 0.658 and recall rises from 0.297 to 0.339.
The paper interprets this as a trade-off between headline predictive performance and interactional granularity. Automated segmentation can preserve aggregate AUROC while producing fewer, broader units that merge multiple contributions. It also reports facet-level AUROC values showing that Maintain is consistently lower than Const and Neg; this suggests that Maintaining Team Function is the hardest facet under the tested feature and model regime.
6. Methodological implications and limitations
The paper argues that WTD supports the development of practical CPS monitoring tools because a fully automated pipeline remains feasible even when both transcription and segmentation are automated. At the same time, it emphasizes that educationally useful systems need to preserve not only whether a CPS behavior occurred, but also what behavior occurred, who produced it, and when. In that respect, segmentation quality is not a secondary preprocessing detail; it directly shapes interpretability and pedagogical actionability (Venkatesha et al., 6 Jul 2025).
Several limitations are made explicit. First, the evaluation is single-dataset: only WTD is used, which limits claims about generalization to other collaborative settings, age groups, or task formats. Second, the task itself is controlled and structured; the balance-scale activity ensures consistency, but may not represent broader classroom collaboration. Third, the study evaluates a specific automated segmentation system, namely Google’s VAD-based segmentation, which is said to under-segment; other pipelines might over-segment or behave differently. Fourth, the label-transfer rule is based on temporal overlap, so alternative mapping rules could change the results. Fifth, the 26.5% reduction in utterance count under automated segmentation entails a real loss of granularity even when aggregate AUROC remains comparatively stable.
These limitations imply that WTD is especially strong as a benchmark for automation sensitivity in a tightly defined collaborative task. A plausible implication is that its greatest value lies in controlled comparisons of Oracle versus automated speech-processing components, rather than in broad claims about CPS detection in all classroom discourse.
7. Relation to prior WTD work and terminological ambiguity
The cited study positions itself relative to earlier WTD-based CPS research, especially Bradford et al. (2023), which showed that CPS markers could be detected using multimodal features in naturalistic small-group settings, as well as WTD dataset papers by Khebour et al. and related discourse-annotation and segmentation-aid work by Terpstra et al. (2023). Its distinctive contribution is not a new WTD release, but a four-condition benchmark for testing how automatic transcription and automatic segmentation affect CPS detection on the same corpus (Venkatesha et al., 6 Jul 2025).
The acronym WTD should also be distinguished from unrelated uses of “weights task” language in other areas. In the broader arXiv literature, similarly named or informally described “weights task” resources refer to very different objects, including body weight estimation from pressure images in “MassNet” (Wu et al., 2023), dataset-size recovery from LoRA weights in “LoRA-WiSE” (Salama et al., 2024), and longitudinal weight prediction from diet diaries in “DietDiary” (Gui et al., 2024). Those works concern physiological weight or model-parameter inference, not the collaborative problem-solving corpus described here. This contrast is useful because it clarifies that, in its established educational-computing usage, the Weights Task Dataset denotes a CPS-annotated triadic interaction dataset built around a balance-scale block-weight task, not a generic benchmark for learning from numerical or neural-network weights.