SCB-ST-Dataset4: Classroom Behavior Corpus

• SCB-ST-Dataset4 is a comprehensive, large-scale spatio-temporal video dataset capturing key classroom behaviors—hand-raising, reading, and writing—with detailed annotations.
• It utilizes a two-stage pipeline that combines manual annotations and machine-generated pseudo-labels to efficiently propagate bounding boxes across 150-frame clips.
• Baseline evaluations with YOLO and SlowFast models highlight high detection accuracy, underscoring the dataset’s value for advancing multi-modal educational analytics.

SCB-ST-Dataset4 is a large-scale, spatio-temporal video corpus specifically constructed for the automatic detection and analysis of student classroom behaviors using deep learning. It provides a high-fidelity foundation for benchmarking temporal action detection and multi-model fusion approaches in educational settings. The dataset is centered on three core behaviors—hand-raising, reading, and writing—and is accompanied by detailed annotations, benchmarking protocols, and practical resources for immediate usage in machine learning pipelines (Yang et al., 2023).

1. Dataset Scope and Statistics

SCB-ST-Dataset4 comprises 757,265 unique video frames and 25,810 labeled spatio-temporal clips distributed across three classroom behaviors: hand-raising (class 0), reading (class 1), and writing (class 2). Each clip is a 5-second window at 30 fps, corresponding to 150 contiguous frames, resulting in approximately 3.87 million total frames when considering replication across clips.

Annotation Distribution:

Behavior	#Annotations
Hand-raising	11,207
Reading	10,841
Writing	3,762

The dataset is a culmination in a series of increasing-scale corpora, with SCB-ST-Dataset4 manifestly outscaling its predecessors both in sample size and diversity:

Dataset	#Classes	#Images	#Annotations
SCB-Dataset1	1	4,001	11,248
SCB-Dataset2	3	4,266	18,499
SCB-Dataset3-U	6	671	19,768
SCB-ST-Dataset4	3	757,265	25,810

Every labeled instance seeds a 150-frame video clip; where contiguous frames are unavailable, the key frame is temporally padded by replication. The class distribution among these clips adheres to the annotation counts specified above.

2. Construction and Annotation Protocol

SCB-ST-Dataset4 was created via a fully automated, two-stage pipeline leveraging both human annotation and machine-generated pseudo-labels, with the explicit aim of avoiding additional manual bounding-box labeling during the spatio-temporal extension phase. The construction proceeds as follows:

Stage A: Image-level Detector Training and Label Expansion

1. Approximately 1,000 full-length classroom videos (∼40 minutes each) were drawn from designated sources ("bjyhjy", "1s1k", "youke", "qlteacher", "youke-smile.shec").
2. 3–15 frames per video were uniformly sampled and manually labeled for one of the three target behaviors.
3. YOLOv7 was trained on these key annotated frames.
4. The trained detector generated labels on remaining frames, followed by rapid human-in-the-loop correction and fine-tuning.
5. The detector was retrained on the expanded annotation set.

Stage B: Spatio-Temporal Clip Generation

• For each annotated frame at time $t$ in video $v$, a consecutive 150-frame clip spanning $[t-60, t+89]$ was extracted, or padded by replicating the key frame when insufficient pre/post frames existed.
• The bounding boxes from the single annotated frame were propagated identically to all frames in the clip, with no additional manual intervention.

Pseudocode Summary:

for each (v, t, boxes) in L: if video v length ≥ t+89 and t ≥ 60: clip_frames = read_frames(v, t-60 … t+89) else: frame_f = read_frame(v, t) clip_frames = [frame_f]*60 + [frame_f]*90 clip_annotations = [boxes] * 150 Clips.append((clip_frames, clip_annotations))

No new manual labeling occurs after initial image-level annotations; all spatio-temporal annotation is obtained by box propagation.

3. Behavior Similarity Index

To quantify the visual ambiguity between target behaviors, the dataset introduces the Behavior Similarity Index (BSI). For two classes $i$ and $j$:

• $N_i$: total samples in class $i$
• $N_j$: total samples in class $j$
• $S_{i,j}$: total number of frames where class $i$ and $j$ boxes overlap.

The pairwise BSI is defined as:

$$\mathrm{BSI}(i,j) = \left( \frac{S_{i,j}}{N_i}, \frac{S_{i,j}}{N_j} \right)$$

$\mathrm{BSI}(i,j).0$ indicates the fraction of $i$'s samples overlapping $j$, and $\mathrm{BSI}(i,j).1$ vice versa. High BSI values (approaching (1,1)) indicate near-indistinguishability. In practice, BSI for reading vs. writing ranged from (4–5%, 12–19%) across evaluated YOLO models, indicating a meaningful degree of visual overlap (Yang et al., 2023).

4. Baseline Benchmarks and Performance

SCB-ST-Dataset4 serves as a rigorous testbed for both image-based and spatio-temporal models. Object detection baselines—YOLOv5 (small/medium), YOLOv7 (standard/x), YOLOv8 (small/medium)—and the dual-stream SlowFast architecture were evaluated.

Aggregate and Per-class Results:

Model	mAP@50 (%)	Hand-raise (AP@50)	Reading (AP@50)	Writing (AP@50)
YOLOv5s	72.9	—	—	—
YOLOv5m	74.7	85.4	75.2	63.3
YOLOv7	77.2	—	—	—
YOLOv7x	80.3	86.8	77.4	70.0
YOLOv8s	74.5	—	—	—
YOLOv8m	77.6	84.6	76.2	66.8
SlowFast	82.3	96.9	89.8	57.8

SlowFast achieves the best overall performance, with $>80\%$ mAP in under 10 epochs, outperforming purely spatial detectors (YOLOv7x attains $\sim 80\%$ mAP after 100 epochs). Class-wise, the highest precision is attained for hand-raising, with the lowest for writing, reflecting the inherent visual ambiguity in these behaviors.

5. Multi-Model Fusion for Student Profiling

To support enriched analytics, a multi-modal fusion strategy augments behavior detection with identity, pose, and affective information using the following parallelized streams:

• Continuous (30 fps):
  • YOLOv7 (CrowdHuman-fine-tuned) for crowd/person boxes.
  • DeepSort for multi-object tracking and persistent IDs.
  • SlowFast for per-frame behavior classification.
• Non-Continuous (1 fps):
  • YOLOv7 for student/teacher role distinction.
  • SynergyNet for head-pose estimation.
  • A facial expression model to render one of seven emotion classes.

All outputs are time-synchronized with per-person IDs and stored in a unified format: {student_ID, timestamp, behavior, pose, identity, expression}

This fusion framework enables complex downstream analytics, such as concentration scoring and behavioral profiling. The YOLOv7 CrowdHuman-fine-tuned model achieves 85.6% mAP@50 on crowd detection (Yang et al., 2023).

6. Data Access and Usage

SCB-ST-Dataset4 is publicly available at https://github.com/Whiffe/SCB-dataset. The repository contains:

• clips/: video segments (e.g., .mp4 or .avi), each 150 frames.
• annotations/: per-clip annotations in text or JSON.
• train/, val/, test/: splits for standard machine learning workflows.

Example: Download and Basic Usage

git clone https://github.com/Whiffe/SCB-dataset.git cd SCB-dataset/SCB-ST-Dataset4 ls

PyTorch Dataset Loader Snippet

class SCBST4(Dataset): def __init__(self, root, split='train', transform=None): self.root = root self.transform= transform split_file = os.path.join(root, f"{split}.txt") self.clips = [line.strip() for line in open(split_file)] def __len__(self): return len(self.clips) def __getitem__(self, idx): clip_name = self.clips[idx] imgs = [] for i in range(150): img_path = os.path.join(self.root, 'clips', clip_name, f"{i:03d}.jpg") img = Image.open(img_path).convert('RGB') if self.transform: img = self.transform(img) imgs.append(img) ann_path = os.path.join(self.root, 'annotations', clip_name + '.txt') boxes, labels = load_yolo_txt(ann_path) return torch.stack(imgs), boxes, labels

Recommended settings for SlowFast training: batch size 2, learning rate 0.1, 43 epochs, input size $640 \times 640$. Pretraining on Kinetics or AVA is advised for optimal convergence. For YOLO models, a typical configuration involves 100 training epochs with pretrained weights (Yang et al., 2023).

7. Research Impact and Benchmark Use

SCB-ST-Dataset4 addresses the challenge of large-scale, annotated, spatio-temporal classroom behavior data in education AI, enabling rigorous analysis and development of multi-modal detection systems without the costly overhead of manual video labeling. It establishes new standards for benchmarking behavioral detection, introduces the Behavior Similarity Index for quantifying class overlap, and supports multi-perspective analytics via model fusion. Its scale, annotation protocol, and accessibility position it as a foundational resource for future research in automated educational assessment and computer vision in classroom environments (Yang et al., 2023).