EvoStruggle: Modeling Skill Struggle Evolution

• EvoStruggle is a research framework that quantifies struggle during human skill tasks by analyzing annotated video data over repeated attempts.
• It employs a temporal action localization approach, using expert-curated annotations to precisely mark struggle intervals across diverse activities.
• Empirical results show a marked reduction in struggle time with practice, offering insights for adaptive system design and evolutionary modeling.

EvoStruggle refers to a research framework, dataset, and associated computational paradigm aimed at quantifying and modeling the evolution of struggle during human skill acquisition, particularly through the lens of temporal action localization on video data. The EvoStruggle dataset and methodology facilitate fine-grained tracking of how struggle occurs, evolves, and eventually diminishes as individuals repeat skill-based tasks. This is situated within a broader research context encompassing adaptive learning, assistive system design, and the evolutionary selection of mechanisms supporting robustness and generalization in both biological and artificial systems (Feng et al., 1 Oct 2025). Further, the theoretical perspective on the evolutionary adaptivity of cooperation and struggle connects to broader inquiries into the long-term robustness of evolving populations and the design of artificial evolutionary algorithms (Ivanko, 2017).

1. Dataset Design and Scope

EvoStruggle provides a large-scale, annotated resource for the temporal localization of struggle in human task performance. The dataset comprises 61.68 hours of high-definition video (1920×1080 resolution at 50 FPS), totaling 2,793 untrimmed videos with 5,385 human-annotated struggle segments. Data were collected from 76 unique participants across 126 recording sessions, systematically distributed among four core activities: tying knots (34), origami (32), tangram puzzles (30), and shuffling cards (30). Each of the 18 distinct task variants—4 knot types, 5 origami shapes, 5 tangram puzzles, and 4 card shuffles—was repeated five times per participant, enabling longitudinal observation of skill development and the evolution of struggle.

A two-stage annotation protocol was used: first, a domain expert labeled struggle instances using rapid playback and explicit key tap marking of perceived hesitation, repeated attempts, prolonged actions, or frustration. Next, keyframe clusters were consolidated, defining precise start and end points for each struggle segment. Consistency was prioritized via a single-expert pipeline after pilot studies revealed non-expert annotations produced noisy segment boundaries. No inter-annotator agreement metrics are reported.

Analyses show that in early task repetitions, participants spend approximately 60% of their time in struggle, decreasing to about 24% by the fifth attempt, capturing quantitative reduction in struggle as skill is acquired (Feng et al., 1 Oct 2025).

2. Temporal Action Localization Formulation

The struggle determination problem in EvoStruggle is cast as a Temporal Action Localization (TAL) task. The input consists of a sequence of video features:

$$X = \{x_1, x_2, \ldots, x_n\}, \quad x_i \in \mathbb{R}^d$$

where $x_i$ are $d$-dimensional features per video snippet obtained using state-of-the-art feature extractors such as I3D or SlowFast.

The target output is a set of temporally localized interval proposals

$$S = \{ s_k \}_k, \quad s_k = (t_k^{\text{start}}, t_k^{\text{end}})$$

where $s_k$ are predicted start and end frame indices corresponding to struggle segments. Each proposal is classified as either ‘struggle’ or ‘non-struggle’. The ground-truth and predicted segments are evaluated using temporal Intersection-over-Union (tIoU)

$\text{IoU}(G, P) = \frac{|G\cap P|}{|G\cup P|}$

with the primary metric being mean Average Precision (mAP) evaluated across thresholds $T = \{0.3, 0.5, 0.7\}$:

$mAP = \frac{1}{|T|} \sum_{\tau\in T} AP(\tau)$

where $AP(\tau)$ is average precision at threshold $\tau$ (Feng et al., 1 Oct 2025).

3. Model Architectures and Experimental Protocols

EvoStruggle establishes baseline performance using three recent TAL models:

• Actionformer: Utilizes pre-extracted snippet features (SlowFast or I3D), a transformer encoder, and a multi-scale proposal head.
• TriDet: Adopts the feature-based pipeline with “Trident” heads for boundary regression.
• Re2TAL: Implements an end-to-end approach using a reversible SlowFast-101 backbone and Actionformer-based detection head, jointly trained.

Standard TAL losses (classification and boundary regression) are used, with AdamW optimizer (learning rate ≈ 1e-4, weight decay ≈ 1e-3), and batch sizes between 16–32 video snippets per GPU.

Three data split paradigms are employed:

• Within-Activity: Train/validation split (70/30 or 80/20) within each activity, with participant disjointness.
• Task-Generalization: Leave-one-task-out within each activity.
• Activity-Generalization: Leave-one-activity-out over all activities.

No special architectural adaptations were introduced for the struggle detection task beyond reliance on standard TAL pipelines (Feng et al., 1 Oct 2025).

4. Experimental Results and Cue Analysis

Empirical results establish baseline mAP values for struggle localization across various settings:

Evaluation Protocol	Actionformer	TriDet	Re2TAL
Within-Activity (mean mAP)	Tying Knots: 39.4%<br>Origami: 28.0%<br>Tangram: 29.9%<br>Shuffle: 50.8%	41.9%<br\>28.4%<br\>30.7%<br\>49.6%	35.9%<br\>32.4%<br\>44.6%<br\>59.8%
Task-Generalization (overall average)	≈29.9%	≈30.9%	≈34.6%
Activity-Generalization (overall average)	—	—	≈19.2%

Key findings demonstrate that struggle is a transferable visual concept across multiple tasks (task-generalization mAP ≈ 34.6%), but performance declines significantly in cross-activity generalization (mAP ≈ 19.2%), evidencing domain and context-dependent difficulties.

Models effectively leverage salient struggle cues: large, abrupt hand motions; extended pauses; and anomalous hand-object dynamics (e.g., repeated dropping of cards). Struggle localized as short, subtle hesitations (particularly in origami) remains a substantial challenge; both very long and transient struggle segments are often mislocalized or incompletely predicted. Quantitative metrics reveal that including frequent struggle events (early learning attempts) is necessary for robust model generalization, as excluding novice-phase data diminishes performance (Feng et al., 1 Oct 2025).

5. Interpretation, Limitations, and Applications

Longitudinal analysis across task repetitions underscores that the time spent struggling, as well as overall task duration, decrease monotonically: from ∼60% to ∼24% average struggle time over five attempts. This demonstrates the value of capturing evolving behaviors for assistive system design, where real-time detection of struggle trajectories enables the inference of learner skill level and adaptation of tutoring or robotic interventions.

EvoStruggle’s scope is currently limited to four desk-based activities. Expansion to other domains (such as assembly or cooking) could introduce qualitatively distinct struggle modalities. The use of a single annotator precludes formal inter-annotator reliability analysis, indicating an axis for further dataset refinement. Additionally, current TAL approaches underperform on extremely short or very long struggle intervals, suggesting potential future directions involving hierarchical temporal modeling or multimodal integration (e.g., force sensing, audio streams).

The authors make EvoStruggle publicly available with the aim of advancing the state of temporal struggle localization, generalization, and the development of adaptive assistive systems (Feng et al., 1 Oct 2025).

6. Broader Evolutionary Perspective: Altruism and Struggle in Artificial Systems

The concept of struggle, as operationalized in EvoStruggle, dovetails with broader evolutionary questions regarding adaptivity and robustness in biological and computational systems. “Should Evolution Necessarily be Egolution?” provides formal models contrasting altruistic and egoistic energy exchange in evolving populations. Altruistic exchanges enable small organisms to “catch up,” maintaining population diversity and parallel search trajectories through genome space, while egoistic exchanges rapidly deplete small organisms, resulting in collapse of genetic diversity and poor adaptability.

Quantitative tests with up to 100 organisms in a 100×100-periodic environment, and complex stress-test scenarios, show that only populations with altruistic energy sharing (never egoistic) achieve the highest level of “dexterous” evolutionary adaptivity (successful and resilient target sequence navigation; e.g., ms=7, ds=6, ns=7 yields ≈9000 steps to final target, minimum diversity ≈4). A regression analysis links size parameters to performance:

$x_i$0

Idyll-tests further indicate that these populations avoid catastrophic overspecialization.

These findings suggest that in both natural and artificial systems, mechanisms that mitigate bottlenecks and promote cooperative adaptation can yield superior resilience and flexibility. In the context of genetic algorithms, introducing diversity-preserving operators and cooperative reproduction probability, or adaptively sharing selection probability, can enhance algorithmic robustness and search efficiency (Ivanko, 2017).

7. Implications for Adaptive System Design and Evolutionary Modeling

The EvoStruggle framework operationalizes struggle as a temporally localizable, observable dynamic whose progression encodes meaningful information about skill acquisition stage. When integrated into real-world tutoring or human-robot interfaces, the detection of struggle patterns offers a principled method for inferring user competence and adapting assistance accordingly.

At the meta-level, explicit modeling of struggle, as opposed to implicit error or reward signals, aligns with broader lessons from evolutionary theory: long-term adaptivity and generalization benefit from cooperative, diversity-maintaining mechanisms. This has direct implications for artificial evolutionary system design, robustness testing of adaptive controllers, and prospective research into the origins of complex, cooperative behaviors in both computational and biological domains (Feng et al., 1 Oct 2025, Ivanko, 2017).