Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation

Abstract: Recent progress in Temporal Action Segmentation (TAS) has increasingly relied on complex architectures, which can hinder practical deployment. We present a lightweight dual-loss training framework that improves fine-grained segmentation quality with only one additional output channel and two auxiliary loss terms, requiring minimal architectural modification. Our approach combines a boundary-regression loss that promotes accurate temporal localization via a single-channel boundary prediction and a CDF-based segment-level regularization loss that encourages coherent within-segment structure by matching cumulative distributions over predicted and ground-truth segments. The framework is architecture-agnostic and can be integrated into existing TAS models (e.g., MS-TCN, C2F-TCN, FACT) as a training-time loss function. Across three benchmark datasets, the proposed method improves segment-level consistency and boundary quality, yielding higher F1 and Edit scores across three different models. Frame-wise accuracy remains largely unchanged, highlighting that precise segmentation can be achieved through simple loss design rather than heavier architectures or inference-time refinements.

• The paper introduces a dual-loss framework that improves fine-grained action segmentation by combining boundary supervision with CDF-based segment regularization.
• The boundary-regression loss enhances temporal localization, while the segment-level regularization reduces over-segmentation and fragmented predictions.
• Experimental results on benchmarks like GTEA and 50Salads show significant improvements in F1 score and edit distance with minimal additional model overhead.

Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation

Introduction

Fine-grained temporal action segmentation (TAS) remains a critical challenge in video understanding, hampered by over-segmentation and boundary misalignment when using standard frame-wise objectives. The paper "Combining Boundary Supervision and Segment-Level Regularization for Fine-Grained Action Segmentation" (2604.01859) proposes a lightweight, architecture-agnostic framework introducing two complementary auxiliary loss functions: a boundary-regression loss for temporal localization and a CDF-based segment shape regularization loss for segment structure. Both are designed for minimal architectural modification and can be applied as training-time objectives to any TAS model.

Figure 1: Overview—left: integration of a single boundary output channel and auxiliary losses into a standard TAS model; right: improved F1/parameter trade-off on GTEA.

Methodology

The proposed method augments existing TAS backbones by adding a single boundary channel and two auxiliary losses, maintaining computational efficiency. The central components are:

1. Boundary-Regression Loss ($\mathcal{L}_{B}$): Introduces explicit supervision at temporal action transitions via an additional boundary channel. The loss is a binary cross-entropy over a class-agnostic boundary mask, encouraging high boundary responses proximate to true segment changes (Figure 2).
2. CDF-Based Segment Shape Regularization Loss ($\mathcal{L}_{S}$): Imposes segment-level structural consistency by penalizing the squared $L_2$ distance between cumulative distributions (CDFs) of predicted versus ground-truth segment label probabilities. This quantifies and reduces over-segmentation and fragmented predictions within segments (Figure 3).

Losses are applied with temporal decoupling to reduce gradient conflicts: $\mathcal{L}_{B}$ is enforced only around boundaries, while $\mathcal{L}_{S}$ is activated within non-boundary regions. The overall loss is added to the backbone's standard objective (e.g., frame-wise cross-entropy), with hyperparameters for the loss weights and warm-up period empirically tuned per dataset and architecture (Figure 4).

Figure 4: Architecture-agnostic framework—boundary-regression loss focuses on boundaries, CDF regularization aligns segment shape, both integrating with arbitrary TAS backbones.

Figure 2: Boundary channel output (orange curve) aligns with binary ground-truth boundaries to localize transitions.

Figure 3: CDF-based segment regularization enforces consistent cumulative structure within predicted and true segments.

Experimental Results

Quantitative Analysis

Evaluation on GTEA, 50Salads, and Breakfast benchmarks, using MS-TCN, C2F-TCN, and FACT backbones, demonstrates that the proposed approach consistently boosts segmental F1 (notably F1@10) and Edit scores, while frame-wise accuracy is mostly unaffected. Gains are particularly significant for MS-TCN on 50Salads (+4.0% Edit, +3.5% F1@10) and GTEA, validating strong performance/complexity trade-offs with negligible parameter increase.

Compared to boundary-aware methods such as ASRF and BCN, the proposed dual-loss configuration achieves competitive or superior results with dramatically less model complexity. For example, BCN’s improvements incur a >16x cost in MACs and parameters versus the almost overhead-free boundary channel in this work (Figure 5).

Figure 6: The proposed method yields smoother, more aligned segmentations across datasets versus baseline MS-TCN.

Figure 5: Superior trade-off—F1@10 vs. model parameters, highlighting minimal complexity increase vs. prior methods.

Ablation and Qualitative Studies

Ablation confirms that both $\mathcal{L}_{B}$ and $\mathcal{L}_{S}$ contribute complementary improvements: boundary regression sharpens transitions, segment regularization enhances segment consistency, and their combination is optimal. Temporally decoupled loss assignment outperforms naive application to all frames, as the targeted strategy mitigates gradient contention and promotes structural accuracy (Figure 7).

Figure 7: Decoupled (“Ours”) loss assignment improves both segment quality and boundary localization near action transitions.

Sensitivity analysis shows applying $\mathcal{L}_{S}$ after initial model convergence is critical, substantiating its effect once coarse frame-level predictions are stable.

Synergy with Post-Processing

The framework is compatible with post-hoc refinements such as ASOT. On GTEA and 50Salads, combining the proposed training losses with ASOT further boosts performance. However, the effect is dataset-dependent: on Breakfast, ASOT degrades results, whereas the proposed method alone consistently improves all metrics, emphasizing its robustness as a standalone regularizer.

Practical and Theoretical Implications

From an application perspective, this framework enables accurate, boundary-aware TAS in deployment-constrained settings, eliminating the need for branch-heavy architectures or inference-time refinement. The method’s architecture-agnostic nature supports indefinite future extensibility to new backbones, including transformers or other sequence models. Theoretically, the empirical separation of boundary localization and segment-level shape encodes desirable inductive biases for temporal structure—suggesting an avenue for further principled investigation, for instance, by connecting the regularization term to explicit optimal transport objectives.

Future Directions

Potential developments include extending the framework to semi-supervised or weakly-supervised TAS, leveraging the regularization effects where full annotation is unavailable. Additionally, integrating this loss strategy into online or real-time segmentation pipelines could further bridge the gap between high-accuracy models and operational constraints. There is also opportunity to refine loss formulations, for example, via adaptive or learned decoupling between boundary and segment regions, or by augmenting the CDF-based regularizer with more expressive structural constraints tailored to hierarchical or overlapping action classes.

Conclusion

Combining explicit boundary supervision with CDF-based segment regularization addresses longstanding challenges in fine-grained action segmentation without incurring significant model complexity. This dual-loss framework advances temporal coherence and boundary fidelity in TAS, equipping standard architectures with structural guidance absent in frame-wise objectives, and achieving strong empirical results with negligible overhead (2604.01859). The architecture-agnostic, lightweight nature of this approach promotes its adoption in both current and future sequence segmentation frameworks.