BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video

Abstract: Multiple existing benchmarks involve tracking and segmenting objects in video e.g., Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS), but there is little interaction between them due to the use of disparate benchmark datasets and metrics (e.g. J&F, mAP, sMOTSA). As a result, published works usually target a particular benchmark, and are not easily comparable to each another. We believe that the development of generalized methods that can tackle multiple tasks requires greater cohesion among these research sub-communities. In this paper, we aim to facilitate this by proposing BURST, a dataset which contains thousands of diverse videos with high-quality object masks, and an associated benchmark with six tasks involving object tracking and segmentation in video. All tasks are evaluated using the same data and comparable metrics, which enables researchers to consider them in unison, and hence, more effectively pool knowledge from different methods across different tasks. Additionally, we demonstrate several baselines for all tasks and show that approaches for one task can be applied to another with a quantifiable and explainable performance difference. Dataset annotations and evaluation code is available at: https://github.com/Ali2500/BURST-benchmark.

• The paper introduces the BURST dataset as a unified benchmark that consolidates object recognition, segmentation, and tracking tasks in video analysis.
• The authors deploy a semi-automated annotation process and HOTA metrics to standardize evaluations across exemplar-guided and class-guided tasks.
• Baseline results highlight performance gaps in long-tail tasks, prompting future research for robust and generalized video analysis methods.

An Expert Analysis of "BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video"

The paper under review, "BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video," presents an ambitious endeavor to consolidate various subfields of video analysis into a coherent framework. This work introduces BURST, a comprehensive dataset and benchmark that aims to bridge gaps between disparate research areas such as Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS). By doing so, it facilitates the development of generalized methods capable of simultaneously addressing multiple video analysis tasks. The authors have provided a dense yet meticulously crafted benchmark dataset that includes a high volume of diverse videos with pixel-level annotations, underscoring their commitment to enhancing comparability and knowledge dissemination across related domains.

Key Contributions

The primary contribution of this work is the creation of the BURST dataset, which serves as a unified platform for evaluating several tasks in object recognition, segmentation, and tracking across video sequences. The dataset includes a significant number of video clips annotated with high-quality object masks, offering a rich playground for assessing complex video tasks previously evaluated in isolation. The unified benchmark encompasses six tasks categorized under two main streams—exemplar-guided and class-guided—each with distinct requirements.

Exemplar-Guided Tasks:

1. Mask Task: Provides initial object masks for guiding subsequent tracking and segmentation.
2. Box Task: Relies on initial bounding boxes, necessitating further refinement to achieve precise segmentation.
3. Point Task: Utilizes single interior points as initial cues, demanding sophisticated object inference techniques.

Class-Guided Tasks:

1. Common Task: Involves object classes commonly studied in standard datasets like COCO.
2. Long-tail Task: Addresses the challenge posed by infrequently occurring object classes, thus testing the robustness of algorithms in recognizing less common instances.
3. Open-World Task: Evaluates the capability of algorithms to generalize beyond trained categories to previously unseen classes, promoting innovation in open-set recognition and tracking.

Evaluation Metrics

The authors adopt HOTA (Higher Order Tracking Accuracy) as a unified evaluation metric across all tasks, praised for its balanced assessment of detection accuracy and temporal association. Importantly, this facilitates quantitative comparisons across different task paradigms within the benchmark. For open-world scenarios, the researchers employ OWTA, a variant of HOTA that excludes false positive penalties, thereby aligning well with the open-set nature of the task.

Dataset and Annotation

The BURST dataset is constructed using videos from the TAO dataset, re-annotated to provide pixel-precise masks crucial for fine-grained video analysis. The authors detail a semi-automated process for densifying the temporal annotations of their training set, merging machine-generated data with meticulous human verification to ensure high-quality labeling. With 2,914 videos spanning diverse environments (e.g., indoor, outdoor, scripted, and non-scripted scenes), the dataset presents a comprehensive experimental environment to foster advancements in video analysis models.

Baseline Results and Insights

The paper provides baseline results using contemporary methods like STCN and tracking-by-detection frameworks. Their findings highlight strengths and weaknesses across various tasks, illustrating the potential for knowledge transfer between class-guided and exemplar-guided methods. The results underscore substantial performance gaps in the long-tail task, indicating future research opportunities for improving recognition performance in infrequent classes.

Implications for Future Research

BURST stands as a seminal contribution towards standardizing evaluation pipelines across related sub-domains, promoting the development of unified approaches that maintain high performance across diverse scenarios. The benchmark not only provides a fertile ground for existing methods to be meaningfully compared but also challenges the research community to innovate and tackle the intricacies associated with long-tail and open-world recognition tasks. Given its comprehensive scope and robust evaluation framework, BURST is poised to influence future directions in video analysis research, encouraging the synthesis of previously isolated methodologies into more holistic systems capable of operating in diverse, real-world settings.

Conclusion

In summary, the BURST paper makes a substantial contribution towards unifying video object recognition, segmentation, and tracking benchmarks. By facilitating cross-task comparisons and promoting comprehensive method development, it sets a new standard for evaluating and advancing the state-of-the-art in video analysis.