End-to-End Referring Video Object Segmentation with Multimodal Transformers (2111.14821v2)

Abstract: The referring video object segmentation task (RVOS) involves segmentation of a text-referred object instance in the frames of a given video. Due to the complex nature of this multimodal task, which combines text reasoning, video understanding, instance segmentation and tracking, existing approaches typically rely on sophisticated pipelines in order to tackle it. In this paper, we propose a simple Transformer-based approach to RVOS. Our framework, termed Multimodal Tracking Transformer (MTTR), models the RVOS task as a sequence prediction problem. Following recent advancements in computer vision and natural language processing, MTTR is based on the realization that video and text can be processed together effectively and elegantly by a single multimodal Transformer model. MTTR is end-to-end trainable, free of text-related inductive bias components and requires no additional mask-refinement post-processing steps. As such, it simplifies the RVOS pipeline considerably compared to existing methods. Evaluation on standard benchmarks reveals that MTTR significantly outperforms previous art across multiple metrics. In particular, MTTR shows impressive +5.7 and +5.0 mAP gains on the A2D-Sentences and JHMDB-Sentences datasets respectively, while processing 76 frames per second. In addition, we report strong results on the public validation set of Refer-YouTube-VOS, a more challenging RVOS dataset that has yet to receive the attention of researchers. The code to reproduce our experiments is available at https://github.com/mttr2021/MTTR

• The paper introduces MTTR, a unified Transformer that extracts both visual and linguistic features for efficient RVOS.
• The paper formulates RVOS as a sequence prediction problem, enabling effective object tracking and segmentation across video frames.
• The paper demonstrates significant mAP and IoU improvements on benchmarks, highlighting its potential for real-time video analytics.

End-to-End Referring Video Object Segmentation with Multimodal Transformers

The paper presents a novel approach to the task of Referring Video Object Segmentation (RVOS), leveraging a Transformer-based architecture referred to as the Multimodal Tracking Transformer (MTTR). This model addresses the challenges of RVOS, which involves segmenting and tracking a specific object described by a textual query across video frames. The complexity of RVOS arises from the need to integrate multiple modalities: natural language understanding, video processing, instance segmentation, and tracking.

Key Contributions

MTTR introduces a simplified architecture compared to existing solutions, which typically require intricate pipelines. The MTTR model processes video and text simultaneously using a single multimodal Transformer. By modeling RVOS as a sequence prediction problem, MTTR circumvents the text-related inductive biases and mask refinement steps traditional methods employ.

1. Transformer Architecture: The model employs a unified Transformer for both linguistic and visual feature extraction, leveraging advancements like the Swin Transformer for visual data and a text encoder based on RoBERTa.
2. Sequence Prediction: MTTR views the task through the lens of sequence prediction, allowing natural tracking by detecting and following object sequences across frames without needing manual alignment.
3. Temporal Segment Voting Scheme (TSVS): This novel inference mechanism scores predicted sequences based on their association with the query text, improving decision accuracy even in challenging conditions where the object may be occluded or absent in some frames.

Performance Evaluation

MTTR's efficacy is evaluated on established benchmarks such as A2D-Sentences, JHMDB-Sentences, and Refer-YouTube-VOS. MTTR demonstrates superior performance over previous state-of-the-art methods, with substantial improvements in mean Average Precision (mAP) and Intersection over Union (IoU) metrics on A2D-Sentences (+5.7 mAP and +5.0 mAP improvements on A2D-Sentences and JHMDB-Sentences respectively). These results highlight MTTR's capability to produce precise instance masks swiftly, processing up to 76 frames per second.

Implications and Future Directions

The MTTR framework greatly simplifies the RVOS process, reducing the need for complex component integration seen in traditional methods. This streamlined approach not only improves performance but also opens pathways for further exploration of Transformer-based solutions in multimodal contexts.

From a theoretical standpoint, this work demonstrates the power of sequence prediction in realizing end-to-end solutions for complex vision-language tasks. Practically, this can lead to more robust video segmentation applications in fields such as autonomous driving, video editing, and augmented reality, where real-time performance is crucial.

Future research may explore scaling up the Transformer architecture, investigating the effects of larger models and training on expansive datasets. Additionally, adapting MTTR for real-time applications in dynamic environments offers an exciting area of exploration.

In conclusion, MTTR provides an effective blueprint for the integration of multimodal data within a single architectural framework, setting a new standard for RVOS tasks and potentially influencing other domains requiring seamless integration of video and textual data.