Invert4TVG: Inversion Tasks for TVG

• Invert4TVG is a framework that reverses the traditional TVG mapping by recovering action-related query information from video segments.
• It introduces three inversion tasks—Verb Completion, Action Recognition, and Video Description—to enhance both action semantics and localization performance.
• The method leverages an 80/20 reinforcement learning schedule to balance direct temporal localization with auxiliary semantic inversion tasks.

Searching arXiv for the specified paper and closely related TVG work to ground the article with citations. [arXiv search] Query: "Invert4TVG Temporal Video Grounding inversion tasks (Chen et al., 10 Aug 2025)" Inversion Tasks for TVG (Invert4TVG) is a framework for Temporal Video Grounding (TVG) that treats grounding not only as temporal localization but as action-centered video understanding. In the standard formulation, TVG maps a video–query pair to a temporal interval, $\text{TVG}(V,q)\to \tau$. Invert4TVG introduces an inverse formulation, $\text{Invert-TVG}(V,\tau)\to q'$, in which the model derives query-related action content from a grounded segment, and it uses this reversal to improve both localization accuracy and semantic robustness without additional data (Chen et al., 10 Aug 2025).

1. Conceptual basis and relation to “inversion”

The central motivation of Invert4TVG is that many modern TVG methods, especially RL-based LVLM methods such as TimeZero and Time-R1, optimize heavily for temporal IoU and formatting correctness, which can cause overfitting to localization metrics rather than understanding the actual action semantics. The framework is built on the claim that TVG is fundamentally a video understanding problem, especially involving actions and verbs in the query. Under this view, a model that localizes “person opens the door” must not merely predict a segment with high overlap, but must also understand “opens” as the semantic anchor of the query (Chen et al., 10 Aug 2025).

The term “inversion” in Invert4TVG denotes reversal of the task mapping rather than reconstruction from gradients or latent variables. This usage differs from the inversion settings in “GradViT: Gradient Inversion of Vision Transformers” (Hatamizadeh et al., 2022), where the objective is to reconstruct hidden inputs from shared gradients, and from “Meta-Auxiliary Network for 3D GAN Inversion” (Jiang et al., 2023), where inversion means mapping a real image to an editable latent code while adapting a 3D-aware generator. Invert4TVG instead inverts TVG annotations into auxiliary segment-to-language tasks that recover action semantics from the ground-truth temporal span.

The paper’s argument is that improved action understanding raises the ceiling on grounding accuracy. A plausible implication is that temporal localization and semantic comprehension are not separable optimization targets in TVG: if action semantics degrade during RL post-training, the upper bound of localization performance may also decline.

2. Formalization and the three inversion tasks

Invert4TVG derives three inversion tasks from existing TVG annotations: Verb Completion (VC), Action Recognition (AR), and Video Description (VD). Each begins from the original TVG sample $(V,q,\tau)$ and uses the ground-truth segment $\tau$ to recover query-related action information (Chen et al., 10 Aug 2025).

Task	Input and output	Reward criterion
Verb Completion (VC)	Input: $\tau$ and a query with the action verb masked; output: the missing verb or a sentence containing it	Reward $1$ if predicted verb equals the ground-truth verb or a tense variant; otherwise $0$
Action Recognition (AR)	Input: $\tau$ and a prompt such as “Use a verb to describe the event based on the video”; output: a single verb	Reward $1$ if the predicted verb appears in the set of verbs from the ground-truth query sentence, including tense variants
Video Description (VD)	Input: $\tau$ and a prompt such as “Describe what people have done based on the video”; output: a free-form description	Reward $\text{Invert-TVG}(V,\tau)\to q'$0 if the ground-truth verb appears in the generated sentence, including tense variants

Verb Completion masks the action verb in the query and asks the model to recover it from the aligned segment. The paper gives the example “Person closed the door” becoming “Person [ ] the door,” together with the prompt “Add a verb for the event ‘Person [ ] the door’ based on the video.” Its reward is binary and treats tense variants as equivalent: $\text{Invert-TVG}(V,\tau)\to q'$1

Action Recognition asks the model to identify the action directly from the segment. It uses the set of verbs extracted from the ground-truth query sentence and rewards the model if the predicted verb belongs to that set, again counting tense variants as correct: $\text{Invert-TVG}(V,\tau)\to q'$2

Video Description asks for a free-form sentence describing the segment and checks whether the ground-truth verb is present in the predicted description: $\text{Invert-TVG}(V,\tau)\to q'$3

The paper characterizes these tasks as complementary. VC reinforces query–action mapping, AR is the most direct segment-to-verb action-perception task, and VD promotes broader segment-level understanding rather than only single-verb prediction.

3. Reinforcement-learning framework and reward design

Invert4TVG fine-tunes an LVLM, specifically Qwen2.5-VL, with reinforcement learning over both the main TVG task and the auxiliary inversion tasks. The framework builds on GRPO (Group Relative Policy Optimization), as used in DeepSeek-R1, and the paper states that the key point is a task-dependent reward signal that optimizes both grounding and semantic inversion behavior (Chen et al., 10 Aug 2025).

For standard TVG, the reward combines IoU reward and format reward. The IoU reward is

$\text{Invert-TVG}(V,\tau)\to q'$4

where $\text{Invert-TVG}(V,\tau)\to q'$5 is the predicted interval and $\text{Invert-TVG}(V,\tau)\to q'$6 is the ground-truth interval. The format reward requires the structured response

$\text{Invert-TVG}(V,\tau)\to q'$7

and assigns $\text{Invert-TVG}(V,\tau)\to q'$8 for wrong format and $\text{Invert-TVG}(V,\tau)\to q'$9 for correct format. The full TVG reward is therefore

$(V,q,\tau)$0

For inversion tasks, the reward is

$(V,q,\tau)$1

where $(V,q,\tau)$2 is one of $(V,q,\tau)$3, $(V,q,\tau)$4, or $(V,q,\tau)$5. This means that every inversion task uses the same format reward together with a task-specific binary semantic reward.

The model is not trained by applying all tasks jointly at every step. Instead, the paper uses probabilistic alternating training: with 80% probability it chooses the main TVG task, and with 20% probability it chooses an Invert-TVG task; if an inversion task is chosen, VC, AR, and VD are sampled uniformly. The combined objective is formalized as

$(V,q,\tau)$6

with

$(V,q,\tau)$7

and

$(V,q,\tau)$8

where $(V,q,\tau)$9.

This scheduling is explicitly intended to avoid memory blow-up, gradient conflict, unstable training, and task dominance by easier tasks. The paper’s interpretation is that the two task families have different optimization dynamics, so alternation is preferable to a fully joint scheme.

4. Optimization target: balancing localization and action understanding

The conceptual contribution of Invert4TVG lies in its treatment of action understanding as a support signal for temporal grounding rather than as an unrelated auxiliary objective. The paper identifies a failure mode in which training only for IoU yields better segment overlap while the model becomes worse at understanding the action semantics behind the query. Under this diagnosis, the model can become a better “segment guesser” but a worse “action reasoner” (Chen et al., 10 Aug 2025).

The three inversion tasks are designed to counter that degradation. VC forces the model to recover verbs from a segment, thereby reinforcing the mapping between query language and observed action. AR requires direct identification of the action shown in the segment, which strengthens action perception. VD asks for a description that includes the action content, which maintains broader awareness of what happened in the segment. Together, they are presented as semantic anchors that preserve or improve the capabilities on which accurate grounding depends.

The paper further argues that accurate temporal localization is not independent of semantic understanding. To localize “opening a door,” the model must understand what “opening” looks like and when it occurs; better verb and action understanding helps the model rank candidate segments more correctly. This suggests that the framework raises localization performance by improving the action semantics that define the grounding target.

The probabilistic schedule is part of this balance mechanism. The paper states that if Invert-TVG is too frequent, the model becomes too focused on semantic generation and neglects grounding; if it is too rare, the semantic benefit is too weak. The best performance occurs when Invert-TVG is sampled about 20% of the time, and higher percentages hurt grounding performance.

5. Experimental configuration and empirical results

The implementation uses Qwen2.5-VL as the backbone and evaluates both 3B and 7B variants. Video frames are sampled at 2 FPS and resized to reduce memory; the paper notes that a 50-second video yields about 100 frames of size $\tau$0. Verb extraction uses SpaCy en_core_web_sm-3.8.0. Optimization uses AdamW with $\tau$1, $\tau$2, $\tau$3, weight decay $\tau$4, learning rate $\tau$5, and approximately 80 hours per epoch (Chen et al., 10 Aug 2025).

The framework is evaluated on three TVG benchmarks: Charades-STA, ActivityNet, and QVHighlights. Charades-STA contains 6,672 indoor human-activity videos, 12,408 training pairs, and 3,720 test pairs. ActivityNet contains 20K long videos, with 37,421 train, 17,505 val, and 17,031 test. QVHighlights contains 10,460 high-resolution YouTube videos and 48K annotated clip-query pairs, converted to single-segment tasks for evaluation. The main metrics are [email protected], [email protected], and [email protected], with mIoU also mentioned as an auxiliary average IoU metric.

The baseline set includes traditional VLP methods such as 2D-TAN, Moment-DETR, EaTR, and SnAG; SFT-based LVLM methods such as VideoChat-Flash, TRACE, HawkEye, and TimeSuite; and RL-based methods including Time-R1 (3B) and Time-R1 (7B).

On Charades-STA, the paper reports the best performance among compared methods. For the 3B model, Invert4TVG achieves [email protected] = 80.8, [email protected] = 69.0, and [email protected] = 44.0, compared to Time-R1* (3B) at 78.7 / 64.1 / 36.9. The headline gain is +7.1 points at [email protected]. For the 7B model, Invert4TVG achieves 83.0 / 72.5 / 51.4, compared to Time-R1* (7B) at 82.8 / 72.2 / 50.1. The paper also states that Invert4TVG outperforms Time-R1 in zero-shot evaluation on ActivityNet and QVHighlights, which it uses to support the claim of better generalization, especially on more complex scenarios.

6. Ablations, interpretation, and limitations

The ablation study is central to the paper’s validation of the inversion-task design. It compares single-task variants with the full combination on Charades-STA. Only-VD achieves 79.1 / 64.3 / 39.4, Only-AR achieves 78.2 / 65.2 / 43.8, Only-VC achieves 78.8 / 68.0 / 42.0, and full Invert4TVG achieves 80.8 / 69.0 / 44.0. The paper interprets these results as evidence that the tasks are complementary: VD helps broader semantic and contextual understanding, AR is strongest at high-precision action recognition and gives the best [email protected] among the single-task variants, VC sits between the two, and the combination is best overall (Chen et al., 10 Aug 2025).

A second ablation varies the Invert-TVG sampling probability. The best performance occurs when Invert-TVG is about 20%. Increasing the inversion-task frequency too much hurts TVG, and at 100% Invert-TVG performance is worst because the model stops optimizing the main grounding objective. This confirms the paper’s principle that auxiliary semantic learning should support, not replace, localization training.

A third ablation compares the proposed binary reward with a cosine-similarity-based reward. Cosine similarity yields 76.2 / 62.2 / 39.8, whereas the binary 0/1 reward yields 80.8 / 69.0 / 44.0. The paper attributes this to the stability and lower noise of binary rewards and notes that cosine similarity can assign nontrivial similarity to weakly related verbs such as “run” and “eat,” which is undesirable for precise action grounding.

The paper implies several limitations. The auxiliary tasks are verb-centric and therefore mainly target action understanding rather than objects, attributes, or spatial relations. The method depends on good verb extraction from queries and descriptions. It is tailored to TVG datasets with sufficiently rich action language and may be less effective for non-action-centric queries. Its training objective remains largely driven by binary rewards, which are stable but relatively coarse. It also requires careful tuning of task probability to avoid hurting grounding performance. These caveats situate Invert4TVG as a method specifically optimized for the action-semantic component of TVG rather than a general solution to all forms of video–language grounding.

A concise synthesis of the framework is that it pairs forward grounding, $\tau$6, with inverse action recovery, $\tau$7. By deriving Verb Completion, Action Recognition, and Video Description directly from existing TVG annotations and integrating them into a GRPO-based RL schedule in which TVG is sampled 80% of the time and inversion tasks 20% of the time, Invert4TVG seeks to preserve action understanding while improving localization accuracy, particularly at stricter IoU thresholds (Chen et al., 10 Aug 2025).