CoTasks Framework Overview

• CoTasks Framework is a family of techniques that decomposes complex multimodal tasks into sequential, semantically coherent sub-tasks, enhancing interpretability and performance.
• It employs chain-of-thought style supervision and explicit task-factorization to improve dense video captioning and entity-centric video QA outcomes, providing measurable gains.
• The approach supports distributed multi-task learning by integrating parallel module training and consistency regularization, yielding significant performance improvements in diverse vision tasks.

The CoTasks framework comprises a family of methodologies for decomposing complex multimodal reasoning tasks—particularly in dense video understanding and multi-task vision—into structured sequences of sub-tasks. This approach enables greater sample efficiency, interpretability, and performance in video LLMs (VideoLLMs) and distributed multi-task computer vision systems by explicit task-factorization and chain-of-thought style supervision. CoTasks implementations are documented in several distinct but conceptually related research lines, including dense video captioning with reasoning path decomposition (Lee et al., 12 Jan 2025), distributed task networks with consistency regularization (Lu et al., 2020), and entity-centric video instruction tuning for compositional video QA (Wang et al., 18 Jul 2025).

1. Conceptual Foundations and High-Level Decomposition

The CoTasks paradigm shares a unifying principle: complex tasks are decomposed into a sequence of semantically coherent, lower-complexity sub-tasks, often reflecting natural reasoning or analytical steps in the source domain. In modern video understanding, this manifests as bifurcating dense descriptions of video events into chains involving event segmentation, temporal grounding, caption generation, and relation extraction.

For dense video captioning, CoTasks factors the joint modeling objective $p(c, t, n \mid v)$—with $c$ as the set of event captions, $t$ as their temporal boundaries, $n$ the segment count, and $v$ the input video—into ordered conditional subproblems. Two canonical “reasoning paths” are employed:

• Time-then-caption path $P_{t\rightarrow c}$: $$p(c, t, n \mid v) = p(n \mid v) \cdot p(t \mid v, n) \cdot p(c \mid v, n, t)$$
• Caption-then-time path $P_{c\rightarrow t}$: $$p(c, t, n \mid v) = p(n \mid v) \cdot p(c \mid v, n) \cdot p(t \mid v, n, c)$$

Analogously, entity-centric video QA decomposes compositional reasoning into: (1) frame localization, (2) entity tracking, (3) spatial relation extraction, and (4) temporal relation extraction (Wang et al., 18 Jul 2025). In distributed vision systems, CoTasks refers to parallelized modules for each separate vision task, with explicit consistency constraints capturing analytic dependencies between them (Lu et al., 2020).

2. Sub-Task Structure, Inputs, and Outputs

CoTasks implementations adhere to well-defined multi-turn task chains, in which each sub-task’s outputs form part or all of the input for subsequent steps. In dense video captioning (Lee et al., 12 Jan 2025), the sequence comprises:

A. Event Number Prediction

• Input: Encoded video $v$ + segmentation prompt
• Output: Scalar $\hat{n}$ (event count)

B. Temporal Segmentation or Captioning (depending on path)

• For temporal path:
  • Input: $v$, $\hat{n}$, dialogue history
  • Output: $\{\text{start}_i, \text{end}_i\}_{i=1}^{\hat{n}}$ (intervals)
• For captioning path:
  • Input: $v$, $\hat{n}$, dialogue history
  • Output: $\{\text{caption}_i\}_{i=1}^{\hat{n}}$

C. Complementary Step

• Inputs: Prior outputs
• Output: Remaining missing sequence component

In entity-centric video instruction tuning (Wang et al., 18 Jul 2025), four canonical CoTasks are defined per video question:

1. Frame localization (outputs: entity list, representative frame indices)
2. Object tracking (outputs: bounding boxes per frame/entity)
3. Spatial relation extraction (outputs: intra-frame relational triples)
4. Temporal relation extraction (outputs: cross-frame action relations)

Each sub-task’s predictions can be expressed in a table for the video QA scenario:

CoTask	Input Modality	Output Structure
1	frames, question, entities	entity list, frames
2	frames, entities	{frame: {entity: box}}
3	prior outputs	{entity, rel, entity, frame(s)}
4	prior outputs	{entity, rel, entity, frame(s)}

3. Mathematical Formulation and Training Objectives

In both dense captioning and entity-centric QA, the training loss is a sum of conditional negative log-likelihoods over each sub-task in the chain, reflecting the multi-turn decomposition:

$$\mathcal{L}(\theta) = -\sum_{i=1}^N \Bigg[ \sum_{j=1}^K \log P(a_{i, j} \mid v_i, q_{i, 0}, a_{i, 1}, \ldots, a_{i, j-1}; \theta) + \log P(a_{i, \text{ans}} \mid v_i, q_{i, 0}, a_{i, 1}, \ldots, a_{i, K}; \theta) \Bigg]$$

where $a_{i, j}$ denotes the answer to CoTask $j$ in sample $i$. For video dense captioning, standard next-token cross-entropy is applied to each prompt/response turn (Lee et al., 12 Jan 2025). No additional terms are introduced beyond decomposition.

When coupled with metric-based Direct Preference Optimization (M-DPO) (Lee et al., 12 Jan 2025), candidate predictions for each sub-task are scored with task-aligned metrics (e.g., temporal IoU, METEOR, SODA$_c$), forming preference pairs with the preference-gap-aware DPO-style loss:

$$L_{M\text{-}DPO} = \mathbb{E}_{\text{pairs}}\left[ 1(m^w - m^l > \gamma) \cdot L_s \right]$$

with $$L_s(\hat{y}^w, \hat{y}^l) = -\log \sigma(\beta [r(\hat{y}^w) - r(\hat{y}^l)])$$ and $$r(\hat{y}) = \log \left[ \frac{\pi_\theta(\hat{y}|\text{context})}{\pi_\text{ref}(\hat{y}|\text{context})} \right]$$.

In distributed multi-task vision, the global loss aggregates each module’s own supervised loss and a set of analytic or logical consistency losses across modules (Lu et al., 2020). For tasks $t_i$ and $t_j$, pairwise consistency is enforced by:

$$L^{\text{consist}}(i,j) = \lambda_{ij} \cdot D(g_{i \to j}(\hat{y}_i), \hat{y}_j)$$

with $g_{i \to j}$ a differentiable mapping (e.g., normal estimation from depth) and $D$ a relevant distance function.

4. Inference and Distributed Training Workflows

Inference in single-model CoTasks typically proceeds as a deterministic sequence of VideoLLM calls, each adapting prompts to the current reasoning state. For example, in dense captioning (time-then-caption path) (Lee et al., 12 Jan 2025):

def CoTasks_DVC_Infer(v, VideoLLM): n_hat = VideoLLM.generate(v, "How many segments…?") t_hat = VideoLLM.generate(v, "Breakdown video into time segments given n=" + n_hat, history=[n_hat]) c_hat = VideoLLM.generate(v, "Generate captions for each segment", history=[n_hat, t_hat]) return n_hat, t_hat, c_hat

In distributed multi-task learning (Lu et al., 2020), each task process runs in parallel on a dedicated node, fetching possibly stale peer predictions from remote procedure call (RPC) servers, enforcing consistency on unsupervised mediator data, and updating only local parameters:

for each training step: update w_i (local) by supervised loss fetch predictions from peers on mediator batch compute consistency loss, backpropagate only through w_i periodically update server with fresh w_i

Staleness in peer predictions is tolerated (on the order of thousands of steps), enabling substantial scalability and asynchronous operation.

5. Empirical Results and Comparative Performance

CoTasks consistently demonstrates large gains in reasoning-intensive video understanding and efficient multi-task vision systems.

In dense video captioning (Lee et al., 12 Jan 2025):

• CoTasks chain-of-subtasks prompts yield +0.9 SODA$_c$ and +0.8 METEOR on ActivityNet over single-prompt baselines
• Augmenting with M-DPO adds +0.6 SODA$_c$ and +0.2 METEOR
• Temporal grounding [email protected] improves from 44.0 to 63.3; mIoU from 30.4 to 44.1

In entity-centric video QA (Wang et al., 18 Jul 2025):

• On NeXT-QA, average GPT-4 score improves by +17.4 for Qwen2.5-VL-3B (esp. descriptive +48.1, temporal +10.9, causal +14.6)
• STAR accuracy (Qwen2.5-VL-3B) rises from 31.1% (baseline) to 65.4% with CoTasks add-on prompting
• Ablation demonstrates that structured grounding (CoTasks 1–2) and relation extraction (CoTasks 3–4) independently confer incremental improvements, with maximum benefit from the full chain

Distributed multi-task CoTasks (Lu et al., 2020) achieves:

• Joint depth, segmentation, and motion training yielding depth AbsRel drop from 0.165→0.125 and segmentation mIoU increase from 0.455→0.478
• 3D detection with 5% labeled data: 3D mAP improves from 17.6 to 23.5 with consistency regularization

6. Advantages, Limitations, and Analysis

Advantages:

• Decomposition substantially reduces per-step complexity, enhances interpretability, and isolates error modes
• Explicit multi-turn reasoning invokes chain-of-thought benefits, supporting stepwise supervision
• Structured supervision (e.g., preference-aligned DPO or CoT embedding) enables easier alignment to task metrics, especially with limited data or architectural constraints
• Distributed architectures offer modularity, scalability, and seamless integration of simulated or unlabeled data via consistency losses

Limitations:

• Dependency on object-level or structured annotations limits applicability in datasets lacking fine-grained labels (Wang et al., 18 Jul 2025)
• Error propagation across chain steps, especially in strictly sequential chains
• In certain implementations, model weights are held fixed during CoTask prompting (“inference-time augmentation”), with full end-to-end fine-tuning as future work

This suggests that CoTasks is most advantageous in settings where structured annotation exists and compositional inference is essential. A plausible implication is that resource-constrained or low-data regimes benefit disproportionately from CoTasks decomposition and consistency supervision.

7. Future Directions and Extensions

Key trajectories indicated by current literature include:

• Direct instruction-tuning of VideoLLMs on CoTasks-formatted data, enabling end-to-end weight adaptation (Wang et al., 18 Jul 2025)
• Automatically generating CoTasks steps for arbitrary video datasets via pretrained detectors/recognizers, reducing annotation costs
• Extending CoTasks to more open-domain video reasoning, multimodal dialogue, or planning agents
• Broader adoption in distributed, large-scale multi-task systems extending beyond vision, potentially integrating with multi-agent cooperative learning paradigms

The CoTasks conceptual family thus unites task decomposition, structured supervision, and inter-task consistency as central tools for advancing compositional reasoning and scalable learning in complex multimodal AI systems (Lee et al., 12 Jan 2025, Wang et al., 18 Jul 2025, Lu et al., 2020).