CoTasks: Structured Task Composition

• CoTasks is a research paradigm defining structured and compositional task regimes, including task-centric perception and cooperative multi-agent planning.
• It employs explicit task decomposition methods such as chain-of-tasks reasoning and affordance prompting to improve detection and overall model performance.
• Applications span computer vision benchmarks, video-language modeling, robotic manipulation, multi-agent reinforcement learning, and code/text task suites.

“CoTasks” is not a single canonical framework. In the literature, the term appears as a shorthand, acronym, or task-family name for several distinct but structurally related ideas: task-centric perception, chain-of-tasks decomposition, cooperative task composition, spatially compositional multi-tasking, and jointly trained task suites. The shared theme is a shift away from single-label prediction toward explicit task structure, whether the task is choosing objects that afford an action, decomposing dense video captioning into sequential sub-problems, organizing bimanual manipulation through affordance-fulfilling primitives, or balancing heterogeneous instruction-tuning datasets (Sawatzky et al., 2019, Lee et al., 12 Jan 2025, Wang et al., 18 Jul 2025).

1. Terminological landscape

In computer vision, “CoTasks” is a shorthand for COCO-Tasks, a benchmark for task-driven object detection that asks which objects in an image are most suitable for a specified task rather than which category labels are present (Sawatzky et al., 2019). In another visual line, “CompositeTasking” is abbreviated as CoTasks and denotes spatial composition of multiple pixel-wise tasks within a single image through a Task Palette (Popovic et al., 2020). In video modeling, VidChain uses “Chain-of-Tasks (CoTasks)” for multi-turn decomposition of dense video captioning into segmentation, grounding, and captioning sub-tasks (Lee et al., 12 Jan 2025), while the later paper “CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks” uses the term for four object-centric intermediate video reasoning tasks—frame localization, object tracking, spatial relation extraction, and temporal relation extraction (Wang et al., 18 Jul 2025).

The term also appears in robotics and action sequencing. OAKINK2 states that its three-level abstraction—Affordance, Primitive Task, and Complex Task—directly supports composing cooperative bimanual sequences (“CoTasks”) by chaining affordance-fulfilling primitives with explicit dependencies (Zhan et al., 2024). In cooperative multi-agent reinforcement learning, the Composite Task Challenge constructs composite tasks whose completion requires division of labor and cooperation rather than treating such specialization as optional (Li et al., 1 Feb 2025). In code-centered settings, the CoTexT overview uses “CoTasks” for a unified suite of code–text tasks such as summarization, refinement, generation, and defect detection (Phan et al., 2021), and the TIKTOC overview situates “CoTasks” as open-ended coding tasks with test-case-level student modeling (Duan et al., 2024). This suggests that the term functions less as a single proper noun than as a recurrent label for structured, compositional, or cooperative task regimes.

2. Task-driven perception: COCO-Tasks, GGNNs, and affordance prompting

The earliest prominent use is COCO-Tasks, abbreviated as CoTasks, introduced as a benchmark for task-driven object detection. It reuses MS COCO 2014 images and instance annotations, then adds per-instance binary suitability labels indicating whether an object is “preferred” for a given task. The dataset contains about 40,000 images in total, with 30,229 train images and 9,495 test images, and covers 14 everyday tasks including “step on something,” “sit comfortably,” “open parcel,” and “serve wine.” Each task is annotated by 5 trained annotators, and an object is labeled preferred if selected by a majority of at least 3 of 5 annotators. No new masks or boxes are created; suitability is attached to existing COCO instances. Evaluation uses COCO [email protected] per task, averaged as mAP@0.5 across tasks (Sawatzky et al., 2019).

COCO-Tasks reframes object detection around suitability rather than category membership. Suitability is task-centric, instance-specific, and context-dependent: it depends on alternatives in the scene, object properties such as size or cleanliness, and global context such as distinguishing a real toilet from a mirror reflection. The baseline method uses a ResNet-101 backbone and a fully connected Gated Graph Neural Network over detected objects. For each candidate object $o_i$, the model predicts a task-specific suitability score $s_i^t \in [0,1]$, and the final confidence is

$\tilde{s}_i^t = d_i \cdot s_i^t,$

where $d_i$ is the detector score. The proposed method with fusion reaches $0.742$ [email protected] with ground-truth boxes, $0.326$ with Faster R-CNN detections, and $0.332$ with YOLOv2 detections, outperforming detector, classification, and ranking baselines. The dataset also shows substantial intra-category differentiation, indicating that instance-level properties rather than category identity alone determine suitability (Sawatzky et al., 2019).

A later extension, CoTDet, retains COCO-Tasks as the evaluation substrate but changes the modeling premise from category-conditioned suitability to affordance-centered reasoning. Instead of enumerating object classes, CoTDet uses Multi-Level Chain-of-Thought prompting to extract affordance knowledge from LLMs through a sequence of task $\rightarrow$ object examples $\rightarrow$ rationales $\rightarrow$ essential visual attributes. These knowledge units are encoded with RoBERTa and used to condition a Deformable-DETR detector by generating knowledge-aware queries and knowledge-conditioned denoising for box regression. On COCO-Tasks, CoTDet reports $s_i^t \in [0,1]$0 mAP_box and $s_i^t \in [0,1]$1 mAP_mask, improving over TOIST by $s_i^t \in [0,1]$2 box AP and $s_i^t \in [0,1]$3 mask AP, and over GGNN by $s_i^t \in [0,1]$4 mAP_box and $s_i^t \in [0,1]$5 mAP_mask. The paper emphasizes that affordances such as “sharp blade,” “pointed tip,” or “handle” bridge tasks to rare or unconventional objects more effectively than fixed object vocabularies (Tang et al., 2023).

3. Chain-of-tasks in video-language modeling

In dense video captioning, VidChain defines CoTasks as a sequential decomposition of a complex objective into smaller sub-tasks that a VideoLLM can solve in multi-turn form. Dense video captioning requires inferring the number of events $s_i^t \in [0,1]$6, their timestamps $s_i^t \in [0,1]$7, and captions $s_i^t \in [0,1]$8. VidChain factors the joint distribution in two ways:

$s_i^t \in [0,1]$9

for the time-first path, and

$\tilde{s}_i^t = d_i \cdot s_i^t,$0

for the caption-first path. The framework pairs this decomposition with Metric-based Direct Preference Optimization, which constructs preference pairs using task metrics such as SODA_c, METEOR, CIDEr, and temporal IoU-based measures. Applied to VTimeLLM and VideoLLaMA2, VidChain improves Dense Video Captioning on ActivityNet and YouCook2 and also transfers strongly to Temporal Video Grounding. For example, VideoLLaMA2 (7B) with VidChain reaches SODA_c $\tilde{s}_i^t = d_i \cdot s_i^t,$1 on ActivityNet, up from a $\tilde{s}_i^t = d_i \cdot s_i^t,$2 baseline, and VTimeLLM (7B) improves TVG performance on ActivityNet from $\tilde{s}_i^t = d_i \cdot s_i^t,$3 to $\tilde{s}_i^t = d_i \cdot s_i^t,$4 for $\tilde{s}_i^t = d_i \cdot s_i^t,$5 (Lee et al., 12 Jan 2025).

A different but related video use appears in “CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks.” Rather than decomposing dense captioning, it decomposes video question answering into four entity-level foundational tasks: object-based frame localization, object tracking with bounding boxes, spatial relation extraction, and temporal relation extraction. The framework reconstructs object-centric supervision for NeXT-QA by aligning it with VidOR annotations and uses STAR directly because STAR already provides object-level annotation. The intermediate outputs are strictly structured JSON objects or lists, such as {"objects": [str], "timestamps": [int]} for CoTask 1 and lists of frame-indexed bounding boxes for CoTask 2. These CoT-style intermediate steps are then inserted into the prompt before the final answer (Wang et al., 18 Jul 2025).

The empirical effect is substantial. On NeXT-QA, CoTasks raises LLaVA-video-7B from an average GPT-4 evaluation score of $\tilde{s}_i^t = d_i \cdot s_i^t,$6 to $\tilde{s}_i^t = d_i \cdot s_i^t,$7, and Qwen2.5-VL-3B from $\tilde{s}_i^t = d_i \cdot s_i^t,$8 to $\tilde{s}_i^t = d_i \cdot s_i^t,$9, with especially large gains in causal, temporal, and descriptive categories. On STAR, Qwen2.5-VL-3B improves from $d_i$0 accuracy without prompting or fine-tuning to $d_i$1 with CoTasks prompting only, exceeding the reported fine-tuning-only baseline of $d_i$2. At the same time, direct generation of the intermediate tasks remains difficult for current VideoLLMs: for LLaVA-video-7B, CoTask 2 has an average score of only $d_i$3, indicating that explicit object-level grounding is still a bottleneck (Wang et al., 18 Jul 2025).

4. Cooperative and compositional task execution

In manipulation, OAKINK2 operationalizes cooperative CoTasks through a three-level object-centric abstraction: Affordance, Primitive Task, and Complex Task. An affordance is a predicate tied to object parts and states; a Primitive Task is the minimal hand–object interaction sequence that fulfills an affordance and has explicit pre-condition and terminal condition; a Complex Task is a composition of primitives constrained by a Primitive Dependency Graph, which is a DAG. The dataset contains 627 total sequences, including 363 Primitive demonstrations and 264 Complex Task demonstrations, amounting to 4.01 million image frames across four views, with 75 objects, 39 affordances, 60 primitive types, and 150 Complex Task instances derived from 38 long-horizon goals. Within the Complex Task Completion framework, LLMs decompose textual goals into primitive sequences, an Oracle retrieves or retargets object trajectories, and TaMF generates bimanual hand motion for each primitive. Planning remains challenging: the overall success rate is $d_i$4, rising to $d_i$5 for tasks with at most 3 primitives and dropping to $d_i$6 for tasks with more than 5 primitives (Zhan et al., 2024).

In cooperative MARL, the Composite Task Challenge formalizes a related but distinct notion of CoTasks. It composes atomic defense and pursuit subtasks into composite missions whose success criterion is “all subtasks succeed” and failure criterion is “any subtask fails.” The benchmark introduces eight heterogeneous-asymmetrical tasks spanning three factors: Information Interference, Subtask Dissimilarity, and Subtask Quantity. It evaluates 10 cooperative MARL methods and finds that all baselines perform poorly on the main tasks. For example, on HEA_P2G-D2 all baselines achieve zero maximum test win rate; on HEA_M2G and HEA_M3G all baselines are also at $d_i$7; and on HEA_D3G and HEA_D4G all baselines again remain at $d_i$8. Simplified variants show that the dynamics are solvable, but the full composite tasks expose deficits in specialization, coordination, robustness to interference, and scaling (Li et al., 1 Feb 2025).

In industrial IoT, CONTASKI addresses cooperative task allocation rather than task reasoning. A task is represented as

$d_i$9

where $0.742$0 is the required capability set, $0.742$1 is task duration, and $0.742$2 is the quorum. Clusters are formed by capability similarity using

$0.742$3

and leaders accept tasks when capability cover and quorum constraints are satisfied. In NS-3 evaluation, CONTASKI achieved 100% of allocated tasks in cases with 75 and 100 nodes, and on average more than 80% clusters performed tasks in a low response time (Pedroso et al., 2020).

A browser-agent perspective appears in cotomi Act, which treats tasks and organizational knowledge as shared first-class artifacts. Its execution scaffold combines adaptive lazy observation, verbal-diff-based history compression, coarse-grained actions, and best-of-$0.742$4 action selection, reaching $0.742$5 on the 179-task WebArena human-evaluation subset and exceeding the reported $0.742$6 human baseline. The system also passively observes browsing behavior and abstracts it into task boards and wiki pages inside a shared workspace; a controlled proxy evaluation reports that task success improves as behavior-derived knowledge accumulates, with gains up to $0.742$7 percentage points over a baseline of about $0.742$8, and one catalog-ordering setting rising from about $0.742$9 to about $0.326$0 after 48 procedural scripts are available (Oyamada et al., 4 May 2026).

5. Spatial composition and balanced multi-task learning

CompositeTasking uses CoTasks in a spatially compositional sense. A single encoder–decoder network receives an image and a dense pixel-wise Task Palette indicating which task should be performed at each location. The network outputs a unified three-channel image-like embedding, and conditional normalization in the decoder routes each pixel to the requested task. Formally, the Task Palette is a discrete map

$0.326$1

and the network produces

$0.326$2

This allows normals on one region, human parts on another, edges elsewhere, and so on, in a single forward pass. The benchmark setting uses PASCAL-MT with 4,998 train and 5,105 validation images and five tasks: semantic segmentation, human body parts, surface normals, semantic edges, and saliency. Under semantic rule training $0.326$3, CTN reports edges $0.326$4, segmentation $0.326$5, parts $0.326$6, normals $0.326$7, and saliency $0.326$8, while a standalone palette predictor reaches $0.326$9 mIoU (Popovic et al., 2020).

CoTBal addresses a different problem: not spatial composition but balancing co-trained tasks in multi-task visual instruction tuning. It defines two performance-based balancing signals. Inter-Task Contribution measures how much one task improves another during joint training, and Intra-Task Difficulty measures how much a task suffers when only a small subset of its data is used. The method introduces Generic Task Weighting, which applies task weights at the token level:

$0.332$0

and combines contribution- and difficulty-based weights into a final vector $0.332$1. On a seven-task visual instruction tuning setup built on LLaVA-v1.5-7B, CoTBal improves average per-task improvement $0.332$2 from $0.332$3 for Equal Weighting to $0.332$4, while keeping average per-task error $0.332$5 comparable at $0.332$6 versus $0.332$7 (Dai et al., 2024).

6. Code-centered and educational uses

In code intelligence, CoTexT uses “CoTasks” for a unified multi-task regime over code–text problems. It is a T5-style encoder–decoder initialized from T5-base with 220M parameters, trained with SentencePiece tokenization and span-corruption denoising on unimodal and bimodal code corpora. Downstream tasks are all cast into a text-to-text interface with task and language prefixes, including code summarization in six languages, code refinement on small and medium Bug2Fix subsets, code generation on CONCODE, and defect detection on Devign. On CONCODE, CoTexT (1-CC) reports EM $0.332$8, BLEU $0.332$9, and CodeBLEU $\rightarrow$0; on overall code summarization it reaches $\rightarrow$1; and on defect detection CoTexT (1-CCG) achieves $\rightarrow$2 accuracy (Phan et al., 2021).

In educational data mining, TIKTOC uses “CoTasks” for open-ended coding tasks. The setting augments CodeWorkout with instructor/developer-authored test cases and models both generated student code and per-test-case pass/fail outcomes. Each student interaction is

$\rightarrow$3

and TIKTOC jointly minimizes a code generation loss and a test-case prediction loss:

$\rightarrow$4

The dataset subset includes 17 unique problems, 246 unique students, 3714 student code submissions, and 305 total test cases. The model uses Llama 3 8B Instruct as the backbone, ASTNN for code embeddings, and a DKT-style LSTM knowledge state aligned to the LLM embedding space. TIKTOC reaches AUC about $\rightarrow$5, F1 about $\rightarrow$6, accuracy about $\rightarrow$7, and CodeBLEU about $\rightarrow$8, improving AUC over Code-DKT-TC by $\rightarrow$9 and CodeBLEU over OKT-TC by $\rightarrow$0 (Duan et al., 2024).

Across these literatures, CoTasks consistently denotes explicit task structure rather than undifferentiated end-to-end prediction. Sometimes that structure is object suitability, sometimes a chain of sub-tasks, sometimes a dependency graph over primitives, sometimes a spatial task map, and sometimes a weighted collection of jointly optimized tasks. A plausible implication is that the term has become a convenient label for research programs that treat task composition, task conditioning, or task cooperation as the central modeling primitive rather than a secondary engineering detail.