Unified Query-Based Task Representation

• Unified query-based task representation is a modeling paradigm that formulates diverse computational tasks as queries to structure inputs, outputs, and supervision.
• It leverages various model classes—including transformers, Bayesian trees, and generative LLMs—to drive multi-task learning and improve state-of-the-art performance.
• The framework promotes interoperability and extensibility across domains like search, 3D scene understanding, and multi-modal perception through unified querying.

A unified query-based task representation is a modeling paradigm in which diverse, often heterogeneous, computational tasks are formulated and solved through a single representation that leverages queries—vector, symbolic, or multimodal—as the central abstraction. This approach subsumes task-specific branches by structuring all task inputs, outputs, supervision, or internal communication through queries, such that the resulting system provides interoperability, cross-task synergy, and flexible extensibility. Unified query-based representations now underpin hierarchical task modeling for search, multi-task learning in perception and language, user and graph embeddings, 3D scene understanding, and end-to-end few-shot transfer.

1. Core Mathematical Formulations and Model Classes

Unified query-based representations are instantiated through a spectrum of model classes:

• Bayesian Rose Tree (BRT) Hierarchies: Each atomic datum (e.g., search query) becomes a leaf in a recursively constructed, nonparametric tree. Node likelihoods are recursively defined by

$$p(D_m \mid T_m) = \pi_{T_m} \cdot f(D_m) + (1-\pi_{T_m}) \prod_{T_i \in \mathrm{ch}(T_m)} p(D_i \mid T_i)$$

where $\pi_{T_m}$ controls granularity and $f(D_m)$ encodes aggregated query–query affinity features (Mehrotra et al., 2017).

• Unified Transformer and Attention Models: Queries are learnable or dynamically sampled vectors; all tasks (e.g., detection, segmentation, pose, Q&A) are formulated as direct set predictions/or as conditioning operations on these shared queries after multi-scale cross- and self-attention. Examples include Multi-task Human Query Transformers (Jin et al., 2023), Mixed-Query Transformer (MQ-Former) (Wang et al., 2024), and UQFormer for instance and boundary query fusion (Dong et al., 2023).
• Unified Generative LLMs: Heterogeneous tasks are re-cast into a shared sequence generation schema, often as structured (e.g., JSON-like) outputs. All task-specific control is handled via in-context prompts and explicit output blocks, with parameter sharing allowing strong cross-task transfer (Huang et al., 10 Feb 2026).
• Embedding-Querying Paradigms (EQ-Paradigm): Feature extraction is decoupled from task heads by a querying stage, where a set of 3D query points (arbitrary locations) interpolate or attend to embedded features. The querying output then serves as the universal interface for any downstream prediction head (object detection, segmentation, shape classification) (Yang et al., 2022).
• Hierarchical Query Structures: Set-level or instance-level queries, often with learned or positional anchors, are updated through stackable Transformer decoder layers to yield content-rich unified query vectors shared by every task head, as in HQNet (Jin et al., 2023) or RepVF (Li et al., 2024).
• Multimodal and Scene-Based Queries: Scene representations unify geometric, semantic, and visual modalities, with queries acting as retrieval or reasoning anchors to extract multimodal object-centric embeddings for planning and situational understanding (Li et al., 24 Sep 2025).

2. Query Instantiation and Task Encapsulation

The core operational principle is the encoding of both data and tasks via query structures, with diverse instantiations:

Model/Domain	Query Instantiation	Output/Prediction Decoding
BRT (Search)	Atomic queries (text); tree-structured	Path from leaf to root = (sub)task path
MQ-Former	Learnable + proposal queries	Direct set prediction (mask, box, class, etc.)
EQ-Paradigm	Arbitrary 3D point sets	Task head predicts at each query point
QP-OneModel	Sequence-to-sequence with JSON prompts	Each field in structure is a sub-task output
HQNet	N_q instance queries + positional enc.	Detection, pose, segmentation, attribute, mesh heads for every query
3D QSR	Object-IDs, cross-modal queries	NeRF/point cloud/graph retrieval by joint embedding

This organization ensures that every prediction, regardless of modality or sub-task, is rooted in or decoded via a unique query representation, yielding fine-grained task disambiguation while supporting task decomposition, augmentation, and joint optimization.

3. Inference, Optimization, and Synergistic Training

Inference and optimization methods vary per instantiation but share unification strategies:

• Greedy Agglomerative Structure Induction: In hierarchical search tasks, merge operations maximizing a Bayes factor produce a tree with each query at exactly one leaf (Mehrotra et al., 2017).
• Unified Transformer Optimization: All queries are batched and updated jointly under shared decoder layers. Hungarian matching or set-prediction is used for one-to-one mapping between queries and ground-truth targets (e.g., DETR, UQFormer) (Jin et al., 2023, Dong et al., 2023, Wang et al., 2024).
• Multi-task Supervision and Reinforcement Learning: Generative LLMs optimize structured output for all tasks jointly, including multi-reward RL (e.g., NER-F1, segmentation F1, taxonomy score) for QP-OneModel (Huang et al., 10 Feb 2026).
• Contrastive and Autoregressive Joint Losses: User-centric models jointly optimize query-anchored contrastive alignment and next-token prediction, with further cluster-based soft prompt tuning for downstream task sensitivity (Yuan et al., 16 Feb 2026).
• Downstream-Task Generalization Mechanisms: For example, multimodal 3D representations enable LLM-driven planners to issue and resolve open-vocabulary object queries, retrieve fine-grained geometry, and trigger precise manipulation/navigational actions (Li et al., 24 Sep 2025).

4. Quantitative Performance and Empirical Evidence

Unified query-based representations consistently deliver or exceed state-of-the-art performance in multi-task and cross-domain evaluations:

• Search Hierarchy Extraction: BRT-affinity achieves flat clustering F₁ ≈ 0.70 (versus 0.60–0.65 for baselines), 81% subtask validity via crowdsourcing, and improved downstream term-prediction (Mehrotra et al., 2017).
• Multi-Task Human Perception: HQNet enables combined detection, segmentation, pose, attribute, and mesh recovery, outperforming multi-head strategies and generalizing to unseen tasks/domains (e.g., face detection and zero-shot tracking) (Jin et al., 2023).
• Queryable 3D Reasoning: In 3D QSR, retrieval precision/recall for scene graph queries reach 0.74/0.72, affordance success 93%, and negation success 90% (SG). End-to-end robot plans for abstract commands are achieved in complex environments (Li et al., 24 Sep 2025).
• Image Segmentation Across Domains: MQ-Former achieves 58.8 Mask AP on COCO instance and 43.4 mAP on open-set SeginW (7+ points over prior art) with a pure shared-weights, mixed-queries architecture (Wang et al., 2024).
• Unified Point Cloud Understanding: EQ-Paradigm delivers consistent 2–6 point mIoU/AP/classification accuracy improvements across semantic segmentation, detection, and shape recognition, with full decoupling between backbone and task head (Yang et al., 2022).
• Graph Embedding: Path-LLM raises macro-F1 from 0.27 (WalkLM) to 0.75, with a 90%+ reduction in path samples needed over baselines (Shang et al., 2024).
• E-commerce Query Classification: SSUF significantly boosts click and recall in multi-task e-commerce categorization versus separate pipelines, using label and knowledge enhancements for semi-supervised training (Yuan et al., 26 Jun 2025).

5. Architectural and Methodological Benefits

The unified query-based paradigm offers several concrete advantages:

• Parameter and Codebase Unification: All tasks share the same backbone, decoder, or generative LLM weights, lowering maintenance, version skew, and training redundancy (Huang et al., 10 Feb 2026, Jin et al., 2023, Wang et al., 2024).
• Seamless Task Addition/Removal: New tasks are accommodated by adjusting heads or prompt entries, with the underlying query representation unchanged (Jin et al., 2023, Yuan et al., 16 Feb 2026).
• Cross-task Semantic Synergy: Intermediate results (e.g., intent descriptions, segmentation/NER blocks) inform downstream tasks, enabling positive transfer (e.g., improved segmentation leading to improved term-weighting) (Huang et al., 10 Feb 2026).
• Gradient and Feature Competition Mitigation: Unified heads and joint attention naturally balance per-task gradients (e.g., RFTR achieving $\Delta \approx 2.47$ with low variance, obviating calibration) (Li et al., 2024).
• Scalability to Open-Set/Domain/Label: Dynamic query allocation (conditional+learnable) and universal decoding enable strong zero/transfer/shots (e.g., open-vocabulary segmentation in MQ-Former, task-conditioned user representation in Q-Anchor) (Wang et al., 2024, Yuan et al., 16 Feb 2026).
• Efficient Inference and Serving: KV-cache reuse amortizes inference cost per user/query in deployment-scale recommendation (Yuan et al., 16 Feb 2026); set-prediction and task-agnostic queries eliminate NMS and hand-crafted post-processing (Dong et al., 2023).

6. Limitations, Open Questions, and Future Directions

Unified query-based representations are not universal solutions, and open challenges remain:

• Interpretability: The semantics of learned queries (especially dynamic or attention-generated) can be opaque, particularly in settings lacking explicit hierarchical structure.
• Task Conditionality vs. Universality: Balancing broad generalization with sharp task sensitivity (e.g., via prompt tuning or label-enhanced modules) requires architectural and training-careful approaches (Yuan et al., 26 Jun 2025, Yuan et al., 16 Feb 2026).
• Data/Compute Efficiency: Although empirical results show reductions (e.g., Path-LLM’s 90% path reduction), training unified architectures for very large or diverse task sets may incur overhead if not carefully managed.
• Explicit Reasoning and Multi-Hop Tasks: Current query-based segmentation/scene frameworks do not support compositional "reasoning segmentation" or multi-hop dialogue/QA, suggesting a need for further fusion with LLMs or explicit symbolic components (Wang et al., 2024, Li et al., 24 Sep 2025, Song et al., 2017).
• Hyperparameter and Loss Balancing: Automated methods for calibration are often less necessary but may be required for further scaling/diversification.

7. Cross-Domain Impact and Generalization

Unified query-based task representations now dominate in multi-task computer vision (Wang et al., 2024, Jin et al., 2023), task/subtask search analysis (Mehrotra et al., 2017), dialogue and retrieval (Hu et al., 2024, Song et al., 2017), point cloud and 3D reasoning (Yang et al., 2022, Li et al., 2024, Li et al., 24 Sep 2025), user modeling (Yuan et al., 16 Feb 2026), graph representation (Shang et al., 2024), and industry e-commerce query systems (Yuan et al., 26 Jun 2025, Huang et al., 10 Feb 2026). The widespread empirical successes suggest that organizing computational tasks and their outputs/inputs as queries—learnable, symbolically structured, or dynamically conditioned—aligns naturally with both data structure and cross-task semantic sharing requirements. The explicit formalizations surveyed here provide a robust foundation and a design space for further extensions to lifelong learning, open-ended multitask agents, and reasoning-centric AI.