Hypergraph-of-Thought Paradigm

• Hypergraph-of-Thought is a reasoning framework that constructs hypergraphs over symbolic triples and visual patches to enable expert-level, multi-hop and multimodal inference.
• The methodology involves building textual and visual hypergraphs, encoding them with the AllSet Transformer, and fusing modalities using a cross-modal co-attention mechanism.
• Empirical results on ScienceQA show that HoT-equipped models achieve higher accuracy with moderate model sizes compared to traditional Chain-of-Thought approaches.

The Hypergraph-of-Thought (HoT) paradigm is a reasoning framework designed to enhance foundation models by explicitly modeling high-order, multi-hop, and multimodal inference. Unlike the conventional Chain-of-Thought (CoT) approach, which enforces a linear, step-by-step reasoning process reminiscent of layperson logic, HoT constructs a hypergraph structure over symbolic thought triples and visual patches, enabling expert-level comparative and integrative reasoning across modalities. Central to HoT are the construction of textual and visual hypergraphs, their encoding via the AllSet Transformer architecture, and a cross-modal co-attention graph mechanism that fuses reasoning signals before sequence generation. Empirical results on the ScienceQA benchmark demonstrate that HoT-equipped models surpass CoT-augmented LLMs at significantly lower model sizes (Yao et al., 2023).

1. Formal Structure of the Hypergraph-of-Thought

The foundation of HoT is the textual Hypergraph-of-Thought, $$\mathcal G_\text{text} = (\mathcal V_\text{text}, \mathcal E_\text{text})$$, where each vertex is a symbolic knowledge triple $v \equiv (h_v, r_v, t_v)$ comprising a head entity, relation, and tail entity. This extends the standard Graph-of-Thought (GoT) by allowing hyperedges that connect arbitrary subsets of triples, thus naturally representing higher-order and multi-hop relationships.

For the visual domain, the visual Hypergraph-of-Thought is defined as $$\mathcal G_\text{img} = (\mathcal V_\text{img}, \mathcal E_\text{img})$$, where vertices correspond to image patches and hyperedges emerge from clustering these patch-level features via k-means, thereby grouping semantically related visual regions for comparative reasoning.

2. Hyperedge Construction Algorithms

Textual hyperedges are generated by multi-hop random walks over the underlying GoT. The procedure is as follows: sample a start node $v_0 \sim \text{Uniform}(V)$, perform a $k$-step walk to select a sequence $\{v_{i_1}, v_{i_2}, \ldots, v_{i_k}\}$, then form a hyperedge $e = \{v_{i_1}, v_{i_2}, \ldots, v_{i_k}\}$. This sampling is repeated $N$ times, populating $\mathcal E_\text{text}$ with sets of triples encoding multi-hop inference paths. For the visual graph, patch-level features extracted by a frozen Swin Transformer are clustered into $m$ groups; each cluster defines a hyperedge linking its constituent patches.

3. Cross-Modal Co-Attention Graph Learning

After encoding the textual and visual hypergraphs with AllSet Transformer layers—which alternately propagate information from vertices to hyperedges and vice versa—textual ($v \equiv (h_v, r_v, t_v)$0) and visual ($v \equiv (h_v, r_v, t_v)$1) hyperedge embeddings are obtained. These are mapped to a common space, and co-attention is computed using a matrix

$v \equiv (h_v, r_v, t_v)$2

where $v \equiv (h_v, r_v, t_v)$3 denotes entrywise multiplication and $v \equiv (h_v, r_v, t_v)$4 is a learned gating matrix. The final fused multimodal representation $v \equiv (h_v, r_v, t_v)$5 is then used for downstream reasoning. Internally, the AllSet Transformer implements multi-head attention and MLPs for multiset exchange as detailed in the core architecture (see equations (3)-(7) of (Yao et al., 2023)).

4. Model Architecture and Reasoning Pipeline

The architecture operates in two stages:

1. Rationale Generation: Given a question and image, frozen encoders (T5 for text, Swin+DETR for images) produce embeddings. The textual and visual HoT modules encode high-order relationships and fuse cross-modal information, enriching the context vector before the T5 decoder generates an explicit rationale.
2. Answer Generation: The model receives the original question plus rationale, re-encodes, and the decoder then predicts the final answer.

HoT modules are interposed between the frozen encoders and the decoder, enabling joint high-order, cross-modal reasoning during both rationale and answer generation.

5. Training Objectives

The model minimizes the sum of two autoregressive cross-entropy losses: $v \equiv (h_v, r_v, t_v)$6 where $v \equiv (h_v, r_v, t_v)$7 are rationale tokens and $v \equiv (h_v, r_v, t_v)$8 are answer tokens. All parameters except those of the frozen encoders are trained. No auxiliary or contrastive objectives are introduced.

6. Comparative Empirical Results

Experiments on the ScienceQA benchmark compare HoT-augmented T5 models against CoT-based LLMs:

Model	Model Size	ScienceQA Accuracy (%)
UnifiedQA-Base w/o CoT	223M	70.12
GPT-3.5 w/ CoT	175B	75.17
ChatGPT w/ CoT	-	78.31
GPT-4 w/ CoT	-	83.99
HoT-T5-Base	223M	81.42
HoT-T5-Large	738M	83.38

Ablation studies show performance drops of 1–2 points when removing the textual HoT, visual HoT, or cross-modal co-attention, confirming the additive value of each component.

7. Implications and Distinctions

HoT generalizes and outperforms CoT by modeling reasoning as a hypergraph rather than as a linear or tree-structured path. This enables complex, expert-level, multi-hop, and multimodal reasoning within a single framework at moderate model scale. A plausible implication is that such hypergraph-based approaches may offer a scalable path to expert-level reasoning in foundation models without relying on extreme increases in model size (Yao et al., 2023).