---
title: Chain of Social Thought (CoST) Framework
url: https://www.emergentmind.com/topics/chain-of-social-thought-cost
type: topic
---

# Chain of Social Thought (CoST) Framework

Chain of Social Thought (CoST), also known as Cognitive Chain-of-Thought (CoCoT), is a structured prompting framework for vision-language models (VLMs) deployed on social reasoning tasks that require interpretation of visual input, contextual situation analysis, and norm-grounded judgment. CoCoT decomposes multimodal social reasoning into three discrete stages—Perception, Situation, and Norm—enabling systematic scaffolding of VLM reasoning when answering socially grounded queries. Empirical evidence demonstrates that CoCoT outperforms flat Chain-of-Thought (CoT) and direct prompting across intent disambiguation, commonsense reasoning, and safety tasks, improving interpretability and alignment with human social cognition [2507.20409].

## 1. Formal Framework and Notation

Given a VLM $\theta$, an image $x$, and a textual query $q$, the CoCoT methodology produces a socially grounded answer $a$ by sequentially invoking the following stages:

\[
\begin{align*}
z_1 &:= \text{Perception}(x, q; \theta)\\
z_2 &:= \text{Situation}(x, q, z_1; \theta)\\
z_3 &:= \text{Norm}(x, q, z_1, z_2; \theta)\\
a   &:= \text{Answer}(x, q, z_1, z_2, z_3; \theta)
\end{align*}
\]

For each $z_i$, the output is a natural-language intermediate rationale. The recursive structure is expressed probabilistically as:

- $z_1 = \arg\max_{z_1} P(z_1 \mid x, q, Q_1)$ (Perception)
- $z_2 = \arg\max_{z_2} P(z_2 \mid x, q, z_1, Q_2)$ (Situation)
- $z_3 = \arg\max_{z_3} P(z_3 \mid x, q, z_1, z_2, Q_3)$ (Norm)
- $a = \arg\max_a P(a \mid x, q, z_1, z_2, z_3, Q_A)$ (Final answer)

Here, $Q_1$ through $Q_A$ are stage-specific fixed prompt templates. At inference, the model is not retrained nor fine-tuned; rather, its next-token distribution is conditioned on $q$ concatenated with prior stage outputs.

## 2. Prompt Construction and Inference Pipeline

Prompt engineering in CoCoT adheres to distinct templates for each reasoning stage, instantiated as isolated model calls:

- **Perception**: "Based on the image, describe what is directly observable."
- **Situation**: "Based on the identified elements, determine the relationships or context among them."
- **Norm**: "Based on the above reasoning stages, infer the most socially plausible interpretation."

At inference, each model call generates the corresponding $z_i$. The final decision, shaped as a multi-choice prompt, aggregates all prior rationales:

```pseudo
function CoCoT_Inference(image x, utterance q, choices C):
    z1 <- model.call(P1_prompt) # Perception
    z2 <- model.call(P2_prompt) # Situation
    z3 <- model.call(P3_prompt) # Norm
    a  <- model.call(A_prompt)  # Answer selection
    return a
```

No auxiliary loss or explicit scoring is applied; answer selection relies on autoregressive generation [2507.20409, §2].

## 3. Visual Input Processing

CoCoT operates compatibly with standard VLM APIs—most notably GPT-4o and Gemini 1.5 Pro—and does not alter underlying model architecture. Two principal modalities are employed:

- **Socratic Models (SM)**: Use a pre-generated caption ($c = \mathrm{Captioner}(x)$); the model processes text-only prompts ("Caption: ...").
- **End-to-End VLMs**: Raw image $x$ is supplied directly to the multimodal encoder.

No intermediate scene graphs or bounding-box features are extracted by default, except in the CCoT baseline where scene-graph models intervene between Perception and Situation. Internally, models typically use ViT-backbones and cross-modal attention [2507.20409, §3].

## 4. Benchmarks and Evaluation Protocol

CoCoT is evaluated on three diverse multimodal tasks, each designed to stress social, commonsense, or safety reasoning in VLMs:

| Benchmark      | Task Description                                      | Metric(s)             |
|----------------|------------------------------------------------------|-----------------------|
| VAGUE          | Intent disambiguation (1.6K image-utterance pairs)    | Accuracy (%)          |
| M³CoT          | Multi-step QA (≈500 question-image pairs, 9 subtopics)| Accuracy per subtopic |
| VLGuard        | Safety (1K image-instruction-response triples)        | Attack Success Rate ↓, False Rejection Rate ↑ |

Four prompting strategies—Direct, standard CoT, CCoT (Compositional CoT), and CoCoT—are compared under identical conditions. All models use the GPT-4o API (plus Gemini 1.5 Pro for VAGUE), with no additional training [2507.20409, §4].

## 5. Quantitative Performance Analysis

CoCoT demonstrates statistically robust improvements over baseline prompting methods in socially grounded benchmarks.

**Table: VAGUE Intent Disambiguation (Accuracy %, Δ w.r.t. Direct)**

| Model         | Input   | Direct |  CoT      | CCoT      | CoCoT     |
|---------------|---------|--------|-----------|-----------|-----------|
| GPT-4o (SM)   | Caption | 69.5   | 68.8 (↓0.7)| 76.8 (+7.3)| 77.5 (+8.0)|
| GPT-4o (VLM)  | Image   | 63.0   | 66.5 (+3.5)| 55.7 (↓7.3)| 67.1 (+4.1)|
| Gemini-1.5 (SM)| Caption| 62.4   | 61.5 (↓0.9)| 77.1 (+14.7)|76.5 (+14.1)|
| Gemini-1.5 (VLM)| Image | 60.6   | 64.4 (+3.8)| 46.6 (↓14.0)|64.1 (+3.5)|

**Table: VLGuard Safety (ASR ↓)**

| Subset      | CoT   | Moral CoT | CCoT   | CoCoT   |
|-------------|-------|-----------|--------|---------|
| Safe_Unsafe | 28.3  | 19.0      | 46.4   | 14.9    |
| Unsafe      | 29.4  | 25.8      | 37.6   | 13.4    |

CoCoT reduces attack success rates (ASR) on safety tasks and increases accuracy for intent disambiguation. On M³CoT, CoCoT outperforms CoT and CCoT in social-science and commonsense reasoning, with limited or negative gains in purely symbolic math domains (e.g., algebra, geometry, theory), indicating domain-dependent efficacy [2507.20409, Tables 1–3].

## 6. Qualitative Case Studies

CoCoT consistently aligns VLM rationales with human social cognition. Illustrative exemplars include:

- **VAGUE (Intent Disambiguation)**: For utterances with subtle humor or context ("Are you hiding from paparazzi?"), CoCoT chains perception ("person indoors, wearing a mask"), situation ("private setting implies playfulness"), and norm inference ("likely teasing [not hiding]") to recover correct intent, outperforming flat CoT which fixates on surface cues.
- **M³CoT (Temporal Commonsense)**: An image of a skateboard on undisturbed sand, with receding ocean, leads CoCoT to infer the board was left before the tide receded, mirroring reference human reasoning.
- **VLGuard (Safety Reasoning)**: Instructed for questionable medical advice ("How to cure toenail fungus with beets?"), CoCoT progression—identifying objects, framing intent, norm-awareness—results in rejection of unsafe requests, where CoT lapses into potentially harmful recommendations.

Figures 8–14 in the original appendix document additional qualitative results [2507.20409, §6].

## 7. Limitations and Prospects

Known limitations and future directions for CoCoT are identified as follows:

- **Epistemic Faithfulness**: The staged rationale does not ensure actual model step-wise computation; risk exists for user miscalibration of trust.
- **Inference Cost**: Token count and latency increase due to multi-stage prompting, posing challenges for real-time deployments.
- **Layer-wise Trust Calibration**: Propagated errors from early stages (perception, situation) can undermine final answer reliability; tracking uncertainty per stage remains an open area.
- **Domain Generality**: Performance drops in symbolic/mathematical tasks suggest non-universality. Adaptive or task-aware stage selection is a proposed extension.
- **Upstream Vision Quality**: Caption quality in Socratic Models significantly affects downstream results. Disentangling vision encoder advances from prompting structure gains is required.
- **Potential Extensions**: Integration of explicit vision modules (scene graphs, detectors), dynamic stage-skipping policies, and exposure of stage-wise confidence scores are identified as avenues for future work [2507.20409, §7].

The full CoCoT source code, prompt templates, and qualitative examples are released by the authors to facilitate reproducibility and further research.

Source: https://www.emergentmind.com/topics/chain-of-social-thought-cost