---
title: 'SO-101 Task Set: Multi-Domain Overview'
url: https://www.emergentmind.com/topics/so-101-task-set
type: topic
---

# SO-101 Task Set: Multi-Domain Overview

Searching arXiv for the cited papers to ground the article.
The expression **SO-101 Task Set** is context-dependent rather than universally standardized. In current arXiv usage, it denotes at least three distinct technical constructions: an introductory Standard Operating Procedure (SOP) task set aligned with the staged training logic of FM SO.P; a real-world manipulation benchmark built on the low-cost SO-101 robot for evaluating Vision-Language-Action and imitation-learning policies; and a possible 101-sequence instantiation inside the SOTVerse user-defined task space for single-object tracking. A related but separate source of nomenclatural overlap is **SOCIAL-CHEM-101**, where “SO-101” refers to Social Chemistry 101 rather than a robotic, tracking, or SOP benchmark [2602.09336][2606.08881][2204.07414][2011.00620].

## 1. Terminological scope and disambiguation

Across the relevant literature, the phrase does not identify a single benchmark family with shared executors, modalities, or metrics. In FM SO.P, SO-101 is an **entry-level SOP task set** designed around a three-stage progression from terminology precision to sequencing and then to conditional graph reasoning. In low-cost robotics, the SO-101 Task Set is a **real-world, execution-centric benchmark** on a physical tabletop manipulator. In SOTVerse, “SO-101 Task Set” is not an official paper term at all; it is a **paper-grounded instantiation** of a 101-sequence user-defined task space derived from SOTVerse construction rules. In Social Chemistry, the closely related label refers to **SOCIAL-CHEM-101**, a corpus of rules-of-thumb and social judgments rather than a benchmark commonly described as an SO-101 task set [2602.09336][2606.08881][2204.07414][2011.00620].

| Usage | Core object | Source |
|---|---|---|
| FM SO.P SO-101 | Introductory SOP task set | [2602.09336] |
| Robotic SO-101 Task Set | Real-world manipulation benchmark on SO-101 robot | [2606.08881] |
| SOTVerse SO-101 instantiation | 101-sequence user-defined tracking task space | [2204.07414] |
| SOCIAL-CHEM-101 | Social and moral norm corpus | [2011.00620] |

This multiplicity creates a recurrent interpretive hazard: identical surface naming can conceal incompatible definitions of task, state, supervision, and evaluation. The most reliable way to interpret the phrase is therefore by domain and source paper, not by the string “SO-101” alone.

## 2. SO-101 as an introductory SOP task set in FM SO.P

In the FM SO.P framework, SO-101 is defined as an **introductory SOP task set** that reflects the progression logic of SOP understanding: **terminology precision $\rightarrow$ sequencing $\rightarrow$ conditional graph reasoning**. The general task formulation is a mapping $f_\theta : (C, Q) \rightarrow A$, where $C$ is contextual information, $Q$ is a query or scenario, and $A$ is the space of valid procedural responses. The task set is explicitly aligned with three capability stages: **Concept Disambiguation**, **Action Sequence Understanding**, and **Scenario-Aware Graph Reasoning** [2602.09336].

| Stage | Task type | Beginner composition |
|---|---|---|
| 1 | Concept Disambiguation | 8–12 contrastive QA pairs per scenario |
| 2 | Action Sequence Understanding | Type A: 1 positive + 4 negatives per pattern; Type B: positive workflow QAs plus negatives violating DAG constraints |
| 3 | Scenario-Aware Graph Reasoning | One or two small DAG scenarios per domain with negative-only workflow QAs |

The first stage targets terminology precision through contrastive QA pairs grounded in SOP constraints and system state. Its formal setup distinguishes semantically similar but contextually distinct terms, with outputs consisting of a correct answer $a^+$ and negatives $\{a^-_j\}^m$. The training signal is a contrastive loss,
$$
L_1(\theta) = - \mathbb{E}_{(q,c,a^+,\{a^-\})} \left[ \log \frac{e^{s^+/\tau}}{e^{s^+/\tau} + \sum_j e^{s^-_j/\tau}} \right].
$$
The second stage teaches procedural correctness over workflows $W=\langle a_1,a_2,\dots,a_m\rangle$ using preconditions and effects, with negatives generated by reorder, omit, and insert operators. The third stage extends to conditional reasoning over directed action graphs $G=(V,E,\phi)$ and introduces violations such as cycles, missing preconditions, and invalid edges. At the curriculum level, FM SO.P uses cumulative data,
$$
D_{\text{train}}(k) = \bigcup_{i=1}^{k} D_i,
$$
and a unified objective
$$
\theta^* = \arg\min_\theta \left[\alpha_1 L_1 + \alpha_2 L_2 + \alpha_3 L_3\right],
$$
with $\alpha_1=\alpha_2=\alpha_3=1/3$.

The paper specifies beginner-oriented templates for SO-101. Stage 1 uses 8–12 contrastive QA pairs per scenario, with emphasis on authentication or authorization logic, threshold-based constraints, and domain-specific term disambiguation. Stage 2 mixes **Type A** constraint understanding and **Type B** workflow sequencing. Type A uses 4–5 scenario patterns with 1 positive and 4 negative QAs; Type B uses 2–3 scenario patterns with positive workflow QAs and several negative workflow QAs that violate order, omit steps, or use wrong prerequisites. Stage 3 introduces one or two small DAG scenarios per domain and focuses on negative-only workflow QAs involving skipped authentication, missing OR-satisfaction, wrong order, or missing parameters.

Evaluation is domain-adaptive rather than fixed. FM SO.P uses a three-agent system: an **Adaptive Rubric Generator** producing rubrics $R(D)=\{(r_i,w_i)\}$ with normalized weights, a **Stratified Test Set Builder** constructing $B=\bigcup_l B_l$ across complexity and question types, and a **Rubric Scorer** aggregating per-dimension scores by
$$
e_{\text{final}} = \sum_i w_i \cdot e_i.
$$
Pass rate is formalized as
$$
\text{PassRate} = \frac{N_{\text{pass}}}{N_{\text{total}}} \times 100\%.
$$

On SOPBench across **Bank, DMV, Healthcare, Market, University, Library,** and **Hotel**, FM SO.P reports **34.33%** overall pass rate for the Qwen-2.5-7B-Instruct variant, **39.67%** for Qwen-2.5-14B-Instruct, and **48.30%** for Qwen-2.5-32B-Instruct. The 7B variant matches **Qwen-2.5-72B-Instruct** at **34.44%** with approximately **10× fewer parameters**, and stage-wise ablations rise from **11.30%** at the 7B base to **17.11%**, **24.50%**, and **34.33%** after Stages 1, 2, and 3. Training is reported with **AdamW**, learning rate **$1\times10^{-4}$**, batch size **256**, **12 epochs**, temperature **$\tau=1$**, mixed-precision **bfloat16**, and **8× NVIDIA H100-80GB GPUs**.

## 3. SO-101 as a low-cost real-world robotic benchmark

In embodied AI, the SO-101 Task Set is a **standardized real-world benchmark** defined on the low-cost, open-source SO-101 tabletop manipulator. Its emphasis is execution-centric: the benchmark is designed to expose robustness gaps that emerge on affordable hardware under embodiment uncertainty, including limited actuator precision, reduced control stability, lower payload capacity, joint backlash, control latency, trajectory jitter, limited positional repeatability, and imperfect camera–arm calibration [2606.08881].

| Task | Goal or instruction | Capability stressed |
|---|---|---|
| Pen Transfer | Move a pen from a left-side start region to a right-side target region | Control fidelity |
| Selective Color Sorting | “Put the pink block in the plate and remove the others” | Grounding robustness |
| Multi-Object Packing | “Place all snacks into the cardboard box” | Temporal consistency and long-horizon stability |
| Precision Pen Placement | Pick up a pen and insert it into a pen holder | Embodiment-noise robustness |

The benchmark standardizes task execution, data collection, and adaptation directly on the physical platform. It uses **100 human teleoperated trajectories per task**, collected with end-effector control and synchronized RGB, instruction, and low-level commands, for a total of **400 demonstrations** across the four tasks. Object poses and layouts are randomized within predefined workspace constraints during both data collection and evaluation. Each model–task pair is evaluated in **20 independent, real-hardware episodes**, yielding **320 episodes** total for four models across four tasks.

The evaluated policies are **$\pi_{0.5}$**, **SmolVLA**, **Wall-X**, and **ACT**. Adaptation is constrained to interface-level alignment between model observations or actions and SO-101; no changes are made to architectures, losses, or optimizers. The benchmark augments binary success with a structured failure taxonomy: **Grasp Instability**, **Repetition Loop**, **State Mismatch**, and **Precision Misalignment**. For aggregate analysis, **State Mismatch** is treated as **semantic-level failure**, while **execution-level failure** is the average of **Grasp Instability** and **Repetition Loop**. **Precision Misalignment** is excluded from that aggregate because it is task-specific.

The metric definitions are deliberately minimal and event-based. Success rate is
$$
S = \frac{N_{\text{success}}}{N_{\text{total}}},
$$
and recovery rate is
$$
R = \frac{N_{\text{successful recovery}}}{N_{\text{recovery opportunity}}}.
$$
A successful recovery is credited when the policy restores a valid task state and resumes progress without external help.

Reported success rates are sharply task-dependent. On **Pen Transfer**, ACT reaches **75%**, SmolVLA **70%**, Wall-X **95%**, and $\pi_{0.5}$ **95%**. On **Selective Color Sorting**, the same models reach **0%**, **5%**, **0%**, and **10%**. On **Multi-Object Packing**, they achieve **10%**, **10%**, **30%**, and **55%**. On **Precision Pen Placement**, they achieve **50%**, **45%**, **80%**, and **65%**. Average success is **33.75%** for ACT, **32.5%** for SmolVLA, **51.25%** for Wall-X, and **56.25%** for $\pi_{0.5}$. Recovery rates further separate the policies: **30.77%** for $\pi_{0.5}$, **20.51%** for Wall-X, **6.45%** for ACT, and **3.23%** for SmolVLA. The dominant empirical pattern is that execution instability remains high across all models, while semantic failure varies more strongly by architecture.

## 4. SO-101 as a SOTVerse user-defined tracking task space

Within the SOTVerse literature, “SO-101 Task Set” is not an official benchmark name. The paper states explicitly that the term **does not appear explicitly anywhere in the paper**. It nevertheless provides a precise construction by which an SO-101 Task Set can be defined as a **user-defined task space comprising 101 sub-sequences** selected from SOTVerse according to the 3E paradigm, challenging-factor filters, and space construction rules [2204.07414].

SOTVerse formalizes a subtask as
$$
S = E \times M_s \times M_p \times T,
$$
and the full task space as
$$
\mathbb{S} = \mathbb{E} \times \mathbb{M} \times \mathbb{T}.
$$
Here, the environment is built from eight representative benchmarks—**OTB2015, VOT2016, VOT2018, VOT2019, GOT-10k, VOTLT2019, LaSOT,** and **VideoCube**—organized into a normal space totaling **12.56 million frames**. SOTVerse automatically labels challenging factors per frame and allows user-defined task generation through explicit construction rules.

For an SO-101 instantiation, the environment consists of **101 sequences** constructed from those benchmarks using per-frame challenging labels and inclusion rules. A sequence is regarded as challenging for an attribute if **more than half of the frames** are challenging for that attribute. Construction also requires **minimum sub-sequence length 100 frames**, start-point screening to exclude tiny or blur targets and frames near target absence, and deduplication that discards overlaps of **$\geq 50\%$**. A suggested design in the paper is **10 sequences per challenging factor** from **$c_1$–$c_{10}$** plus **one normal sequence** from the short-term, long-term, or GIT normal spaces.

The challenging factors are divided into static and dynamic attributes. Static attributes include ratio, relative scale, illumination, and blur bbox; dynamic attributes include delta ratio, delta relative scale, delta illumination, delta blur bbox, fast motion, and low corrcoef. The paper provides explicit thresholds, for example **$c_9$ fast motion: $\epsilon_t \geq 0.16$** and **$c_{10}$ low corrcoef: $\rho_t \leq 0.75$**. Evaluation uses **OPE** and optionally **R-OPE**, with failure in R-OPE triggered by **10 consecutive failures** and re-initialization at the next start point. Reported indicators include **precision**, **normalized precision**, **success**, **challenging plot**, **attribute plot**, and **robust plot**.

The significance of this construction is methodological rather than nominal. SOTVerse treats the task space as compositional and user-defined, so the SO-101 label functions here as a specific selection protocol over a much larger metaverse of single-object tracking environments, not as a canonical benchmark released under that exact name.

## 5. Relation to SOCIAL-CHEM-101

A separate line of work uses “SO-101” to refer to **Social Chemistry 101**, formally **SOCIAL-CHEM-101**. This resource is part of the broader Social Chemistry formalism for reasoning about social and moral norms in natural-language situations. It is not a robotic or SOP benchmark, and its canonical object is a corpus consisting of **104k real-life situations**, **292k rules-of-thumb**, **365k structured “breakdown” annotations**, and **over 4.5M categorical and free-text annotations** [2011.00620].

The resource is built around four variables: situation $s$, rule-of-thumb $r$, action transcription $a$, and attribute sets $b=\{b_r,b_a\}$. It defines **12 judgment dimensions** spanning RoT-level attributes and action-level attributes. These include anticipated agreement, RoT category, moral foundations, RoT targeting, action transcription, agency, social judgment, legality, cultural pressure, action candidate, and taking action. Label spaces are explicit: for example, social judgment takes values **{Very bad, Bad, Expected/OK, Good, Very good}**, legality uses **{Illegal, Depends/Tolerated, Legal}**, and cultural pressure uses **{Strong pressure against, Pressure against, Discretionary, Pressure for, Strong pressure for}**.

The task family is correspondingly language-centric. It includes attribute classification, moral foundation classification, legality and cultural pressure prediction, controlled conditional generation, model-choice generation, retrieval or matching, and free-text explanation generation. Representative objectives include controlled generation
$$
L = -\sum_{(s,b_y,y)\in D}\log p_\theta(y\mid s,b_y),
$$
model-choice generation
$$
L = -\sum_{(s,y,b_y)\in D}\log p_\theta(y,b_y\mid s),
$$
and attribute-only prediction
$$
L = -\sum_{(s,b_y)\in D}\log p_\theta(b_y\mid s).
$$
The paper’s Neural Norm Transformer instantiates these objectives with decoder-only and encoder–decoder LMs, including GPT, GPT-2, BART-Large, and T5-Large, using an **80/10/10** train/dev/test split by situations.

This usage matters because it explains why “SO-101” may be encountered outside SOPs, tracking, or robotics. In this setting, however, the object is a structured norm-reasoning corpus rather than a task set centered on physical execution, graph constraints, or benchmark episodes.

## 6. Comparative evaluation regimes and recurring misconceptions

The main senses of SO-101 differ most sharply in what they count as an **environment**, an **executor**, and a **successful outcome**. FM SO.P evaluates language models against domain-adaptive SOP rubrics and reports **PassRate**, rubric-weighted scores, and **Borda counts**. The robotic benchmark evaluates VLA and imitation-learning policies on physical tasks and reports **success**, **recovery**, and failure-code distributions. SOTVerse instantiates an SO-101 space over tracking sequences and uses **precision**, **normalized precision**, **success**, **challenging plot**, **attribute plot**, and **robust plot**. SOCIAL-CHEM-101 evaluates social-norm models through classification losses, human relevance judgments, **BLEU-4**, and attribute adherence such as **micro-F1** or **Attr. pF1** [2602.09336][2606.08881][2204.07414][2011.00620].

| Context | Executor | Primary evaluation |
|---|---|---|
| FM SO.P SOP SO-101 | Language model | $e_{\text{final}}$, PassRate, Borda count |
| Robotic SO-101 | VLA or imitation-learning policy | $S$, $R$, failure taxonomy |
| SOTVerse SO-101 instantiation | Tracker | Precision, normalized precision, success, robust plot |
| SOCIAL-CHEM-101 | Language model | Classification metrics, BLEU-4, human relevance, micro-F1 |

Several misconceptions follow from the name collision. First, the robotic SO-101 benchmark and the FM SO.P SO-101 task set are not related by modality or supervision: one is a real-hardware benchmark with teleoperated demonstrations, while the other is a staged SOP understanding curriculum with contrastive QA pairs, workflow negatives, and DAG reasoning. Second, the SOTVerse usage is a **constructible instantiation**, not an official benchmark title in the paper. Third, SOCIAL-CHEM-101 is a corpus of social and moral norms, not a physical or procedural benchmark. A plausible implication is that “SO-101 Task Set” functions less as a standardized benchmark identifier than as a local naming convention whose meaning must be recovered from the surrounding research program.

Under that interpretation, the most stable encyclopedia-level characterization is not a single definition but a typology: **SO-101** names beginner or diagnostic task configurations in multiple subfields, yet each version is anchored to a different formalism—procedural reasoning over SOPs, execution robustness on low-cost robots, challenge-aware subsequence selection in tracking, or norm-conditioned text modeling.

Source: https://www.emergentmind.com/topics/so-101-task-set