Cortical Messaging Protocol (CMP)

• CMP is a modality-agnostic protocol that standardizes the exchange of spatial features and object hypotheses across sensor, learning, and motor modules.
• It supports bottom-up, lateral, and top-down communications, enabling rapid consensus and efficient sensorimotor fusion in modular systems.
• Inspired by neocortical communication, CMP offers plug-and-play interoperability, enhancing the scalability and performance of distributed sensorimotor architectures.

The Cortical Messaging Protocol (CMP) is the modality-independent interface by which all components of a Thousand Brains system—namely, sensor modules (SMs), learning modules (LMs), and motor modules—exchange spatial and object-centric information, specifically “features at poses.” CMP affords plug-and-play modularity in distributed sensorimotor architectures by permitting architectural elements to operate and communicate in standardized, serializable message formats. CMP is biologically inspired, drawing from the long-range cortico-cortical and thalamo-cortical information exchanges found in the neocortex, and implements a high-level “wire protocol” to support hierarchical, lateral (voting), and top-down exchanges of sensorimotor hypotheses, object identities, and goal states (Clay et al., 2024).

1. Purpose and Biological Motivations

CMP provides a unified, modality-agnostic protocol to manage the exchange of spatially anchored features and object hypotheses throughout Thousand Brains systems. The protocol standardizes the communication required for:

• Bottom-up inference: relaying sensed or inferred features and their poses (positions and orientations) from lower to higher levels.
• Lateral integration (voting): rapidly reaching consensus among parallel or redundant modules via hypothesis sharing.
• Top-down context: transmitting goal-states or priors, for example, desired poses and features, to bias inference and action selection.
• Sensor/actuator cohesion: coordinating multi-modal sensor integration and translating goals into actuator commands.

CMP is specifically motivated by biological findings that cortical columns in the neocortex exchange high-level hypotheses (e.g., object identities and pose information) without sharing their full internal models. This design reflects evidence that location-tagged hypotheses broadcast between cortical and thalamic regions facilitate efficient sensorimotor inference (Clay et al., 2024).

2. Structure of CMP Messages

A CMP message encapsulates information about features and their spatial context, enabling seamless integration across disparate modules. Each message comprises the following:

Field	Type/Structure	Purpose/Example
x	Vector $\in \mathbb{R}^3$	Position in body-centric space (e.g., $(x, y, z)$)
R	Matrix $\in SO(3)$	Orientation: 3 orthonormal axes (e.g., $n$, $c_1$, $c_2$)
f	Discrete ID or Vector	Feature/object label (e.g., “mug”) or embedding
$\{h_1, \ldots, h_F\}$	Array of floats (optional)	Non-morphological features (e.g., color histogram)
confidence	Float $[0,1]$	Belief measure for hypothesis
use_flag	Boolean	$u=1$ for processing downstream
sender_id	String	Module identifier
sender_type	Enum {SM, LM, Motor}	Module category

Not all fields are mandatory; optional fields can be omitted or set to null. This flexible structure allows diverse sensors and modules to participate without superfluous overhead.

struct CMPMessage { Vector3 x; Matrix3 R; FeatureID f; float h[F]; float confidence; bool use_flag; string sender_id; enum {SM, LM, Motor} sender_type; };

3. Communication Topology and Connectivity

CMP supports three principal connection modalities:

• Bottom-up (Hierarchical): SM $\rightarrow$ LM $\rightarrow$ higher-level LM $\rightarrow$ …, with each LM treating the object ID output of its input module as a feature for its own inference.
• Lateral (Voting, Non-hierarchical): LM$_i \leftrightarrow$ LM$_j$, where modules exchange alternative object–pose hypotheses to speed consensus formation and error correction.
• Top-down (Goal/Context): LM$_n \rightarrow$ LM$_k$, transmitting expectations, priors, or goal-states to guide lower-level computation.

There is no fixed hierarchy or mandatory layering: modules connect according to simple adjacency lists, enabling arbitrary heterarchies. A single LM may interface with multiple SMs, several LMs, and the motor system simultaneously.

No routing tables or packet headers are implemented beyond sender_id, favoring architectural transparency and minimal communication overhead.

4. Operational Semantics and Protocol Dynamics

CMP communication is event-driven and synchronized to discrete time steps, usually coinciding with the agent’s movement/observation cycle. Protocol rules include:

• Messaging Schedule: At each time step $t$, any SM or LM with fresh information emits zero or more CMPMessages to downstream recipients. There is no global clock aside from synchronization at time steps.
• Conflict and Consensus: Each module integrates “evidence” for competing hypotheses, summing weighted votes. Inconsistent or low-confidence votes reduce net evidence and are pruned if below acceptance thresholds.
• Temporal Synchronization: Modules strictly process input messages from time $t$ before constructing outputs for time $t+1$. No asynchrony or message “lookahead” is supported.
• Learning Isolation: CMP carries hypotheses only—no transfer of internal parameters occurs over the protocol. Modules perform learning (e.g., Hebbian or STDP updates) internally, typically episodically after evidence integration.

5. Illustrative Pseudocode

A minimal forward-feedback CMP messaging loop involving two LMs can be outlined as:

def step_module_A(sensor_obs, prev_obs_A): disp_A = sensor_obs.pose - prev_obs_A.pose A.update_local_hypotheses(sensor_obs.features, disp_A) # send current most likely hypothesis to downstream B msg_A = CMPMessage( x=A.best_pose, R=A.best_orientation, f=A.best_object_id, confidence=A.best_confidence, sender_id=A.id, sender_type='LM' ) send(to=B, msg=msg_A) def step_module_B(msgs_from_A, sensor_obs_B, prev_obs_B): disp_B = sensor_obs_B.pose - prev_obs_B.pose B.update_local_hypotheses(sensor_obs_B.features, disp_B) for m in msgs_from_A: pA_in_B = transform_pose(m.x, m.R, A_to_B_displacement) B.update_hypotheses_with_vote(m.f, pA_in_B, m.confidence) msg_B = B.compose_vote() send_to(peers_of_B, msg_B)

This pseudocode demonstrates context-driven hypothesis updating, evidence integration, and propagation of current best hypotheses in a fully modular and decentralized manner (Clay et al., 2024).

6. Multimodal and Hierarchical Applications

CMP enables several canonical neuro-inspired computation scenarios:

• Multimodal Sensor Fusion: E.g., a camera patch (SM$_1$) emits “feature=pointNormalType3” at a 3D pose; a touch patch (SM$_2$) emits “feature=curvatureHigh” at a nearly coincident pose, both with high confidence. Associated LMs exchange “mug” object hypotheses via CMP, rapidly reaching consensus after lateral voting.
• Hierarchical Composition: A low-level LM recognizes a “tire” at pose $p_t$, passing this via CMP to a higher-level LM, which concurrently ingests raw features (e.g., from a wide-span SM) and learns that a “car” is composed of “tire”, “window”, etc., at specific relative poses.

The protocol’s architectural neutrality allows modules representing distinct modalities, spatial fields, or abstraction levels to interoperate seamlessly.

7. Empirical Performance and Constraints

Empirical assessment in simulated object-recognition (Habitat environment) indicates:

• Single-LM systems required 4–6 steps to converge to an object hypothesis.
• Voting among five LMs reduced recognition to 2–3 steps on average.
• Tuning the evidence threshold parameter enables controllable tradeoff between speed (fewer steps) and classification accuracy.

No specific measurements of bandwidth or bit-rate are reported; CMP message payloads are typically 20–50 floats plus metadata, presenting negligible overhead for Python-based implementations. CMP does not specify or constrain network-level packetization, latency, nor scheduling at a bitwise or lowest-level cognitive cycle granularity.

---

All structural, procedural, and empirical descriptions are sourced directly from “The Thousand Brains Project: A New Paradigm for Sensorimotor Intelligence” (Clay et al., 2024).