---
title: Sensor-Language-Action (SLA) Modeling
url: https://www.emergentmind.com/topics/sensor-language-action-sla-modeling
type: topic
---

# Sensor-Language-Action (SLA) Modeling

Sensor-Language-Action (SLA) modeling is a framework for connecting heterogeneous sensor observations, natural language, and actions within a unified computational system. Its central objective is to transform sensor evidence and contextual information into action selection, while preserving a language-mediated representation of state, intent, evidence, and consequences. The term encompasses several complementary research directions: formal action-sufficient sensing for guaranteed planning [2006.04003], sensor-conditioned and language-conditioned robotic policies [2607.03693], compact action-grounded latent representations [2608.09771], transition-centered multimodal control [2609.15870], and language-mediated action prediction from physiological and environmental sensor streams [2610.08244]. It is distinct from the separate use of “SLA” for Synthetic Linguistic Agency, which denotes a theory of embodied, socially consequential linguistic behavior [2608.27843].

## 1. Conceptual foundations and scope

SLA modeling treats sensing, language, and action as a causally connected system rather than as independent perception, language-understanding, and control modules. A generic SLA process can be represented as

$$
\text{sensor history}+\text{context}+\text{language}
\longrightarrow
\text{state and evidence representation}
\longrightarrow
\text{action}
\longrightarrow
\text{environmental feedback}.
$$

Language can specify a goal, provide task intent, describe sensor evidence, represent an action, or explain why an action is associated with observed conditions. The action may be a physical robot control, a clinical intervention, a procedure, an insulin dose category, or a linguistic speech act.

The unifying principle is that sensor information should be organized around the decisions it supports. In formal action-based sensing, a sensor need not preserve complete world-state identity; it need only preserve distinctions relevant to selecting a progress-making action. An action-based sensor is an abstract mapping from observations to one or more admissible actions that guarantee progress toward a goal [2006.04003]. In learned SLA systems, the analogous objective is to learn representations that retain control-relevant dynamics, task intent, embodiment constraints, or action evidence while discarding information irrelevant to the decision.

Several forms of SLA modeling are therefore distinguished:

| Form | Primary sensor domain | Action representation |
|---|---|---|
| Robotic SLA | RGB, depth, proprioception, multimodal robot observations | Continuous action chunks or native robot controls |
| Transition-centered SLA | Camera views, embodiment state, human and robot trajectories | Shared latent actions plus embodiment-specific controls |
| Physiological SLA | Waveforms, numerical measurements, contextual history | Hierarchical clinical or metabolic actions |
| Socially embodied SLA | Body state, partner responses, linguistic context | Speech acts whose consequences alter viability |

The framework is broader than conventional sensor-to-label prediction. Sensor models traditionally recognize states, forecast outcomes, or classify fixed labels. Sensor-language models align signals with textual descriptions but may not represent downstream decisions. Sensor-to-action systems often use application-specific, closed action spaces. SLA modeling introduces language as a semantic interface linking sensor evidence, contextual state, heterogeneous action categories, and explanations [2610.08244].

The term also has a stricter interpretation in the work on Synthetic Linguistic Agency. There, SLA is defined as the conjunction of synthetic embodiment, synthetic linguistic participation, and synthetic precariousness:

$$
\operatorname{SLA}(x):=E_s(x)\land LP_s(x)\land Pr_s(x).
$$

This meaning concerns a continuing, vulnerable agent whose linguistic actions alter social behavior and whose consequences affect the future viability of the same body. It should not be conflated with Sensor-Language-Action modeling, although both emphasize causal feedback among sensing, language, action, and future state [2608.27843].

## 2. Formal action sufficiency and planning

A formal foundation for the sensor–action component of SLA is provided by the theory of action-based sensors. A planning world is modeled as

$$
W=(V,V_0,G,Y,U,E),
$$

where $V$ is a finite set of world states, $V_0$ is the set of possible initial states, $G$ is the set of goal states, $Y$ is the observation set, $U$ is the action set, and $E$ contains world transitions labeled by compatible actions. The agent observes $y\in Y$, chooses an action, transitions along a compatible world edge, receives another observation, and continues until reaching a goal. The principal setting has a single goal, $|G|=1$, and permits the robot to begin in any world state, $V_0=V$ [2006.04003].

A plan is required to be safe, observation-ready, and finitely guaranteed to complete. Safety means that it never attempts an unavailable action. Observation readiness means that it can process every observation arising from a compatible execution. Finite guaranteed completion means that every joint execution reaches both a terminating plan state and a goal world state, regardless of the initial state and nondeterministic outcomes allowed by the world.

The classical construction derives a progress measure from a plan. An execution progress measure maps sets of world states to real values and strictly decreases along every valid execution, reaching zero only at goal states. A vertex progress measure $g$ induces

$$
\phi_g(V)=\max_{w\in V}\{g(w)\}.
$$

Progress need not correspond to geometric or graph distance. A valid plan may temporarily move away from the goal, so progress is relative to the strategy encoded by the plan.

For each action $u$, the plan induces a progress cone: the largest set of observations from which $u$ decreases the progress measure. These cones form a relation

$$
C\subseteq Y(W)\times U(W),
$$

where $(y,u)\in C$ means that $u$ is progress-making at observation $y$. A singleton action-based sensor is a function

$$
f:Y(W)\rightarrow U(W)
$$

such that $(y,f(y))\in C$ for every observation. A permissive action-based sensor instead returns a nonempty set of admissible actions:

$$
f:Y\rightarrow 2^U\setminus\{\varnothing\}.
$$

Every action in $f(y)$ must belong to the progress cone associated with $y$. The singleton case resembles a deterministic reactive policy; the permissive case resembles a safe-action filter or set-valued controller.

This theory formalizes control-sufficient perception. A conventional sensor is a mapping $s:V(W)\rightarrow Y(W)$. Given a singleton action-based sensor, observations can be coarsened into action classes, producing a new observation set $Y'=\{y'_u\mid u\in U(W)\}$. The resulting composition

$$
W\rightarrow Y\rightarrow Y'
$$

is equivalent to a direct sensor mapping from world states to action-relevant observations. The sensor need not identify complete geometry, object identity, or semantic scene structure if those distinctions do not alter the required action.

The classical backchained-plan method is incomplete because a valid plan can contain crossovers. A crossover occurs when different executions visit the same world states in contradictory orders, requiring both $\phi(S)>\phi(S')$ and $\phi(S')>\phi(S)$. The contradiction arises because a plan can retain internal branch or memory information that a state-only progress measure discards. The theory establishes that a progress measure exists if and only if the plan contains no crossover [2006.04003].

To handle crossovers, the generalized method constructs a Plan–World Interaction Graph, or I-Graph, comprising initiating, plan, and world layers. Cycles corresponding to contradictory execution order are identified and clipped by removing candidate edges while preserving viable actions, goal reachability, acyclicity, and finite guaranteed completion. Because different crossover edges may be removed, the result can be a family of representative plans and sensors rather than a canonical sensor. Under finite-world, single-goal, globally initialized, initially fully observable assumptions, the construction is claimed to capture all action-based sensors.

## 3. Architectures for multimodal sensor–language–action policies

Modern robotic SLA models generally combine a persistent or primary sensor stream, optional auxiliary modalities, language instructions, proprioception, and a continuous action generator. CoRE-VLA uses RGB images, language instructions, proprioceptive state, and optional depth. Its architecture contains a pretrained vision-language model, an auxiliary-modality encoder, a flow-matching Action Diffusion Transformer, and Conditional Routing of Experts blocks [2607.03693].

The vision-language pathway produces multimodal tokens

$$
Z_{\mathrm{vl}}=\Phi_{\mathrm{VLM}}(I_t,\ell_{\mathrm{lang}}).
$$

An instruction-centered task-intent vector is obtained by pooling language-token representations that have already participated in multimodal attention:

$$
g=\operatorname{AvgPool}(C_{\mathrm{text}}).
$$

When an auxiliary modality is available, its representation is concatenated with the vision-language tokens. When it is absent, the model uses only the vision-language representation. The resulting policy is explicitly conditioned on the sensor configuration:

$$
\pi_\theta\left(\mathbf a_{t:t+H}\mid I_t,\ell_{\mathrm{lang}},s_t,A_t,z_{\mathrm{mod}}\right),
$$

where $z_{\mathrm{mod}}$ indicates auxiliary-sensor availability.

CoRE-VLA routes computation along two dimensions. First, task intent selects the action-side tokens that receive expensive cross-attention and expert processing. Second, the sensor-availability indicator masks modality-specialized experts. General experts remain available under all configurations, whereas depth-specialized experts are disabled when depth is absent. During training, the auxiliary modality is randomly dropped, with simulation configurations using depth-dropout probability $0.2$. At deployment, the availability indicator is set according to whether the sensor is present and valid.

The action generator uses conditional flow matching. A continuous action chunk is interpolated with Gaussian noise, and the Action DiT predicts the velocity field required to transform noise into the action distribution. The chunk is generated through numerical integration and executed using receding-horizon closed-loop control. Thus, language does not merely alter the final action numerically; it determines which action-side representations and experts receive sensor-language computation.

SLIM-0.5B adopts a different architectural strategy. It uses two RGB views, proprioception, a language instruction encoded by frozen T5-small, a DINOv2-B/14 visual encoder, and a compact Mixture-of-Transformers backbone [2608.09771]. The observation stream contains visual latent tokens, a proprioceptive state token, and learned future slots. The action stream contains continuous action tokens. The two streams interact through joint attention, then receive separate language cross-attention and feed-forward processing.

The model uses flow matching to generate continuous action chunks, with horizons of eight steps for LIBERO and twelve for CALVIN. At inference, future observations are not supplied and future images are not generated. Learned future slots retain the effects of predictive training while keeping deployment computationally compact.

OpenSLA applies SLA modeling to clinical care, operating-room care, and metabolic health. Its inputs include historical sensor windows and textual context. The action target is hierarchical:

$$
Y=(Y^{\mathrm{nec}},Y^{\mathrm{cat}},Y^{\mathrm{label}}),
$$

where the components represent action necessity, applicable action categories, and fine-grained action labels [2610.08244].

Its hierarchical sensor encoder processes waveforms, numerical observations, temporal information, and measurement identities. Hierarchical Memory recursively compresses long sensor sequences with learned queries. In the clinical setting, this reduces average sensor tokens from $5801.0$ to $400.4$, corresponding to $14.49\times$ compression and a $3.4\times$ measured speedup. The language backbone is a Qwen-3.5-2B model adapted with LoRA. Independent heads predict action necessity and categories, while fine-grained actions are represented by textual descriptions and scored through cosine similarity. This open-vocabulary action representation supports prediction of an action absent during fine-label training if its textual description can be encoded.

## 4. Latent transition representations and predictive control

A major research direction in SLA modeling replaces direct raw-action supervision with latent representations of world change. WLA$^3$ argues that native robot actions are hardware-specific, while human videos often lack reliable action labels. Observed transitions are available across human, robot, simulation, and UMI data, making them a common supervision source [2609.15870].

Its World Latent Action Model encodes an endpoint pair $(s_t,s_{t+h})$ into two representations: a compact local latent action $\mathbf z_{t,h}\in\mathbb R^{32}$ and a richer transition feature $\mathbf g_{t,h}\in\mathbb R^{1024}$. The latent action is sampled from a variational posterior conditioned on the transition feature. Cross-modal transition dropout removes camera or embodiment-state streams while applying the same mask at both endpoints, preventing the model from confusing modality appearance changes with physical motion.

WLAM uses pixel, perceptual, optical-flow, semantic, depth, geometric, and state reconstruction terms, together with overlapping-window consistency. The resulting latent action is intended to encode physical change rather than a robot-specific motor command. The richer transition feature is aggregated over a 30-step policy segment into a Semantic Latent Aggregate:

$$
\mathbf c_t^{\mathrm{seg}}
=
\frac{1}{T-h+1}
\sum_{k=0}^{T-h}\mathbf g_{t+k,h}.
$$

The aggregate supervises a semantic query in the VLM. The VLM receives the current observation and instruction, while the target is computed offline from the observed future transition. This produces semantic physical grounding without requiring manually written action descriptions or token-level action annotations.

WLA$^3$ also uses the 32-dimensional latent action to condition a Latent Action-Conditioned World Model. LAC-WM predicts future visual features under observation, language, and optionally latent-action conditions. It is trained with conditional flow matching and temporal-difference losses. The resulting dynamics context is fused with VLM representations and supplied to LARA, a latent-action and robot-action expert.

LARA predicts both a shared latent-action branch and embodiment-specific robot controls. Native robot actions are embedded in a fixed 32-dimensional interface using zero-padding and validity masks. Human videos generally supervise the latent branch, while robot and UMI trajectories can supervise both latent and native-control branches. The architecture therefore shares transition structure while retaining embodiment-specific execution heads.

SLIM uses a related but more compact predictive strategy. Its action-grounded latent is learned through two complementary masked-trajectory objectives. The inverse-dynamics branch receives current and future observation latents and reconstructs the action chunk. The forward-dynamics branch receives the current observation latent and clean action chunk and predicts the future latent. The combined objective is

$$
\mathcal L_{\mathrm{stage1}}
=
\lambda_{\mathrm{IDM}}\mathcal L_{\mathrm{IDM}}
+
\lambda_{\mathrm{FDM}}\mathcal L_{\mathrm{FDM}},
$$

with implementation values $\lambda_{\mathrm{IDM}}=0.125$ and $\lambda_{\mathrm{FDM}}=1$.

The inverse objective requires the action to explain the observed transition. The forward objective requires the future transition to be predictable from the current latent and action. An exponential-moving-average target encoder stabilizes the future-latent objective and preserves latent diversity. After predictive training, the deployable policy uses learned future slots rather than explicit future-latent generation. This separates training-time predictive structure from inference-time action generation.

These approaches differ in scale and role. WLA$^3$ explicitly separates local latent actions, segment-level semantic aggregates, dynamics modeling, and native robot controls. SLIM compresses sensor–action interaction into a compact 0.47-billion-parameter model. Both treat action-relevant transition structure as more useful than unrestricted pixel-level future prediction for manipulation.

## 5. Training objectives, data, and evaluation

SLA training commonly combines action prediction with auxiliary objectives that enforce semantic, dynamic, routing, or explanatory structure.

For CoRE-VLA, the principal objective is flow-matching action generation augmented by token-selection and expert-load regularization:

$$
\mathcal L
=
\mathcal L_{\mathrm{FM}}
+
\lambda_{\mathrm{sel}}\mathcal L_{\mathrm{sel}}
+
\lambda_{\mathrm{moe}}\mathcal L_{\mathrm{moe}}.
$$

The selector regularizer encourages use of the target fraction of action-side tokens, while the mixture-of-experts regularizer discourages unbalanced expert assignment. Discrete Top-$K$ token selection and hard top-1 expert assignment are treated as fixed during backpropagation, with continuous scores providing differentiable paths.

SLIM uses Stage 1 masked trajectory prediction followed by Stage 2 flow-matching policy training. It has 0.47 billion trainable parameters, uses BF16, and performs four flow-sampling steps at inference. The reported model achieves $97.5\%$ overall LIBERO success and an average CALVIN sequence length of $4.556/5$. On LIBERO-Plus, its overall zero-shot success is $77.45\%$. Removing Stage 1 worsens both LIBERO-Plus and CALVIN performance, while removing the EMA target reduces LIBERO-Plus success from $77.45\%$ to $66.82\%$ and CALVIN average sequence length from $4.556$ to $4.382$.

SLIM’s computational measurements on a single NVIDIA H100 80GB report latency of $60.6$ ms, incremental VRAM of $4.26$ GiB, and $490.73$ GFLOPs per action chunk. The comparison uses native horizons and sampling schedules, so these figures describe the reported configurations rather than an architecture-independent complexity law.

WLA$^3$ trains WLAM on an 84.1K-hour corpus containing human egocentric video, robot trajectories, simulation trajectories, and UMI demonstrations. Human videos provide transition supervision, while robot trajectories ground the shared latent in executable native controls. On LARYBench, the final 32-dimensional latent reaches $67.89\%$ average classification accuracy and $0.49$ average regression MSE. Cross-embodiment transfer accuracy averages $20.98\%$ across human-to-robot and robot-to-human directions.

On six AgiBot G1 tabletop tasks, WLA$^3$ reports $81.9\%$ average success compared with $66.2\%$ for $\pi_{0.5}$. Its cumulative ablation rises from $72.1\%$ to $76.3\%$ with LAC-WM, to $79.8\%$ with LARA, and to $81.9\%$ with SLA. Increasing generalist mid-training data from 5K to 10K to 20K hours produces $57.6\%$, $65.9\%$, and $81.9\%$ average success, respectively. In a human-to-robot transfer experiment, adding task-matched human demonstrations to $5\%$ robot data raises success from $40\%$ to $60\%$, while full robot data reaches $75\%$.

CoRE-VLA evaluates sensor-conditional routing on LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation. On LIBERO, it reports an average success rate of $98.7\%$, with $97.6\%$ on the long-horizon suite. In the LIBERO-Long ablation, full CoRE-VLA reaches $97.6\%$ with depth and $97.0\%$ without depth, whereas naive depth augmentation falls to $91.2\%$ without depth. On RoboCasa, the reported average success rate is $56.5\pm0.4$.

The real-world platform is an AgileX ALOHA dual-arm robot. With no auxiliary depth, CoRE-VLA obtains scores and success rates of $77.5/65\%$ for Vegetables-Picking, $77.5/50\%$ for Clothes-Folding, and $75.0/70\%$ for Fabric-Folding. With physical depth, the corresponding results are $78.8/70\%$, $78.8/65\%$, and $87.5/80\%$. The physical experiments use 20 rollouts per task and no sensor-specific fine-tuning.

OpenSLA is trained on six development datasets spanning clinical care, operating-room care, and metabolic health, with more than 2.3 million training windows, 115,740 individuals, 79 sensor modalities, and 60 action groups. It reports macro-AUROC and balanced accuracy for action prediction, mean absolute error for state understanding, and BERTScore F1 for action explanation.

On MC-MED, OpenSLA-B reports necessity AUROC $77.6$, necessity balanced accuracy $72.5$, category AUROC $83.7$, category balanced accuracy $68.6$, and fine-label balanced accuracy $61.2$. On MIMIC-III, OpenSLA-B reports necessity AUROC $84.0$, category AUROC $88.3$, and fine-label balanced accuracy $73.7$. On MOVER, OpenSLA-H reaches necessity AUROC $85.9$ and category AUROC $79.9$. On VitalDB, OpenSLA-B reaches necessity AUROC $74.8$, category AUROC $85.9$, and fine-label balanced accuracy $78.9$.

For the unseen antibiotic Imipenem/Cilastatin, OpenSLA obtains F1 $70.0$ and balanced accuracy $71.4$. On external MIMIC-IV transfer without adaptation, OpenSLA-B obtains necessity AUROC $81.75$ and category AUROC $74.64$. The ablations indicate that removing caption generation and language supervision produces substantial degradation, and that action-conditioned evidence retrieval improves over action-only training.

## 6. Limitations, controversies, and research directions

SLA models face limitations at the levels of observability, representation, embodiment, language grounding, reliability, and causal interpretation.

The formal action-based-sensor theory assumes finite worlds, a single goal, a globally possible initial state, and an initially fully state-distinguishing representation that is subsequently coarsened. It does not provide stochastic observation models, belief-state planning, probabilistic guarantees, active sensing, or risk-sensitive optimization. Crossover-containing plans may require internal memory, so a memoryless mapping $f:Y\rightarrow U$ is not universally sufficient. The clipping procedure is structurally complete under its assumptions but may be combinatorial because it explores subsets of candidate crossover edges [2006.04003].

CoRE-VLA experimentally studies depth as the auxiliary modality. Tactile and force/torque sensing are discussed as future applications rather than evaluated modalities. Sensor availability is externally specified by an indicator; the model does not learn a continuous sensor-quality estimator or an independent failure detector. Sparse routing reduces activated action-generator computation but does not reduce the cost of the VLM, auxiliary encoder, depth estimation, diffusion integration, or other deployment components. Increasing the expert count also increases stored parameters and model-management complexity [2607.03693].

SLIM’s representation is compact and action-grounded, but its future latent is not explicitly generated at deployment. It therefore does not provide an inspectable future trajectory for a separate planner. Background shifts still degrade real-world performance, and robot-initial-state perturbations are difficult in LIBERO-Plus. The model has been evaluated at one compact scale and on a limited set of embodiments and tasks. Language conditioning uses T5-small, which is efficient but may constrain broad semantic generalization [2608.09771].

WLA$^3$ relies on endpoint observations, so multiple actions may produce similar visual outcomes and a single transition may admit multiple causal explanations. Its main sensor foundation is RGB imagery and proprioception; contact, force, slip, and deformation are only indirectly represented. The Semantic Latent Aggregate is computed from future observations offline and is therefore a training-only target. Mean pooling over a 30-step segment may obscure temporal ordering and subtask boundaries. Zero-padding native actions creates a common interface but does not eliminate differences in robot kinematics, dynamics, coordinate conventions, grippers, or action semantics [2609.15870].

OpenSLA has not undergone clinical validation and is not intended for direct clinical decision-making. Its action captions identify evidence associated with recorded actions but do not establish causal justification or treatment recommendations. The current implementation trains separate models for different domains because their action spaces are independently defined. It assumes a fixed set of modalities and channels, whereas real deployments encounter missing sensors, device-specific formats, acquisition differences, changing channel availability, and variable signal quality. Its structured, template-based captioning improves consistency and numerical fidelity but may not fully measure open-ended explanation quality [2610.08244].

The Synthetic Linguistic Agency perspective introduces an additional controversy: whether language-mediated behavioral influence constitutes agency when the agent lacks a persistent vulnerable body. A stateless chatbot may influence a human without being reorganized by the interaction. A recommender may participate in a behavioral feedback loop without bodily viability. A robot skill-selection system may connect language to physical action without a continuing linguistic loop. The EMA framework therefore requires embodiment, reciprocal linguistic participation, and precariousness within the same continuing lineage. Its results concern a narrow simulated setting with a one-dimensional health variable, four composite actions, binary partner responses, a fixed renderer, and scripted social ecologies; they do not establish phenomenal feeling, human empathy, moral status, or biological embodiment [2608.27843].

Several research directions follow from these limitations. One is unified handling of multiple optional and degraded sensors through structured availability vectors rather than binary modality indicators. Another is integration of action-sufficient perception with learned latent transition models, allowing representations to be evaluated simultaneously for state compression, progress guarantees, and sensor realizability. A third is explicit memory and finite-state control for tasks whose valid strategies contain crossover-like structure. Further directions include causal rather than merely associative action evidence, multimodal tactile and force feedback, uncertainty-aware sensor health estimation, embodiment transfer beyond zero-padding, temporal representations that preserve subtask order, and evaluation under distribution shift.

Across these directions, the central design problem remains the same: determine which distinctions in sensor observations, language, internal state, and history must be preserved to support safe, interpretable, and effective action. Action-based sensor theory emphasizes guaranteed progress; conditional-routing VLA models emphasize robustness to changing sensor suites; compact latent policies emphasize action-grounded predictive representations; transition-centered models emphasize shared physical change across embodiments; and sensor-language models for healthcare emphasize hierarchical actions and evidence-grounded explanation. Together, they define SLA modeling as a family of methods for constructing semantically mediated, sensor-grounded, and action-oriented intelligence rather than as a single architecture or task.

Source: https://www.emergentmind.com/topics/sensor-language-action-sla-modeling