OpenSLA: Sensor-Language-Action Model

• OpenSLA is a unified Sensor-Language-Action (SLA) model used for connecting multimodal sensor observations with natural language context and heterogeneous actions.
• OpenSLA benefits health and medical research by supporting hierarchical action prediction, in which the necessity, category, and fine-grained label of an action are predicted from sensor and context data.
• Formalized with action-conditioned evidence retrieval, OpenSLA can produce explanations grounded in sensor evidence, offering clinical insights by linking recorded actions to supporting sensor data.

OpenSLA is a unified Sensor-Language-Action (SLA) model that connects multimodal sensor observations, natural-language context, heterogeneous actions, and action-relevant explanations. It is introduced as an implementation of SLA modeling, a framework intended to extend sensor intelligence beyond perception and state recognition toward action prediction and explanation. OpenSLA supports hierarchical action prediction, sensor-state understanding through generated language, action-evidence explanation, language-guided evidence grounding, and zero-shot transfer to an unseen fine-grained action and an external clinical cohort (Xu et al., 6 Oct 2026).

1. Conceptual foundations and scope

SLA modeling addresses the limitations of conventional sensor models, which generally learn mappings from sensor signals to fixed state labels or task-specific outcomes. Such systems typically use closed action vocabularies, separate perception from decision-making, have limited mechanisms for incorporating user or patient context, and provide limited connections between predicted actions and the sensor evidence supporting them.

OpenSLA instead treats language as a semantic interface between sensing and acting:

$$\text{multimodal sensor observations} \;\longleftrightarrow\; \text{context and physiological meaning} \;\longleftrightarrow\; \text{heterogeneous actions}.$$

An SLA sample is anchored at a decision time $T_0$ and contains sensor observations preceding $T_0$, individual or user context, an action occurring at or after $T_0$, and language descriptions of sensor state and action-relevant evidence. A generic representation is:

$$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$

where $\mathbf X_i$ denotes multimodal sensor observations, $\mathbf C_i$ denotes textual or structured context, $\mathbf y_i$ denotes hierarchical action targets, and $\mathbf T_i$ denotes caption or language supervision.

The action space is hierarchical:

1. Action necessity: whether an intervention or action is needed.
2. Action category: the type of action required.
3. Fine-grained label: the specific medication, procedure, or action.

For example, the action Imipenem can be represented as:

$$\text{Imipenem} \Rightarrow (\text{Yes},\text{antibiotics},\text{Imipenem}).$$

For continuous glucose monitoring (CGM), the hierarchy differs: necessity indicates whether an insulin bolus occurs, the category is a dose bin, and there is no separate fine-grained label vocabulary.

OpenSLA therefore supports both sensor-to-action inference and action-conditioned explanation:

$T_0$0

and

$T_0$1

The language component is not limited to rendering predictions in textual form. It provides semantic supervision for local signal characteristics, cross-channel relationships, user-specific context, action semantics, and action-relevant evidence.

2. Benchmark, domains, and data construction

The OpenSLA benchmark covers clinical care, operating-room care, and metabolic health. It contains seven datasets: MC-MED, MIMIC-III, MIMIC-IV, MOVER, VitalDB, MetaboNet, and PEDAP. Six datasets are used for development, while MIMIC-IV is withheld as an external clinical cohort.

The benchmark comprises 115,740 individuals in the detailed cohort table, approximately 116,000 individuals in the headline description, 79 sensor modalities, and 60 action groups. The development data contain more than 2.3 million training windows.

Dataset	Domain	Individuals
MC-MED	Clinical	49,196
MIMIC-III	Clinical	35,276
MIMIC-IV	External clinical	122
MOVER	Operating room	26,329
VitalDB	Operating room	3,657
MetaboNet	CGM	1,095
PEDAP	CGM	65

The evaluation windows are individual-disjoint. Test windows are balanced with respect to action necessity by sampling action-positive and action-negative windows. The external MIMIC-IV evaluation uses 1,000 windows, balanced into 500 positive and 500 negative examples, from 87 individuals.

Input modalities include waveform and numerical measurements. Waveforms comprise ECG, plethysmography, respiration, arterial and other invasive pressure, capnography, airway pressure, EEG, and glucose traces. Numerical measurements include heart rate, respiratory rate, oxygen saturation, blood pressure, temperature, cardiac output, hemodynamic measures, ventilator measurements, oxygen delivery, laboratory values, neurological and pain scores, urine output, historical basal insulin, and related CGM events.

Input-history lengths are domain-specific:

• Clinical care: 30-minute sensor history.
• Operating-room care: 5-minute sensor history.
• CGM: 2-hour history represented as 24 five-minute slots.

Action targets are defined after or at $T_0$2. For MC-MED and MIMIC-III, the target action occurs in $T_0$3. For MOVER and VitalDB, the target action or event occurs in approximately the following two minutes. For CGM, the target is bolus onset at the decision slot.

The benchmark contains 39 clinical action categories, 17 operating-room categories, and four CGM dose categories, producing 60 domain-specific action groups. CGM dose bins are:

$T_0$4

Clinical and operating-room samples may contain multiple category and fine-grained targets within a target window.

3. Captioning and language supervision

OpenSLA uses a structured, mostly rule-based captioning pipeline to create language supervision. The captions are not free-form clinical notes and are not presented as causal explanations. They associate pre-decision observations with recorded actions while preserving measurement identity, numerical summaries, temporal information, source modality, signal quality, context, and action relation.

The pipeline creates four caption facets.

Observational captions

Observational captions describe channel-level properties. For numerical measurements, they can include the median, range, number of observations, state descriptor, and latest value. For waveforms, they can include rate, periodicity, amplitude, morphology or pattern, and signal-quality qualification. CGM captions can describe glucose level, range, slope, latest reading, variability, and time-in-range information.

Relational captions

Relational captions describe temporal trends, persistence or stability, cross-channel agreement, relationships between numerical measures and waveforms, and prior-event timing. The pipeline uses predefined physiologically related channel pairs rather than unrestricted correlation discovery.

Global captions

Global captions combine local findings with individual-level context, including background information, presentation, prior history, prior interventions, patient attributes, and overall physiological state. Their purpose is to place local sensor observations within a broader decision context.

Action captions

Action captions select pre-decision evidence associated with a recorded action. A finding may be described as supporting an action, providing indirect or incomplete evidence, serving as context only, lacking a label-specific rationale, providing evidence against a CGM bolus, or providing no immediate treatment evidence.

The paper explicitly characterizes action captions as evidence associations rather than causal explanations or treatment recommendations. A recorded treatment may depend on information unavailable in the sensor window. Consequently, captions use qualification language such as “indirect,” “possible,” “context-specific,” or “not explicitly documented.”

This distinction is important for interpreting OpenSLA explanations. High similarity between a generated caption and a reference caption does not establish causal faithfulness, clinical correctness, or treatment appropriateness.

4. Model architecture

OpenSLA consists of a pretrained language-model backbone with structured action heads, a hierarchical sensor encoder, and a multimodal fusion decoder for action-conditioned, sensor-grounded captioning. Two principal variants are defined:

• OpenSLA-B: the base configuration.
• OpenSLA-H: the hierarchical-memory configuration.

Both use a Qwen-3.5-2B language backbone adapted with LoRA.

Sensor encoding

Clinical and operating-room waveforms are processed with a frozen pretrained DINO-style signal encoder adapted from OSF. Each channel is independently encoded. Because CGM histories are shorter, CGM uses a trainable convolutional patchifier instead of the larger waveform encoder.

Numerical observations are processed by a trainable Transformer. Each numerical representation combines a value embedding, measurement-identity embedding, and temporal embedding. This allows identical numerical values from different measurements, such as heart rate and temperature, to remain semantically distinct.

The initial sensor representation is:

$T_0$5

where $T_0$6 is the number of initial sensor tokens and $T_0$7 is the feature dimension.

Hierarchical Memory

OpenSLA-H compresses long sensor histories at multiple resolutions. Starting with $T_0$8, it applies $T_0$9 query layers:

$T_0$0

where $T_0$1 contains $T_0$2 learned queries, $T_0$3, and $T_0$4.

The hierarchical representation includes local-memory, temporal-memory, and global-memory tokens. Local queries aggregate features within channel-time groups. Temporal queries aggregate local memories across time for a channel. Query parameters are shared across groups or channels, channel embeddings are added at both stages, and time embeddings are added to local queries. Numerical observations are grouped by measurement identity and relative timestamp, processed through the query hierarchy, and appended to the numerical memory.

Domain	Tokens before $T_0$5 after	Compression	Speedup
Clinical	5801.0 $T_0$6 400.4	$T_0$7	$T_0$8
Operating room	3915.8 $T_0$9 838.0	$T_0$0	$T_0$1
CGM	33.0 $T_0$2 21.0	$T_0$3	approximately $T_0$4

Hierarchical memory is consequently most useful for long clinical and operating-room sequences; CGM histories are already short.

Language backbone and fusion

Compressed sensor tokens are supplied to the language backbone. The paper compares LLaVA-style projection or adapter training, Flamingo-style cross-attention injection, and direct LoRA adaptation of the LLM. Direct LoRA performs best in the reported operating-room ablation.

The backbone hidden state at the final input position is:

$T_0$5

OpenSLA uses LoRA rank 16, LoRA $T_0$6, and LoRA dropout $T_0$7. The pretrained signal encoders and other specified pretrained modules remain frozen.

5. Action prediction and grounded explanation

Hierarchical action heads

Necessity prediction uses a binary classifier:

$T_0$8

Category prediction uses independent binary classifiers:

$T_0$9

Independent category heads allow multiple simultaneous categories in clinical and operating-room examples. Category loss uses positive-class-weighted binary cross-entropy because of class imbalance.

Fine-grained actions are represented through textual descriptions rather than fixed classifier weights. If $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$0 is the language embedding of candidate action $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$1, the model projects the decision representation and action embedding into a shared space:

$$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$2

The action score is cosine similarity:

$$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$3

Within category $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$4, the predicted action is:

$$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$5

This formulation permits an action absent from supervised classifier training to be introduced through its textual descriptor.

Action-conditioned evidence retrieval

The action profile $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$6 contains an action-necessity token and action-category tokens. Each token combines an action embedding with its predicted score. Given language hidden states $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$7 and the pooled action profile $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$8, OpenSLA retrieves relevant waveform and numerical sensor tokens, denoted by $$\mathcal{D} = \left\{ \left( \mathbf X_i,\mathbf C_i,\mathbf y_i,\mathbf T_i \right) \right\}_{i=1}^{M},$$9.

The caption representation is refined through a gated residual update:

$\mathbf X_i$0

where $\mathbf X_i$1 is a cross-attention branch over predicted actions, $\mathbf X_i$2 is a cross-attention branch over retrieved sensor evidence, and $\mathbf X_i$3 and $\mathbf X_i$4 are learnable gates. Each branch contains cross-attention, a feed-forward block, and an output projection. The refined representation is passed through the original language-model head for autoregressive caption generation.

This decoder explicitly conditions language generation on both the predicted action and the sensor evidence supporting it.

Training objective

The total objective is:

$\mathbf X_i$5

where $\mathbf X_i$6 is the hierarchical action-prediction loss and $\mathbf X_i$7 is autoregressive caption-generation loss.

The action loss is:

$\mathbf X_i$8

The caption loss is autoregressive cross-entropy:

$\mathbf X_i$9

Signal-related caption tokens receive higher weights in the implementation. Training uses teacher-forced ground-truth caption prefixes but predicted actions for action-conditioned decoding.

Optimization uses AdamW with learning rate $\mathbf C_i$0, weight decay $\mathbf C_i$1, 10% warmup, linear decay, gradient-clipping norm $\mathbf C_i$2, effective batch size 16, and a 24-hour training cap on H200 GPU nodes.

6. Evaluation, capabilities, and limitations

OpenSLA is evaluated against supervised models, time-series foundation models, sensor-LLMs, and zero-shot multimodal LLMs. Metrics include macro-AUROC, balanced accuracy, fine-grained balanced accuracy, caption-derived mean absolute error, and BERTScore F1 for action explanations. The reported uncertainty intervals are half-widths of 95% percentile intervals obtained from 2,000 individual-level cluster bootstrap resamples.

Action prediction

For MC-MED, OpenSLA-H obtains necessity AUROC 77.0, necessity balanced accuracy 72.2, category AUROC 83.7, category balanced accuracy 68.7, and fine-label balanced accuracy 65.2. For MIMIC-III, it obtains necessity AUROC 81.9, necessity balanced accuracy 73.3, category AUROC 88.1, category balanced accuracy 74.5, and fine-label balanced accuracy 76.5.

For MOVER, OpenSLA-H obtains necessity AUROC 85.9, necessity balanced accuracy 80.6, category AUROC 79.9, category balanced accuracy 69.9, and fine-label balanced accuracy 67.6. For VitalDB, it obtains necessity AUROC 77.1, necessity balanced accuracy 66.3, category AUROC 86.3, category balanced accuracy 80.8, and fine-label balanced accuracy 78.5.

For CGM, OpenSLA-B obtains necessity AUROC 73.3 and category AUROC 66.5 on MetaboNet; OpenSLA-H obtains necessity AUROC 71.9 and category AUROC 67.2. On PEDAP, OpenSLA-B obtains necessity AUROC 81.2 and category AUROC 65.2, while OpenSLA-H obtains necessity AUROC 80.9 and category AUROC 59.7.

CGM category prediction is difficult because insulin decisions may depend on future carbohydrate intake unavailable in the observed history. This represents an information limitation rather than solely a modeling limitation.

State understanding and explanations

OpenSLA generates captions from which physiological quantities are extracted. Representative OpenSLA-H mean absolute errors include cardiac index 1.01, ECG-II rate 15.9, respiratory rate 1.17 in the clinical domain, respiratory rate 1.41 in the operating-room domain, temperature 0.47, ETCO$\mathbf C_i$3 3.18, basal rate 0.35, and carbohydrate amount 0.38.

BERTScore F1 values for OpenSLA-B and OpenSLA-H include 81.5 and 82.0 on MC-MED, 78.6 and 80.9 on MIMIC-III, 81.5 and 81.0 on MOVER, 80.9 and 80.7 on VitalDB, 85.6 and 80.4 on MetaboNet, and 87.6 and 76.7 on PEDAP.

These results indicate that language supervision preserves physiological information and supports action-evidence descriptions. They do not establish that generated explanations are causally valid, clinically sufficient, or suitable for autonomous decision-making.

Ablations and grounding

Removing caption supervision produces the largest degradation in the reported ablations. On MOVER, the action-only model obtains necessity AUROC 78.1 versus 83.4 for OpenSLA-B and category AUROC 72.1 versus 80.3. On VitalDB, the corresponding necessity AUROC values are 54.1 versus 74.8 and category AUROC values are 72.8 versus 85.9.

A context-only model performs worse than the full sensor-plus-context model. On MOVER, necessity AUROC is 79.8 for context-only versus 83.4 for OpenSLA, while category AUROC is 75.7 versus 80.3. On VitalDB, necessity AUROC is 64.9 versus 74.8 and category AUROC is 83.4 versus 85.9.

A qualitative controlled-substitution experiment holds patient context fixed while replacing the sensor recording with a higher-ABP example. The generated caption changes accordingly, including a reported pleth-rate proxy change from approximately 76.1 to 82.1 per minute. This demonstrates input sensitivity, but not causal faithfulness in the stronger sense.

Zero-shot action and cohort transfer

Imipenem/Cilastatin is held out from supervised fine-grained action targets while the broader antibiotic category remains present during training. OpenSLA represents the unseen action through its textual description and obtains Imipenem F1 70.0 and balanced accuracy 71.4, compared with lower results for OpenTSLM, GPT-5.6-Luna, and DeepSeek-V3.2.

On the unseen MIMIC-IV cohort, OpenSLA-B obtains necessity AUROC 81.75, necessity balanced accuracy 74.40, category AUROC 74.64, and category balanced accuracy 56.35. OpenSLA-H obtains necessity AUROC 80.52, necessity balanced accuracy 73.00, category AUROC 74.90, and category balanced accuracy 58.49.

These experiments support transfer to an unseen but related clinical cohort and semantic generalization to one held-out action. They do not establish generalization to unseen action categories, entirely new domains, novel action modalities, or actions without semantically related training examples.

Limitations

OpenSLA has not undergone prospective clinical validation, safety evaluation, deployment testing, or regulatory review. The paper states that it is not intended for direct clinical decision-making.

The current system trains separate models by domain because action spaces are defined independently. It is therefore not a single generalist agent operating across clinical care, operating-room control, and metabolic health.

The implementation assumes a fixed set of expected channels and modalities. Missing sensors, altered sampling rates, different devices, acquisition artifacts, and changing channel availability remain challenges.

The captioning pipeline depends on predefined extraction rules, templates, and action mappings. Captions may omit unavailable information, associate evidence with an action without proving causality, inherit signal-quality errors, or use indirect evidence when no explicit rationale is recorded.

Recorded interventions may reflect unobserved factors, and short target windows may not capture longer-term planning. BERTScore evaluates semantic similarity to reference captions rather than factual correctness, causal validity, or clinician usefulness.

OpenSLA also requires substantial sensor harmonization, signal-quality filtering, waveform feature extraction, context processing, action mapping, and caption construction. Although hierarchical memory reduces the cost of long sequences, the system still requires a large language backbone, domain-specific encoders, and extensive paired sensor-language-action supervision.

OpenSLA’s principal contribution is a unified representational and modeling framework in which sensing, contextual interpretation, action prediction, and evidence explanation are trained together:

$\mathbf C_i$4

The reported experiments support this framework in controlled clinical, operating-room, and metabolic-health benchmarks. They do not yet establish safe autonomous action selection. Further work is required on cross-domain generalist modeling, missing and changing modalities, prospective validation, causal and factual grounding, robustness, and deployment safeguards.