Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfer

Abstract: Health foundation models (FMs) learn useful representations from wearable sensors, but interpreting what they encode and transferring that knowledge across modalities after training remains difficult. We present a post-training framework that decomposes frozen embeddings into interpretable directions, referred to as symbols, and use these symbols to align the embedding spaces without retraining. We evaluate the framework on three FMs for photoplethysmography (PPG) and accelerometer data, independently pretrained on ~20M minutes of unlabeled data from ~172K participants, and analyzed on a held-out cohort of 30K subjects. We find that extracted symbols associate selectively with health conditions and physiological attributes, and these associations are partially shared across modalities and architectures. Cross-modal transfer via symbols retains more than 95% of in-domain performance, is nearly symmetric across domain directions, and saturates with limited paired data, together indicating that alignment recovers a shared low-dimensional subspace rich in physiological information. Overall, these results suggest that health FM embeddings contain an interpretable symbolic organization that is shared across modalities and supports cross-domain transfer without joint training.

• The paper introduces a post-training framework that extracts symbolic representations using PCA, ICA, and NMF from health sensor embeddings.
• It employs bijective assignment and sparse CCA to align symbols across modalities, achieving high transfer performance with AUC retention up to 98.7%.
• The study confirms that emergent symbols are selectively grounded in health indicators, supporting neurosymbolic objectives in clinical applications.

Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfer

Overview and Motivation

Health foundation models (FMs) trained on wearable biosignals, such as photoplethysmography (PPG) and accelerometry, encode rich representations that correlate with physiological and clinical attributes. However, these deep models lack explicit symbolic structuring and interpretability crucial for clinical scenarios and neurosymbolic AI objectives. The paper proposes a rigorous post-training framework to decompose frozen FM embeddings into interpretable directions—denoted as "symbols"—and demonstrates that these symbols facilitate cross-modal alignment and transfer without joint retraining (2605.07407).

The approach leverages theoretical underpinnings such as the Linear Representation Hypothesis (LRH) and the Platonic Representation Hypothesis, which posit that neural networks, regardless of architecture or modality, converge toward similar latent structures. The work empirically tests whether independently trained health FMs admit symbol-like decompositions consistent across modalities, whether these symbolic factors demonstrate selective, interpretable health associations, and whether they enable efficient cross-modal knowledge transfer.

Symbol Extraction and Alignment Framework

The pipeline utilizes three separately trained FMs: PPG~ViT, PPG~EfficientNet, and Accel~ViT, each learned from approximately 20 million minutes of sensor data collected from $\sim$172,000 participants in the Apple Heart and Movement Study. To standardize subject-level analysis, embeddings are temporally averaged per participant.

Symbol Extraction

Linear projections—including Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Non-negative Matrix Factorization (NMF)—are applied to frozen embeddings, yielding candidate symbol spaces. These methods offer distinct inductive biases: ICA promotes independence and sparsity, NMF produces parts-based additive factors, and PCA identifies orthogonal axes of maximal variance.

Symbol Alignment

The alignment step employs:

• Bijective Assignment: Hungarian algorithm based one-to-one matching for maximal interpretability, preserving symbol identity.
• Canonical Correlation Analysis (CCA): Linear projection to a shared canonical space, optimizing cross-domain correlation and enabling flexible multi-dimensional mapping. Sparse CCA introduces interpretability constraints via L1 regularization.

Cross-modal alignment metrics are computed for each combination of extractor and alignment method.

Figure 1: Cross-modal symbol alignment analysis between PPG~EfficientNet and Accel~ViT, illustrating correspondence in raw embeddings, concentrated peaks via ICA, broader alignment via NMF, and strong shared linear structure recovered by CCA.

Emergence and Meaningfulness of Symbols

Semantic Grounding and Selectivity

Symbol-target association is quantified using the Wasserstein-1 (W1) distance across 23 binary health and physiological targets. The Selectivity Index (SI), a product of Hoyer sparsity and the L1 norm of the condition profile, captures the quality of symbol-target associations—measuring both peak sharpness and magnitude.

ICA-extracted symbols exhibit high selectivity, activating for specific health conditions while negative controls (e.g., blood type) remain silent. Embeddings and extracted symbols show consistent, modality-shared profiles for cardiovascular and physiological targets, while less direct associations (hepatorenal, mental health) show weaker cross-modal correspondence.

Strong cross-modal cosine similarity (median 0.89 with ICA, 0.84 with raw embeddings) confirms the genuineness of shared physiological structure and that symbol extraction sharpens, but does not manufacture, meaningful alignment.

Cross-Domain Symbolic Transfer

Transfer Efficacy

Symbolic transfer is implemented via logistic regression classifiers trained in source symbol spaces and evaluated after alignment in target spaces. CCA-based alignments achieve best transfer performance:

• Physiological marker transfer: AUC 0.88–1.00, $\approx$100% retention.
• Health condition transfer: AUC 0.71–0.83, $98.7\% \pm 1.6\%$ retention.
• Negative control (blood type): AUC $\approx 0.56$, near chance.

Raw embeddings without symbol extraction and alignment yield near-random transfer, and demographic confounder baselines are consistently outperformed, demonstrating non-trivial learned physiological structure.

Figure 2: Symbolic transfer analysis showing $>$98\% AUC retention after cross-modal alignment, high symmetry across domain directions, and rapid retention plateau with limited paired data, indicating a low-dimensional shared subspace.

Symmetry and Efficiency

Transfer performance is nearly symmetric ($r=0.995$, mean $|\Delta\text{AUC}|=0.009$), suggesting approximate isomorphisms between independently trained symbol spaces. Transfer retention saturates rapidly with respect to paired participant overlap, reaching $>95\%$ by $\sim$7.5K paired samples, implying low intrinsic dimensionality and substantial practical utility for real-world deployment with limited paired datasets.

Practical and Theoretical Implications

The results show that frozen health FM embeddings already contain emergent symbol-like structure, accessible via post hoc linear projections, and partially shared across modalities. The existence of a physiologically meaningful, low-dimensional shared subspace confirms theoretical predictions (Platonic Representation Hypothesis) regarding inter-modal representational convergence. Sparse CCA demonstrates that most of the shared information is captured by a small set of directions, enabling interpretable, data-efficient transfer.

Practically, the symbolic layer enables modular cross-modal reasoning, efficient unlearning of targeted biases, and interpretability transfer from opaque sensor modalities (e.g., accelerometry) to more physiologically tangible domains (e.g., PPG or ECG). Causally linking symbols to FM predictions, expanding analyses to nonlinear extractors, and exploring population-stratified meaning are pertinent future directions.

Limitations include the linearity of extraction methods (potentially missing nonlinear shared structure) and the shared training population across modalities (possible confounding via common demographics, although negative controls and confounder baselines argue against spurious alignment). Extension to independent populations would further validate physiological convergence.

Conclusion

The paper establishes that health FM embeddings admit interpretable, symbol-like decompositions that are partially shared across modalities. These symbols are selectively grounded in health conditions, enable highly efficient cross-modal transfer with limited paired supervision, and support practical neurosymbolic objectives without retraining. This paradigm opens pathways for the structured, interpretable integration and deployment of health FMs in clinical and wearable sensor contexts.