Tongue Image Segmentation Model (TOM)

• TOM is a computational model designed for delineating tongue regions in various medical images, aiding both clinical diagnosis and research.
• It employs modality-specific architectures—such as U-Net and transformer-based SAM—with techniques like multi-task learning and knowledge distillation.
• TOM frameworks optimize preprocessing, augmentation, and loss functions to achieve precise segmentation across imaging types like RGB, ultrasound, MRI, and CT.

A Tongue Image Segmentation Model (often abbreviated as TOM) refers to any computational model designed for the automatic delineation of tongue regions within medical or clinical-grade images. TOMs support both foundational research and clinical applications in areas including speech production, disease diagnosis, and, most prominently, traditional Chinese medicine (TCM) tongue diagnosis. Modern TOMs span classic convolutional neural networks (CNNs), advanced transformer-based architectures leveraging prompt-based segmentation, multi-task pipelines incorporating classification, and domain-adaptive or zero-shot segmentation frameworks. Distinct imaging modalities—RGB photography, ultrasound, CT, and MRI—necessitate tailored TOM architectures, loss functions, and evaluation protocols.

1. Model Architectures and Core Methodologies

TOM design is modality- and task-dependent, but prevailing architectures derive primarily from fully convolutional networks, with the U-Net structure being foundational. For planar RGB tongue images, deep encoder–decoder networks perform pixelwise semantic segmentation, often integrating spatial priors or attention mechanisms.

Notable TOM frameworks and their architectural highlights include:

Model/Approach	Backbone/Key Blocks	Salient Features/Strategies
BGHNet (Zeng et al., 2020)	Lightweight DSS-nbt + HRFFM	Context + boundary hybrid loss
Ammonia-Net (Shi et al., 2023)	U-Net, ShuffleNetV2	Multi-task with class correction
TOM (Distillation) (Xie et al., 19 Aug 2025)	SAM, U-Net, DeepLabV3, TinyViT	Multi-teacher KD, diffusion augmentation
TongueSAM (Cao et al., 2023)	ViT-H/16 (SAM) + YOLOX	Auto-prompt, zero-shot adaptation
Memory-SAM (Chae et al., 17 Oct 2025)	DINOv3 + SAM2 (Hiera-L)	Retrieval-to-prompt, training-free
BowNet (Mozaffari et al., 2019)	Encoder–Decoder + dilation	Parallel multi-scale features

Ultrasound and MRI tongue segmentation demand fine structure preservation with minimal annotated data. U-Net and its derivatives (e.g., Dense U-Net (Zhu et al., 2019), wBowNet (Mozaffari et al., 2019)) are routinely employed, with modifications such as batch normalization, dropout regularization, and domain-adaptive fine-tuning protocols (Mozaffari et al., 2019). TOMs for 4D-CT rely on volumetric nnU-Net architectures, exploiting 3D contextual cues and self-configurable preprocessing (Michiwaki et al., 29 Jan 2025).

2. Data Modalities, Preprocessing, and Augmentation

Editor's term: “Imaging-adaptive TOM” refers to the fact that TOMs adjust design and preprocessing to the characteristics of the input modality.

RGB Photographs

• Datasets typically include several hundred to several thousand images, spanning laboratory-controlled, web-harvested, and in-the-wild smartphone photos (Xie et al., 19 Aug 2025, Cao et al., 2023).
• Preprocessing: resizing/cropping (often to 1024×1024 or 512×512), standard normalization to ImageNet statistics, and, when required, artifact suppression.
• Augmentation: color jitter, random rotations/scales, synthetic images via diffusion models (Xie et al., 19 Aug 2025).

Ultrasound

• Frames downsampled to 128×128, with manual splines or binary contours as ground truth (Zhu et al., 2019, Mozaffari et al., 2019, Mozaffari et al., 2019).
• Augmentation: horizontal flip, narrow-range rotations (±5–15°), zooms; “informed undersampling” for representativeness (Mozaffari et al., 2019).
• Postprocessing: skeletonization and spline smoothing for tongue edge extraction.

MRI and CT

• MRI: center-cropping, histogram equalization, manual contour annotation by consensus across annotators (Isaieva et al., 2024).
• 4D-CT: intensity clipping, z-axis resampling to isotropic spacing, z-score normalization of patches (Michiwaki et al., 29 Jan 2025).

Diffusion-based Augmentation (for robust generalization in photometric conditions):

• Forward and reverse denoising, mask-to-image synthesis with Stable Diffusion, and text-to-image synthesis to expand clinic-relevant appearance diversity (Xie et al., 19 Aug 2025).

3. Loss Functions, Evaluation Metrics, and Objective Design

Loss function selection reflects the severe class imbalance (contour pixels << background) and the need to jointly optimize for region and boundary precision.

Common formulations:

• Binary Cross-Entropy (BCE):

$$L_{\mathrm{BCE}} = -\frac{1}{N}\sum_{i=1}^N [y_i\log p_i + (1-y_i)\log(1-p_i)]$$ Used for binary contour extraction (MRI, ultrasound, boundary maps).

• Categorical Cross-Entropy:

For multi-class tasks (RGB: tongue body, tooth-marks, background), as in Ammonia-Net (Shi et al., 2023).

• Dice Loss/Coefficient:

$\mathrm{Dice}(X,Y) = \frac{2|X\cap Y|}{|X| + |Y|}$ Augments or replaces BCE to directly penalize false negatives/positives (segmentation overlap).

• Compound, Weighted, and Hybrid Losses:
  • Map-level (F1), boundary-level (BF1), pixel (BCE), and patch-level (SSIM) losses in BGHNet (Zeng et al., 2020).
  • Weighted BCE for severe class imbalance (Isaieva et al., 2024, Zhu et al., 2019).
  • Multi-task loss for segmentation + classification (Ammonia-Net):

  $$L_{\text{total}} = L_{\text{seg}} + \lambda_B L_B + \lambda_Q L_Q$$

• Knowledge Distillation Losses:

Logits-based KL divergence, mask-level MSE, plus BCE between student and teacher(s) for parameter-efficient TOMs (Xie et al., 19 Aug 2025).

Evaluation Metrics:

• Mean Intersection-over-Union (mIoU): primary metric for region overlap (both binary and multi-class) (Xie et al., 19 Aug 2025, Cao et al., 2023).

• Mean Pixel Accuracy (mPA), Dice coefficient, and Boundary F1 (BF1): important for evaluating boundary-specific improvements (Zeng et al., 2020).

• Mean Sum of Distances (MSD): for contour-tracing benchmarks in ultrasound/MRI (Isaieva et al., 2024, Zhu et al., 2019, Mozaffari et al., 2019).

4. Innovations in Automated Prompting and Zero-/Few-Shot Segmentation

The advent of large prompt-driven segmentation models (SAM, MedSAM, Hiera-SAM2) has yielded a class of TOMs with novel mechanisms for automatic and robust tongue region localization.

Automatic Prompt Generation and Zero-Shot Transfer:

• TongueSAM (Cao et al., 2023) integrates a YOLOX object detector for box prompt generation, routed to a frozen SAM ViT-H backbone with only Mask Decoder fine-tuning. This enables “zero-shot” adaptation across data domains.

  • mIoU: ~95.23% zero-shot on web-harvested data, significantly outperforming classic DeepLabv3+ (Cao et al., 2023).
• Memory-SAM (Chae et al., 17 Oct 2025) eliminates human-prompt dependency entirely by using DINOv3 ViT-L/16 feature retrieval and mask-constrained foreground/background matching; images are segmented without training or manual input, attaining mIoU = 0.9863 on mixed controlled/wild datasets.

Knowledge Distillation for Model Compression:

• TOM (Xie et al., 19 Aug 2025) introduces a multi-teacher distillation regime: SAM (LoRA-tuned), U-Net, and DeepLabV3 teacher models guide a parameter-efficient TinyViT-based student. The student TOM reduces parameter count by 96.6% compared to the teacher, retaining 95.22% mIoU.

5. Segmenting Specialized Structures and Multi-task Learning

TOMs address both holistic tongue boundary segmentation and the delineation of intra-tongue pathologies (e.g., tooth-marks in TCM). For example:

• Ammonia-Net (Shi et al., 2023): Simultaneous segmentation of tongue body and tooth-marks. Embeds pixel-level outputs back into the classifier pipeline to boost 4-class TCM diagnosis accuracy (F1 = 76.85%; 2-class tooth-marked/non-tooth-marked, accuracy = 99.06%).
• BGHNet (Zeng et al., 2020): Achieves real-time inference speeds (≈30–40 fps) and high mIoU (0.9911) in clinical images using a lightweight design, with explicit architectural pathways and loss terms for precise boundary focus—critically relevant for TCM or any application sensitive to tongue region demarcations.
• nnU-Net for 4D-CT (Michiwaki et al., 29 Jan 2025): Deploys automatic preprocessing and a 3D U-Net “cascade” for organ motion analysis in swallowing studies, with tongue median Dice = 0.85, enabling dynamic tongue/bone/bolus tracking in volumetric data.

6. Failure Modes, Limitations, and Remediation Strategies

Failure modes recurrent in TOM development include:

• Boundary ambiguity due to contact with adjacent tissue or bolus (CT: palate, barium (Michiwaki et al., 29 Jan 2025)).
• Motion blur and rapid movement artifacts (notably in functional MRI/CT or ultrasound) leading to “smearing” and minor under-segmentation.
• Interference from metal artifacts (e.g., dental work).
• In ultrasound/MRI, low contrast and speckle noise limit delineation; solutions are batch normalization, dropout, and careful contour post-processing (Dijkstra path smoothing, skeletonization) (Isaieva et al., 2024, Zhu et al., 2019).
• Supervised models overfit when training-data distribution is narrow; zero-shot and distillation-based TOMs generalize better but can still be sensitive to prompt misalignment (if auto-generated boxes overlap non-tongue regions) (Cao et al., 2023).

Suggested remedies, as identified within the literature:

• Expanding and diversifying annotated datasets, especially for rare or ambiguous presentations.
• Multi-stage segmentation (bones, cartilages as spatial priors).
• Interactive correction tools for efficient manual refinement.
• Incorporation of synthetic augmentation (diffusion modeling) to address domain shift (Xie et al., 19 Aug 2025).

7. Practical Deployment, Tool Availability, and Impact

TOM deployment spans from standalone offline executables to cloud-based batch segmentation services:

• TOM (2025) Tool (Xie et al., 19 Aug 2025): Provides an online web UI and downloadable offline tool requiring no programming. Capable of processing bulk uploads of clinical tongue photographs, exporting both masks and patch-extracted tongue regions for downstream diagnosis.
• Open Source Implementations:
  • CNN-based tracking tool in ultrasound: full pipeline and pre-trained weights publicly available (Zhu et al., 2019).
  • Memory-SAM codebase at https://github.com/jw-chae/memory-sam (Chae et al., 17 Oct 2025).
• Impact on Downstream Diagnosis:
  • Precise segmentation improves explainability and classification accuracy for TCM constitution typing; isolating the tongue surface from background artifacts enhances feature learning and clinician interpretability (Xie et al., 19 Aug 2025).
• Real-Time and Clinical Relevance:

Lightweight and high-speed TOMs (e.g., BGHNet) occupy a crucial niche in point-of-care settings, balancing region accuracy, edge fidelity, and computational cost to support clinical workflows (Zeng et al., 2020).

Tongue Image Segmentation Models represent a technologically diverse, rapidly maturing field, with cross-fertilization between classical dense architectures, large prompt-based models, and diffusion-driven augmentation pipelines. Advances in data efficiency, generalization, and tool usability underpin their increasing role in both research and applied clinical diagnostics.