Spinal Keypoint Prompting in AIS Imaging

• Spinal keypoint prompting is a method that overlays structured anatomical markers on X-rays to guide multimodal LLMs in detecting spinal deformities in AIS.
• It employs distinct visual prompts—CSM, VCL, and SVM—to encode spine geometry, enhancing diagnosis, regional localization, and direction detection.
• Integrated in a Divide and Conquer framework, the approach acts as a diagnostic scaffold to reveal model limitations and improve clinical assessments.

Searching arXiv for the cited papers and closely related prompt-based keypoint/medical imaging work to ground the article. Spinal keypoint prompting is a visual prompting strategy for multimodal LLMs (MLLMs) in adolescent idiopathic scoliosis (AIS) self-management, introduced in "Adapting and Evaluating Multimodal LLMs for Adolescent Idiopathic Scoliosis Self-Management: A Divide and Conquer Framework" (Wu et al., 15 Sep 2025). It provides extra anatomical visual cues derived from a spine keypoint detection model by overlaying structured spine landmarks or derived lines onto anteroposterior X-rays, with the aim of improving reasoning about curve location and direction without retraining the MLLM end-to-end. In that study, spinal keypoint prompting is the visual adaptation mechanism inside a broader Divide and Conquer framework, motivated by the finding that current MLLMs are far from capable in realizing personalized assistant in AIS care, with the greatest challenge lying in accurate detection of spinal deformity locations, where the best accuracy is 0.55, and directions, where the best accuracy is 0.13 (Wu et al., 15 Sep 2025).

1. Definition and representational forms

In the terminology of the AIS study, spinal keypoint prompting consists of visual prompting strategies supplied by a spinal keypoint detection system. The prompts are intended to encode spinal geometry that the base MLLM may otherwise miss when inspecting raw AP spine radiographs (Wu et al., 15 Sep 2025).

Prompt type	Construction	Intended cue
CSM	Curved Spine Midline	A curved centerline following the spinal curve
VCL	Vertebral Connection Line	Lines connecting vertebral points, emphasizing alignment across vertebrae
SVM	Segmented Vertebrae Marks	Marks placed on segmented vertebral locations

These prompt forms are not language-side instructions. They are visual overlays placed on the original X-ray. For region-localization tasks, the prompt also includes color coding for thoracic, thoracolumbar, and lumbar regions. The resulting input is therefore a visually enriched radiograph that injects geometry and localization into the MLLM’s visual stream (Wu et al., 15 Sep 2025).

The paper’s formulation is operational rather than formal. It does not provide explicit mathematical equations for spinal keypoint prompting, nor an explicit prompt template in LaTeX. Its central claim is instead representational: structured anatomical landmarks or derived lines can act as a visual adapter for MLLMs when the underlying task depends on fine-grained spinal anatomy.

2. Construction pipeline and system integration

The construction workflow has four stages. First, a spine keypoint detection model is run on the AP X-ray. Second, its output is converted into one of the three visual prompt types, CSM, VCL, or SVM. Third, for spinal deformity location detection, the prompt may be color coded by anatomical region. Fourth, the prompt-augmented image is passed to the MLLM together with a task-specific question concerning diagnosis, deformity location, or deformity direction (Wu et al., 15 Sep 2025).

The keypoint detector used in the study is SpineHRNet+, described as a spine-specific keypoint model based on HRNet and UNet architectures. The MLLM side includes five models: Qwen2.5-VL-7B, InternVL3-8B, InternVL3-14B, Llama 3.2-Vision, and Phi-3-Vision. Reported implementation details are 4 × RTX 3090 GPUs, temperature = 0, bfloat16 quantization, and flash attention disabled (Wu et al., 15 Sep 2025).

A recurring misconception is to treat spinal keypoint prompting as model retraining. In the paper, it is explicitly used as a visual adapter, not as a language-side change. This distinction matters because the method probes whether performance limitations arise from missing anatomical cues rather than from the absence of additional domain text or end-to-end optimization. The study therefore uses prompting as a targeted intervention on the visual understanding bottleneck.

3. Position within the Divide and Conquer framework

Spinal keypoint prompting is one component of a broader Divide and Conquer framework for AIS self-management. Its specific locus is the Visual Spinal Assessment (VSA) branch, which is organized as three sequential tasks: AIS Diagnosis (AD), Spinal Deformity Location Detection (SDLD), and Spinal Deformity Direction Detection (SDDD). AD is binary classification for AIS present or absent. SDLD is multi-class classification over thoracic, thoracolumbar, and lumbar. SDDD is multi-class classification over leftward curvature and rightward curvature (Wu et al., 15 Sep 2025).

The paper emphasizes that these VSA tasks reflect how clinicians actually inspect spinal X-rays: diagnose, localize, then determine direction. This clinical ordering is important because the value of prompting depends on which subproblem is being addressed. Diagnosis requires coarse discriminative recognition; location detection requires region discrimination; direction detection requires a more demanding synthesis of spatial geometry.

The framework also contains two non-visual branches. Domain Knowledge Assessment (DKA) uses multiple-choice questions spanning six categories: basic knowledge; etiology and pathophysiology; clinical presentation and diagnosis; assessment and monitoring; treatment options; and complications and prognosis. Patient Education and Counseling Assessment (PECA) uses open-ended patient-facing question answering on disease explanation, treatment options, daily life management, long-term prognosis, and follow-up and monitoring, stratified by mild, moderate, and severe cases. In this design, spinal keypoint prompting is reserved for visual understanding, whereas retrieval augmented generation is used for DKA and PECA (Wu et al., 15 Sep 2025).

4. Quantitative effects on diagnosis, localization, and direction detection

On AIS diagnosis, spinal keypoint prompting changes performance substantially for some architectures and negligibly for others. Qwen2.5-VL-7B improves from a baseline of F1 0.83 and AUC 0.74 to 0.93 and 0.79 with CSM, 0.96 and 0.71 with VCL, and 0.92 and 0.52 with SVM. InternVL3-14B improves from 0.60 and 0.57 at baseline to 0.95 and 0.65 with CSM, 0.96 and 0.83 with VCL, and 0.92 and 0.84 with SVM. InternVL3-8B remains at 0.94 and 0.50 across baseline, CSM, VCL, and SVM. Llama 3.2-Vision remains at 0.94 and 0.50 except for VCL, where it yields 0.80 and 0.77. Phi-3-Vision shifts from 0.84 and 0.57 at baseline to 0.94 and 0.50 with CSM, 0.67 and 0.74 with VCL, and 0.94 and 0.60 with SVM. The strongest diagnosis result reported is InternVL3-14B + VCL, with F1 0.96 and AUC 0.83; the paper’s interpretation is that VCL is often the strongest diagnosis prompt for discriminative behavior (Wu et al., 15 Sep 2025).

On deformity location detection, the reported metric is overall accuracy together with region-specific F1 and AUC for thoracic, thoracolumbar, and lumbar. Qwen2.5-VL-7B rises from baseline OA 0.15 to 0.33 with CSM, 0.43 with VCL, and 0.35 with SVM; with color, these become 0.31, 0.44, and 0.40. For this model, thoracic F1 improves strongly with prompts and color, including CSM from 0.68 to 0.79 and VCL from 0.81 to 0.81, whereas thoracolumbar remains harder, around the 0.55–0.67 F1 range, and lumbar is often weaker and does not improve much. InternVL3-14B rises from baseline OA 0.26 to 0.17 with CSM, 0.21 with VCL, and 0.22 with SVM; with color, these become 0.28, 0.32, and 0.50. One of the clearest gains is therefore InternVL3-14B + SVM + color = 0.50 OA, up from 0.26 baseline. The paper also highlights large thoracic improvements for this model, including CSM thoracic F1 from 0.52 to 0.79 and SVM thoracic F1 from 0.63 to 0.82 (Wu et al., 15 Sep 2025).

On deformity direction detection, the task is explicitly identified as the hardest. The best overall performance without color is InternVL3-14B + SVM = 0.16 OA, and with color it is InternVL3-14B + SVM = 0.38 OA. Qwen2.5-VL-7B has baseline OA 0.09, then 0.09 with CSM, 0.08 with VCL, and 0.03 with SVM; with color, the corresponding values are 0.13, 0.08, and 0.12. InternVL3-14B moves from baseline 0.10 to 0.11 with CSM, 0.14 with VCL, and 0.16 with SVM; with color these become 0.16, 0.21, and 0.38. The study notes that Qwen2.5-VL-7B can sometimes identify regional direction patterns but struggles to combine them into the correct overall bending direction (Wu et al., 15 Sep 2025).

Across the three tasks, the empirical pattern is uneven but consistent in one respect: prompt effectiveness varies by model architecture. Some models benefit strongly from line-based prompts or from color-coded region encoding, whereas others remain effectively non-discriminative.

5. Limitations, failure modes, and interpretive cautions

The paper introduces spinal keypoint prompting because current MLLMs are weak at detecting whether AIS is present, locating the deformity region, and determining deformity direction. It also reports that many models can be overly biased toward predicting positive findings or fail to discriminate between spinal regions. This explains why some apparently strong F1 values coexist with poor AUC or poor task-level discrimination (Wu et al., 15 Sep 2025).

Several configurations illustrate this limitation directly. InternVL3-8B, much of Llama 3.2-Vision, and much of Phi-3-Vision frequently behave like near-constant positive predictors, with AUC = 0.50 on diagnosis. In SDLD, InternVL3-8B, Phi-3-Vision, and many Llama 3.2 settings stay near OA around 0.05–0.11, with F1 near zero for some regions and AUC around 0.50. In SDDD, InternVL3-8B remains near-zero OA with color across prompts, Llama 3.2-Vision with SVM reaches OA 0.00, and Phi-3-Vision is about 0.11 OA with almost no improvement (Wu et al., 15 Sep 2025).

The paper also warns that some apparently high regional accuracies are misleading. For InternVL3-14B and Phi-3-Vision, thoracolumbar accuracy can look high because the models are simply outputting negative results rather than correctly detecting deformity. Direction and location metrics therefore have to be interpreted alongside the task structure, not as isolated summary statistics.

The authors’ bottom-line interpretation is that visual prompting alone is not enough. It improves models only when the underlying MLLM has sufficient capability; otherwise the model may still fail, especially on direction detection and region-specific discrimination. In that sense, the paper characterizes spinal keypoint prompting less as a complete solution and more as a “diagnostic scaffold”: it improves interpretability, provides anatomical priors, reveals where models fail, and partially adapts general MLLMs toward scoliosis self-management (Wu et al., 15 Sep 2025).

6. Relation to prompt-driven landmark detection and morphology-aware spine prompting

Within the broader literature, spinal keypoint prompting can be situated between two neighboring strands: general prompt-driven keypoint detection and spine-specific morphology-aware prompting. "GKDT: General Keypoint Detection Transformer" (Lu et al., 1 Jul 2026) is relevant because it is explicitly designed for arbitrary keypoint sets specified by prompts and supports visual prompts, text prompts, or both. Its central mechanism converts prompt-derived prototypes into convolution kernels for heatmap prediction, uses self-attention among textual keypoint prototypes and visual keypoint prototypes, and applies cross-attention to aggregate context from the query image. The model is trained with mix-modal prompted training over $\{\text{visual}, \text{text}, \text{both}\}$ and with dynamic importance sampling for long-tailed data, and it includes medical landmark datasets such as Cephalometric and Hand X-ray (Lu et al., 1 Jul 2026). This suggests a more formal prompt-conditioned route for spinal keypoint detection than the overlay-based operational design used in AIS radiograph prompting.

A second neighboring strand is prompt-based spine segmentation. "MorphSAM: Learning the Morphological Prompts from Atlases for Spine Image Segmentation" (Fan et al., 16 Jun 2025) is not a classic spinal keypoint-prompting paper, because it does not define explicit anatomical keypoints, landmark coordinates, or point-based anchors. Instead, it learns two automatic morphological prompts from multi-atlas segmentation outputs: anatomical prompts derived from atlas label maps and semantic prompts derived from text descriptions converted from atlas content. On the spine dataset, MorphSAM reports DC 0.9298 ± 0.0167, IoU 0.8739 ± 0.0232, ASD 1.0979 ± 0.5066, and HD95 3.9261 ± 3.0217, outperforming SAM2_mask by +0.0751 DC and +0.1013 IoU; its ablations show that combining anatomical and semantic prompts is better than either alone, and that the ratio component in semantic descriptions is the most influential (Fan et al., 16 Jun 2025).

Taken together, these adjacent works indicate that prompt design for spinal imaging is not restricted to manual points or boxes. One line of work uses explicit keypoint-conditioned or prototype-conditioned detection; another uses atlas-derived morphological and semantic priors. A plausible implication is that spinal keypoint prompting belongs to a wider class of anatomy-aware prompt mechanisms in which structural priors, regional encoding, and task decomposition are introduced to compensate for the mismatch between general-purpose visual-LLMs and the fine-grained geometry of spinal imaging.