OrthoInsight: Modular Orthopaedic AI & AR

• OrthoInsight is a modular framework that unifies diverse orthopaedic workflows by converting sparse clinical data into detailed biomechanical, geometric, or textual insights.
• Its modules span Brillouin microscopy for early cartilage changes, monocular arthroscopy reconstruction with AR guidance, 4D dynamic joint imaging, standalone holographic navigation, and CT-based rib fracture reporting.
• The system emphasizes integration of multimodal data to enhance intraoperative guidance and post-acquisition diagnostics while addressing limitations like shallow penetration and model speed.

Across the cited work, the name OrthoInsight is attached to multiple orthopaedic workflows, including Brillouin-microscopy-based cartilage assessment, monocular arthroscopy reconstruction and augmented reality (AR) guidance, four-dimensional joint imaging from cone-beam CT (CBCT) and fluoroscopy, standalone holographic navigation, AR-based C-arm repositioning, and multi-modal rib-fracture reporting (Wu et al., 2018, Shu et al., 2024, Tang et al., 22 Aug 2025, Liebmann et al., 2020, Unberath et al., 2018, Wu et al., 18 Jul 2025). This suggests that OrthoInsight is best understood as a modular research label spanning sensing, registration, visualization, kinematic analysis, and language-generation pipelines rather than as a single invariant implementation.

1. Scope and research profile

The systems associated with OrthoInsight address distinct but adjacent orthopaedic tasks: biochemical sensing of early osteoarthritis (OA), intra-articular reconstruction from monocular arthroscope video, weight-bearing dynamic joint analysis, image-guided navigation, intraoperative imaging logistics, and radiological report generation. The unifying theme is the conversion of sparse, noisy, or operationally constrained clinical data into higher-level geometric, biomechanical, or textual representations.

Context	Core components	Representative source
Early OA cartilage assessment	Brillouin microscopy, enzymatic PG depletion model, histological correlation	(Wu et al., 2018)
Arthroscopic AR reconstruction	OneSLAM, Depth Anything, 3D Gaussian splatting, Unity AR tools	(Shu et al., 2024)
Dynamic joint imaging	Dual robotic arm CBCT, DeepDRR, 3D-2D registration, kinematic metrics	(Tang et al., 22 Aug 2025)
AR navigation and imaging workflow	HoloLens registration, holographic guidance, AR C-arm pose recall	(Liebmann et al., 2020, Unberath et al., 2018)
Rib fracture diagnosis and reporting	YOLOv9, medical knowledge graph, LLaVA	(Wu et al., 18 Jul 2025)

Taken together, these systems occupy different layers of the orthopaedic information stack. Some modules infer tissue composition from optical phonon interactions; others reconstruct explicit geometry for AR anchoring; others fuse static and dynamic radiography for 4D kinematics; still others transform CT findings into structured diagnostic prose. The research corpus therefore spans both intraoperative assistance and post-acquisition interpretation.

2. Brillouin microscopy for extracellular-matrix sensing

In the OA-focused work, Brillouin microscopy is used to detect proteoglycan (PG) loss from articular cartilage by measuring the Brillouin frequency shift $\Delta\nu_B$, which arises from the interaction of light with thermally excited acoustic phonons. In back-scattering geometry, the shift is given by

$\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$

and the longitudinal elastic modulus satisfies

$M_L = \rho v^2.$

Because articular cartilage is $\sim 70$–$85\%$ water, small changes in PG content, hydration, and permeability strongly affect $v$ and therefore $\Delta\nu_B$. PG depletion reduces fixed charge density, allows more water to partition into the extracellular matrix, lowers the effective longitudinal modulus, and reduces $\Delta\nu_B$ (Wu et al., 2018).

The optical implementation uses a single-frequency diode laser at $\lambda = 561$ nm and 30 mW, a single-stage VIPA spectrometer with a Rayleigh-suppression filter, and a microscope objective with $\mathrm{NA}=0.5$. The reported lateral resolution is $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$0m and the axial resolution is $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$1m. The scanning geometry is confocal back-scatter, and penetration depth in articular cartilage is $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$2m before multiple scattering suppresses the signal. Data acquisition consists of $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$3-scans perpendicular to the surface in $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$4–$\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$5m steps with five spectra per point, and $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$6 scans in $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$7m steps.

The validation protocol uses 10 mm porcine cartilage-on-bone punches digested with trypsin at either $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$8 mg/ml or $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$9 mg/ml in PBS at $M_L = \rho v^2.$0 for $M_L = \rho v^2.$1–$M_L = \rho v^2.$2 h. After digestion, samples are washed in PBS and bisected, with one half allocated to Brillouin imaging and the other to histology. Safranin-O staining at $M_L = \rho v^2.$3 labels GAGs, and Fast Green at $M_L = \rho v^2.$4 counterstains bone in 5 $M_L = \rho v^2.$5m sections. Brillouin $M_L = \rho v^2.$6 decreases in the same zones where Safranin-O staining intensity drops, establishing correlation with PG depletion and associated hydration changes.

Quantitatively, control cartilage shows deep-tissue $M_L = \rho v^2.$7 GHz, compared with 7.2 GHz in PBS. Low-dose trypsin at $M_L = \rho v^2.$8 mg/ml produces no statistically significant change over 4 h, with variations $M_L = \rho v^2.$9 GHz and within noise. High-dose trypsin at $\sim 70$0 mg/ml produces a progressive decline reaching a maximum relative shift of $\sim 70$1 MHz after 4 h; two-way ANOVA shows statistically significant decreases at 3 h ($\sim 70$2) and 4 h ($\sim 70$3). With instrument noise of approximately $\sim 70$4 MHz, the system is reported to detect $\sim 70$5 changes corresponding to $\sim 70$6 shifts in water content, and to distinguish between $\sim 70$7 PG loss, which shows no change, and $\sim 70$8 PG loss, which yields a measurable drop.

The principal significance of this module is its sensitivity to early compositional cartilage changes that are described as small ($\sim 70$9) and difficult to register with magnetic resonance and ultrasound imaging. Its non-destructive character also motivates minimally invasive arthroscopic translation. At the same time, the reported limitations are substantial: shallow penetration restricts assessment to superficial cartilage zones; the two-phase hydration model ignores local changes in $85\%$0 and $85\%$1 and possible digestion by-products; the study uses porcine tissue under ex vivo conditions; and the spectrometer footprint is not yet clinically compatible.

3. Monocular arthroscopy reconstruction and AR guidance

The arthroscopic AR pipeline processes monocular arthroscope video through four sequential modules: monocular SLAM, monocular depth estimation, 3D Gaussian splatting, and AR rendering with interaction. OneSLAM produces camera poses $85\%$2 and a sparse 3D point cloud by windowed bundle adjustment minimizing reprojection error. Depth Anything produces pixel-wise disparity maps $85\%$3, which are converted to pseudo-depth by a frame-specific affine inverse mapping,

$85\%$4

fit against visible SLAM point depths. Normals are then computed via a depth-to-normal translator. The scene is represented explicitly as a set of Gaussians with 3D mean, covariance, spherical harmonic color coefficients, and opacity, and trained with a loss combining photometric, SSIM, depth-alignment, normal-consistency, normal-prior, and opacity-regularization terms (Shu et al., 2024).

AR deployment uses a real-time Unity renderer that superimposes the learned Gaussian model onto the live arthroscope feed. For correct overlay, camera extrinsic calibration provides $85\%$5, optical tracking provides $85\%$6, and the world-to-camera transform is

$85\%$7

With an additional fixed transform $85\%$8, a model point $85\%$9 is rendered by

$v$0

The interface is explicitly human-in-the-loop and supports point selection, distance measurement, and region annotation. For notch sizing, two image taps are unprojected via $v$1, 3D neighbors are averaged to reduce noise, and the Euclidean distance $v$2 is reported.

The reported performance is dense 3D reconstruction and competitive rendering fidelity with explicit 3D representation in 7 minutes on average, directly from monocular arthroscopy and without additional data and/or hardware. On four phantom datasets, the abstract reports reconstruction $v$3 mm, $v$4, and $v$5 on average, while the details add an AR measurement accuracy within $v$6 mm and AR annotation $v$7. Computationally, dense reconstruction takes approximately 7 minutes on an RTX TITAN GPU plus Ryzen 7 system, while AR rendering and measurement run at video framerates on a laptop GPU.

The significance of this pipeline lies in addressing the restricted field of view and lack of depth perception that characterize arthroscopy. Its explicit 3D Gaussian representation supports direct interaction and viewpoint changes in a way that is operationally different from implicit NeRF-style reconstruction. A plausible implication is that such a representation is particularly well matched to intraoperative measurement and anchoring tasks, where geometric editability matters as much as photorealistic rendering.

4. Four-dimensional dynamic joint imaging and 3D-2D registration

The 4D Virtual Imaging Platform combines a gantry-free CBCT system, deep-learning-based preprocessing, and single-plane fluoroscopy registration to quantify dynamic, weight-bearing joint motion. The hardware uses two ultra-lightweight $v$8 kg 6-degree-of-freedom robotic arms carrying the X-ray source and flat-panel detector, mounted on a floor-fixed base with integrated vertical lift of max $v$9-travel $\Delta\nu_B$0 cm and rotary bearings. The detector is 61 cm wide with $\Delta\nu_B$1 pixels and 0.3 mm pixel pitch. The absence of a traditional C- or O-shaped gantry allows arbitrary programmable source-detector trajectories; a typical acquisition uses a reverse-spiral $\Delta\nu_B$2 source trajectory around the knee (Tang et al., 22 Aug 2025).

The preprocessing stack includes 2D fluoroscopy segmentation, 3D CBCT segmentation, and DRR generation. The 2D module is an nnU-Net-style 2D U-Net with residual blocks, skip-connections, and instance normalization. The first frame is manually refined, and subsequent frames are seeded by the previous frame’s segmentation through a ConvLSTM-inspired propagation module; the loss is combined Dice plus binary cross-entropy. Training uses 500 manually annotated fluoroscopy sequences of knee flexion, with augmentation by in-plane rotations of $\Delta\nu_B$3, translations of $\Delta\nu_B$4 px, Gaussian intensity noise with $\Delta\nu_B$5 up to 0.05, and random gamma or contrast. Reported held-out performance is Dice coefficient $\Delta\nu_B$6 and pixel-wise accuracy $\Delta\nu_B$7. Static CBCT segmentation uses TotalSegmentator with a 3D U-Net backbone, without fine-tuning, and reaches mean Dice $\Delta\nu_B$8 over 20 clinical scans. DeepDRR then produces synthetic projections through a material decomposition CNN, spectrum-aware ray tracer, CNN-based scatter estimator, and noise simulator in $\Delta\nu_B$9 s per frame.

Registration seeks the rigid transform $\Delta\nu_B$0 maximizing normalized cross-correlation between the DRR and segmented fluoroscopy image. Optimization proceeds in three stages: global initialization by Differential Evolution along each bone’s PCA mechanical axis; a Kinematic Priority Module that restricts updates to physiologically plausible directions; and local refinement by a hybrid Powell–Nelder–Mead strategy. Although full Jacobians are not computed in closed form, the paper notes a Levenberg–Marquardt form as an underlying model for small updates. In simulation, the full BoneAxis-Reg + KPM pipeline reaches mean target registration error $\Delta\nu_B$1 mm and success rate $\Delta\nu_B$2 for an RSR threshold of 1.5 mm, outperforming Nelder–Mead, Powell + Nelder–Mead, PEHL, ProST, and BoneAxis-Reg without KPM. In a lamb knee joint experiment with RSR threshold 3 mm, the reported values are $\Delta\nu_B$3 mm and $\Delta\nu_B$4.

The clinical protocol described for post-total knee arthroplasty uses one patient standing on a weight-bearing platform. Static CBCT is acquired at $\Delta\nu_B$5, 110 kVp, 51 mA, 15 ms, and $\Delta\nu_B$6 reverse spiral; dynamic fluoroscopy is a single-plane lateral view at knee flexion angles $\Delta\nu_B$7, $\Delta\nu_B$8, $\Delta\nu_B$9, and $\lambda = 561$0, with 0.3 mm pixels, $\lambda = 561$1, and 30 fps during controlled step-ascent or lunge. Quantification is built around the tibial plateau plane $\lambda = 561$2, the perpendicular femoral-condyle contact distances $\lambda = 561$3, the medial-lateral difference $\lambda = 561$4, and the Distance Difference Variance $\lambda = 561$5. The reported findings include most condyle contact points lying below the diagonal in the $\lambda = 561$6 plot, peak $\lambda = 561$7 mm, elevated DDV, and shortened trajectory lines near hyperextension, interpreted as medial-lateral asymmetry consistent with varus/valgus malalignment and residual ligament imbalance.

This module shifts OrthoInsight from surface visualization toward biomechanical inference under functional loading. Its distinctive claim is not merely volumetric imaging, but fusion of static 3D CBCT with dynamic 2D X-rays into a clinically quantified 4D representation under upright, weight-bearing conditions.

5. Standalone AR navigation and intraoperative imaging workflow assistance

The standalone navigation system based on Microsoft HoloLens uses the first-generation device as a self-contained AR headset running a Universal Windows Platform app for marker tracking, registration, and holographic rendering. The headset provides inertial measurement, SLAM, and two front-facing environment cameras in Research Mode at $\lambda = 561$8 px each. Sterile rigid fiducial markers based on AprilTags or ArUco patterns are detected in the stereo views, corner noise is smoothed by a Kalman filter, stereo triangulation yields 3D corner positions, and Horn’s absolute orientation computes the marker pose. The workflow includes a custom pointing device for intraoperative surface digitization and a custom navigation tool for K-wire or pedicle screw insertion. Registration begins with PCA-based extreme-point matching under two bilateral-symmetry hypotheses and is refined by ICP minimizing

$\lambda = 561$9

After registration, planned entry points are shown as semi-transparent blue spheres and planned axes as green lines, while angular deviation is visualized by a colored triangular indicator with thresholds at $\mathrm{NA}=0.5$0 and $\mathrm{NA}=0.5$1 (Liebmann et al., 2020).

On spine phantoms, the reported means for $\mathrm{NA}=0.5$2 K-wires are trajectory error $\mathrm{NA}=0.5$3, entry-point error $\mathrm{NA}=0.5$4 mm, registration RMSE $\mathrm{NA}=0.5$5 mm, surface digitization time $\mathrm{NA}=0.5$6 s, and $\mathrm{NA}=0.5$7 collected points. The paper relates these values to common clinical targets such as trajectory error $\mathrm{NA}=0.5$8 and entry error $\mathrm{NA}=0.5$9 mm. At the same time, the described limitations include hologram drift due to head motion and lack of fiducial compensation for patient motion, interference between Research Mode sensors and built-in SLAM stability, and the use of rigidly fixed phantoms without soft-tissue motion or breathing.

A separate AR workflow targets technician-in-the-loop C-arm repositioning. Here an optical see-through HMD with on-board SLAM and integrated near-infrared depth sensing records desired C-arm poses as point clouds and re-displays them as a “ghost” object for later alignment. The software chain comprises SLAM-based HMD localization, depth-capture and point-cloud generation, transform chaining and storage, AR rendering, and optional ICP-based quantitative feedback. In the simulated orthopedic trauma setting, a board-certified X-ray technician defines target poses around a draped Sawbones pelvis phantom, and the primary reported operational result is a reduction of mean X-rays per pose from 2.76 in the conventional workflow to zero during AR-based repositioning (Unberath et al., 2018).

The geometric trade-off is explicit. Reported translational error is $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$00 mm for the AR method versus $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$01 mm conventionally, while rotational error is $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$02 versus $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$03. Projection-domain keypoint displacement is $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$04 px for the AR approach, $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$05 px for the conventional first try, and $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$06 px after conventional refinement. A common misconception would be to read radiation reduction as equivalent to improved pose accuracy; the reported numbers do not support that equivalence. What they do support is a different optimization target: fewer trial-and-error fluoroscopic acquisitions during repositioning.

6. Multi-modal rib fracture diagnosis and report generation

In the rib-fracture system, OrthoInsight denotes a multi-modal framework that integrates a YOLOv9 detector, a medical knowledge graph, and a fine-tuned LLaVA LLM for diagnostic report generation from CT scans. The architecture maps CT image features $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$07 through a linear projection $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$08 into the LLM embedding space, obtains textual embeddings $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$09 from prompt text, fuses visual and textual representations by element-wise product or concatenation, and generates a report $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$10. Training uses a joint objective

$\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$11

where $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$12 is the YOLOv9 detection loss, $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$13 is the knowledge retrieval loss, and $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$14 is the teacher-forcing language-model loss (Wu et al., 18 Jul 2025).

The detection stage uses a CSPDarknet backbone with P5/P6 output, a BiFPN neck with three scales $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$15, and three detection branches with 3 anchors each. Anchor scales are $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$16 and aspect ratios are $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$17. Fine-tuning uses input size $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$18, batch size 16, 150 epochs, AdamW with initial learning rate 0.01 and cosine annealing, binary cross-entropy for objectness and classification, and CIoU for boxes. The knowledge graph contains approximately six node types—fracture classification, etiology, injury mechanism, clinical manifestations, treatment plan, and complication management—with relations such as “hasEtiology”, “hasMechanism”, “suggestsTreatment”, and “complicationOf”, for roughly 42,000 facts derived from a surgery textbook and treatment guidelines. Retrieval forms a query from detected fracture types and expert report text, embeds the query and graph nodes with a text encoder, ranks by cosine similarity, and appends the top-$\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$19 context vectors to the prompt.

The report-generation stage uses 28,675 CT scans with preliminary reports and retrieved facts to produce approximately 50,000 instruction pairs via GPT-4O-enhanced reports. Training proceeds in two phases: Stage 1 concept alignment freezes the LLM and vision encoder and trains the projection $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$20 for 5 epochs, while Stage 2 instruction tuning applies LoRA to LLaVA layers and the projection for 20 epochs, with rank 64, $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$21, dropout 0.05, batch size 10, learning rate $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$22, and AdamW. The compute environment is Ubuntu 20.04, Intel Xeon Platinum 8255C, three NVIDIA RTX 3090 GPUs, Python 3.8, PyTorch 1.11.0, and CUDA 11.3, with a session time of approximately 60 hours.

The dataset contains 28,675 CT images at $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$23 px with five fracture subtypes and YOLO annotations, one-to-one radiologist reports from five experts, and a train/validation/test split of 70%/10%/20%. On the test split, YOLOv9 achieves 98.3% precision, 89.0% recall, 74.2% $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$24, and 61.5% $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$25. On 200 expert-rated samples, the report generator scores $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$26 for Diagnostic Accuracy, $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$27 for Content Completeness, $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$28 for Logical Coherence, and $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$29 for Clinical Guidance, with an average score of 4.28 and paired $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$30-tests against GPT-4 and Claude-3 showing $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$31 and Cohen’s $\Delta\nu_B = \frac{2n}{\lambda} v \sin(\theta/2),$32. Failure cases remain important: low-contrast fractures near the mediastinum may be missed, rare buckle or segmental variants are underrepresented, knowledge retrieval can return tangential facts when query text is sparse, and the system is described as hardware-heavy for real-time deployment.

This branch of OrthoInsight differs qualitatively from the intraoperative systems. Its contribution is not spatial registration or tracking, but coupling lesion detection with clinically structured language generation and externalized medical context. It therefore occupies the reporting and decision-support end of the orthopaedic pipeline.

7. Validation status, limitations, and interpretive cautions

The evidence base is heterogeneous. The Brillouin module is ex vivo and porcine, with superficial penetration depth and a spectrometer footprint not yet clinically compatible (Wu et al., 2018). The arthroscopy AR pipeline shows strong phantom and cadaver-sequence metrics and video-rate rendering, but its dense model is still reconstructed in approximately 7 minutes rather than continuously online, and its measurement and annotation results remain tied to the reported evaluation setup (Shu et al., 2024). The 4D CBCT platform combines simulation, animal-joint validation, and a single post-TKA clinical example, so its sub-voxel registration claim is best read in the context of the stated protocols and thresholds rather than as a universal bound (Tang et al., 22 Aug 2025).

The AR navigation systems also require careful reading. In the HoloLens study, clinically relevant phantom results coexist with hologram drift, SLAM instability under Research Mode, and the absence of soft-tissue or respiratory motion (Liebmann et al., 2020). In the C-arm repositioning study, zero repositioning X-rays do not imply superior translational accuracy, and the reported AR pose error remains substantially larger than conventional refinement (Unberath et al., 2018). These are not contradictions; they indicate that workflow metrics, radiation metrics, and geometric metrics are partially independent objectives.

The rib-fracture reporting system similarly combines strong aggregate performance with identifiable failure modes. High expert-rated scores and statistically significant gains over GPT-4, Claude-3, and LLAVA1.5-7B coexist with detector misses in low-contrast regions, rare-class underrepresentation, imperfect knowledge retrieval, and substantial compute requirements (Wu et al., 18 Jul 2025). More generally, the cited work does not support treating OrthoInsight as a clinically standardized end-to-end product. A more precise interpretation is that it names a family of orthopaedic AI and AR modules whose maturity ranges from proof-of-principle to clinically motivated prototype, and whose common research agenda is to enrich limited clinical observations with reconstructed geometry, biomechanical context, or report-level semantics.