Holographic MIMO Empowered NOMA-ISAC for 6G: Rate-Splitting Enhanced Near-Field Modeling, Multi-Objective Optimization, and Statistical Performance Validation

Abstract: Holographic multiple-input multiple-output (MIMO) systems with extremely large apertures enable transformational capabilities for sixth-generation (6G) integrated sensing and communications (ISAC). However, existing non-orthogonal multiple access (NOMA) ISAC works inadequately address: (i) holographic near-field propagation with sub-wavelength antenna spacing; (ii) rate-splitting multiple access (RSMA) integration for interference management; (iii) statistical validation under realistic impairments. This paper presents a comprehensive holographic MIMO NOMA-ISAC framework featuring: \textbf{(1)} Unified near-field modeling incorporating spatially-correlated Rayleigh fading, spherical wavefront propagation, and sub-wavelength antenna coupling effects; \textbf{(2)} Novel rate-splitting enhanced NOMA (RS-NOMA) architecture enabling flexible interference management between sensing and communication; \textbf{(3)} Multi-objective optimization suite comparing hybrid alternating optimization with successive convex approximation (HAO-SCA), weighted minimum mean square error (WMMSE), semidefinite relaxation (SDR), fractional programming (FP), and deep reinforcement learning (DRL); \textbf{(4)} Rigorous statistical validation over 5000 Monte Carlo runs with significance testing across massive MIMO scenarios (up to 1024 antennas). Results demonstrate that RS-NOMA achieves \SI{11.7}{\percent} higher sum-rate than conventional NOMA and \SI{18.8}{\percent} over WMMSE at matched sensing utility. Sensing CRLB improvements of \SI{2.4}{\decibel} are confirmed with 99\% statistical confidence. The framework establishes rigorous foundations for practical 6G holographic MIMO ISAC deployment.