Enabling Performant and Flexible Model-Internal Observability for LLM Inference

Abstract: Today's inference-time workloads increasingly depend on timely access to a model's internal states. We present DMI-Lib, a high-speed deep model inspector that treats internal observability as a first-class systems primitive, decoupling it from the inference hot path via an asynchronous observability substrate built from Ring^2, a GPU-CPU memory abstraction for capturing and staging tensors, and a policy-controlled host backend that exports them. DMI-Lib enables the placement of observation points across a rich space of internal signals and diverse inference backends while preserving serving optimizations and adhering to tight GPU memory budgets. Our experiments demonstrate that DMI-Lib incurs only 0.4%--6.8% overhead in offline batch inference and an average of 6% in moderate online serving, reducing latency overhead by 2x-15x compared to existing baselines with similar observability features. DMI-Lib is open-sourced at https://github.com/ProjectDMX/DMI.

• The paper presents DMI-Lib, a flexible framework that decouples internal observability from LLM inference for real-time interpretability and control.
• The paper demonstrates minimal overheads (0.4%–6.8% in offline and below 10% in online settings) compared to traditional synchronous extraction methods.
• The paper introduces innovative components like HookPoint, Ring² staging, and an asynchronous exporter to capture internal signals efficiently and scale across GPUs.

Model-Internal Observability as a First-Class Systems Primitive: DMI-Lib

Motivation and Background

Inference-time access to internal states in LLMs—including residual streams, attention patterns, activations, and control signals—enables high-value applications such as mechanistic interpretability, real-time safety monitoring, test-time alignment, and speculative decoding. However, conventional extraction mechanisms either couple observability to model APIs or specific engines, or impose significant performance penalties due to synchronous host intervention and nontrivial memory overheads. This paper presents DMI-Lib, a model- and inference-agnostic high-performance deep model inspector that decouples internal observability from inference execution, thus treating it as a first-class systems concern.

DMI-Lib resolves a critical gap by allowing practitioners to instrument arbitrary points in the inference pipeline with negligible overhead, without violating memory constraints or interfering with serving optimizations typical of production LLM deployments. This capability unlocks core interpretability, alignment, and control use-cases previously infeasible at scale or in online settings.

System Design and Architecture

DMI-Lib achieves performant and flexible internal observability through three principal components:

• HookPoint: Minimal instrumentation module compatible with CUDA Graphs, installable at arbitrary points in the PyTorch computation graph. At execution, HookPoint uses a custom C++/CUDA operator to transfer tensor snapshots to a dedicated staging layer without breaking graph execution assumptions.

  Figure 1: HookPoint's execution stack enables efficient capture of arbitrary internal tensors, avoiding synchronous host callbacks.

• Ring^2 GPU-CPU Memory Staging: A split ring-buffer abstraction coordinating device-side payload storage and host-side metadata management. Raw tensor bytes are staged in device memory; compact descriptors are placed in managed CPU-preferred memory; runtime metadata flows through a host FIFO, enabling zero-interrupt tensor capture and asynchronous draining.

  Figure 2: The Ring^2 workflow allows decoupled tensor staging and metadata management, minimizing cross-device coordination overhead.

• Asynchronous Data Exporter: Host-side engine that drains staged tensors in batches, reconstructs records with semantic metadata, and exports them to downstream storage or analytic backends. The exporter pipeline is multi-threaded and decoupled for high throughput and minimal backpressure propagation.

DMI-Lib is further equipped with a policy manager offering runtime selection of observation completeness versus best-effort monitoring under hardware bottlenecks. Users can configure hook granularity or request-dropping strategies to balance fidelity and latency overhead.

Figure 3: DMI-Lib maintains near-original inference speed below PCIe bandwidth saturation, gracefully degrading to synchronous offloading under overload.

Figure 4: High-level schematic of DMI-Lib's modular pipeline, showing HookPoint instrumentation, Ring^2 tensor staging, and asynchronous export.

Performance Evaluation

Empirical results demonstrate DMI-Lib's low-impact instrumentation:

• Offline batch inference: Overhead remains between 0.4% and 6.8% (average 3.6%) for flexible custom hooks across Qwen3-4B/14B, Llama3.1-8B, and batch sizes up to 64, substantially outperforming baselines such as PyTorch hooks (up to 62.3% overhead) and NNsight.
• Online serving: Median TTFT and TPOT overheads are consistently below 10% up to moderate throughput rates, with 2–15x latency reduction compared to engine-bound observability tools that serialize per-step D2H transfers.
• Multi-GPU scaling: Maintains batch-size and throughput scaling nearly identical to vanilla Hugging Face in tensor-parallel mode; supports up to 98% of baseline maximum batch size.
• Overload management: Hook filtering and request-dropping policies can restrict observation to critical signals or requests, reducing overhead under PCIe saturation by up to 12.4x.

  Figure 5: Minimal overhead for limited hooks in offline inference, demonstrating DMI-Lib's efficiency versus baselines.

  

  Figure 6: DMI-Lib sustains low overhead with extensive custom hook coverage, unlike conventional approaches.

  

  Figure 7: (a) DMI-Lib maintains near-ideal scaling under tensor parallelism. (b) Storage backend ablation confirms pipelined exporter efficiency.

  

  Figure 8: DMI-Lib achieves near-native TTFT across request rates, outperforming synchronous D2H strategies in online serving.

  

  Figure 9: DMI-Lib maintains low TPOT overhead up to high throughput in online inference environments.

Observability Tradeoffs and Overload Policy

DMI-Lib's hierarchically organized overload control—hook filtering combined with runtime request-dropping—enables users to prioritize either completeness or latency isolation. Empirical results show:

• Hook filtering: Reduces observation volume, maintaining overhead between 0.4%–16.9% under varying hook sets.
• Best-effort policy: Drops recent or pattern-matched requests to prevent memory saturation, reducing overhead by an order of magnitude under bandwidth constraints.

  Figure 10: Overload onset as a function of hook count and ring buffer size; DMI-Lib outperforms synchronous extraction even under heavy observation load.

  

  Figure 11: (a) Hook filtering trades observability for performance. (b) Request-granular dropping keeps inference throughput stable under overload.

Microbenchmark Analysis

Per-step ablation studies validate DMI-Lib's architectural choices:

• Ring^2 staging: Reduces compute-path perturbation to 1.36% (prefill) and 5.05% (decode), with asynchronous draining amortizing export costs.
• Synchronous extraction: PyTorch hooks and Hugging Face built-in APIs induce >100% overhead per step, confirming incompatibility with optimized inference graphs.
• Memory footprint: DMI-Lib supports near-maximal batch sizes (<4% reduction) even with full internal state extraction.

Figure 12: Overhead breakdown confirms asynchronous Ring^2 staging hides most export cost.

Use Cases and Practical Implications

Mechanistic interpretability and knowledge distillation are two practical demonstrations:

• Attention pattern analysis: DMI-Lib enables one-pass capture of all per-(layer, head) attention matrices, facilitating offline analysis of functional circuit structures such as Duplicate Token and attention-sink patterns, aligned with interpretability literature.

  Figure 13: DMI-Lib direct capture exposes functional attention-head patterns in Qwen3-0.6B.

• Layerwise distillation: Efficient capture and persistence of teacher model hidden states enables repeated student training regimes, with layerwise supervision yielding 2–3% accuracy gains over logits-only distillation, and 4–9% over standard supervised fine-tuning.

Implications and Future Directions

The formalization of model-internal observability as a systems primitive opens new avenues for real-time interpretability, safety, and control in large-scale deployments. DMI-Lib’s architecture suggests a general template for asynchronous, low-overhead capture of arbitrary intermediate signals during inference, decoupled from the execution path and tailored to heterogeneous resource constraints. The policy layer enables practical tradeoffs between completeness and latency adherence, critical for production environments.

In the broader context, such infrastructure can serve as the foundation for advanced safety interventions, distributed analytics, and model governance paradigms where internal activation dynamics are central. Integration with speculative decoding and test-time alignment may yield further serving optimizations, and tight coupling with system-level observability tools promises unified monitoring solutions spanning hardware and model internals.

Conclusion

DMI-Lib provides performant, flexible, and portable model-internal observability for LLM inference. By decoupling internal signal extraction from inference execution through CUDA-compatible hooks and asynchronous memory staging, DMI-Lib achieves minimal overhead in both offline and online contexts, robust overload control, and broad applicability in interpretability and control tasks. The work lays foundational systems infrastructure for next-generation model monitoring and intervention capabilities, aligning observability requirements with practical performance constraints.

(2605.11093)