European Deep Inference Fabric (eDIF)
- European Deep Inference Fabric (eDIF) is an NDIF-compatible infrastructure that enables remote mechanistic interpretability of large language models by providing access to internal model parameters and activations.
- It facilitates collaborative experimentation through causal tracing, activation patching, and representation analysis across distributed GPU clusters in European institutions.
- The pilot study demonstrated increased user engagement, significant time savings, and enhanced community formation while addressing scalability and resource accessibility challenges.
European Deep Inference Fabric (eDIF) is an NDIF-compatible infrastructure for mechanistic interpretability research on LLMs, developed as a feasibility study on how to provide widespread accessibility of LLM interpretability infrastructure in Europe through remote model inspection via the NNsight API (Guggenberger et al., 14 Aug 2025). It is motivated by the observation that black-box APIs limit access to model internals such as weights and activations, while open-weight models require considerable compute resources that are often unavailable to smaller or less-funded academic institutions in Europe. In this setting, eDIF is positioned as a shared research fabric for causal interventions, probing, tracing, and related forms of model analysis, with an explicit emphasis on democratized access, reproducibility, and community formation.
1. Origins, scope, and research rationale
eDIF is motivated by the need to democratize access to advanced model interpretability infrastructure for European researchers, lowering both technical and institutional entry barriers. The project is also intended to enable remote, collaborative experimentation on LLMs without the need to host large models locally and to support the development of a pan-European research community around model transparency, reproducibility, and open science in AI.
The initiative builds on prior work from the US-based NDIF project and adapts that paradigm to the European context regarding institutional, funding, and regulatory constraints. NDIF provides shared, remote access to LLMs through a PyTorch-compatible deferred execution API, NNsight, enabling safe co-tenancy, model introspection, and causal experimentation. eDIF is fully NDIF-compatible, with interoperability and portability presented as design goals rather than incidental properties.
A common misconception is that remote interpretability infrastructure is equivalent to a generic inference service. In the eDIF design, the emphasis is not merely on remote execution of prompts, but on access to intermediate activations, parameters, and intervention pathways needed for mechanistic interpretability. This distinction is central to the project’s stated purpose.
2. System architecture and deployment model
The primary host is Ansbach University of Applied Sciences, Germany. The core hardware consists of 8 × NVIDIA RTX A6000 GPUs, each with 48GB VRAM, selected to support concurrent multi-model deployments and maximize per-card memory for efficient co-tenancy. Additional deployments include experimental clusters at Friedrich-Alexander University (FAU), using AMD Instinct MI300X GPUs, and at Heilbronn University. Together, these deployments are presented as supporting distributed research across institutions (Guggenberger et al., 14 Aug 2025).
The software stack is explicitly layered. The base operating system is Ubuntu LTS 22.04. Containerization uses Docker and Docker Compose for isolated, portable deployments, while Apptainer (Singularity) is used for non-root environments on HPC clusters. Ray coordinates request scheduling and resource allocation using a FIFO-based policy. FastAPI handles front-end user requests, and the backend pipeline processes model inference and experimental interventions. For storage, MinIO provides S3-compatible object storage. Monitoring and analytics are handled by Loki for log aggregation, Prometheus and InfluxDB for time-series metrics, and Grafana for visual dashboards.
Remote model inspection is mediated by the NNsight API. User code specifies experiments locally, while actual computation and model execution happen on the remote eDIF backend. The paper’s architecture figures describe the request path as flowing from local user scripts, through NNsight and FastAPI, to the Ray-managed GPU backend, with results returned over the same channel. In practical terms, this design allows local scripting with remote execution semantics.
The project also reports successful porting and operation on FAU’s HPC system with AMD MI300X GPUs. This required conversion of Docker containers to Apptainer and switching from CUDA/NCCL to ROCm/RCCL, which is presented as demonstrating multi-vendor compatibility and codebase portability. The paper is descriptive and technical rather than mathematical: there are no key equations or formal algorithms detailed, and resource allocation is instead expressed operationally, including in the model table.
3. Model portfolio and interpretability operations
The deployed model portfolio in the pilot includes GPT-2 small, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Llama-70B. Llama 4 Maverick was also deployed on an FAU external server for advanced testing. All models were downloaded from public Hugging Face repositories, with NNsight integration described as mimicking standard Hugging Face code interfaces.
| Model | Size | GPUs Req. |
|---|---|---|
| GPT-2 small | 124M parameters | 1 × RTX A6000 |
| DeepSeek-R1-Distill-Llama-8B | 8.03B parameters | 2 × RTX A6000 |
| DeepSeek-R1-Distill-Llama-70B | 70.6B parameters | 5 × RTX A6000 |
The interpretability operations emphasized in the pilot are activation patching, causal tracing, and representation analysis. Activation patching is described as swapping or modifying activations to test the causal contribution of intermediate states. Causal tracing follows and manipulates computation paths across layers or neurons. Representation analysis includes methods such as probe training, logit lens, and attribution circuits. Participant usage also included circuit analysis and sparse autoencoder work, indicating that the platform supported a heterogeneous set of mechanistic workflows rather than a single canonical protocol.
Structured notebooks were provided for onboarding. These covered gender bias and protection circuits and gave stepwise guidance for backend and API access. This onboarding strategy suggests that eDIF was conceived not only as infrastructure, but also as a reproducible experimental environment with shared entry points for new users.
4. Pilot study design, participants, and usage patterns
The feasibility study recruited 29 applicants, of whom 16 were selected based on expertise and need for compute. The selected researchers came from the UK, Netherlands, Spain, Italy, Germany, Austria, and France, with affiliations spanning several universities and institutions. The represented research programs included gender bias analysis, protective instinct circuit tracing, sparse autoencoder exploration, and mechanism “white-boxing” (Guggenberger et al., 14 Aug 2025).
The pilot reports a gradual increase in user engagement. User activity showed a clear upward trend, peaking at over 700 requests per day by the end of the study. Onboarding and ease of use received a mean rating of 4.2/5, while intuitiveness received 4.0/5. Platform integration into ongoing work was substantial: 83.3% partially integrated eDIF/NNsight into their workflows during the study, and 100% expressed interest in continued access.
Collaborative features were also evaluated positively. The study reports consensus on the value of shared notebooks and open discussion spaces, and 83.4% rated collaborative capabilities as 4 or 5 out of 5. This marked what the paper describes as the starting point for building a user community around the platform.
Scientific utility was tied to both flexibility and execution model. Remote execution was strongly preferred over local hosting for resource-intensive tasks, and NNsight’s deferred execution and PyTorch compatibility were rated highly. Participants used causal tracing, activation patching, logit lens, probe training, circuit analysis, and sparse autoencoder methods, which the study interprets as evidence of platform versatility. Efficiency gains were also reported: approximately half of users reported 10–30% time savings, 33% reported 30–50%, and at least one reported over 50% savings compared to local workflows.
5. Operational performance, bottlenecks, and failure modes
The overall platform was reported as stable throughout daily operation. The Docker-based stack comprising Ray, FastAPI, MinIO, Loki, Prometheus, InfluxDB, and Grafana is described as robust in routine use, and monitoring dashboards indicated that the majority of requests were successful even during higher-load periods (Guggenberger et al., 14 Aug 2025).
At the same time, the study identifies several limitations. Activation data downloads sometimes exceeded 30 minutes, attributed to bandwidth limitations. Execution interruptions and model freezes occurred, especially under high load or in multi-GPU deployments, and these required restarts and manual error handling. GPU utilization was also described as inefficient in some cases because of the lack of multi-user GPU sharing: resources could remain underutilized when per-model allocations were not fully consumed.
Another limitation was the absence of auto-profiling. Model-to-GPU assignment required manual tuning, and no automated estimation was available. On HPC systems, portability introduced further friction. Docker root requirements were incompatible with academic HPC environments, while Apptainer porting was complex because of missing Docker features and shorter compute slots.
These findings clarify that democratized access did not eliminate systems-level constraints. A plausible implication is that eDIF’s significance lies as much in exposing the operational requirements of remote interpretability at institutional scale as in demonstrating that such a service can function reliably enough for sustained research use.
6. Development roadmap, governance, and broader European significance
The roadmap addresses the observed bottlenecks directly. Planned work includes network optimization, output compression, and improved VRAM tracking to address bandwidth and throughput issues. Enhanced error management, retry logic, and push-based alerts are under development. A GPU profiling tool with semiautomatic estimation and dynamic allocation is planned for larger-scale deployment (Guggenberger et al., 14 Aug 2025).
Model support is also slated to expand. The roadmap names newer and European-origin models including Qwen, Gemma3, Salamander, and Teuken, along with hot-swapping and session-based model switching. Strategic exploration includes quantized and sharded models and multimodal systems such as Qwen-2.5-VL. Usability work includes a planned no-code or low-code UI, compatibility with Jupyter and VS Code, modular tools, and integration of high-level analytical packages.
Community and governance mechanisms are treated as part of the infrastructure rather than as ancillary features. Planned measures include enhanced documentation, tutorials, a Discord and GitHub ecosystem, interactive events, token-based authentication, usage quotas, GDPR-aligned data handling, and scalable user management frameworks. Confirmed next steps include continuation of pilot resources at Ansbach, expansion to H100/H200 clusters at Heilbronn, and continued AMD pilots.
The paper frames the broader significance of eDIF in strategic terms. It is presented as filling a critical gap in European AI infrastructure for interpretability, enabling equitable and reproducible research access while fostering pan-European collaboration. By being open, NDIF-compatible, and multi-vendor ready, it is also described as helping reduce dependence on commercial or non-EU providers. The long-term vision is to position eDIF as a cornerstone for responsible, transparent, European AI research across technical, scientific, regulatory, and industrial settings. This suggests that eDIF should be understood not simply as a cluster deployment, but as an institutional model for remote mechanistic interpretability in Europe.
