Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems

Abstract: Modern HPC file systems can contain billions of files and hundreds of petabytes of data, making even simple questions increasingly intractable to answer. Traditional file system utilities such as find and du fail to scale to these sizes. While external indexing tools like GUFI and Brindexer improve query performance, they remain batch-oriented and unsuitable for heterogeneous, rapidly evolving environments. We present Icicle, a scalable framework for continuous file system metadata indexing and monitoring. Icicle maintains a unified, up-to-date, and queryable view of file system state while supporting both periodic snapshot-based ingestion for bulk metadata updates and event-based ingestion for real-time synchronization from production systems such as Lustre and IBM Storage Scale. Built on Apache Kafka and Apache Flink, Icicle provides high-throughput, fault-tolerant, and horizontally scalable ingestion of metadata events into two complementary search indexes, enabling both individual file discovery and aggregate summary statistics by user, group, and directory. This architecture enables efficient support for both coarse-grained administrative queries and interactive analytics over billions of objects. Our experimental evaluation on production-scale HPC datasets demonstrates order-of-magnitude throughput improvements over existing monitoring and indexing approaches, with tunable options for balancing consistency, latency, and metadata freshness.

• The paper introduces a dual-mode framework that combines batch snapshots and real-time event streams for efficient metadata indexing in HPC systems.
• It leverages Apache Kafka, Apache Flink, and mergeable quantile sketches to significantly improve throughput and scalability over traditional approaches.
• The dual-index architecture supports both detailed and aggregate queries, enabling rapid analytics and effective metadata observability across large-scale file systems.

Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems

Motivation and Requirements

Contemporary HPC storage infrastructures routinely span hundreds of petabytes with billions of files, fragmented across heterogeneous, tiered environments. While this scale is foundational for scientific workloads and AI-driven analyses, it creates substantial challenges for metadata observability, discovery, and management. Traditional tools such as find and du are operationally infeasible at this scale due to their poor scalability and prohibitive latency—metadata traversals can take days to complete on modern systems. Batch-oriented indexing systems like GUFI and Brindexer offer improved queryability but remain insufficient for heterogeneous, dynamic HPC deployments where near-real-time views and flexibility are paramount.

The paper introduces the Icicle framework to address these gaps, outlining technical requirements for modern metadata monitoring solutions:

• Mixed-mode ingestion: Efficiently process both periodic, bulk metadata snapshots and real-time event streams.
• Heterogeneity: Integrate across diverse storage backends, supporting multiple vendors and file system architectures.
• Configurable consistency/freshness: Allow operators to tune the tradeoff between latency and resource consumption.
• Multi-granular queries: Support both per-object and aggregate queries, enabling analytics ranging from anomaly detection to usage reporting.

Icicle System Architecture and Metadata Model

Icicle is structured as a horizontally scalable, modular system built atop Apache Kafka and Apache Flink to implement both batch and streaming ingestion patterns. The core architectural elements—snapshot pipeline, event monitor, and unified web interface—collectively maintain comprehensive, up-to-date metadata indexes and expose them to end users and administrators for interactive analysis.

Figure 1: Icicle architecture overview, showing ingestion via snapshots and event streams, streaming aggregation, and unified queryable indexes.

At the data model layer, Icicle maintains two primary classes of indexes:

• Primary index: An authoritative mapping from fully-resolved paths to comprehensive POSIX metadata, enabling fine-grained file- and directory-level queries.
• Aggregate index: Precomputed statistical summaries—counts, quantiles, totals—by user, group, and directory, optimized for high-performance, coarse-grained analytics.

This dual-index architecture is crucial. The primary index ensures correctness and supports low-latency querying for detailed administrative and forensic use cases, while the aggregate index yields instant access to usage distributions and percentiles for large slices of the namespace, addressing typical operational and policy queries.

Ingestion, Aggregation, and Stream Processing

Snapshot Pipeline

The snapshot pipeline efficiently transforms heterogeneous metadata dumps (from GUFI, mmapplypolicy, etc.) into uniform representations, exploiting parallel/distributed map-reduce computation using Apache Flink. The pipeline consists of:

• Primary ingestion: Mapping CSV-encoded snapshot rows into normalized, queryable JSON objects.
• Counting/aggregate computation: Sharding workload and aggregating per principal for scalable, memory-bounded distributional statistics.

Quantile computations leverage mergeable sketches (DDSketch, KLL, ReqSketch, t-Digest), ensuring fixed memory cost and high accuracy even on terabyte-to-petabyte metadata sources. Results demonstrate order-of-magnitude reductions in preprocessing time and index size over naive ingestion, especially when using fine-grained CSV partitioning and hashing to control skew.

Real-Time Event Monitor

The event monitor consumes file system event feeds—e.g., Lustre changelogs and GPFS mmwatch—not only for continuous index maintenance but also to enable near-instant metadata consistency.

The monitor adheres to three-level processing: event ingestion (Kafka/gRPC consumers, format normalization), metadata reduction and path reconstruction (with stateful, rule-based consolidation and cancellation), and emission to update notification backends (either MSK or direct search-engine ingestion). The state manager tracks directory hierarchies in-memory, exploiting parent-child relationships for high-throughput path resolution.

Scalability and Performance Evaluation

Empirical evaluation spans both synthetic and production-scale workloads, leveraging AWS-based Lustre and GPFS testbeds as well as multi-petabyte, billion-file dataset snapshots.

Snapshot Ingestion:

• For full pipeline execution (FS-small, FS-medium, FS-large), the aggregate pipeline is the dominant processing stage due to cross-shard shuffling, but resource scaling (e.g., increasing Flink's KPU allocation) yields significant speedups, especially with sufficient parallel CSV partitioning.
• Aggregate index construction with DDSketch attains sub-1% mean relative value error and eliminates the memory bottlenecks associated with exact quantile aggregation for high-cardinality aggregations.
• Index scaling is linear in file count, with aggregate indexes (user/group/directory) typically <1 GB—even for a billion-file dataset—enabling millisecond-scale analytic queries in the web interface.

Real-Time Monitoring:

• On Lustre, Icicle achieves 32k–41k changelogs/s single-MDT throughput versus ~400–500/s for baselines using synchronous fid2path lookups (FSMonitor), a strong numerical improvement (57–83× faster).
• Scaling to multiple MDTs/clients yields nearly linear throughput scaling; at four MDTs, >23k events/sec is processed.
• GPFS event ingestion supports >300k operations/s when scaling clients and filesets, primarily bottlenecked by consumer throughput and state manager capacity.
• Event reduction strategies (coalescing, cancellation) boost throughput by 10–20% where applicable.

  Figure 2: Filebench throughput scaling on Lustre with increased MDT count using c5a.large/xlarge clients.

  

  Figure 3: Filebench throughput scaling on GPFS as number of clients/filesets increases.

  

  Figure 4: Icicle throughput versus Kafka partitions on GPFS, with and without event reduction; saturation beyond two partitions.

Web Interface and Analytics

Icicle incorporates a web-based front-end exposing both low-level and high-level query modalities:

• Graphical query builder: Construction of pattern-based or attribute-constrained searches over the primary index.
• Interactive summaries and reporting: Visualization of usage, top-k users, directory distributions, and full-text search.
• API access: Both RESTful and Elasticsearch-compatible endpoints for integration with policy engines, dashboards, or workflow orchestrators.

This interface demonstrably reduces operator workload for administrative and compliance tasks (e.g., stale file remediation, permission audits) and enables actionable real-time insights for user support and billing.

Implications and Future Directions

Icicle advances the state of the art in scalable, federated metadata indexing for HPC file systems. The streaming-first design, unification of batch and real-time ingestion, and adoption of efficient quantile sketching set a template for future cross-system indexing efforts, especially as HPC moves toward cloud/elastic deployments and tighter coupling with workflow automation.

The extensible, modular metadata schema facilitates integration with non-POSIX sources (e.g., cloud object stores with IAM policies). Anticipated future developments include:

• Transparent cross-site federation: Aggregating metadata views across institutional/cloud boundaries.
• Integration with anomaly detection and enforcement engines: Real-time triggers for data movement, tiering, and compliance.
• Support for custom, user-extensible analytics: Enabling end users to define novel semantic aggregations or composite queries at scale.

The practical impact is significant for data-intensive science—improved data discoverability, compliance, and lifecycle management can reduce costs, optimize storage utilization, and enhance scientific reproducibility.

Conclusion

Icicle constitutes a robust, high-throughput, and highly configurable framework for unified metadata indexing and monitoring in HPC environments. Its architectural design balances the operational demands of administration, user interactivity, and advanced analytics, demonstrating strong empirical results on both synthetic benchmarks and real-world, petabyte-scale workloads. The adoption of mergeable quantile sketches and distributed, stateful streaming sets a high bar for performance and extensibility, providing new avenues for cyberinfrastructure innovation in large-scale scientific computing.

Reference: "Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems" (2604.10295).