Distributed Semi-Speculative Parallel Anisotropic Mesh Adaptation

Abstract: This paper presents a distributed memory method for anisotropic mesh adaptation that is designed to avoid the use of collective communication and global synchronization techniques. In the presented method, meshing functionality is separated from performance aspects by utilizing a separate entity for each - a multicore cc-NUMA-based (shared memory) mesh generation software and a parallel runtime system that is designed to help applications leverage the concurrency offered by emerging high-performance computing (HPC) architectures. First, an initial mesh is decomposed and its interface elements (subdomain boundaries) are adapted on a single multicore node (shared memory). Subdomains are then distributed among the nodes of an HPC cluster so that their interior elements are adapted while interface elements (already adapted) remain frozen to maintain mesh conformity. Lessons are presented regarding some re-designs of the shared memory software and how its speculative execution model is utilized by the distributed memory method to achieve good performance. The presented method is shown to generate meshes (of up to approximately 1 billion elements) with comparable quality and performance to existing state-of-the-art HPC meshing software.

• The paper introduces DM_CDT3D, a distributed anisotropic tetrahedral mesh adaptation method that combines speculative cavity-level parallelism with asynchronous runtime support while eliminating collective communication and global synchronization.
• The method generates a billion-element mesh in 3.46 hours on 512 cores, outperforming shared-memory CDT3D and NASA refine in key large-scale tests while maintaining metric conformity comparable to the shared-memory implementation.
• The approach improves scalability through pre-adapted frozen interfaces, pseudo-active elements, point locking, and buffer zones, but interface processing on one node, sequential preprocessing, load imbalance, and parameter tuning remain important limitations.

This paper presents a distributed memory method for anisotropic tetrahedral mesh adaptation that eliminates collective communication and global synchronization from the meshing pipeline (2602.15204). The work combines two previously independent components: CDT3D, a shared-memory anisotropic adaptation code exploiting fine-grain speculative parallelism at the cavity level, and PREMA, a parallel runtime system providing asynchronous communication, mobile objects, and load balancing within a globally addressable namespace. The central design decision is an a priori interface adaptation strategy: subdomain interfaces are fully adapted on a single multicore node before distribution, after which interface elements are frozen while interior elements are adapted in parallel across cluster nodes.

Motivation and design context

The work is positioned against NASA's CFD Vision 2030 identification of mesh generation as a workflow bottleneck, and within the authors' "Telescopic Approach," which assigns different meshing kernels to different levels of the memory hierarchy. A recurring finding from prior "functionality-first" black-box parallelization efforts (VGRID, TetGen, AFLR, Netgen/PMSH, PODM) is that codes not designed for scalability cannot be integrated into parallel frameworks without substantial redesign; notably, a distributed Delaunay implementation spent more than 50% of its runtime migrating data and ran 7× slower than its shared-memory counterpart. This paper embraces that lesson: rather than treating CDT3D as a black box, the authors modify it deliberately.

The avoidance of collectives is motivated by the DoE Exascale Computing Project studies showing that although most MPI calls are point-to-point, runtime is dominated by collectives (MPI_Allreduce, MPI_Alltoall), with many proxy applications spending over half their time in communication. State-of-the-art anisotropic adaptors such as Feflo.a and refine rely on re-partitioning, all-to-all global ID assignment, and implicit synchronization points whose cost grows with core count. The presented method avoids these entirely, at the cost of constraining interface adaptation to a single node — a trade-off that dominates the scalability analysis later in the paper.

Algorithmic structure

The distributed method proceeds in seven steps: coarse mesh generation at a fraction of target metric complexity; PQR decomposition; interface preprocessing; interface adaptation on one node; making subdomains simply connected; distribution; and interior adaptation across nodes. Conformity is maintained because interface points are preemptively locked and their elements deactivated during interior adaptation.

The key mechanism enabling this separation is a combination of three techniques layered onto CDT3D's speculative execution model:

• Pseudo-active/pseudo-inactive classification: elements processed during interface adaptation are marked pseudo-inactive so they are not re-processed during interior adaptation, avoiding redundant work.
• Preemptive point locking: points defining frozen regions are locked before operations begin, which controls edge collapse and vertex smoothing (operations that would otherwise modify inactive elements) via CDT3D's existing lock-based correctness model.
• Buffer zones around frozen interfaces: point creation adjacent to pseudo-inactive elements was found to prevent convergence, generating slivers; prohibiting insertion on elements sharing a vertex with frozen elements resolves this.
• Order-of-operations discipline: edge collapse serves as pre-refinement and post-refinement in CDT3D's pipeline; skipping post-refinement collapse during interface adaptation prevents interior adaptation from regenerating already-removed edges.

The ablation study is striking: respecting order of operations alone reduces interface adaptation time by 4.25× (321 s to 75 s), while adding the pseudo-active modification and preemptive locking reduces per-subdomain interior adaptation times by roughly 1.6× for essentially identical output meshes (~49M tets per subdomain). Local reconnection — the bottleneck operation — is approximately halved. This directly quantifies the penalty of naive black-box usage and substantiates the paper's claim that integration requires code-level understanding of each operation's purpose.

Two additional infrastructure details deserve note: subdomains must be simply connected before distribution since CDT3D assumes manifold input, and a sequential $O(nm^2)$ face-traversal routine enforces this after interface adaptation; and duplicate data between subdomains are tracked via global identifiers assigned at decomposition, enabling final mesh merging without all-to-all communication. The authors also document an earlier a posteriori variant (interiors first, then iterative interface shifts with data migration), which incurs 30–35% communication overhead and an implicit master/worker synchronization per shift iteration; it is shown inferior to the a priori approach throughout the evaluation.

Evaluation

Experiments were run on Wahab (dual Xeon Gold 6148, 32 cores/node, up to 512 cores) and Anvil (dual EPYC 7763, 96 cores/node, up to 1,536 cores). Comparisons are against the original shared-memory CDT3D and NASA's refine, using UGAWG-style metrics: edge length in the target metric (unit-length optimal) and mean ratio shape measure (1.0 optimal).

For the delta wing geometry at 10 million complexity (~100M elements), the a priori method achieves the best runtime up to 256 cores (23 min vs. refine's 31 min on Wahab); refine is slightly faster at 512 cores (21 vs. 22 min). On Anvil, DM_CDT3D outperforms refine at every tested configuration (6 min vs. 15 min at 384 cores), and refine crashes at 768 cores and hangs at 1,536 cores. At 20 million complexity, refine fails sequentially and on 2 cores, and DM_CDT3D again wins through 256 cores (50 min both at 512). For the cube geometry scaled to 100 million complexity (~1B elements), the headline result emerges: DM_CDT3D generates the billion-element mesh in 3.46 hours on 512 cores versus refine's 5.98 hours and SM_CDT3D's 8.12 hours on 32 cores. Adaptation rates reach ~232,523 elements/s (DM_CDT3D, 384 cores on Anvil) versus refine's 124,555 on the same benchmark. These numbers support the ECP-derived thesis that collective-communication overhead becomes dominant at scale — though the caveat is that refine generates 10–15% more elements and exhibits somewhat better quality histograms, so part of its runtime disadvantage reflects a different operating point on the quality/cost trade-off.

Quality results show DM_CDT3D matching SM_CDT3D's metric conformity distributions across all cases, satisfying stability and weak reproducibility (meshes vary with core count but remain of similar quality). refine consistently achieves the best minimum mean ratio and tightest edge-length deviation, so the proposed method does not dominate on quality — a concession the authors state plainly.

Limitations and open questions

Several constraints bound the reported performance. Interface adaptation is confined to one multicore node, causing its share of end-to-end time to grow from ~20% (32 cores) to over 40% (256 cores); this is the principal scalability ceiling, and the authors identify a fully speculative simultaneous interior/interface scheme as the needed remedy. Scalability is further limited by sequential components (simply-connected enforcement, data structure conversion) and by load imbalance at 16 nodes where each node receives exactly one subdomain; PREMA's load balancing is disabled because overdecomposition is not yet used, and overdecomposition itself conflicts with the requirement that the coarse mesh be dense enough for decomposition. Parameter settings (decomposition configuration, unlocked layer counts, operation schedules) were found empirically and are geometry- and complexity-dependent — adapting the same delta wing at doubled complexity required a different decomposition and additional quality improvement iterations — motivating a machine-learned parameter model as future work. PQR also produces highly uneven subdomains (20K–2M tets under a 4×2×2 split). Finally, robustness in the presence of CAD-based geometry via EGADS remains unaddressed, and validation inside a full CFD simulation pipeline has not yet been performed.

Conclusion

The paper demonstrates that an anisotropic, tightly-coupled adaptation kernel can be extended from single-node execution to ~1-billion-element generation on hundreds of cores without collectives or global synchronization, provided the kernel exposes its lock-based speculative execution model for external control. The quantitative ablations make a concrete case that black-box integration wastes nearly half the runtime, and the end-to-end comparisons show competitive-to-superior runtimes relative to refine, particularly on large meshes and wide nodes. The remaining questions — parallelizing interface adaptation beyond one node, eliminating the sequential simply-connected routine, automating parameter selection, and demonstrating solver-integrated value — define the immediate agenda for this line of work.