DDCNN V2: Efficient Domain Decomposition CNN-DNN

• DDCNN V2 is a deep learning framework that applies domain decomposition with iterative feedback and adaptive partitioning for parallel CNN-DNN training.
• It segments input images into overlapping subdomains processed by local CNNs, whose outputs are aggregated by a coarse DNN, enabling scalability and speedup.
• The approach leverages parallel training, parameter sharing, and iterative coupling to improve accuracy while significantly reducing training time in image recognition tasks.

A Domain Decomposition-Based CNN-DNN Architecture, known as DDCNN, is a two-level model-parallel training framework for deep learning architectures, specifically convolutional neural networks (CNNs) and deep neural networks (DNNs) applied to complex image recognition tasks. DDCNN decomposes the input domain into subdomains, each processed by a dedicated "local" CNN, whose outputs are subsequently aggregated by a "coarse" DNN, yielding a final global prediction. This approach is motivated by the scalability challenges of single-model deep networks trained on ever-expanding datasets and hardware, aiming to achieve both acceleration of training and improved accuracy by leveraging parallelism inspired by domain decomposition methods in numerical analysis (Klawonn et al., 2023).

1. Domain Decomposition and Data Splitting

Every sample, such as an input image $X \subset \Omega$, is partitioned into $N = p^d$ subdomains $\{\Omega_i\}_{i=1}^N$ for $d=2$ or $3$ (2D or 3D data), with $p$ and overlaping parameter $\delta \geq 0$ fixed per spatial dimension. Two primary tiling patterns are supported:

• Type-A (Regular): Regular, grid-aligned subdomains with optional pixel/voxel overlap to increase shared local context. For 2D images,

$$\Omega_{i,j} = \Bigl[ (i-1)\frac{H}{p}-\delta,\,i\frac{H}{p}+\delta \Bigr] \times \Bigl[ (j-1)\frac{W}{p}-\delta,\,j\frac{W}{p}+\delta \Bigr],\quad i,j=1,\dots,p$$

truncated to valid image indices.

• Type-B (Mixed): Enriches with overlapping tiles of larger size (e.g., $2 \times 2$) to increase context.

Each $X$ is thus split into subimages $N = p^d$0, with boundaries adjusted and zero-padding as required. Overlapping regions result in pixel/voxel repetition across multiple subimages.

2. Local CNN Subnetworks

Each subimage $N = p^d$1 is mapped by an independently trained local CNN $N = p^d$2 to a class probability vector $N = p^d$3 ($N = p^d$4 = number of classes). For the 2D case (CIFAR-10, TF-Flowers), the architecture follows a compact variant of VGG-style CNNs:

• Input: $N = p^d$5
• Three convolutional blocks (filters: 32, 64, 128); each includes:
  • Two Conv2D layers ($N = p^d$6), ReLU, and MaxPool ($N = p^d$7)
  • Dropout (20%) after each block
• Flatten $N = p^d$8 Dense layer ($N = p^d$9 units, $\{\Omega_i\}_{i=1}^N$0 proportional to image area)
• Dense layer with $\{\Omega_i\}_{i=1}^N$1 units + softmax

For 3D data (e.g., chest CT), layers become Conv3D/MaxPool3D and binary sigmoid prediction. Each local CNN is trained on its assigned subdomain independently.

3. Output Aggregation via a Coarse DNN

The collection of outputs $\{\Omega_i\}_{i=1}^N$2 from the local CNNs is concatenated into a "supervector" $\{\Omega_i\}_{i=1}^N$3:

$\{\Omega_i\}_{i=1}^N$4

A fully-connected aggregation DNN $\{\Omega_i\}_{i=1}^N$5 computes the global prediction:

$\{\Omega_i\}_{i=1}^N$6

Typical aggregation networks use 4 hidden layers (sizes: 128, 64, 32, $\{\Omega_i\}_{i=1}^N$7) with ReLU/nonlinearities and optional dropout. This coarse net is trained using cross-entropy loss, frozen local CNN weights, and class labels.

4. Model Training Workflow and Parallelization

The DDCNN framework enables a highly parallel training regime:

1. Subimage Extraction: Each training sample is split into $\{\Omega_i\}_{i=1}^N$8 subimages.
2. Local Stage: All local CNNs $\{\Omega_i\}_{i=1}^N$9 train independently and in parallel on their respective subdomain data using standard optimizers (e.g., Adam, SGD).
3. Output Sync: Post local convergence, compute $d=2$0 for all samples, generating datasets for the coarse net.
4. Coarse Stage: Freeze all local CNN weights; train only the coarse DNN on the assembled supervectors $d=2$1 and sample labels.
5. Inference: For new samples: crop to subimages, evaluate all local CNNs, concatenate, and evaluate aggregation DNN.

No inter-process communication occurs during local CNN training beyond the final output synchronization. The approach is thus resilient to hardware heterogeneity and straggler processes.

5. Empirical Performance and Speedup

DDCNN demonstrates substantial speed-ups and accuracy retention across diverse image recognition tasks. Example results:

Dataset & Network	Global CNN Acc (%)	Avg Local Acc (%)	DDCNN Global Acc (%)	Speedup
CIFAR-10 VGG3, $d=2$2 overlap	78.2	59.8	79.7	Not given
TF-Flowers $d=2$3 non-overlap	64.6	55.9	84.7	$d=2$4
Yale Faces $d=2$5 non-overlap	99.3	56.0	97.6	Not given
Chest CT $d=2$6 overlap	76.7	63.5	89.9	$d=2$7

In the most challenging cases (large 3D volumes, $d=2$8), DDCNN reduces wall-clock training time by up to $d=2$9-$3$0 on 8×V100 clusters, while matching or exceeding the monolithic model's test accuracy (Klawonn et al., 2023).

6. Bottlenecks and Proposed Extensions for DDCNN V2

Several key limitations of the initial DDCNN (v1) are identified:

• Absence of feedback from the global coarse DNN to local CNNs (non-iterative).
• Each local CNN trained from scratch in isolation, without parameter sharing or transfer.
• Rigid spatial and uniform image partitioning—no adaptive or content-aware domain splitting.

Authors propose the following extensions for a plausible "DDCNN V2":

• Iterative Schwarz-type coupling: Enable two-way communication, where coarse DNN outputs can guide retraining of local CNNs, potentially improving global coherence.
• Hierarchical/multilevel decompositions: Insert intermediate aggregation layers ("sub-coarse" nets), enabling multi-scale data assimilation.
• Learnable decomposition: Utilize data-driven, perhaps attention-based, mechanisms to segment images into more semantically meaningful or functionally optimal subdomains.
• Parameter sharing: Introduce schemes to share early-layer parameters across local CNNs, reducing memory footprint and potentially enabling greater regularity.
• Asynchronous/adaptive training: Implement straggler-tolerance and locally adaptive stopping criteria, further leveraging hardware.
• Task-specific extensions: Adapt aggregation procedures for segmentation/detection or incorporate alternative heads.
• Optimized hardware mapping: Adapt subdomain-to-processor assignment dynamically to maximize load efficiency based on overlap-aware data partitioning.

A plausible implication is that realizing these extensions could yield further speedups and better accuracy-hardware trade-offs, particularly on modern multi-GPU/TPU clusters and on more complex, large-scale vision tasks (Klawonn et al., 2023).