Optimization of Model Splitting, Placement, and Chaining for Multi-hop Split Learning and Inference

Abstract: Service Function Chaining (SFC) establishes efficient communication paths by ensuring that traffic traverses a predefined sequence of network functions in a specified order to meet particular service requirements. Inspired by this concept, we have proposed an SFC-based architecture for multi-hop split learning (MSL) and split inference (MSI), facilitating distributed AI applications to effectively route smashed data across multi-hop networks. However, the multi-hop environment presents new challenges, including (1) determining optimal cut points, (2) deploying split sub-models on appropriate computing nodes, and (3) routing smashed data through the underlying communication networks while adhering to service requirements. To address these challenges, we formulate an Integer Linear Programming (ILP) model to jointly optimize model splitting, placement, and chaining (data routing) in the SFC-based MSL/MSI architecture, aiming to minimize end-to-end inference or training latency. Additionally, we propose a Block Coordinate Descent (BCD)-based heuristic algorithm to efficiently solve the problem. Comprehensive evaluations demonstrate the effectiveness and characteristics of the proposed formulation and algorithm.

• The paper introduces a joint ILP formulation and a BCD-based heuristic to minimize end-to-end latency by optimizing model splitting, placement, and chaining for multi-hop split learning and inference.
• It demonstrates near-optimal performance with the heuristic achieving latencies within 1% of the ILP benchmark while drastically reducing computational overhead.
• The results highlight scalability for distributed AI deployments, enabling efficient model partitioning across edge and cloud resources under realistic resource constraints.

Optimization of Model Splitting, Placement, and Chaining for Multi-hop Split Learning and Inference

Introduction and Context

This work systematically addresses joint optimization for split learning (SL) and split inference (SI) in multi-hop, distributed environments, extending previous single-hop SL/SI paradigms to a general Service Function Chaining (SFC)-based architecture. The primary contributions are a rigorous Integer Linear Programming (ILP) formulation for minimizing latency via joint model splitting, placement, and chaining (data routing), and a Block Coordinate Descent (BCD)-based heuristic that achieves near-optimal performance with vastly reduced computational complexity.

Traditional SL/SI approaches operate under the premise of direct client-server communications, failing to account for the complex routing and placement decisions inherent in multi-hop, resource-constrained networks. The SFC-based MSL/MSI (Multi-hop Split Learning/Multi-hop Split Inference) framework conceptualizes partitioned sub-models as chained network functions over arbitrary directed graphs representing physical networks, bringing the problem into alignment with network function virtualization literature but introducing unique challenges due to neural model resource demands and chain execution order.

System Model and Problem Formulation

SFC-based MSL/MSI Architecture

The physical network is modeled as a directed graph $\mathcal{G} = (V, E)$, with computing resources (CPU and GPUs) distributed across vertices. Each node is annotated with finite memory and storage, and links carry asymmetric bandwidth and propagation delays. Sub-model placements are encoded by an augmented network $\mathcal{G}^+$, adding 'imaginary' nodes representing candidate sub-model deployments and their feasible mappings.

A service chain request $R = (id, s, d, b, mode)$ specifies a global model $F$ (composed of $L$ layers), source/dest nodes, mini-batch size $b$, and execution mode (inference or training). The model is to be partitioned (split) at optimal points ($K$ splits), with each sub-model assigned to feasible nodes.

Figure 1: System model of SFC-based MSL/MSI, depicting split points, sub-model assignments, and routing via virtualized nodes and links.

Joint Optimization via ILP

The ILP objective is to minimize total end-to-end latency (aggregate computation and communication, potentially bidirectional for training), subject to:

• Binary variables for split points ($y_{\hat{v}_k, l}$) and placement/routing ($x_{i, j}^k$)
• Resource constraints: memory, storage, bandwidth, and per-node model processing capability
• Layer assignment and contiguity: every layer is allocated to exactly one sub-model, and splits must yield contiguous model fragments
• Path constraints: sub-paths must begin and terminate in correct order, traversing selected nodes/links only

Novel constraints ensure that model assignment adheres strictly to feasible resource envelopes and sequential order across splits. The objective function intricately models both computation (layer-wise FLOPs, I/O) and communication (activation/gradient size, transmission delays, propagation delays per path segment).

Figure 2: Relationship between model splitting variable $y_{\hat{v}_k, l}$ and valid model segmentation.

Algorithmic Framework

While ILP delivers ground-truth optimality, its scaling limitations (combinatorial explosion in binary variables with $\mathcal{G}^+$0, $\mathcal{G}^+$1, $\mathcal{G}^+$2) necessitate heuristics for practical deployment. The BCD-based method alternates between optimizing model splitting (minimizing per-segment latency for a fixed path/placement via $\mathcal{G}^+$3-Sequence Segmentation) and placement/chaining (optimizing placement given a fixed split using an augmented DFTS/Dijkstra search). Iteration continues until convergence (change in latency below threshold $\mathcal{G}^+$4), guaranteeing local optimality in practice.

Evaluation and Results

Experimental Setting

• Model: ResNet101 ($\mathcal{G}^+$5), metrics measured per building block and executed on a mixed CPU/GPU NSFNET topology (14 nodes, 42 links)
• Parameters: Mini-batch size $\mathcal{G}^+$6, chain length $\mathcal{G}^+$7
• Benchmarks: Full ILP ('oracle'), BCD, computation-minimizing (COMP-MS), communication-minimizing (COMM-MS)
• Constraints: Practical per-node resources, realistic PyTorch-computed FLOPs, activations, gradients

Latency/Scaling Results

Both inference and training latencies are strongly superlinear in mini-batch size $\mathcal{G}^+$8, with communication overheads dominating at higher $\mathcal{G}^+$9 due to cascading smashed data transmissions. Notably, the optimal $R = (id, s, d, b, mode)$0 is not monotonic: minimal $R = (id, s, d, b, mode)$1 is best for small $R = (id, s, d, b, mode)$2, but $R = (id, s, d, b, mode)$3 is empirically superior for large $R = (id, s, d, b, mode)$4 or training, balancing computation parallelism against communication bloat.

Figure 3: Inference latency per batch across K and b, highlighting the computation/communication tradeoff landscape.

Figure 4: Training latency per batch dissects computation, transmission, and propagation contributions.

Model splitting points (Fig.~6 and Fig.~7) adapt to minimize the largest per-node delay, with early layers (large smashed data) typically retained on resource-constrained ingress nodes, while deep-layer, parameter-rich blocks migrate to GPUs nearer the egress. This behavior is systematically observed for both inference and training, but training imposes tighter memory constraints, amplifying the need for careful split point selection.

Figure 5: Optimal service path and model splitting for MSI ($R = (id, s, d, b, mode)$5, $R = (id, s, d, b, mode)$6) shows dominant allocation of early layers to source CPU and mid-late layers to GPUs.

Figure 6: Training setting ($R = (id, s, d, b, mode)$7, $R = (id, s, d, b, mode)$8) shifts more layers to GPUs due to computation/memory bottlenecks.

Heuristic Performance

BCD achieves latencies within 1% of ILP solutions across all configurations, at $R = (id, s, d, b, mode)$9 lower runtime. COMP-MS and COMM-MS, in contrast, incur major penalties: COMP-MS overlooks communication, leading to oversized smashed activations; COMM-MS overloads ingress nodes, yielding excessive computation and memory usage—undesirable in realistic resource-constrained devices.

Scalability analysis underscores the BCD approach: as $F$0 and $F$1 increase, execution time for BCD grows polynomially, while ILP scaling is exponential, making BCD the only practical choice for large-scale deployments.

Figure 7: Execution time as a function of $F$2, demonstrating the scalability gulf between ILP and BCD.

Implications and Future Work

The ILP abstraction captures the fundamental tradeoff in SFC-based distributed inference/training between latency-critical computation and inter-node communication, accounting for both placement flexibility and heterogeneous resource constraints. The BCD heuristic's performance indicates that near-optimal chain construction is achievable well beyond small, toy networks, laying the groundwork for scalable edge/cloud collaborative learning infrastructures.

Practically, these results facilitate the deployment of large models on federated or resource-constrained edge domains, reducing latency without requiring homogeneous hardware. The methodology can be extended to incorporate privacy preservation (e.g., differentially private split assignment), multi-path chain layouts for load balancing/fault tolerance, and federated model update or aggregation in hybrid FL-SL settings.

Conclusion

This study presents a rigorous and general joint optimization framework for multi-hop split learning and inference over arbitrary networks. The presented ILP models the complex interplay of computation, communication, and resource constraints, while the BCD-based algorithm demonstrates state-of-the-art scalability with near-optimal quality. This work provides both a theoretical foundation and a practical algorithmic toolset for scalable AI deployment in heterogeneous, distributed settings.