Experimental validation of scaling PP/TP for very large models and CP for very long sequences

Experimentally validate the expected benefits of further increasing pipeline or tensor parallelism for extremely large models and increasing context parallelism for very long sequences, quantifying their impact on Model FLOPs Utilization, throughput, and memory usage in large-scale settings.

Background

The empirical study identifies best and worst hybrid strategies for LLaMA and Mamba at 1B and 7B scales. The authors suggest that larger degrees of model parallelism (PP/TP) should help for extremely large models, and that higher context parallelism (CP) should help for very long sequences. However, they explicitly state that due to resource limitations, these scenarios have not been experimentally validated and are left for future work.

References

Further increases in the degree of model parallelism (PP/TP) are expected to benefit extremely large models. Increasing context parallelism (CP) is expected to be advantageous for very long sequences; however, due to resource limitations, experimental validation of these scenarios is left for future work.

— Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide  (2602.09109 - Amer et al., 9 Feb 2026) in Section 6.4, Summary of Empirical Insights (end of section)

Nevertheless, determining the optimal hybrid parallelization strategy for a given training scenario remains an open research problem.

— The Life of a Token: from Words to Bits on the Wire  (2609.19924 - Avesani et al., 17 Sep 2026) in Section 5, subsection “Hybrid Parallelism and communication summary”

Finally, our benchmarks cover a representative but not exhaustive set of QKV tensor configurations; a systematic sweep over batch sizes and query/key-value head ratios beyond those reported in \S\ref{sec:exp-cp-scalability} and \S\ref{sec:exp-e2e} is left to future work.

— FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention  (2610.17901 - Wang et al., 13 Oct 2026) in Appendix, Section "Limitations and Broader Impact" (sentence beginning "Finally, our benchmarks...")