Error propagation for non-smooth operators

Develop fundamentally different error-propagation techniques for non-smooth operators such as ReLU, max, and min, potentially using piecewise local rules near thresholds or propagated distribution-shape information such as skewness and kurtosis.

Background

TOPIQ propagates compression-error bias and variance through compositions of smooth primitive operators using a second-order Taylor expansion. The paper identifies truncation-style operators, including ReLU, max, and min, as outside the current framework because they lack useful second derivatives. Extending propagation to these operators is explicitly identified as an open direction, with piecewise threshold rules and richer statistical summaries suggested as possible approaches.

References

Standard ReLU ($\alpha{=}0$) is not used in this paper; a concentrated-input bound analogous to Thm.~\ref{thm:product-bwd-bound} is left for future work.

— Direct Message Approximation (DMA): A Consistency-Based Framework for Tractable Approximate Inference on Factor Graphs  (2609.29466 - Herbrich et al., 24 Sep 2026) in Appendix B, Section 'Theorem Coverage by Factor and Direction', footnote marked $^{\dagger}$

Several directions remain open. On the operator side, the current second-order Taylor expansion targets smooth functions; truncation-style operators such as ReLU, max, and min lack useful second derivatives and would require fundamentally different propagation techniques, such as piecewise local rules near thresholds or enriching the propagated summaries with distribution-shape information (e.g., skewness and kurtosis).

— TOPIQ: Statistical Error Propagation for Quantity-of-Interest Prediction under Lossy Compression  (2608.26912 - Liu et al., 27 Aug 2026) in Section Conclusion and Future work