Graphical Calculus for products and convolutions (1903.01366v1)

Abstract: Graphical calculus is an intuitive visual notation for manipulating tensors and index contractions. Using graphical calculus leads to simple and memorable derivations, and with a bit of practice one can learn to prove complex identities even without the need for pen and paper. This manuscript is meant as a demonstration of the power and flexibility of graphical notation and we advocate exploring the use of graphical calculus in undergraduate courses. In the first part we define the following matrix products in graphical language: dot, tensor, Kronecker, Hadamard, Kathri-Rao and Tracy-Singh. We then use our definitions to prove several known identities in an entirely graphical way. Despite ordinary proofs consist in several lines of quite involved mathematical expressions, graphical calculus is so expressive that after writing an identity in graphical form we can realise by visual inspection that it is in fact true. As an example of the intuitiveness of graphical proofs, we derive two new identities. In the second part we develop a graphical description of convolutions, which is a central ingredient of convolutional neural networks and signal processing. Our single definition includes as special cases the circular discrete convolution and the cross-correlation. We illustrate how convolution can be seen as another type of product and we derive a generalised convolution theorem. We conclude with a quick guide on implementing tensor contractions in python.

• The paper introduces a visual notation that uses mediating tensors to simplify tensor contractions and proofs of complex identities.
• It details graphical representations for six common tensor products and extends the approach to depict convolution operations intuitively.
• The study offers practical guidance for translating graphical tensor diagrams into efficient numpy.einsum code for real-world applications.

The paper "Graphical Calculus for products and convolutions" (1903.01366) presents graphical calculus as an intuitive visual notation for manipulating tensors and understanding operations like matrix products and convolutions. The core idea is to represent tensors as shapes with wires representing indices. Connecting wires signifies tensor contraction (summation over repeated indices), while unconnected wires represent the indices of the resulting tensor. This visual approach aims to make complex tensor algebra more accessible and memorable, facilitating easier derivation and proof of identities.

The paper introduces two fundamental "mediating" tensors crucial for defining various operations graphically:

1. The Kronecker tensor ($\delta$): Represented as a black circle with wires. A rank-$N$ Kronecker tensor $\delta_{i_1 i_2 \dots i_N}$ is 1 if all indices $i_1, \dots, i_N$ are equal, and 0 otherwise. Graphically, connecting multiple wires to a single Kronecker tensor implies these indices must be equal. This tensor can be used to extract diagonals ($A_{ij}\delta_{ijk}$ extracts the diagonal of matrix $A$), compute traces ($A_{ij}\delta_{ij}$ computes $\mathrm{Tr}(A)$), or represent identity operations.
2. The Vectorization tensor ($\gamma$): Represented as a triangle. It serializes multiple indices into a single index, effectively reshaping a tensor (like a matrix) into a vector. $\gamma_{i_1i_2\dots i_N, m}$ is 1 if $m$ is the combined index value derived from $i_1, \dots, i_N$ according to a specific serialization formula ($$m = \sum_{j=1}^Ni_j\left(\prod_{\ell=1}^{j-1}I_\ell\right)$$), and 0 otherwise. Graphically, multiple wires entering the flat side of the triangle merge into a single wire exiting the pointed side.

Using these mediating tensors and the basic rules of connecting wires (contraction), the paper defines the graphical representations of six common matrix/tensor products:

• Dot Product: $A_{ij}B_{jk}$ is shown by connecting the 'j' wires of $A$ and $B$. The visual connection directly represents the index contraction.
• Tensor Product: $A_{ij}B_{kl}$ is represented by simply drawing the two tensors ($A$ and $B$) next to each other with all their wires exposed, indicating no index contraction.
• Kronecker Product: $$(A \otimes B)_{mn} = A_{ij}B_{k\ell}\gamma_{ik,m}\gamma_{j\ell,n}$$. Graphically, this involves the tensor product of $A$ and $B$, followed by connecting the respective row indices ('i' and 'k') to a vectorization tensor and the respective column indices ('j' and 'l') to another vectorization tensor.
• Hadamard Product: $(A \circ B)_{ij} = A_{ij}B_{ij}$. This is defined using the Kronecker tensor: $A_{ij}B_{kl}\delta_{ik}\delta_{jl}$. Graphically, it involves drawing $A$ and $B$ and connecting their corresponding row wires with a rank-2 Kronecker tensor (or just connecting the wires directly, as $\delta_{ik}$ represents equality) and their column wires similarly.
• Kathri-Rao Product: $$(A \odot B)_{mn} = A_{ij}B_{k\ell}\gamma_{ik,m}\delta_{j\ell n}$$. This involves a Kronecker product on row indices and a Hadamard product on column indices. Graphically, this combines the vectorization of row wires (using $\gamma$) and the connection of column wires (using $\delta$ or direct connection).
• Tracy-Singh Product: $$(A \star B)_{mn} = A_{ijk\ell}B_{pqrs}\gamma_{ipkr,m}\gamma_{jq\ell s,n}$$. This is described as a double Kronecker product using vectorization tensors to merge outer and inner indices. Graphically, it shows a more complex arrangement of vectorization tensors combining indices from two higher-rank tensors.

A significant strength highlighted is the use of graphical calculus for proving tensor identities. The visual rules for manipulating the diagrams often make complex identities immediately obvious or dramatically simplify the derivation process. The paper demonstrates proofs for various properties, including bisymmetry relations between different products (e.g., $$(A \otimes B) \circ (C \otimes D) = (A \circ C) \otimes (B \circ D)$$) and identities involving vectorization (e.g., $\mathrm{col}(ABC) = (C^T \otimes A)\mathrm{col}(B)$ for non-diagonal $B$). The visual nature allows spotting patterns like bisymmetry, often related to swapping Kronecker and vectorization tensors (Fig. 6).

The paper extends graphical calculus to convolutions, crucial operations in signal processing and convolutional neural networks. It defines a generalized convolution tensor $\chi_{ijk}^{(\pm\pm\pm)}$, which is 1 if $\pm i \pm j \pm k = 0 \pmod D$ and 0 otherwise, where the signs determine the type of convolution. Graphically, this is a diamond shape with three wires. Contracting this tensor with two vectors $(\mathbf{a}, \mathbf{b})$ implements their convolution or cross-correlation, depending on the signs. This shows how convolution can be viewed as another type of tensor product (Fig. 13). The definition naturally extends to higher-rank tensors like matrices (Fig. 15).

The graphical notation also provides insight into the convolution theorem. The standard theorem $\widehat{f(x)g(x)} = \widehat f(k)*\widehat g(k)$ translates to a relationship involving Hadamard product and convolution in the graphical representation (Fig. 16). The paper shows that this identity arises from contracting the convolution tensor with Fourier transform matrices (or their inverses), resulting in a Kronecker tensor (Fig. 17). This generalizes: indices on the convolution tensor with opposite signs are contracted by opposite Fourier transforms (Forward vs. Inverse Fourier Transform matrices).

For practical implementation, the paper strongly advocates using graphical calculus drawings as a direct guide for writing code, specifically using Python's numpy.einsum function. einsum allows performing tensor contractions using Einstein summation convention, either by specifying the index notation in a string or by providing tensors and lists of integers representing their indices. The graphical diagram of a desired tensor contraction can be directly translated into the index notation required by einsum. For example, matrix multiplication $A_{ij}B_{jk} \rightarrow C_{ik}$ becomes np.einsum("ij,jk->ik", A, B) or np.einsum(A, [0, 1], B, [1, 2], [0, 2]). The graphical representation clarifies which indices are contracted (repeated in the notation) and which remain free (appear in the output specification).

In summary, the paper positions graphical calculus as a powerful pedagogical and practical tool for linear algebra and tensor operations. It provides a visual language for defining diverse products and convolutions, simplifying the understanding and derivation of complex identities, and offering a reliable method for translating tensor operations into efficient code using libraries like numpy.einsum. The approach emphasizes visual intuition as a complement or alternative to traditional index notation, making advanced tensor manipulations more accessible.