---
title: Locally Refined Convolution in 3D and Speech
url: https://www.emergentmind.com/topics/locally-refined-convolution-lrc
type: topic
---

# Locally Refined Convolution in 3D and Speech

Searching arXiv for relevant papers on “Locally Refined Convolution” and related uses of the term.
Locally Refined Convolution (LRC) denotes a point-cloud feature-learning operator introduced in LRC-Net to encode fine-grained contexts within local regions of a 3D point set by applying variable-size convolution filters over multi-scale local-region features [2003.08240]. In that formulation, a point cloud is treated not as an unordered bag of points, but as a collection of local regions whose internal geometry and mutual spatial arrangement both matter. The operator appears as the core of an intra-region context encoding module, and is paired with an inter-region context encoding module that integrates spatial relationships among local regions based on spatial similarity measures. Experimental results reported for LRC-Net show that this combination is competitive with state-of-the-art methods in shape classification and shape segmentation applications [2003.08240]. The same term, “locally refined convolution,” was later used in a distinct speech-enhancement architecture, LORT, where it denotes a local-detail modeling block for time-frequency spectrogram refinement rather than a point-cloud operator [2509.23832].

## 1. Conceptual motivation and problem setting

LRC was introduced to address limitations of point-cloud networks that use fixed-size CNN or MLP operators together with simple max pooling over local features [2003.08240]. The motivating premise is that point clouds are irregular and sparse, unlike images, so traditional CNN assumptions about a fixed spatial neighborhood with a regular grid do not hold naturally. In the formulation used by LRC-Net, many prior methods either use fixed-size filters that are too rigid for irregular, sparse point sets or use permutation-invariant pooling that collapses local structure too aggressively [2003.08240].

Within this framework, the central claim is that a single fixed-size filter is insufficient because the geometry visible inside a local region depends on how densely and how sparsely points are sampled. LRC therefore uses variable-size convolution filters over multi-scale local-region features so that the network can adapt to irregular point distributions by considering multiple scale areas around each centroid [2003.08240]. The intended effect is to capture fine-grained geometry inside local regions, correlations across different scales within a region, and spatial relationships among neighboring local regions.

A plausible implication is that LRC should be understood less as a direct analogue of image convolution than as a structured operator over learned local-scale descriptors. In LRC-Net, the convolution is not applied to raw 3D coordinates on a grid; it is applied to the feature map formed by multiple neighborhood scales inside each local region [2003.08240].

## 2. Local-region construction and feature preparation

The input point cloud is defined as
\[
\mathbf{P} = \{ p_i \in \mathbb{R}^3, i=1,2,\cdots,N \}.
\]
A subset of \(M\) points is sampled as centroids,
\[
\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},
\]
and each sampled point \(p'_j\) defines a local region \(\mathbf{R}_j\) [2003.08240].

Around each centroid, the network constructs \(T\) scale areas,
\[
\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},
\]
where each scale area contains \(K_t\) points. The construction pipeline consists of a sampling layer that selects centroids using farthest point sampling (FPS), a searching layer that finds nearest neighbors around each centroid, and a grouping layer that groups the searched points into multi-scale areas [2003.08240]. The searching is performed with k-nearest neighbors (kNN), though ball query is mentioned as an alternative; the paper explicitly prefers kNN because it “can guarantee the information inside local regions and is robust to the input point cloud with different sparsity” [2003.08240].

Before feature extraction, each point in a local region is converted to centroid-relative coordinates,
\[
p_l = p_l - p'_j,
\]
which gives translation/rotation robustness in the local feature extraction stage [2003.08240]. Each scale area \(\mathbf{A}^t_j\) is then passed through a PointNet-style block consisting of an MLP applied independently to points in the area followed by max pooling over the points in that area, yielding a \(D\)-dimensional feature for each scale,
\[
\bm{s}^{t}_j.
\]
These scale-wise features form the input to the locally refined convolution operator [2003.08240].

The implementation reported for the point-cloud experiments uses multi-scale neighborhoods with \(K_1=16, K_2=32, K_3=64, K_4=128\), together with \(M=384\) sampled centroids, \(T=4\) scales, local-region feature dimension \(D=128\), ReLU after FC layers, batch norm, dropout 0.4, and the Adam optimizer with learning rate 0.001 [2003.08240].

## 3. Mathematical formulation of the locally refined convolution operator

For a given local region \(\mathbf{R}_j\), the scale features are concatenated as
\[
\bm{S}^{1:T}_j = \bm{s}^1_j \oplus \bm{s}^2_j \oplus \cdots \oplus \bm{s}^T_j ,
\]
where \(\oplus\) denotes concatenation. More generally,
\[
\bm{S}^{a:a+b}_j
\]
denotes concatenation of consecutive scale features \(\bm{s}^a_j,\bm{s}^{a+1}_j,\dots,\bm{s}^{a+b-1}_j\) [2003.08240].

A convolution filter is then defined as \(\bm{w} \in \mathbb{R}^{hD}\), where \(h\) is the filter window size in the scale dimension. Applied to a window of \(h\) scale features, it produces
\[
c_k = f(\bm{w} \cdot \bm{S}^{a:a+h-1}_j + b),
\]
where \(b \in \mathbb{R}\) is bias and \(f\) is a nonlinearity such as ReLU [2003.08240]. Sliding this filter across all valid windows of the region’s scale-feature sequence gives
\[
\bm{c} = [c_1, c_2, \cdots, c_{T-h+1}].
\]
Max pooling over this response vector then produces the output for that filter,
\[
\hat{\bm{c}} = \max \{ \bm{c} \}.
\]

The paper states that this process is carried out with multiple filter sizes. In general, there are \(T\) kinds of convolution filters with different sizes, and \(\frac{D}{T}\) filters for each kind, so that the output per local region is a \(D\)-dimensional vector \(\bm{r}_j\) [2003.08240]. In implementation terms, this is a 1D convolution over the scale axis, but the filter width changes with the number of scales it spans, which is the variable-size convolution idea.

This formulation differs from simple feature concatenation or max pooling because it is designed to learn the “intrinsic structure and correlation of multi-scale areas from their feature maps, rather than simple feature concatenating or max pooling” [2003.08240]. This suggests that the essential novelty of LRC lies in explicitly modeling interactions among adjacent scales before the final local-region aggregation step.

## 4. Intra-region and inter-region context encoding

In LRC-Net, Locally Refined Convolution constitutes the intra-region context encoding module [2003.08240]. That module operates on the multi-scale descriptors within each local region and produces the local region output feature \(\bm{r}_j\). The corresponding design goal is to capture the geometric correlation inside each local region by variable-size convolution filters over the scale-feature sequence.

The network then applies an inter-region context encoding module to model relationships among regions. For centroids \(\{p'_j\}\), the method defines an \(M \times M\) squared-distance matrix
\[
\bm{U} = \left[ \begin{matrix} (p'_1-p'_1)^2 & (p'_1-p'_2)^2 & \cdots & (p'_1-p'_M)^2\\
(p'_2-p'_1)^2 &(p'_2-p'_2)^2 & \cdots & (p'_2-p'_M)^2 \\
\vdots &\vdots & \ddots &\vdots\\
(p'_M-p'_1)^2 &(p'_M-p'_2)^2 & \cdots & (p'_M-p'_M)^2
\end{matrix} \right],
\]
which is converted to a similarity matrix via an exponential kernel,
\[
\bm{V} = e^{- \gamma \bm{U}}.
\]
Here \(\gamma\) controls how strongly distance affects similarity [2003.08240].

Each region feature is enhanced by a weighted sum of all region features,
\[
\bm{r}'_j = \sum_{b=1}^{M} \bm{V}_{j,b} \cdot \bm{r}_b,
\]
followed by normalization,
\[
\bm{r}''_j = \frac{\bm{r}'_j}{ \sum_{b=1}^{M} \bm{V}_{j,b} }.
\]
The output \(\bm{r}''_j\) is the final region feature, enriched with global spatial context [2003.08240]. The paper describes this module as a “greedy strategy” for encoding inter-region structure.

The full LRC-Net architecture is organized into six parts: multi-scale area establishment, area feature extraction, intra-region context encoding, inter-region context encoding, shape classification, and shape segmentation [2003.08240]. For classification, all region features are aggregated into a global feature \(\bm{g}\), and another PointNet layer is used to obtain a 1024-dimensional shape descriptor. For segmentation, interpolation layers propagate global and region-level features back to points using inverse-distance weighted kNN interpolation,
\[
\phi(p) = \frac{\sum_{i=1}^{k}{w(p_i)\phi(p_i)}}{\sum_{i=1}^{k}{w(p_i)}},
\qquad
w(p_i)=\frac{1}{(p-p_i)^2}.
\]
This enables point-wise semantic labels [2003.08240].

## 5. Relation to standard convolution and PointNet-style aggregation

LRC-Net explicitly distinguishes its locally refined convolution from standard convolution on regular grids and from PointNet-style pooling [2003.08240]. A standard CNN convolution assumes a regular grid and a fixed kernel size over an ordered spatial neighborhood. LRC-Net does not convolve on point positions directly like an image CNN; instead, it convolves over a sequence of multi-scale region features inside a local region.

Relative to PointNet-style pooling, the difference is more structural. PointNet and PointNet-like local modules typically map each point independently through an MLP and then aggregate with max pooling, which is permutation invariant but loses context between points or scales. LRC-Net instead first extracts features at multiple scales, then explicitly learns correlations between scale areas via variable-size filters, and only then aggregates [2003.08240].

The contrast can be summarized as follows.

| Comparison target | What it assumes or does | How LRC differs |
|---|---|---|
| Standard convolution | Regular grid and fixed kernel size | Convolves over multi-scale region features rather than raw coordinates on a grid |
| PointNet-style pooling | Independent pointwise MLP plus pooling | Learns interactions across scales before aggregation |
| Simple concatenation | Stacks features without learned scale interaction | Uses variable-size filters to model intrinsic structure and correlation of multi-scale areas |

This framing clarifies a common misconception: the “convolution” in Locally Refined Convolution is not a direct 3D convolution over irregular coordinates. Rather, it is a learned 1D convolutional operator over the scale dimension of region-level descriptors [2003.08240]. A plausible implication is that LRC should be categorized as a context-encoding mechanism for local descriptors rather than as a geometric convolution in the strict grid-convolution sense.

## 6. Empirical findings and ablation evidence

The experimental evidence reported in LRC-Net directly tests the contribution of multi-scale local context, variable-size filters, and inter-region fusion on ModelNet40 classification [2003.08240]. When varying the number of scales \(T\), the reported accuracies are \(92.26\%\) for \(T=1\), \(92.42\%\) for \(T=2\), \(92.54\%\) for \(T=3\), \(93.07\%\) for \(T=4\), and \(92.18\%\) for \(T=5\). The authors interpret this as support for the claim that multi-scale local context matters, while too many scales can add redundancy.

When using 4 scale areas and varying the number of filter types, the reported accuracies are \(92.38\%\) for \(h=1\), \(92.42\%\) for \(h=2\), \(92.67\%\) for \(h=3\), and \(93.07\%\) for \(h=4\) [2003.08240]. This is presented as direct evidence that the variable-size convolution module is effective in aggregating multi-scale area features.

Ablation on the intra-region context encoding module compares the full model against simpler aggregation methods. The reported results are \(93.07\%\) for the full model, \(92.50\%\) for mean pooling, \(92.38\%\) for max pooling, and \(92.30\%\) for concatenation [2003.08240]. These results are used to support the claim that variable-size convolution is better than plain pooling or concatenation for fusing multi-scale local features.

Ablation on the inter-region context encoding module reports \(93.07\%\) for Y(max), \(92.26\%\) for N(max), \(91.57\%\) for Y(mean), \(91.37\%\) for N(mean), and \(91.41\%\) for N(sum) [2003.08240]. The same study varies the spatial similarity parameter \(\gamma\), yielding \(91.33\%\) for \(\gamma=0\), \(91.61\%\) for \(1\), \(92.02\%\) for \(10^2\), \(93.07\%\) for \(10^4\), and \(92.54\%\) for \(10^5\). The paper notes that \(\gamma=0\) degenerates to simple summation, which is taken as confirmation that the learned spatial weighting is important.

The paper also states that LRC-Net outperforms or matches strong multi-scale baselines such as PointNet++, Point2Sequence, and Point2SpatialCapsule on classification, showing that the joint local refinement strategy is competitive [2003.08240]. Since the article’s detailed numerical comparisons are not reproduced in the supplied source block, the precise comparative margins are not stated here.

## 7. Terminological scope and later reuse of “LRC”

The term “Locally Refined Convolution” was later reused in a different technical context by the speech-enhancement model LORT, “Locally Refined Convolution and Taylor Transformer for Monaural Speech Enhancement” [2509.23832]. In that work, LRC denotes the local-detail modeling block in LORT and is designed to complement a Taylor Transformer branch that mainly performs efficient global modeling. The LRC block is built around a gated unit that combines a Convolutional Feedforward Network (CFN), Time-Frequency Dense Local Convolution (TF-DLC), and a residual connection around the gated unit [2509.23832].

Despite the shared name, this later usage is architecturally distinct from the point-cloud operator in LRC-Net. In LORT, the local refinement target is the time-frequency structure of noisy speech spectrograms rather than the multi-scale geometry of local point-cloud neighborhoods. The paper states that the block is used to “capture fine-grained local details” that the Taylor Transformer alone does not model well, and places it inside each locally refined Taylor transformer block rather than as a point-cloud feature extractor [2509.23832].

This reuse of terminology can produce ambiguity in the literature. In strict historical terms, the term is anchored to the point-cloud formulation introduced by LRC-Net, where it specifically refers to variable-size convolution over the scale axis of multi-scale local-region features [2003.08240]. The later speech-enhancement use preserves the broader idea of local refinement but changes the underlying data domain, internal block structure, and mathematical object on which the operator acts [2509.23832]. A plausible implication is that “Locally Refined Convolution” now functions as a family name for locality-enhancing modules in different domains, rather than denoting a single invariant operator across tasks.

Source: https://www.emergentmind.com/topics/locally-refined-convolution-lrc