Locally Refined Convolution in 3D and Speech

• Locally Refined Convolution is a feature-learning operator that applies variable-size convolution filters over multi-scale local-region features in irregular point clouds.
• It enhances intra-region and inter-region context encoding by capturing fine-grained geometric details and spatial relationships to boost classification and segmentation performance.
• The approach has been adapted for distinct domains, including speech enhancement, by refining local time-frequency details, demonstrating its versatility across applications.

Searching arXiv for relevant papers on “Locally Refined Convolution” and related uses of the term. Locally Refined Convolution (LRC) denotes a point-cloud feature-learning operator introduced in LRC-Net to encode fine-grained contexts within local regions of a 3D point set by applying variable-size convolution filters over multi-scale local-region features (Liu et al., 2020). In that formulation, a point cloud is treated not as an unordered bag of points, but as a collection of local regions whose internal geometry and mutual spatial arrangement both matter. The operator appears as the core of an intra-region context encoding module, and is paired with an inter-region context encoding module that integrates spatial relationships among local regions based on spatial similarity measures. Experimental results reported for LRC-Net show that this combination is competitive with state-of-the-art methods in shape classification and shape segmentation applications (Liu et al., 2020). The same term, “locally refined convolution,” was later used in a distinct speech-enhancement architecture, LORT, where it denotes a local-detail modeling block for time-frequency spectrogram refinement rather than a point-cloud operator (Wang et al., 28 Sep 2025).

1. Conceptual motivation and problem setting

LRC was introduced to address limitations of point-cloud networks that use fixed-size CNN or MLP operators together with simple max pooling over local features (Liu et al., 2020). The motivating premise is that point clouds are irregular and sparse, unlike images, so traditional CNN assumptions about a fixed spatial neighborhood with a regular grid do not hold naturally. In the formulation used by LRC-Net, many prior methods either use fixed-size filters that are too rigid for irregular, sparse point sets or use permutation-invariant pooling that collapses local structure too aggressively (Liu et al., 2020).

Within this framework, the central claim is that a single fixed-size filter is insufficient because the geometry visible inside a local region depends on how densely and how sparsely points are sampled. LRC therefore uses variable-size convolution filters over multi-scale local-region features so that the network can adapt to irregular point distributions by considering multiple scale areas around each centroid (Liu et al., 2020). The intended effect is to capture fine-grained geometry inside local regions, correlations across different scales within a region, and spatial relationships among neighboring local regions.

A plausible implication is that LRC should be understood less as a direct analogue of image convolution than as a structured operator over learned local-scale descriptors. In LRC-Net, the convolution is not applied to raw 3D coordinates on a grid; it is applied to the feature map formed by multiple neighborhood scales inside each local region (Liu et al., 2020).

2. Local-region construction and feature preparation

The input point cloud is defined as

$$\mathbf{P} = \{ p_i \in \mathbb{R}^3, i=1,2,\cdots,N \}.$$

A subset of $M$ points is sampled as centroids,

$$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$

and each sampled point $p'_j$ defines a local region $\mathbf{R}_j$ (Liu et al., 2020).

Around each centroid, the network constructs $T$ scale areas,

$$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$

where each scale area contains $K_t$ points. The construction pipeline consists of a sampling layer that selects centroids using farthest point sampling (FPS), a searching layer that finds nearest neighbors around each centroid, and a grouping layer that groups the searched points into multi-scale areas (Liu et al., 2020). The searching is performed with k-nearest neighbors (kNN), though ball query is mentioned as an alternative; the paper explicitly prefers kNN because it “can guarantee the information inside local regions and is robust to the input point cloud with different sparsity” (Liu et al., 2020).

Before feature extraction, each point in a local region is converted to centroid-relative coordinates,

$p_l = p_l - p'_j,$

which gives translation/rotation robustness in the local feature extraction stage (Liu et al., 2020). Each scale area $\mathbf{A}^t_j$ is then passed through a PointNet-style block consisting of an MLP applied independently to points in the area followed by max pooling over the points in that area, yielding a $M$0-dimensional feature for each scale,

$M$1

These scale-wise features form the input to the locally refined convolution operator (Liu et al., 2020).

The implementation reported for the point-cloud experiments uses multi-scale neighborhoods with $M$2, together with $M$3 sampled centroids, $M$4 scales, local-region feature dimension $M$5, ReLU after FC layers, batch norm, dropout 0.4, and the Adam optimizer with learning rate 0.001 (Liu et al., 2020).

3. Mathematical formulation of the locally refined convolution operator

For a given local region $M$6, the scale features are concatenated as

$M$7

where $M$8 denotes concatenation. More generally,

$M$9

denotes concatenation of consecutive scale features $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$0 (Liu et al., 2020).

A convolution filter is then defined as $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$1, where $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$2 is the filter window size in the scale dimension. Applied to a window of $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$3 scale features, it produces

$$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$4

where $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$5 is bias and $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$6 is a nonlinearity such as ReLU (Liu et al., 2020). Sliding this filter across all valid windows of the region’s scale-feature sequence gives

$$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$7

Max pooling over this response vector then produces the output for that filter,

$$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$8

The paper states that this process is carried out with multiple filter sizes. In general, there are $$\mathbf{P}^{'} = \{ p'_j \in \mathbb{R}^3, j=1,2,\cdots,M \},$$9 kinds of convolution filters with different sizes, and $p'_j$0 filters for each kind, so that the output per local region is a $p'_j$1-dimensional vector $p'_j$2 (Liu et al., 2020). In implementation terms, this is a 1D convolution over the scale axis, but the filter width changes with the number of scales it spans, which is the variable-size convolution idea.

This formulation differs from simple feature concatenation or max pooling because it is designed to learn the “intrinsic structure and correlation of multi-scale areas from their feature maps, rather than simple feature concatenating or max pooling” (Liu et al., 2020). This suggests that the essential novelty of LRC lies in explicitly modeling interactions among adjacent scales before the final local-region aggregation step.

4. Intra-region and inter-region context encoding

In LRC-Net, Locally Refined Convolution constitutes the intra-region context encoding module (Liu et al., 2020). That module operates on the multi-scale descriptors within each local region and produces the local region output feature $p'_j$3. The corresponding design goal is to capture the geometric correlation inside each local region by variable-size convolution filters over the scale-feature sequence.

The network then applies an inter-region context encoding module to model relationships among regions. For centroids $p'_j$4, the method defines an $p'_j$5 squared-distance matrix

$p'_j$6

which is converted to a similarity matrix via an exponential kernel,

$p'_j$7

Here $p'_j$8 controls how strongly distance affects similarity (Liu et al., 2020).

Each region feature is enhanced by a weighted sum of all region features,

$p'_j$9

followed by normalization,

$\mathbf{R}_j$0

The output $\mathbf{R}_j$1 is the final region feature, enriched with global spatial context (Liu et al., 2020). The paper describes this module as a “greedy strategy” for encoding inter-region structure.

The full LRC-Net architecture is organized into six parts: multi-scale area establishment, area feature extraction, intra-region context encoding, inter-region context encoding, shape classification, and shape segmentation (Liu et al., 2020). For classification, all region features are aggregated into a global feature $\mathbf{R}_j$2, and another PointNet layer is used to obtain a 1024-dimensional shape descriptor. For segmentation, interpolation layers propagate global and region-level features back to points using inverse-distance weighted kNN interpolation,

$\mathbf{R}_j$3

This enables point-wise semantic labels (Liu et al., 2020).

5. Relation to standard convolution and PointNet-style aggregation

LRC-Net explicitly distinguishes its locally refined convolution from standard convolution on regular grids and from PointNet-style pooling (Liu et al., 2020). A standard CNN convolution assumes a regular grid and a fixed kernel size over an ordered spatial neighborhood. LRC-Net does not convolve on point positions directly like an image CNN; instead, it convolves over a sequence of multi-scale region features inside a local region.

Relative to PointNet-style pooling, the difference is more structural. PointNet and PointNet-like local modules typically map each point independently through an MLP and then aggregate with max pooling, which is permutation invariant but loses context between points or scales. LRC-Net instead first extracts features at multiple scales, then explicitly learns correlations between scale areas via variable-size filters, and only then aggregates (Liu et al., 2020).

The contrast can be summarized as follows.

Comparison target	What it assumes or does	How LRC differs
Standard convolution	Regular grid and fixed kernel size	Convolves over multi-scale region features rather than raw coordinates on a grid
PointNet-style pooling	Independent pointwise MLP plus pooling	Learns interactions across scales before aggregation
Simple concatenation	Stacks features without learned scale interaction	Uses variable-size filters to model intrinsic structure and correlation of multi-scale areas

This framing clarifies a common misconception: the “convolution” in Locally Refined Convolution is not a direct 3D convolution over irregular coordinates. Rather, it is a learned 1D convolutional operator over the scale dimension of region-level descriptors (Liu et al., 2020). A plausible implication is that LRC should be categorized as a context-encoding mechanism for local descriptors rather than as a geometric convolution in the strict grid-convolution sense.

6. Empirical findings and ablation evidence

The experimental evidence reported in LRC-Net directly tests the contribution of multi-scale local context, variable-size filters, and inter-region fusion on ModelNet40 classification (Liu et al., 2020). When varying the number of scales $\mathbf{R}_j$4, the reported accuracies are $\mathbf{R}_j$5 for $\mathbf{R}_j$6, $\mathbf{R}_j$7 for $\mathbf{R}_j$8, $\mathbf{R}_j$9 for $T$0, $T$1 for $T$2, and $T$3 for $T$4. The authors interpret this as support for the claim that multi-scale local context matters, while too many scales can add redundancy.

When using 4 scale areas and varying the number of filter types, the reported accuracies are $T$5 for $T$6, $T$7 for $T$8, $T$9 for $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$0, and $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$1 for $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$2 (Liu et al., 2020). This is presented as direct evidence that the variable-size convolution module is effective in aggregating multi-scale area features.

Ablation on the intra-region context encoding module compares the full model against simpler aggregation methods. The reported results are $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$3 for the full model, $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$4 for mean pooling, $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$5 for max pooling, and $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$6 for concatenation (Liu et al., 2020). These results are used to support the claim that variable-size convolution is better than plain pooling or concatenation for fusing multi-scale local features.

Ablation on the inter-region context encoding module reports $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$7 for Y(max), $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$8 for N(max), $$\mathbf{A}_j = \{ \mathbf{A}^t_j, t=1,2,\cdots,T\},$$9 for Y(mean), $K_t$0 for N(mean), and $K_t$1 for N(sum) (Liu et al., 2020). The same study varies the spatial similarity parameter $K_t$2, yielding $K_t$3 for $K_t$4, $K_t$5 for $K_t$6, $K_t$7 for $K_t$8, $K_t$9 for $p_l = p_l - p'_j,$0, and $p_l = p_l - p'_j,$1 for $p_l = p_l - p'_j,$2. The paper notes that $p_l = p_l - p'_j,$3 degenerates to simple summation, which is taken as confirmation that the learned spatial weighting is important.

The paper also states that LRC-Net outperforms or matches strong multi-scale baselines such as PointNet++, Point2Sequence, and Point2SpatialCapsule on classification, showing that the joint local refinement strategy is competitive (Liu et al., 2020). Since the article’s detailed numerical comparisons are not reproduced in the supplied source block, the precise comparative margins are not stated here.

7. Terminological scope and later reuse of “LRC”

The term “Locally Refined Convolution” was later reused in a different technical context by the speech-enhancement model LORT, “Locally Refined Convolution and Taylor Transformer for Monaural Speech Enhancement” (Wang et al., 28 Sep 2025). In that work, LRC denotes the local-detail modeling block in LORT and is designed to complement a Taylor Transformer branch that mainly performs efficient global modeling. The LRC block is built around a gated unit that combines a Convolutional Feedforward Network (CFN), Time-Frequency Dense Local Convolution (TF-DLC), and a residual connection around the gated unit (Wang et al., 28 Sep 2025).

Despite the shared name, this later usage is architecturally distinct from the point-cloud operator in LRC-Net. In LORT, the local refinement target is the time-frequency structure of noisy speech spectrograms rather than the multi-scale geometry of local point-cloud neighborhoods. The paper states that the block is used to “capture fine-grained local details” that the Taylor Transformer alone does not model well, and places it inside each locally refined Taylor transformer block rather than as a point-cloud feature extractor (Wang et al., 28 Sep 2025).

This reuse of terminology can produce ambiguity in the literature. In strict historical terms, the term is anchored to the point-cloud formulation introduced by LRC-Net, where it specifically refers to variable-size convolution over the scale axis of multi-scale local-region features (Liu et al., 2020). The later speech-enhancement use preserves the broader idea of local refinement but changes the underlying data domain, internal block structure, and mathematical object on which the operator acts (Wang et al., 28 Sep 2025). A plausible implication is that “Locally Refined Convolution” now functions as a family name for locality-enhancing modules in different domains, rather than denoting a single invariant operator across tasks.