---
title: 'ReCoSeg: Weakly Supervised CT Lesion Segmentation'
url: https://www.emergentmind.com/topics/recoseg
type: topic
---

# ReCoSeg: Weakly Supervised CT Lesion Segmentation

Searching arXiv for exact and closely related names to disambiguate “ReCoSeg”.
ReCoSeg is a weakly supervised lesion segmentation framework for computed tomography (CT) scans that uses RECIST measurements as supervision and refines the resulting masks through attention-based co-segmentation. In the cited work, the method addresses the practical setting in which full lesion contours are rarely available, whereas RECIST diameters are routinely stored in clinical archives. The pipeline first converts RECIST annotations into coarse lesion masks and then exploits lesion-to-lesion similarity through a Siamese co-segmentation network to improve mask quality, followed by dense CRF post-processing [2001.08590]. The name should be distinguished from several similarly named segmentation methods, including ReSeg, Co-Seg, and CoReSeg, which address different problems and are not presented as the same method [1511.07053], [2102.00523], [2203.01368].

## 1. Definition and nomenclature

ReCoSeg, in the source that explicitly uses this name, denotes a two-stage weakly supervised lesion segmentation method for 2D CT lesion masks derived from RECIST annotations [2001.08590]. Its central premise is that RECIST diameters provide an abundant but imprecise supervisory signal: they indicate lesion extent on the axial slice where the lesion is largest, yet do not provide a full contour. ReCoSeg therefore treats RECIST as weak supervision and augments it with co-segmentation over similar lesion pairs.

The term is easily conflated with other segmentation models whose names are orthographically similar. ReSeg is a recurrent semantic segmentation architecture built from pre-trained convolutional layers, ReNet-style recurrent sweeps, and transposed-convolution upsampling [1511.07053]. Co-Seg is a label-corruption-robust training framework based on two peer networks, corruption-score ranking, label correction, and retraining [2102.00523]. CoReSeg is an open-set semantic segmentation method based on class-conditional reconstruction and pixelwise FiLM conditioning [2203.01368]. A plausible implication is that the label “ReCoSeg” should be interpreted cautiously unless the weakly supervised DeepLesion/RECIST setting is explicit.

## 2. Problem setting and supervision model

The method is motivated by lesion segmentation on CT scans, where manual delineation is described as prohibitively time-consuming, expensive, and requiring professional knowledge [2001.08590]. Clinical workflows instead often rely on response evaluation criteria in solid tumors (RECIST), which provide two orthogonal diameters on the axial slice where the lesion is largest. ReCoSeg uses these routine measurements as weak supervision.

The supervision model is therefore not pixel-accurate at the outset. RECIST-derived signals are first transformed into rough masks, which serve as pseudo-ground-truth for training. This design acknowledges that RECIST supplies lesion localization and approximate scale but not a precise boundary. The method’s technical significance lies in converting a cheap, clinically prevalent annotation format into a denser supervisory signal suitable for convolutional segmentation models [2001.08590].

This setting differs materially from standard fully supervised semantic segmentation. The weak labels are noisy and structurally incomplete, and the method attempts to recover missing shape detail by exploiting cross-image commonality among similar lesions rather than relying only on single-image segmentation cues. This suggests that ReCoSeg belongs to the broader class of weakly supervised structured prediction systems in which inter-sample relations compensate for sparse annotation.

## 3. Two-stage pipeline

The ReCoSeg pipeline has two stages [2001.08590]. First, the RECIST slice is converted into an initial lesion mask. This is done with GrabCut, using RECIST diameters to define foreground and background seeds. The lesion region around the RECIST cross is initialized as foreground, surrounding areas as background, and GrabCut optimizes an energy function to produce a coarse mask. These masks are explicitly not the final output; they are used as pseudo-ground-truth for the second stage.

Second, the method trains an attention-based co-segmentation network on pairs of lesion images. The key assumption is that lesions within the same semantic cluster share discriminative structure. By presenting lesion pairs to a Siamese encoder-decoder with shared weights, the model learns to emphasize what is common between the two lesions and suppress lesion-specific noise or background clutter. After mask prediction, a densely connected CRF is applied as post-processing to sharpen boundaries and obtain the final refined lesion masks [2001.08590].

The resulting workflow can be summarized as follows.

| Stage | Operation | Role |
|---|---|---|
| 1 | RECIST-to-mask initialization with GrabCut | Produces coarse pseudo-ground-truth |
| 2 | Attention-based Siamese co-segmentation | Refines masks via shared lesion structure |
| 3 | Dense CRF post-processing | Sharpens boundaries |

A plausible implication is that the framework is best understood not as a single segmentation backbone, but as a weak-supervision pipeline in which pseudo-label bootstrapping and pairwise relational learning are equally central.

## 4. Co-segmentation architecture and attention design

The co-segmentation model is adapted from the semantic-aware attention co-segmentation architecture of Chen et al. and uses a Siamese encoder-decoder design [2001.08590]. Two images are processed by shared-weight encoders to obtain feature maps, which are then fused by attention modules that emphasize common semantic content and reduce distracting differences. The decoder upsamples the attended features to produce a two-channel segmentation output for each input image, corresponding to foreground and background.

The paper describes two attention variants: channel attention and channel-spatial attention (CSA). Channel attention focuses on feature channels that are strongly activated in both images, thereby retaining shared lesion-relevant semantics while downweighting channels that are not consistently present across the pair. The spatial component further models where these shared patterns occur, with the stated effect of preserving lesion structure and spatial localization [2001.08590].

Architecturally, the authors evaluate VGG-16, ResNet-101, and a dilated ResNet-101 backbone. The strongest model uses a dilated ResNet-101 encoder with atrous convolutions in the last two residual blocks, yielding an output stride of 8. This improves feature-map resolution and context capture, which matters for lesion boundaries and for small or irregular lesions [2001.08590]. The Siamese structure is not incidental: the method’s learning signal depends on comparing what is shared between two related lesions, rather than segmenting each image independently.

## 5. Pair construction, clustering, and optimization

A critical design choice is how lesion pairs are formed. Since co-segmentation presupposes some commonality between the paired images, the method first computes lesion feature vectors and clusters the entire DeepLesion set into 200 groups using K-means, following a prior lesion-graph strategy [2001.08590]. Pairs are then sampled within each cluster. The paper emphasizes that this clustering stage is important because lesions vary widely in appearance, size, and anatomical background; pairing arbitrary lesions would make co-segmentation harder and could be harmful.

This pairing strategy functions as a similarity prior. By constraining pair sampling to within-cluster lesions, the attention mechanism is more likely to highlight lesion structure rather than shared but irrelevant context. The worse performance of the “without clustering” variant directly supports this interpretation [2001.08590].

Training uses pixel-wise cross-entropy loss on the pseudo-label masks generated by GrabCut [2001.08590]. The method description does not introduce a custom multi-term loss for the co-segmentation network. Instead, robustness is attributed to the pairwise attention mechanism and the CRF refinement at inference time. The training configuration is explicitly given as Adam with learning rate \(1 \times 10^{-5}\), weight decay \(5 \times 10^{-4}\), batch size 20, and two epochs with 12,000 iterations per epoch. Inputs are padded, resized to \(128 \times 128\), and normalized, and the best model is selected by validation Dice [2001.08590].

## 6. Dataset, evaluation protocol, and reported performance

Experiments are conducted on NIH DeepLesion, which contains 32,735 lesion images with RECIST annotations [2001.08590]. The dataset is split using stratified sampling after clustering into 200 groups: 80% training, 10% validation, and 10% testing. From this, paired co-segmentation datasets are constructed, yielding 270,470 training pairs, 28,136 validation pairs, and 3,866 test pairs. Quantitative evaluation uses 1,000 manually annotated segmentations treated as ground truth.

The reported metrics are recall, precision, Dice similarity coefficient, averaged Hausdorff distance (AVD), and volumetric similarity (VS) [2001.08590]. The FCN-16 model with VGG-16 backbone trained directly on the initial masks is the primary weak-supervision-only baseline and achieves a Dice score of 85.8%. The best ReCoSeg model, DRN-101 + CSA, reaches 89.8% Dice, which the paper describes as an increase about 4.0% from 85.8% to 89.8% [2001.08590].

The reported comparison is concise below.

| Model | Dice |
|---|---:|
| FCN-16 | 85.8% |
| VGG-16 co-segmentation | 87.7% |
| VGG-16 + CSA | 88.0% |
| ResNet-101 | 87.9% |
| ResNet-101 + CSA | 88.3% |
| DRN-101 | 87.7% |
| DRN-101 + CSA | 89.8% |

For the best model, the paper reports recall 0.915, precision 0.895, Dice 0.898, AVD 0.349, and VS 0.942 [2001.08590]. The authors state that channel-spatial attention is especially helpful for improving recall, because it helps prevent lesion pixels from being mislabeled as background. They also state that the dilated backbone improves spatial detail by preserving higher-resolution feature maps.

## 7. Interpretation, limitations, and relation to similarly named methods

The main technical interpretation of ReCoSeg is that lesion similarity is used explicitly as a supervisory resource. The network does not rely only on noisy pseudo-masks derived from RECIST; it also learns what multiple similar lesions have in common. This shared-structure signal appears to reduce false positives and improve boundary quality relative to a standard FCN trained directly on weak labels [2001.08590].

The method has several explicit limitations. It depends on RECIST-derived pseudo masks, so inaccurate RECIST slice selection or poor GrabCut initialization can propagate errors into co-segmentation training [2001.08590]. It also assumes that paired lesions are sufficiently similar for co-segmentation to help; poor clustering can cause the attention mechanism to emphasize irrelevant background or fail to discover useful commonality. The framework is fundamentally 2D, operating on RECIST slices rather than full 3D volumes, and future work is described in terms of slice propagation toward volumetric segmentation [2001.08590].

A common misconception is to treat ReCoSeg as interchangeable with other “Re/Co-Seg” methods. That is not supported by the cited records. ReSeg is a recurrent structured prediction architecture for semantic segmentation based on four-directional RNN sweeps over image patches [1511.07053]. Co-Seg addresses noisy labels through collaborative sample selection, agreement-based label correction, and final retraining [2102.00523]. CoReSeg addresses open-set semantic segmentation through conditional reconstruction, frozen closed-set encoder features, and per-pixel reconstruction error thresholding [2203.01368]. These methods share the broad topic of segmentation but not the same supervision regime, architecture, or target problem.

In that narrower and exact sense, ReCoSeg refers to a RECIST-driven weakly supervised lesion segmentation framework whose defining contribution is the combination of GrabCut-based pseudo-mask generation, clustered lesion pairing, attention-based Siamese co-segmentation, and CRF refinement on DeepLesion [2001.08590].

Source: https://www.emergentmind.com/topics/recoseg