AnydoorMed: Reference-Guided Mammography Inpainting

• AnydoorMed is a reference-guided latent diffusion inpainting model designed to synthesize counterfactual mammograms through controlled lesion insertion.
• It employs a mammography-adapted VAE, DINOv2 reference encoding, and high-frequency conditioning to preserve subtle details such as microcalcifications.
• The model offers controlled generation via an input tuple (target scan, lesion patch, spatial mask) and achieves improved FID and LPIPS metrics compared to natural-image methods.

Searching arXiv for the primary paper and a few directly related methods mentioned in the provided data.

AnydoorMed is a reference-guided latent diffusion inpainting model specialised for mammography, introduced in “Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation” (Buburuzan, 30 Jul 2025). It adapts an AnyDoor-style natural-image pipeline to single-channel medical images with minimal changes to the backbone, but with domain-specific components: a mammography-adapted VAE, DINOv2 reference encoding, and a detail or high-frequency conditioning path intended to preserve subtle structures such as microcalcifications. The model addresses controllable, realistic counterfactual generation in mammography: given a target mammogram, a reference anomaly patch cropped from another mammogram, and a spatial edit mask or bounding box, it generates a mammogram in which the anomaly has been inserted into a healthy scan, reinserted where it was removed, or used to replace another anomaly while blending with surrounding tissue (Buburuzan, 30 Jul 2025).

1. Problem formulation and imaging scope

AnydoorMed is designed for controllable counterfactual generation in 2D mammograms. Its basic input tuple consists of a target mammogram, a reference anomaly patch, and a spatial specification of where the anomaly should appear. The output is an edited mammogram in which the anomaly is synthesized within the designated region while preserving the surrounding breast tissue context (Buburuzan, 30 Jul 2025).

The target structures are breast lesions in mammograms from VinDR-Mammo. The lesion set includes masses, architectural distortion, asymmetries, focal or global asymmetry, suspicious calcifications, nipple or skin retraction, skin thickening, and suspicious lymph node, among others. These lesions have associated BI-RADS malignancy scores from 1 to 5, although the conditioning is not explicitly on BI-RADS (Buburuzan, 30 Jul 2025).

Conceptually, the model is analogous to reference-guided inpainting in natural images, but the adaptation is nontrivial because mammography is single-channel, clinically relevant cues often reside in fine high-frequency structure, and plausible lesion insertion requires semantic consistency with breast parenchyma rather than generic object compositing. The core operational question posed by the model is: “How would this breast look if it contained that lesion?” (Buburuzan, 30 Jul 2025).

A recurrent misconception is to treat AnydoorMed as a generic medical-image generator. In the formulation introduced here, it is more specific: it is a reference-guided inpainting system for mammography, not a text-conditioned general-purpose generator, and not a segmentation-driven simulator. The lesion appearance is controlled primarily by a visual reference patch, while location is controlled by a 2D box or mask (Buburuzan, 30 Jul 2025).

2. Latent diffusion architecture

AnydoorMed uses a latent diffusion model in the Stable Diffusion style (Rombach et al., 2021). A $512\times512$ mammogram crop is encoded by a mammography-adapted VAE into a latent tensor

$$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$

which is subsequently decoded back to image space (Buburuzan, 30 Jul 2025).

The latent forward process follows DDPM-style Gaussian corruption: $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$ with $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$ under a predefined noise schedule $\{\beta_t\}$. A U-Net denoiser $\epsilon_\theta$ predicts the added noise,

$$\epsilon_\theta : (\mathbf{z}_t, t, C) \mapsto \hat{\epsilon},$$

under conditioning $C$ supplied by the reference anomaly and mammography-specific context streams (Buburuzan, 30 Jul 2025).

Training uses the standard conditional noise-prediction MSE: $$\mathcal{L} = \mathbb{E}_{\mathbf{z}_0, t, \epsilon, \mathbf{c}} \Big[ \lVert \epsilon - \epsilon_\theta(\mathbf{z}_t, \mathbf{c}, t)\rVert^2 \Big],$$ with conditioning bundle $$\mathbf{c} = \{\mathbf{c}_{\text{ref}}, \mathbf{c}^{(\text{M})}\}$$. At inference, a DDIM sampler with $$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$0 steps deterministically traverses the latent trajectory from $$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$1 to $$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$2, which is then decoded by the VAE (Buburuzan, 30 Jul 2025).

The conditioning structure is split between reference anomaly encoding and mammography context/detail encoding. Reference information is supplied by DINOv2 tokens (Oquab et al., 2023), while mammography context is supplied by latent context, an edit mask, and a high-frequency map. These streams are injected into the decoder half of the U-Net in two ways: through a detail encoder whose outputs are added to decoder feature maps à la ControlNet, and through newly inserted gated cross-attention layers in which mammography latent features attend to reference tokens (Buburuzan, 30 Jul 2025).

A notable design decision is the parameter-efficient fine-tuning regime. The base U-Net and AnyDoor’s detail encoder remain frozen; only the newly added gating and cross-attention parameters are trained. This preserves pretrained AnyDoor behavior at initialization and constrains adaptation to mammography-specific reference fusion (Buburuzan, 30 Jul 2025).

3. Reference-guided conditioning and lesion synthesis

Reference guidance begins with anomaly extraction from a source mammogram. The lesion bounding box

$$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$3

is used to crop a reference patch that tightly encloses the lesion. The patch is then augmented with horizontal flip and brightness or contrast changes to improve robustness. It is encoded by DINOv2, a self-supervised ViT (Oquab et al., 2023), into a token set projected into reference conditioning

$$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$4

These tokens are injected into several U-Net layers through cross-attention (Buburuzan, 30 Jul 2025).

The cross-attention mechanism is written as

$$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$5

$$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$6

followed by a zero-initialised gated residual: $$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$7 Because the gate is zero-initialised, fine-tuning begins from unchanged pretrained behavior and progressively learns reference-specific residual modulation (Buburuzan, 30 Jul 2025).

Target mammograms are processed through a mammography-specific context pathway. Original VinDR-Mammo DICOMs are approximately $$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$8. For each anomaly or insertion location, a square zoom-in crop is taken around the box center, with side length $$\mathbf{z}_0 \in \mathbb{R}^{64 \times 64 \times 4},$$9–$$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$0 the bounding-box side length, padded if necessary, resized to $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$1, and intensity-normalised to $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$2 (Buburuzan, 30 Jul 2025).

An edit mask $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$3 specifies the lesion region. The context image is formed by zeroing the lesion region: $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$4 To preserve fine structures, the model constructs a high-frequency map

$$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$5

where $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$6 and $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$7 are Sobel kernels and $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$8 is the eroded edit mask. The high-frequency map for the edit region is collaged into the context and passed through the detail encoder, whose features are injected into the U-Net decoder. Context $$\mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\quad \epsilon \sim \mathcal{N}(0,\mathbf{I}),$$9 is separately encoded through the mammography VAE into a latent context representation (Buburuzan, 30 Jul 2025).

This conditioning design assigns distinct roles to the three streams. DINOv2 tokens promote structural integrity of the reference lesion; latent context and high-frequency guidance enforce semantic and intensity consistency with surrounding breast tissue. A plausible implication is that the model’s fidelity depends less on direct copying than on constrained regeneration under local anatomical context (Buburuzan, 30 Jul 2025).

4. Data, optimization, and evaluation protocol

The training and evaluation data are drawn from VinDR-Mammo. For AnydoorMed, the public train or validation split is used with 4000 images for training, 1000 for validation, and 426 positive images for test or evaluation. Annotations include lesion bounding boxes, lesion categories across 10 classes, and BI-RADS scores from 1 to 5 (Buburuzan, 30 Jul 2025).

The preprocessing pipeline converts DICOM to 8-bit PNG via

$\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$0

followed by windowing and normalisation to $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$1, and then mapping to $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$2. Zoom-in cropping, mask construction, and context-image formation follow the lesion-centered procedure described above (Buburuzan, 30 Jul 2025).

The mammography VAE is derived from the Stable Diffusion image VAE (Rombach et al., 2021) by replacing the input and output convolutions with residual blocks to handle 1-channel input. It is fine-tuned at $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$3 with an image reconstruction objective plus perceptual loss from “Taming Transformers for High-Resolution Image Synthesis” (Esser et al., 2020): $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$4 where $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$5. VAE training uses 16 epochs, approximately 7k steps, batch size 4, and Adam with learning rate $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$6 (Buburuzan, 30 Jul 2025).

Diffusion training uses only the denoising MSE objective; there is no adversarial loss and no extra explicit reconstruction, adversarial, or anatomy-regularising loss at the diffusion level. Anatomical plausibility is enforced implicitly by masked-context conditioning, high-frequency guidance, and the inductive bias of the pretrained AnyDoor backbone and DINOv2 features (Buburuzan, 30 Jul 2025).

During diffusion fine-tuning, the VAE encoder or decoder, AnyDoor’s original U-Net weights, and the detail encoder are frozen. The trainable parameters are the newly added gated cross-attention layers. Training is performed at resolution $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$7, latent size $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$8, for 30 epochs, approximately 4000 steps, with batch size 16, Adam, and learning rate $\bar{\alpha}_t = \prod_{i=1}^{t}(1-\beta_i)$9. Model selection keeps the top 3 checkpoints by validation loss and chooses the final checkpoint by FID on a test set of 426 reinsertion samples (Buburuzan, 30 Jul 2025).

The evaluation protocol comprises three tasks:

Task	Target context	Reference use
Reinsertion	Original lesion removed from its scan	Same lesion patch reinserted
Replacement	Existing lesion removed from a scan	Different lesion of similar size inserted
Insertion	Healthy scan with sampled candidate box	Lesion from another scan inserted

For insertion, the procedure samples approximately 20 random candidate boxes in breast tissue and chooses the best overlap, or best of 20, as the insertion location. Metrics are computed on patches centered on the lesion region after compositing back into the full scan. Reported measures are FID, LPIPS, CLIP-I, and DINOv2 similarity. Baselines are direct copy-and-paste and the original natural-image AnyDoor applied without mammography-specific adaptation (Buburuzan, 30 Jul 2025).

The released implementation is announced at https://github.com/alexbuburuzan/AnydoorMed. The practical inference sequence is: select a target mammogram and 2D box, select a reference lesion patch, prepare context image and high-frequency map, encode reference and context, sample via DDIM for 50 steps, decode to image space, and composite the edited $\{\beta_t\}$0 crop back into the full-resolution mammogram, optionally with Gaussian blending along edges (Buburuzan, 30 Jul 2025).

5. Empirical behavior, controllability, and comparative position

Quantitative results are reported as averages over three runs. In reinsertion, AnydoorMed achieves FID $\{\beta_t\}$1, LPIPS $\{\beta_t\}$2, CLIP-I $\{\beta_t\}$3, and DINOv2 similarity $\{\beta_t\}$4. The corresponding AnyDoor scores are FID 6.80, LPIPS 0.19, CLIP-I 89, and DINOv2 34. In replacement, AnydoorMed reports FID $\{\beta_t\}$5, LPIPS $\{\beta_t\}$6, CLIP-I $\{\beta_t\}$7, and DINOv2 $\{\beta_t\}$8, versus AnyDoor at 7.42, 0.20, 88, and 32; copy-and-paste gives FID 4.39 and LPIPS 0.08. In insertion into healthy scans, AnydoorMed reports FID $\{\beta_t\}$9, LPIPS $\epsilon_\theta$0, CLIP-I $\epsilon_\theta$1, and DINOv2 $\epsilon_\theta$2, versus AnyDoor at 7.93, 0.21, 89, and 31; copy-and-paste gives FID 4.64 and LPIPS 0.10 (Buburuzan, 30 Jul 2025).

These results indicate a consistent pattern. Relative to direct application of AnyDoor, the mammography-specific adaptation reduces FID and LPIPS while improving semantic alignment to the reference lesion. Relative to copy-and-paste, the model trades exact pixel reuse for more natural blending. The paper’s qualitative examples further report that calcification patterns and spiculated margins are reasonably preserved, lesions are integrated with consistent breast-tissue intensity and noise, and reinserted lesions are visually similar but not pixel-identical to originals, suggesting regeneration of plausible variants rather than direct copying (Buburuzan, 30 Jul 2025).

AnydoorMed supports controllability along three explicit axes. First, location is controlled by a user-provided or automatically selected bounding box or mask, and inpainting is confined to this region. Second, the reference lesion controls lesion type, morphology, and approximate size, since the reference crop is scaled to fit the mask. Third, the target scan determines the surrounding tissue background and density pattern. This enables healthy-scan counterfactuals, lesion replacement counterfactuals, and augmentation of rare cases or high BI-RADS categories, although structured attribute controls such as explicit BI-RADS, shape, or size conditioning are not yet implemented (Buburuzan, 30 Jul 2025).

The model is positioned against two neighboring lines of work. Relative to MObI, introduced in the same paper, AnydoorMed shares the latent diffusion backbone, reference-guided inpainting, zero-initialised gated cross-attention, and frozen-backbone plus small-adapter strategy, but differs in domain, conditioning, and objective: MObI addresses multimodal autonomous-driving scenes with RGB, lidar, and 3D box conditioning, whereas AnydoorMed addresses single-modality mammography with 2D bounding boxes, high-frequency maps, and lesion-level medical counterfactuals (Buburuzan, 30 Jul 2025). Relative to prior medical diffusion systems that use segmentation maps or text prompts, AnydoorMed uses a visual reference patch for fine-grained control over texture and shape and explicitly leverages high-frequency maps to preserve microstructures (Buburuzan, 30 Jul 2025).

6. Limitations, risk profile, and prospective extensions

Several limitations are explicit. Structural fidelity is not guaranteed: generated lesions may differ from the reference in subtle ways such as shape or calcification count. Poorly chosen bounding boxes, especially those extending outside breast tissue, can produce anatomically implausible content. Some outputs exhibit copy-paste-like or stitching artefacts at mask boundaries. The model is trained only on VinDR-Mammo and may not generalise to other mammography vendors, populations, or other modalities such as MRI, CT, or ultrasound (Buburuzan, 30 Jul 2025).

The empirical validation is also bounded. There is no radiologist user study; realism is assessed by image metrics and visual inspection. Failure cases include under-preservation of microcalcifications, lesions that appear smoother than desired, and anatomically implausible generations when the spatial prior is violated (Buburuzan, 30 Jul 2025).

The synthetic-data risk profile is treated cautiously. Synthetic lesions could introduce bias if specific patterns or BI-RADS categories are over- or under-represented, and downstream models trained heavily on synthetic data might learn non-clinical artefacts. The authors explicitly caution against direct clinical use without rigorous validation and expert oversight, and recommend that synthetic images augment rather than replace real data in training and evaluation. They also note possible misuse, including fabrication of medical evidence, and recommend transparency about synthetic versus real data (Buburuzan, 30 Jul 2025).

Future directions outlined in the discussion include downstream task evaluation on lesion detection and classification, extension to MRI, CT, ultrasound, and 3D volumetric inpainting, incorporation of stronger anatomical priors such as organ or tissue segmentation and density maps, addition of structured attribute conditioning for lesion type, BI-RADS, or size, and radiologist-in-the-loop assessment of realism and control. The paper also proposes interpretability-oriented use cases in which counterfactual lesions probe how diagnostic systems respond to changes in lesion type or location (Buburuzan, 30 Jul 2025).

In the current form, AnydoorMed is best understood as a domain-specialised, reference-guided inpainting framework for mammographic counterfactual generation. Its distinctive technical contribution lies in adapting a natural-image inpainting backbone to mammography through a 1-channel VAE, DINOv2-based reference encoding, and explicit high-frequency conditioning, while retaining a lightweight fine-tuning regime and strong control over lesion identity and placement (Buburuzan, 30 Jul 2025).