DiffBMP: Bitmap Primitive Rendering

• DiffBMP is a differentiable rendering framework that treats arbitrary 2D bitmap primitives as the basis for optimized image composition.
• It employs a custom CUDA tile-and-bin pipeline to efficiently compute gradients and scale optimization over thousands of raster elements.
• The system integrates soft rasterization and structure-aware initialization to enhance both static image and video inverse graphics performance.

DiffBMP is a differentiable rendering framework for scenes composed of arbitrary 2D bitmap primitives—raster assets such as fingerprints, logos, flowers, textures, or handwritten marks—rather than vector shapes. In the named arXiv work, the term denotes “Differentiable Rendering with Bitmap Primitives”, a scalable rendering engine that makes the position, rotation, scale, color, and opacity of bitmap elements differentiable and optimizable. The system is positioned as the bitmap-primitive counterpart to vector-focused differentiable renderers, with a custom CUDA implementation intended to support optimization over thousands of raster elements on consumer GPUs (Hong et al., 26 Feb 2026).

1. Definition and representational scope

DiffBMP addresses a limitation identified in prior differentiable rendering practice: most differentiable 2D renderers have focused on vector primitives, whereas “most images in the world are bitmaps.” The framework therefore treats arbitrary raster cutouts as the atomic primitives of a scene rather than attempting vectorization of real-world assets. The paper characterizes this as the first general-purpose, scalable differentiable renderer for arbitrary 2D bitmap images used as primitives, and argues that bitmap primitives impose distinct computational burdens because they are discrete, high-dimensional arrays rather than analytic shapes (Hong et al., 26 Feb 2026).

A DiffBMP scene is represented as a set of $N$ bitmap primitives. For primitive $i$, the learnable parameters are the center position $(x_i,y_i)$, scale $s_i$, rotation $\theta_i$, opacity logit $\nu_i$, and RGB color logit $c_i\in\mathbb{R}^3$. Each primitive also carries a fixed bitmap template $P_i$ of spatial size $(H_i,W_i)$. Rendering proceeds by spatially transforming each primitive onto the canvas, sampling its bitmap values by bilinear interpolation, modulating them by learnable opacity and color, and compositing all primitives front-to-back.

This scope is important because it makes the framework suitable for assemblage-style inverse graphics with real raster assets. The paper’s examples include fingerprints, autographs, logos, flowers, and foreground video elements, which indicates that DiffBMP is intended less as a specialized benchmarking renderer than as a general bitmap composition system.

2. Forward rendering model

The forward model maps each canvas pixel $(x,y)$ into the normalized local coordinates of primitive $i$0 through a rotation-and-scale transform: $i$1 These local coordinates are then mapped to bitmap coordinates

$i$2

Because $i$3 is generally non-integer, DiffBMP samples the primitive by bilinear interpolation. If $i$4, $i$5, $i$6, and $i$7, then the bilinear weights are

$i$8

with sampled template values

$i$9

and interpolated contribution

$(x_i,y_i)$0

Opacity is defined per pixel as

$(x_i,y_i)$1

where $(x_i,y_i)$2 is sigmoid. Front-to-back Porter-Duff “over” compositing is then expressed through the transmittance

$(x_i,y_i)$3

and the final image

$(x_i,y_i)$4

The alpha image is

$(x_i,y_i)$5

The framework also supports an optional color-preservation formulation in which the effective primitive color blends a fixed original color $(x_i,y_i)$6 and a learnable variation $(x_i,y_i)$7: $(x_i,y_i)$8 with $(x_i,y_i)$9. According to the paper, this is used, for example, in the logo-based Marilyn Monroe assemblage (Hong et al., 26 Feb 2026).

3. Differentiability, objective functions, and optimization aids

DiffBMP is differentiable because every stage of the pipeline is differentiable: the affine coordinate transform, bilinear sampling, sigmoid parameterization of opacity and color, and alpha compositing. The paper emphasizes that the backward pass is not an approximate visibility surrogate, but an exact chain-rule gradient through raster sampling and compositing. For example, the derivative with respect to a primitive’s $s_i$0 coordinate is written as

$s_i$1

The paper also provides explicit expressions for color and alpha gradients, including the dependence of a primitive’s opacity on downstream transmittance terms.

The global optimization problem is cast as

$s_i$2

where $s_i$3 contains all primitive parameters and $s_i$4 is the differentiable renderer. The paper mainly uses pixel reconstruction objectives, especially MSE/PSNR-oriented losses. For spatially constrained rendering it defines

$s_i$5

with the target condition

$s_i$6

This loss restricts RGB reconstruction to the target foreground while separately supervising the rendered alpha matte.

Three optimization aids are central. First, soft rasterization via Gaussian blur is applied to primitives before optimization. The implementation details specify do_gaussian_blur = true and blur_sigma = 1.0. The stated purpose is to enrich spatial gradients, because without blur bilinear interpolation yields meaningful position gradients mainly near primitive edges. Second, structure-aware initialization allocates primitives according to image structure. Local variance is computed over RGB channels with a $s_i$7 sliding window, normalized to $s_i$8, and primitive centers are sampled with probability

$s_i$9

Scale is then initialized by

$\theta_i$0

so high-variance regions receive smaller primitives and low-variance regions larger ones. Colors are initialized near the target,

$\theta_i$1

rotation uses

$\theta_i$2

and opacity is initialized to a low but nonzero value. The main text states $\theta_i$3, while the supplementary common hyperparameters list v_init_bias=-4.0, so the exact initialization varies across configurations. Third, noisy canvas blending renders the foreground over a random background

$\theta_i$4

which prevents the optimizer from leaving target-colored regions uncovered merely because they match the static canvas color (Hong et al., 26 Feb 2026).

4. CUDA system design and computational scaling

A major contribution of DiffBMP is its custom tile-and-bin CUDA rendering pipeline, adapted from tile-based differentiable splatting ideas to 2D bitmap primitives. The image plane is partitioned into $\theta_i$5 tiles, with default tile size $\theta_i$6. On the CPU, each primitive is assigned to every tile whose bounding box it overlaps, with small padding. On the GPU, one CUDA thread block is launched per tile, with $\theta_i$7 threads per block and one thread per pixel in the tile. Threads cooperatively stage primitive parameters in shared memory and composite tile-local primitive lists in front-to-back order.

The paper presents this as the key scalability mechanism. A naive implementation has complexity $\theta_i$8, whereas the tiled renderer behaves as $\theta_i$9, where $\nu_i$0 is the average number of primitives affecting a tile or pixel after culling. The supplementary comparison states the naive PyTorch implementation has memory per iteration $\nu_i$1, while the CUDA implementation reduces this to $\nu_i$2.

The backward kernel mirrors the forward structure and uses packed __half2 atomic accumulation for parameter pairs $\nu_i$3, $\nu_i$4, $\nu_i$5, and $\nu_i$6. The paper reports that plain __half atomicAdd hurt accuracy, __half atomicCAS restored accuracy but was too slow, and packed __half2 atomicAdd preserved accuracy while improving throughput and reducing memory traffic. The implementation uses FP16 or mixed precision for texture fetches, temporaries, and several arithmetic paths.

The reported runtime and memory figures are correspondingly large. At $\nu_i$7 resolution on an RTX 3090, the naive PyTorch baseline requires 1360 ms forward, 2337 ms backward, and 6.4 GB VRAM, whereas CUDA-16bit requires 2.3 ms forward, 6.2 ms backward, and 1.1 GB VRAM. At $\nu_i$8, CUDA-16bit reports 4.3 ms forward, 5.5 ms backward, and 1.6 GB VRAM. At $\nu_i$9, CUDA-16bit reports 9.0 ms forward, 6.4 ms backward, and 3.8 GB VRAM. The supplementary summary describes the CUDA renderer as roughly 30–50× faster than the naive PyTorch baseline, and the abstract states that the system can optimize the attributes of thousands of bitmap primitives “all in under 1 min using a consumer GPU” (Hong et al., 26 Feb 2026).

5. Task-specific extensions and workflow integration

DiffBMP is extended beyond static image fitting in several ways. For videos, the system uses sequential frame optimization with warm starts,

$c_i\in\mathbb{R}^3$0

so each frame is initialized from the optimized parameters of the previous frame. The paper then introduces two heuristics to improve temporal behavior. A primitive is marked “stuck” if it satisfies a large-scale threshold

$c_i\in\mathbb{R}^3$1

a high-opacity threshold

$c_i\in\mathbb{R}^3$2

and a front-depth criterion based on percentile $c_i\in\mathbb{R}^3$3 in its spatial region. Per region, the top-$c_i\in\mathbb{R}^3$4 stuck primitives are selected by

$c_i\in\mathbb{R}^3$5

and their opacity logits are decayed by

$c_i\in\mathbb{R}^3$6

Typical hyperparameters are $c_i\in\mathbb{R}^3$7, $c_i\in\mathbb{R}^3$8, $c_i\in\mathbb{R}^3$9, $P_i$0, $P_i$1, and $P_i$2 or $P_i$3, depending on the experiment. A second heuristic freezes primitives in unchanged regions by computing

$P_i$4

and freezing any primitive whose bounding box $P_i$5 does not intersect the support of $P_i$6: $P_i$7

For spatially constrained rendering, low-opacity primitives are not permanently discarded but re-initialized when $P_i$8, typically 0.3, every prune_iterations, typically 50, outside warmup and optional final phases. The paper states that this preserves primitive budget for downstream editing while improving coverage of constrained foreground shapes.

The framework also includes a separate PSD export kernel. Because optimization kernels rely on atomics and are therefore unsuitable for isolated editable layers, DiffBMP exports each primitive into its own cropped layer using a primitive-level parallel kernel. The paper explicitly highlights low-resolution optimization followed by high-resolution export, including $P_i$9 or $(H_i,W_i)$0 PSD export, and describes Photoshop and After Effects as target editing environments. It also states that the system is publicly accessible through an easy-to-hack Python package (Hong et al., 26 Feb 2026).

6. Empirical behavior, applications, and limitations

The paper compares DiffBMP conceptually and experimentally against DiffVG. Using 2000 primitives at 512 px and 100 iterations, the reported results are: for a simple SVG target, DiffVG achieves 24.26 dB in 67 s and DiffBMP 25.04 dB in 51 s; for a complex SVG target, DiffVG achieves 14.35 dB in 477 s and DiffBMP 23.35 dB in 115 s; for a bitmap primitive target, DiffVG is listed as N/A and DiffBMP achieves 24.26 dB in 36 s. The stated interpretation is that DiffBMP is not only applicable to bitmaps, but more robust when primitive complexity becomes large (Hong et al., 26 Feb 2026).

Ablations support the optimization design. For soft rasterization and structure-aware initialization, the reported PSNRs are 24.4/20.6/25.9 with both off, 24.7/21.5/26.5 with blur on only, 25.5/21.0/27.1 with structure-aware initialization only, and 25.7/21.7/27.4 with both on. Primitive count scaling from 1000 to 4000 improves PSNR from 22.5 dB to 25.4 dB on a portrait-like example, from 19.6 dB to 21.3 dB on a natural scene example, and from 15.4 dB to 17.8 dB on a poster-like example. For dynamic video, the combined warm-start, stuck-removal, and freeze-unchanged configuration yields PSNR 24.38, SSIM 0.630, tLP 3.49, and tOF 1.84, while warm start plus stuck removal gives the best fidelity at PSNR 24.66 and SSIM 0.647.

The demonstrated applications are correspondingly broad. The paper includes Seurat-like compositions using thousands of fingerprints and autographs, a Marilyn Monroe portrait assembled from 300 logos while preserving intrinsic logo colors, spatially constrained flower and foreground compositions, sequential video optimization, and CLIP-guided text-based creation. For the CLIP-based experiments it reports 500–1000 primitives, $(H_i,W_i)$1 optimization resolution, 500 iterations, 16 augmentations, ViT-B/32 CLIP, and a negative prompt “blurry” with weight 0.1.

Several limitations are also explicit. DiffBMP fundamentally depends on GPU/CUDA because of the memory and computation demands of bitmap primitives. Performance is sensitive to hyperparameters such as initialization, primitive library, blur settings, opacity bounds, learning-rate gains, and background treatment, and local minima remain a concern. The paper also notes the absence of automatic hyperparameter selection, video-specific failure modes requiring heuristics, and the expectation that extension to 3D bitmap primitives would be substantially harder. A plausible implication is that DiffBMP is best understood as a practical inverse-graphics and creative optimization system for raster assets rather than a turnkey renderer with architecture-agnostic defaults.