End-to-End Rendering Integration

• End-to-end rendering is a framework that integrates differentiable graphics into trainable pipelines to optimize scene representations through image-space losses.
• It encompasses methods from neural view synthesis and compression to simulation and robotics, emphasizing the interplay between representations and task-specific objectives.
• The approach extends beyond image fidelity by incorporating computational efficiency, synchronization, and robust control within modern rendering systems.

Searching arXiv for recent and representative papers on end-to-end rendering across differentiable rendering, neural rendering, simulation, and systems. End-to-end rendering denotes a family of formulations in which rendering is no longer treated as an isolated terminal stage, but is embedded within a learnable or optimizable pipeline whose upstream variables remain trainable through image-space, task-space, or system-level objectives. Across the literature, the term covers several distinct but related usages: differentiable analysis-by-synthesis for 3D layout and inverse problems, neural rendering pipelines that map structured scene representations directly to images, coupled simulation-and-rendering systems for robotics and autonomous driving, compression or reconstruction pipelines optimized for final rendered quality, and networked XR stacks in which rendering is part of an end-to-end latency budget rather than a local graphics procedure (Luo et al., 2020, Wiles et al., 2019, Zhu et al., 2024, Hu et al., 2024, Yeregui et al., 2024). The common technical motif is that rendering becomes an internal operator whose outputs are compared to target observations or downstream criteria, so that optimization is driven by the rendered result itself rather than by detached intermediate surrogates.

1. From terminal graphics stage to differentiable computational graph

A central shift in end-to-end rendering is the replacement of fixed, non-differentiable rendering stages with differentiable operators that permit gradients to flow from image-space losses back to latent scene variables, geometry, pose, or control parameters. In "End-to-End Optimization of Scene Layout" (Luo et al., 2020), a conditional variational autoencoder generates 3D layouts from scene graphs, and a differentiable rendering module renders semantic and depth maps,

$$\widetilde{S} = R_S(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$

$$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$

so that losses on depth and semantics can update both latent vectors and decoder weights in an analysis-by-synthesis loop. The paper explicitly characterizes this as end-to-end optimization because scene graph conditioning, latent layout generation, 3D parameters, rendering, and 2D loss computation are connected by a single differentiable graph (Luo et al., 2020).

A closely related formulation appears in inverse rendering from a single image. "SynSin: End-to-end View Synthesis from a Single Image" (Wiles et al., 2019) predicts a depth map and a latent 3D point cloud of features from one input image, then uses a differentiable point-cloud renderer to project those features into a target view before a refinement network inpaints missing regions. The renderer splats each point into a disk of influence, keeps the $K$ nearest points per pixel, and blends them with alpha over-compositing; the final image loss therefore supervises both geometry and appearance without requiring ground-truth 3D supervision (Wiles et al., 2019). "LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation" (Park et al., 2019) similarly builds a latent 3D object from reference views and renders depth and mask differentiably from arbitrary viewpoints so that pose parameters $(R,t,c)$ can be refined directly by gradient descent.

In another branch of the literature, the differentiable stage is inserted into feature learning rather than scene synthesis. "End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds" (Li et al., 2020) integrates multi-view rendering into the descriptor network itself. A differentiable SoftRas-based renderer generates view patches from learnable viewpoints, a 2D CNN extracts per-view features, and a soft-view pooling module fuses them into a descriptor. Because the rendered images depend on viewpoint parameters, descriptor loss gradients can update those viewpoints jointly with the rest of the network (Li et al., 2020). This shifts rendering from offline preprocessing to an internal trainable layer.

These works share a precise operational meaning of end-to-end: supervision is imposed on rendered outputs, but the optimized variables are latent and geometric quantities upstream of rendering. A plausible implication is that “end-to-end rendering” is best understood not as a specific renderer class, but as a systems property of the optimization graph.

2. Representation choices: points, voxels, Gaussians, meshes, and latent assets

End-to-end rendering is strongly shaped by the chosen scene representation, because differentiability, memory use, rendering speed, and suitability for downstream tasks all depend on how geometry and appearance are encoded.

Point-based representations are prominent in view synthesis and reconstruction. SynSin represents a scene as a point cloud of learned features rather than a voxel grid, which the paper presents as more flexible than voxels and naturally compatible with varying output resolutions (Wiles et al., 2019). The 2017 single-image inverse-rendering framework "End-to-end 3D shape inverse rendering of different classes of objects from a single input image" (Kamyab et al., 2017) uses latent 2D and 3D representations connected by a mapping network; for faces it reconstructs dense point clouds directly in coordinate space, while for objects it uses binary voxel volumes. The paper explicitly argues that point clouds preserve more detail, whereas binary volumes are easier for nonlinear and convolutional decoding but can be blurrier (Kamyab et al., 2017).

Voxel and latent-volume representations remain important where explicit 3D reasoning is needed. LatentFusion reconstructs a canonical latent voxel volume from reference views and renders from that volume through differentiable warps and trilinear sampling (Park et al., 2019). DROID-Splat, although centered on 3D Gaussian Splatting, inherits a dense disparity-and-pose map from DROID-SLAM as the tracking state and then initializes Gaussians from the tracked map (Homeyer et al., 2024). This separation of tracking geometry from rendering geometry reflects a recurring tension between robust localization and photorealistic image synthesis.

Gaussian representations increasingly dominate work that seeks both differentiability and efficient rasterization. "Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering" (Hu et al., 2024) compresses point clouds so that the decoder directly reconstructs renderable 3D Gaussians rather than recovered points and colors. Each Gaussian is parameterized by center, scale, quaternion orientation, opacity, and RGB color, and the whole codec is trained with an image-based rate–distortion objective defined on rendered views rather than on point-wise reconstruction (Hu et al., 2024). DROID-Splat renders Gaussian hyperprimitives with positions, rotations, scales, density, and spherical-harmonic colors (Homeyer et al., 2024). "NeAR: Coupled Neural Asset-Renderer Stack" (Li et al., 23 Nov 2025) decodes a Lighting-Homogenized SLAT into a relightable 3D Gaussian field conditioned on target view and HDR environment lighting, explicitly avoiding spherical harmonics as the primary lighting carrier because they tend to bake illumination into appearance (Li et al., 23 Nov 2025).

Hybrid mesh-plus-Gaussian representations emerge when rendering must coexist with physics. "Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data" (Moran et al., 4 Jun 2025) introduces SplatMesh, in which an explicit triangle mesh supplies geometry for MuJoCo while 3D Gaussians supply photorealistic appearance. The Gaussians are attached to mesh faces by barycentric sampling, and surfel-like covariance constraints keep them surface aligned (Moran et al., 4 Jun 2025). "One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering" (Zhu et al., 2024) instead combines a differentiable point-based geometry representation $\mathcal{P}=\{(p,n)\}$, Poisson reconstruction on a $128 \times 128 \times 128$ grid, differentiable marching cubes, and a grid-based appearance field $\psi$ to obtain a world model that is both simulation-ready and rendering-ready (Zhu et al., 2024).

A recurrent conclusion across these papers is that end-to-end rendering is inseparable from representational co-design. This suggests that the renderer, the scene parameterization, and the loss must be designed as a coupled system rather than as independent modules.

3. Objective functions: image fidelity, geometry, rate–distortion, and task losses

The optimization criteria used in end-to-end rendering vary widely, but they almost always combine image-based terms with task-specific or physically motivated regularization.

In scene-layout refinement, the total loss

$$\mathcal{L}_{\text{total}}=\alpha \mathcal{L}_{\text{size}}+\beta \mathcal{L}_{\text{depth}}+\gamma \mathcal{L}_{\text{sem}}$$

balances object-size regularization, class-wise depth consistency, and semantic cross-entropy (Luo et al., 2020). In SynSin, the total training objective

$$\mathcal{L}=\lambda_{GAN}\mathcal{L}_{GAN}+\lambda_{l1}\mathcal{L}_{l1}+\lambda_c\mathcal{L}_c$$

combines adversarial, reconstruction, and content losses so that the renderer learns geometry while the refinement network handles disocclusion and realism (Wiles et al., 2019). LatentFusion adds depth, mask, IoU, and latent losses in a render-and-compare objective for pose refinement (Park et al., 2019).

Several works optimize directly for rendered perceptual quality. Bits-to-Photon defines a rate–distortion objective in which the distortion term is not point-wise reconstruction error but an image-space combination of $L_1$, SSIM, and LPIPS measured on rendered views: $$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$0 with $$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$1, $$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$2, and $$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$3 (Hu et al., 2024). In "End-to-End Adaptive Monte Carlo Denoising and Super-Resolution" (Wei et al., 2021), the final loss

$$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$4

combines edge, temporal, spatial, and perceptual terms to map a low-resolution, low-spp rendered image plus gBuffers to a high-resolution clean image, thereby reframing rendering post-processing as a jointly optimized super-resolution-and-denoising problem (Wei et al., 2021).

Inverse design problems add fabrication or physics constraints to image fidelity. "End-to-end Surface Optimization for Light Control" (Sun et al., 2024) minimizes a weighted sum of pixel loss, image-gradient loss, imaging-region loss, barrier terms, and smoothness regularization. The rendered light distribution is computed face-wise from triangle projections and overlap areas, while geometric constraints enforce height-field validity, prevent total internal reflection, and promote piecewise smoothness compatible with CNC milling and polishing (Sun et al., 2024). In real-to-sim identification, One-Shot Real-to-Sim defines a stage-1 loss with RGB, depth, point-cloud, penetration, balance, regularization, and smoothness terms, and a stage-2 loss for point-cloud and robot-joint consistency (Zhu et al., 2024). Splatting Physical Scenes adds photometric, mask, smoothed mask, normal, and Laplacian losses, with appendix weights of 1, 10, $$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$5, 3, and 3 respectively (Moran et al., 4 Jun 2025).

The literature also uses end-to-end rendering to optimize non-visual task objectives. "UNDREAM: Bridging Differentiable Rendering and Photorealistic Simulation for End-to-end Adversarial Attacks" (Phute et al., 19 Oct 2025) formulates adversarial texture optimization as

$$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$6

so that gradients flow from model loss through a differentiable renderer back to the texture on a 3D object embedded in a photorealistic scene (Phute et al., 19 Oct 2025). "PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning" (Gao et al., 16 Mar 2026) uses the term in a different operational sense: rendering is removed altogether in favor of a vector-space pseudo-simulation, and the resulting reward-driven loop optimizes closed-loop driving behavior without the rendering gap (Gao et al., 16 Mar 2026). This provides a useful contrast: some systems make rendering differentiable and central, whereas others eliminate rendering because it is the bottleneck or source of domain shift.

4. Neural rendering, single-step generation, and coupled asset–renderer stacks

A major contemporary use of end-to-end rendering concerns neural renderers that directly map structured inputs—G-buffers, latent assets, or compressed geometry—to final shaded images without explicit light transport simulation.

"RenderFlow: Single-Step Neural Rendering via Flow Matching" (Zhang et al., 11 Jan 2026) recasts neural rendering as a deterministic, single-step conditional generation problem. Instead of starting from Gaussian noise, it begins from the albedo latent and predicts a velocity field in latent space so that the final rendered latent is obtained by

$$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$7

The model is conditioned on albedo, normals, depth, material buffers, and a tone-mapped HDR environment map, and it uses an optional sparse keyframe guidance module for higher physical plausibility and better generalization (Zhang et al., 11 Jan 2026). The paper reports 512×512 inference times of about 0.19 s without keyframes and about 0.24 s with keyframe guidance, compared to about 2.19 s for RGB-X and about 1.40 s for DiffusionRenderer, while also improving PSNR, SSIM, and LPIPS (Zhang et al., 11 Jan 2026).

NeAR pursues a complementary strategy: instead of treating asset generation and rendering as separate problems, it couples them into a single learnable stack. A casually lit input image is first mapped to a Lighting-Homogenized SLAT latent, and a lighting-aware neural renderer then decodes that latent into a relightable Gaussian field conditioned on target view and lighting (Li et al., 23 Nov 2025). The first stage is trained with a conditional flow-matching objective, while the second stage jointly trains the Intrinsic Aware Decoder, the Lighting Aware Decoder, and the lighting tokenizer with losses on HDR reconstruction, PBR attributes, shadows, Gaussian volume, and opacity (Li et al., 23 Nov 2025). The paper frames this explicitly as an end-to-end learnable graphics stack and reports that a 12+6 block configuration reaches about 30 FPS while achieving 32.54 PSNR, 0.9649 SSIM, and 0.0442 LPIPS in ablation (Li et al., 23 Nov 2025).

Not all end-to-end neural rendering is fully image-synthesis-centric. The malaria diagnosis paper "End-to-end Malaria Diagnosis and 3D Cell Rendering with Deep Learning" (Ramesh, 2021) describes a DNN-based 3D rendering module that maps a 2D parasitic cell image to a point cloud, saves it as PCD, converts it to OBJ, and deploys it through echoAR and Sketchfab for AR and web visualization. The paper does not provide explicit equations, reconstruction losses, or rendering accuracy benchmarks, but it clearly frames the component as an integrated rendering stage within a diagnostic workflow (Ramesh, 2021). At the other end of the control spectrum, "MangaFlow: An End-to-End Agentic Framework for Controllable Story to Manga Generation" (Wang et al., 27 May 2026) uses “end-to-end rendering” to mean not one-shot page synthesis but a structured pipeline from story planning through layout generation, reference-conditioned panel rendering, composition, and lettering. Here the rendered outputs are panel images,

$$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$8

and controllability is achieved by making layout, references, and text placement explicit intermediate variables rather than latent entanglements (Wang et al., 27 May 2026).

A common misconception is that end-to-end rendering always implies direct image generation from raw prompts or raw sensors. The cited literature shows the opposite: many successful systems increase the number of explicit intermediate variables, but keep them inside a trainable end-to-end graph.

5. Rendering coupled with simulation, robotics, and autonomous systems

A distinctive expansion of end-to-end rendering places the renderer inside a simulation or control loop, often alongside differentiable physics. In these settings, rendering is both a source of supervision and a bridge between perception and dynamics.

One-Shot Real-to-Sim formulates world-model identification as

$$\widetilde{D} = R_D(\hat{y}_1,M_1,\hat{y}_2,M_2,\ldots,\hat{y}_N,M_N),$$9

where $K$0 includes geometry, appearance, and physical properties, $K$1 is a differentiable rigid-body simulator, and $K$2 is the rendered observation (Zhu et al., 2024). The same mesh recovered from oriented points and differentiable marching cubes is used both for collision detection and for rendering through PyTorch3D, so geometry affects both contacts and image loss (Zhu et al., 2024). Splatting Physical Scenes pushes this integration further by combining MuJoCo MJX and Gaussian splatting in a single optimization over robot poses, camera extrinsics, mesh vertices, and Gaussian appearance, driven by raw robot trajectories with noisy calibration and occlusion (Moran et al., 4 Jun 2025).

UNDREAM occupies a different point in the design space. Rather than making a full photorealistic simulator differentiable, it bridges Unreal Engine and Mitsuba so that adversarial optimization can run “through” a realistic 3D simulation. Unreal handles LevelSequence capture, object motion, weather, and lighting; Mitsuba provides the differentiable replica used for backpropagation to textures on arbitrary 3D objects (Phute et al., 19 Oct 2025). This architecture addresses the mismatch between rich but non-differentiable simulators and differentiable but less controllable renderers.

Autonomous driving work reveals both the importance and the limits of rendering. PerlAD argues that rendering-based RL introduces a “rendering gap,” defined as the discrepancy between training in a rendered simulator and deployment on real-world camera observations (Gao et al., 16 Mar 2026). The proposed solution is not better rendering but a rendering-free vector-space pseudo-simulation with a prediction world model and a hierarchical planner. The paper reports a Driving Score of 78.70 and Success Rate of 57.27 on Bench2Drive, surpassing Raw2Drive’s 71.36 DS and 50.24 SR, and reaching an average DS of 86.83 on the DOS occlusion benchmark (Gao et al., 16 Mar 2026). This is a significant counterexample within the broader topic: end-to-end optimization may benefit from removing rendering when rendering itself becomes the dominant source of inefficiency or domain shift.

Related work on sensing rather than simulation also complicates the picture. "LiDAR-as-Camera for End-to-End Driving" (Tampuu et al., 2022) uses LiDAR images with depth, intensity, and ambient radiation channels from a single sensor, thereby avoiding spatial and temporal misalignment that arises in multimodal camera-depth setups. This is not an end-to-end rendering paper in the differentiable-graphics sense, but it illustrates another operational meaning of rendering-like representations: sensor-native image formation can simplify end-to-end learning when it preserves alignment and geometric content (Tampuu et al., 2022).

6. Systems-level end-to-end rendering: compression, streaming, and XR latency

End-to-end rendering also appears in systems work where the central objective is not only visual fidelity but also decode time, network performance, or end-user latency.

Bits-to-Photon is exemplary in this regard. The codec generates a scalable bitstream whose decoded output is directly renderable 3D Gaussians, so the decoder need not reconstruct a conventional point cloud or mesh before rendering (Hu et al., 2024). On 8iVFB, the paper reports for B2P (SpPCGC), level 8: PSNR 32.1, color decode 26 ms, render 4 ms; and for level 9: PSNR 34.0, color decode 73 ms, render 5 ms (Hu et al., 2024). By contrast, G-PCC plus Gaussian rendering at similar rate has much lower PSNR and requires 2–4 seconds for geometry and color decoding, while Pointersect has about 1 second of rendering time (Hu et al., 2024). Here end-to-end means that the compression target is the final rendered image rather than intermediate point accuracy.

The same systems perspective appears in Monte Carlo rendering acceleration. End-to-End Adaptive Monte Carlo Denoising and Super-Resolution renders low-resolution, low-spp input—typically 540p at 8 spp with albedo, normal, and roughness gBuffers—and learns a two-stage network that first upsamples and then denoises (Wei et al., 2021). The paper reports that 1080p 8spp rendering takes 19.33s, while 540p 8spp takes 4.45s and neural post-processing is under 1s, yielding roughly a fivefold total speedup (Wei et al., 2021). The architectural argument is that super-resolution and denoising require different receptive fields, which motivates deformable convolution in the denoising stage (Wei et al., 2021).

At the networked XR level, "Edge Rendering Architecture for multiuser XR Experiences and E2E Performance Assessment" (Yeregui et al., 2024) uses end-to-end in a media systems sense. Rendering is offloaded to a GPU-enabled edge server; clients transmit video, audio, and 6DoF data, and receive rendered 2D video streams back through WebRTC (Yeregui et al., 2024). The paper introduces QR-code and audio-tone based methods to measure end-to-end video latency, end-to-end audio latency, inter-device synchronization, and intra-media synchronization directly from streams rather than from RTT proxies (Yeregui et al., 2024). Reported results show that with Ethernet, average video and audio latencies stay below 250 ms; inter-device asynchrony for 5 users is about 54.2 ms for video and about 45 ms for audio; with 5G-SA, video asynchrony is about 71.8 ms and audio about 38.8 ms; Wi‑Fi rises to 514.2 ms for video and 69.2 ms for audio (Yeregui et al., 2024). In this domain, end-to-end rendering refers to a delivery path whose quality is determined as much by network and synchronization behavior as by graphics algorithms.

A plausible implication is that “end-to-end rendering” now spans the full stack from asset representation and differentiable graphics to transport and display. The term is therefore partly contextual: in graphics papers it often refers to differentiable image formation, while in XR systems it often refers to motion-to-photon or mouth-to-ear performance across a distributed pipeline.

7. Conceptual boundaries, recurring misunderstandings, and emerging directions

The literature supports several clarifications about what end-to-end rendering is and is not.

First, end-to-end rendering is not synonymous with inverse rendering. Inverse rendering is one important application class, evident in single-image 3D shape reconstruction (Kamyab et al., 2017), pose estimation with differentiable reconstruction and rendering (Park et al., 2019), and optical surface design (Sun et al., 2024). But the concept also includes forward neural rendering (Zhang et al., 11 Jan 2026), descriptor learning (Li et al., 2020), compression for direct rendering (Hu et al., 2024), and networked XR delivery (Yeregui et al., 2024).

Second, end-to-end does not require the elimination of structure. MangaFlow demonstrates the opposite by exposing story sections, section memory, layouts, references, panel images, and text placement as editable intermediate controls, and it reports deterministic layout metrics of Count 100%, IoU 100%, Coverage 99.98%, and Overlap 0.62% precisely because composition is explicit rather than implicit (Wang et al., 27 May 2026). Likewise, 3D-SLN relies on scene graphs, meshes retrieved from SUNCG, and a differentiable renderer, rather than collapsing everything into a single unstructured network (Luo et al., 2020).

Third, differentiability is not always the goal; sometimes the rendering stage is removed because it harms transfer or efficiency. PerlAD is explicit that rendered simulation introduces both high computational cost and an input-domain shift between synthetic visuals and real sensor data (Gao et al., 16 Mar 2026). This suggests that the proper design question is not whether to include rendering, but whether rendering is the right internal representation for the downstream objective.

Across recent work, several trajectories are visible. One is the movement from iterative generation toward single-step deterministic rendering, exemplified by RenderFlow’s flow-matching formulation (Zhang et al., 11 Jan 2026). Another is the co-design of scene representation and renderer, seen in NeAR’s coupled asset–renderer stack (Li et al., 23 Nov 2025) and SplatMesh’s mesh-plus-Gaussian representation (Moran et al., 4 Jun 2025). A third is the increasing fusion of differentiable rendering with differentiable simulation for world-model identification and robot calibration (Zhu et al., 2024, Moran et al., 4 Jun 2025). A fourth is the extension of end-to-end criteria beyond image fidelity to include rate–distortion, synchronization, and safety-critical task performance (Hu et al., 2024, Yeregui et al., 2024, Gao et al., 16 Mar 2026).

Taken together, these developments indicate that end-to-end rendering has evolved from a narrow term for differentiable graphics into a broader methodological principle: rendered outputs, or the decision to avoid rendering, are optimized in direct relation to the final task. This suggests that future progress will depend less on treating rendering as a self-contained graphics primitive and more on integrating it, or strategically bypassing it, within larger optimization loops spanning perception, simulation, interaction, compression, and control.