---
title: 'Object-AVEdit: Object-Level Audio-Visual Editor'
url: https://www.emergentmind.com/topics/object-avedit
type: topic
---

# Object-AVEdit: Object-Level Audio-Visual Editor

Object-AVEdit is an object-level audio-visual editing model introduced for video post-production and film making, designed to perform object addition, replacement, and removal across both audio and visual modalities while preserving the structural information of the source instances during editing [2510.00050]. The system is built on an inversion-regeneration paradigm, combines a word-to-sounding-object well-aligned audio generation model with video editing on Mochi-1, and targets fine audio-visual semantic alignment rather than independent unimodal manipulation [2510.00050]. Related summaries place the term alongside object-centric appearance-shape video editing and audio-sync instance editing, which suggests a broader family of object-grounded editing problems across video and audiovisual generation [2401.02473; 2512.10571].

## 1. Problem formulation and scope

Object-AVEdit addresses a task that is more constrained than generic prompt-based generation and more demanding than single-modality editing. The goal is to edit an audiovisual clip at the level of semantic objects, such as “Add rain to this dog bark video,” “Replace the cat with a dog,” or “Remove the lion roar,” while leaving the rest of the scene as close as possible to the source and keeping the edited audio and video semantically matched [2510.00050].

The method is organized around three simultaneous requirements. The first is **semantic correctness**: the requested object must actually be added, replaced, or removed. The second is **structural preservation**: unedited content in both audio and video should remain close to the original. The third is **cross-modal alignment**: edited audio and video should still correspond to one another after editing [2510.00050]. This combination distinguishes Object-AVEdit from vision-only editors, audio-only editors, and loose audiovisual pipelines that concatenate independent components.

A central difficulty identified by the method is asymmetry between video and audio controllability. In video diffusion editing, prompt-to-prompt style attention manipulation already offers a mechanism for relatively fine spatial control. In audio, by contrast, existing generators such as AudioLDM, AudioLDM2, Stable Audio Open, and JavisDiT audio are described as lacking sufficiently word-level controllable attention maps for precise object editing [2510.00050]. Object-AVEdit therefore treats audio controllability, not merely video controllability, as a first-class design problem.

## 2. Inversion-regeneration formulation

Object-AVEdit uses an inversion-regeneration pipeline in latent space for both modalities. The original audio or video sample is encoded by a VAE, edited in latent space, and decoded back to the output domain:

$$
\boldsymbol{z} = \text{VAEencoder}(\boldsymbol{x}), \qquad
\boldsymbol{x}^* = \text{VAEdecoder}(\boldsymbol{z}^*)
$$

The generator is written in a flow-matching form. Given noisy latent $\boldsymbol{z}_{t_i}$, denoising toward $\boldsymbol{z}_{t_{i-1}}$ is approximated by a first-order update:

$$
\boldsymbol{z}_{t_{i-1}} = \boldsymbol{z}_{t_i} + (t_{i-1}-t_i)\hat{\boldsymbol{\epsilon}}_\theta(\boldsymbol{z}_{t_i},t_i,c)
$$

The learned velocity field is trained with

$$
\hat{\boldsymbol{\epsilon}}_\theta = \arg\min_\theta \mathbb{E}_{\boldsymbol{z}_0,\boldsymbol{z}_1,t}
\left\| \hat{\boldsymbol{\epsilon}}_\theta(\boldsymbol{z}_t,t,c) - (\boldsymbol{z}_1-\boldsymbol{z}_0) \right\|^2
$$

and inversion reverses the same flow:

$$
\boldsymbol{z}_{t_i} = \boldsymbol{z}_{t_{i-1}} + (t_i-t_{i-1}) \hat{\boldsymbol{\epsilon}}_\theta(\boldsymbol{z}_{t_{i-1}},t_{i-1},c)
$$

These equations formalize the paper’s claim that successful editing depends on both nearly information-lossless inversion and high-quality controlled regeneration [2510.00050].

The resulting system is not an end-to-end jointly trained audiovisual editor in the later flow-matching sense. Rather, the same inversion-regeneration logic is applied to audio and video separately, and the resulting outputs are then assessed for semantic alignment [2510.00050]. This architectural choice is one of the key distinctions between Object-AVEdit and later jointly trained audiovisual editing models.

## 3. Word-to-sounding-object aligned audio generation

A major contribution of Object-AVEdit is a dedicated audio generation model intended to make object-level editing possible in the acoustic domain. The model is explicitly described as a **word-to-sounding-object well-aligned audio generation model**, introduced to bridge the gap in object controllability between audio and current video generation models [2510.00050].

Its architecture includes a **VAE / AutoencoderKL** for mel-spectrogram latent representation, a **T5 text encoder** for word-level text conditioning, a **DiT** for flow-matching denoising, and a **HiFi-GAN vocoder** for waveform reconstruction from mel spectrograms [2510.00050]. The design objective is not only generation quality but also an explicit correspondence between word-level embeddings and sounding objects in the denoising attention process, so that the editor can access attention maps for the object to be changed and for objects that should remain unchanged [2510.00050].

The training objectives are reported as a VAE loss

$$
\mathcal{L}_{\text{VAE}} = \lambda_1\mathcal{L}_1 + \lambda_{\text{KL}}\mathcal{L}_{\text{KL}} + \lambda_{\text{GAN}}\mathcal{L}_{\text{GAN}}
$$

and a DiT flow-matching loss

$$
\mathcal{L}_{\text{DiT}} = \mathbb{E}_{\boldsymbol{z}_0,\boldsymbol{z}_1,t}
\left\| \hat{\epsilon}_{\theta}(\boldsymbol{z}_t,t,c) - (\boldsymbol{z}_1-\boldsymbol{z}_0) \right\|^2
$$

The appendix-level model details reported for the audio generator are also unusually explicit: **16 kHz** sampling rate, **64** mel channels, mel frequency range **0–8 kHz**, **96 layers** in the DiT, hidden size **1024**, **1.62B** DiT parameters, **716.8M** parameters for the **T5-large** encoder, **8** VAE latent channels, and **4×4** downsampling [2510.00050]. These choices position the audio branch as a large-scale generative model rather than a lightweight edit head.

## 4. Holistic optimization of inversion and regeneration

The second core contribution is the **inversion-regeneration holistically-optimized editing algorithm**, which explicitly improves both source preservation during inversion and edit quality during regeneration [2510.00050].

For inversion, the method uses **repeated inversion**. At each inversion step, it initializes

$$
\boldsymbol{z}_{t_{i+1}}^0 = \boldsymbol{z}_{t_i}
$$

then iterates $K$ times:

$$
\boldsymbol{z}_{t_{i+1}}^{k+1} = \boldsymbol{z}_{t_i}^{k} + (\sigma_{t_{i+1}} - \sigma_{t_i}) \hat{\boldsymbol{\epsilon}}_\theta(\boldsymbol{z}_{t_{i+1}}^k,t_{i+1})
$$

and averages the estimates:

$$
\boldsymbol{z}_{t_{i+1}} = \frac{1}{K}\sum_{k=1}^{K}\boldsymbol{z}_{t_{i+1}}^k
$$

This repeated-estimation-and-averaging step is presented as a mechanism for reducing inversion noise and preserving source geometry and background structure [2510.00050].

For regeneration, the method replaces left-endpoint velocity estimation with a **midpoint velocity estimate**. It defines

$$
t_{\text{mid}} = \frac{1}{2}(t_i+t_{i-1})
$$

then computes

$$
z_{t_{\text{mid}}} = z_{t_i} + (t_{\text{mid}}-t_i)\hat{\epsilon}(z_{t_i},t_i,C)
$$

followed by

$$
z_{t_{i-1}} = z_{t_i} + (t_{i-1}-t_i)\hat{\epsilon}(z_{t_{\text{mid}}},t_{\text{mid}},C)
$$

The practical claim is that midpoint estimation better approximates the integrated velocity field and thereby improves regeneration quality [2510.00050].

The final editing stage uses **Prompt-to-Prompt style attention control**. Let $M_t^s$ and $M_t^c$ be source-consistent self- and cross-attention maps, and let $(M_t^s)^*$ and $(M_t^c)^*$ denote the corresponding edited attention maps. The controlled maps are

$$
\overline{M_t^c} =
\begin{cases}
(M_t^c)^* & \text{if } t < \tau^c \\
(M_t^c)_{A(j)} & \text{otherwise}
\end{cases}
$$

$$
\overline{M_t^s} =
\begin{cases}
(M_t^s)^* & \text{if } t < \tau^s \\
M_t^s & \text{otherwise}
\end{cases}
$$

where $A(j)$ maps the $j$-th target token to the corresponding source token position, and $\tau^s,\tau^c$ set preservation strength [2510.00050]. Early denoising thus preserves source structure, while later denoising allows target-prompt content to dominate.

The preservation strengths differ by modality and task. For **video**, addition and replacement use $\tau_s = 0.42, \tau_c = 0.42$, while removal uses $\tau_s = 1.00, \tau_c = 0.42$. For **audio**, addition and replacement use $\tau_s = 0.75, \tau_c = 0.75$, while removal uses $\tau_s = 1.00, \tau_c = 0.75$ [2510.00050]. The paper also reports **64** inversion/sampling steps for **Mochi-1** and **100** denoising steps for the audio generator [2510.00050].

## 5. Datasets, evaluation protocol, and reported performance

Object-AVEdit uses two levels of data. For training the audio generation model, it draws on **FSD50K**, **ClothoV2**, **AudioCaps**, **MACS**, and **VGGSound** [2510.00050]. For editing evaluation, it constructs a new **Object-AVEdit dataset** consisting of **3-second audio-video pairs** with **complex scenes** and **addition / replacement / removal** tasks, mainly from **VGGSound**, together with an **Object-AVEdit-Alignment** subset for cases requiring strong changes in both audio and video [2510.00050].

The baselines are modality-specific and cross-modal. Audio editing is compared against **DDIM Inversion**, **ZEUS**, and **SDEdit**; video editing against **RAVE** and **RF-Edit**; and audio generation against **AudioLDM**, **AudioLDM2**, and **JavisDiT audio** [2510.00050]. Evaluation uses **CLAP** and **LPAPS** for audio editing, **CLIP-T**, **CLIP-F**, and **MUSIQ** for video editing, and **SAS**—the mean cosine similarity between **ImageBind** embeddings of edited audio-video pairs—for audiovisual semantic alignment [2510.00050].

The reported numbers are summarized below.

| Setting | Metric(s) | Reported result |
|---|---|---|
| Audio editing, addition | CLAP, LPAPS | -0.0800, 2.5700 |
| Audio editing, replacement | CLAP, LPAPS | 0.2646, 2.6330 |
| Audio editing, removal | CLAP, LPAPS | -0.0866, 2.4294 |
| Audio-visual alignment | SAS | 0.3500 |
| Audio generation | CLAP, KL, FAD | 0.3473, 1.4125, 0.0945 |

For comparison, the paper reports lower SAS values for mixed baseline combinations, including **0.1340** for DDIM/RAVE, **0.1311** for DDIM/RF-Edit, **0.2496** for DDPM/RAVE, **0.2374** for DDPM/RF-Edit, **0.2110** for SDEdit/RAVE, and **0.2003** for SDEdit/RF-Edit [2510.00050]. This is presented as evidence that object-level editing requires more than independently strong unimodal editors.

The video-side results are also task-specific. Reported examples include **CLIP-F = 0.9971** and **MUSIQ = 62.1520** for addition, **CLIP-T = 25.5853** and **CLIP-F = 0.9956** for replacement, and **CLIP-F = 0.9971** for removal [2510.00050]. Qualitatively, the method is described as preserving backgrounds and non-target regions, correctly inserting, replacing, or removing target objects, and yielding cleaner spectrogram changes than baselines [2510.00050].

## 6. Relation to neighboring work, misconceptions, and later developments

Object-AVEdit sits at the intersection of two previously distinct lines of work: object-centric visual editing and joint audio-visual generation. On the visual side, **OIR** argued that diffusion editing should move from fixed whole-image inversion toward per-object inversion-step search, disassembly, and latent reassembly for fine-grained multi-object editing [2310.12149]. **VASE** then formulated real-video object-centric appearance and shape manipulation for one foreground object at a time, with reference-image appearance control and user-specified keyframe masks for shape propagation; the paper explicitly identifies joint appearance-shape editing as its main “Object-AVEdit” setting [2401.02473]. **EVA** addressed zero-shot multi-attribute and multi-object video editing through spatial-temporal layout-guided attention, emphasizing precise text-to-attribute binding and leakage suppression in human-centric videos [2403.16111].

On the audiovisual side, **“Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation”** introduced language-guided joint AV editing from a single reference audio-image pair, using multimodal one-shot adaptation and cross-modal semantic enhancement to counter catastrophic neglect in the vision branch [2410.07463]. **AV-Edit** later focused on sound-effect editing by jointly leveraging visual, audio, and text semantics through CAV-MAE-Edit and MM-DiT, but its output domain is edited audio rather than full bidirectional audiovisual object editing [2511.21146]. **AVI-Edit** shifted the task toward audio-sync video instance editing with a granularity-aware mask refiner and self-feedback audio agent, and it explicitly argues that inversion/regeneration-style Object-AVEdit lacks temporal controllability and cannot precisely control when an event should happen [2512.10571]. **SAVE** moved further toward end-to-end joint flow matching in latent space, using a Schrödinger Bridge to learn direct transport from source to target audiovisual mixtures and reporting stronger temporal synchronization and audiovisual semantic correspondence than pairwise combinations of an audio editor and a video editor [2512.12875].

A common misconception is that object-level audiovisual editing is merely the composition of an audio editor and a video editor. Object-AVEdit itself argues against this by introducing a new audio model for word-to-object controllability and by evaluating joint semantic alignment with SAS rather than only unimodal quality metrics [2510.00050]. Later work reinforces this point from different angles: AVI-Edit emphasizes temporal control and audio guidance [2512.10571], while SAVE treats editing as a coupled multimodal transport problem rather than two separate regeneration problems [2512.12875]. Other later editors pursued different control interfaces altogether, including mask-free text-only inference in **LoVoRA** and albedo-guided unified instance editing in **AlbedoEdit**, again indicating that “object-level” editing can be instantiated through substantially different conditioning and optimization paradigms [2512.02933; 2606.01362].

The available description of Object-AVEdit does not include an explicit limitations section. A plausible implication is that performance remains sensitive to inversion accuracy, preservation-strength hyperparameters, and prompt-to-token correspondence, because these variables are structurally central to repeated inversion, midpoint regeneration, and attention replacement [2510.00050]. Later critiques around temporal controllability and the move toward end-to-end multimodal transport suggest that Object-AVEdit is best understood as a transitional model: it extends prompt-based diffusion editing to object-level paired audio and video with strong reported semantic alignment, while leaving open the broader question of whether inversion-regeneration remains the most effective foundation for fully synchronized, temporally precise audiovisual editing [2510.00050; 2512.10571; 2512.12875].

Source: https://www.emergentmind.com/topics/object-avedit