---
title: 'OmniText-Bench: 1D Controllable Image-Text Testbed'
url: https://www.emergentmind.com/topics/controllable-1d-image-text-testbed
type: topic
---

# OmniText-Bench: 1D Controllable Image-Text Testbed

A controllable 1D image-text testbed, exemplified by the OmniText-Bench protocol, provides a structured experimental environment for research in controllable text-image manipulation (TIM). This testbed is designed to evaluate methods handling fine-grained, spatially localized text insertion, removal, editing, style transfer, and more, using precise annotation schemas and standardized evaluation routines. The architecture centers on diffusion-based generative models augmented for advanced attention manipulation and latent optimization. It includes task pipelines for developing, benchmarking, and comparing TIM systems under unified and reproducible conditions [2510.24093].

## 1. Dataset Structure and Annotation Protocol

Each sample within the OmniText-Bench testbed consists of four primary components:

- **Input Image $(I)$**: A $512 \times 512$ RGB image or cropped region that may contain existing text relevant to the specified TIM operation.
- **Target Mask $(M)$**: A binary mask ($M(p) = 1$ for pixels $p$ to be removed, edited, or inserted; $0$ otherwise), registered to the input image to identify operative regions.
- **Target Text $(T)$**: The ground-truth string, e.g., "FLASH", specifying the intended textual content of the region of interest.
- **Style Reference $(I_{\mathrm{ref}}, m_{\mathrm{ref}})$**: Optional. An image and mask denoting the region(s) whose text style (font, color, stroke) serve as a target for style-aware manipulation. For tasks without explicit style transfer, $(I_{\mathrm{ref}}, m_{\mathrm{ref}}) = (I, M)$.

Annotations are organized by sample directory, with files `input.png` (image), `mask.png` (mask), `text.txt` (target text), and—if necessary—`ref.png` and `ref_mask.png` for style reference.

**Task mappings** to these elements are as follows:

| Task                 | Input $I$          | Target $T$      | Mask $M$          | Style Ref $(I_{\mathrm{ref}}, m_{\mathrm{ref}})$        |
|----------------------|--------------------|-----------------|-------------------|----------------------------------------------------------|
| Text Removal         | Contains text      | "" (empty)      | Where text is     | None                                                     |
| Text Insertion       | Blank in $M=1$     | New string      | Where to insert   | $(I, M)$                                                 |
| Text Editing         | Old text in $M=1$  | New string      | Where to edit     | $(I, M)$                                                 |
| Text Rescaling       | Old text present   | New string      | $M_{\text{old}}, M_{\text{new}}$ | $(I, M_{\text{old/new}})$                |
| Text Repositioning   | Original text      | Unchanged       | $M_{\text{old}}, M_{\text{new}}$ | $(I, M_{\text{old/new}})$                |
| Style-Based Operation| As above           | Varies          | Varies            | $(I_{\mathrm{ref}}, m_{\mathrm{ref}})$                   |

This structure enables isolation and control of content and style manipulations in both input and ground-truth specifications [2510.24093].

## 2. Evaluation Metrics

Evaluation is conducted using standardized, image-informed and text-informed metrics:

- **Text Removal**: Assessed over the full $512 \times 512$ output using
  - PSNR (Peak Signal-to-Noise Ratio, $\uparrow$)
  - MS-SSIM ($\times 10^{-2}$, $\uparrow$)
  - FID (Fréchet Inception Distance, $\downarrow$)

- **Text Insertion, Editing, Rescaling, Repositioning**: Assessed on cropped regions defined by $M$ or $M_{\text{new}}$, using
  - **Content Accuracy**: Word-level accuracy (ACC, $\%$, $\uparrow$), Normalized Edit Distance (NED, $\uparrow$) via scene-text OCR.
  - **Style Fidelity**: Pixel-wise MSE ($\times 10^{-2}$, $\downarrow$), PSNR ($\uparrow$), MS-SSIM ($\times 10^{-2}$, $\uparrow$), FID ($\downarrow$).

For style-aware tasks, fidelity metrics are computed over style-referenced insertions, ensuring that both semantic content and style attributes are jointly evaluated [2510.24093].

## 3. Model Architecture and Attention Mechanisms

OmniText builds upon a pre-trained, latently diffused text-inpainting U-Net backbone (TextDiff-2), incorporating two dedicated operational modules:

- **Text Removal (TR)** performed via attention modulation at sampling time:
  - **Self-Attention Inversion (SAI)**: Inverts self-attention activations over masked regions in early sampling steps to suppress residual focus on removed text:
    $$
    S^{l}_{i,j} = \max_{j'} S^{l}_{i, j'} + \min_{j'} S^{l}_{i, j'} - S^{l}_{i, j}
    $$
    where $S^l$ is the self-attention matrix for U-Net decoder block $l$.
  - **Cross-Attention Reassignment (CAR)**: Constrains cross-attention to align only with start/end tokens:
    $$
    C^l_{i,j} =
      \begin{cases}
        1, & (i \in m \wedge j = E_d) \lor (i \notin m \wedge j = S_d) \\
        0, & \text{otherwise}
      \end{cases}
    $$
    SAI is applied during the first $50\%$ of sampling steps; CAR is applied at all steps.

- **Controllable Inpainting (CI)** enabled by latent-space optimization:
  1. Grid construction: $G=\mathrm{concat}([z_{I_r \cdot (1-M_{\mathrm{shr}})};z_{I_{\mathrm{ref}}}], [m_{\mathrm{shr}};0])$ where $m_{\mathrm{shr}}$ is a possibly shrunk mask for editing.
  2. Latent optimization proceeds during the first $0-40\%$ of diffusion steps. At each step, losses are computed and gradients are backpropagated to $z_t$.

The combined use of direct attention manipulation and optimization-driven inpainting delivers fine-grained control over both appearance and semantics in the manipulated text regions [2510.24093].

## 4. Latent Optimization: Loss Formulations and Update Procedure

Two novel loss functions stabilize and guide latent inpainting:

- **Cross-Attention Content Loss ($\mathcal{L}_C$)**: Ensures content-accurate rendering of each character by maximizing cross-attention activations for each target token over sub-mask regions. The loss for each character sub-mask $m_{c_k}$ and token $c_k$ is formulated:
  $$
  \mathcal{L}_{C} = \sum_{k=1}^N \sum_{i,j} FL\left( C^l_{i, j = c_k},\ 1_{[i\in m_{c_k}]} \right)
  $$
  employing Focal Loss with $\gamma=2$.

- **Self-Attention Style Loss ($\mathcal{L}_S$)**: Aligns the distribution of self-attention within the masked region to a normalized reference mask $GT$ by minimizing the KL divergence:
  $$
  GT_i = \frac{(m_{\mathrm{ref}})_i}{\sum_{i'} (m_{\mathrm{ref}})_{i'}},\quad
  \mathcal{L}_S = D_{\mathrm{KL}}(GT\;\|\;S^l_{i \in m, \bullet})
  $$

The total loss is $\mathcal{L} = \lambda_C \mathcal{L}_C + \lambda_S \mathcal{L}_S$ (default weights: $\lambda_C=5, \lambda_S=10$). Latent variables $z_t$ are updated by Adam with a learning rate of $1\mathrm{e}{-2}$ during the specified optimization stages ($0\%$, $20\%$, $40\%$ along the diffusion timeline) [2510.24093].

## 5. End-to-End Testbed Instantiation and Usage

The testbed runtime consists of well-defined pipelines and hyperparameters:

- **Sampling steps**: 20 total (TR + CI)
- **SAI steps**: first 10 (for TR tasks)
- **Latent optimization**: performed at $0\%$, $20\%$, $40\%$ of the timeline
- **Adam LR**: $1 \times 10^{-2}$
- **Loss weights**: $\lambda_C=5$, $\lambda_S=10$

Reference pseudocode specifies the operational logic for all task types, leveraging a function such as `denoise_step(z, mask, grid, token_embed)` to carry out diffusion steps with attention hooks. The pipeline branches depend on the task—TR for erasure, CI for insertion/edit/transfer—using the provided sample masks, text, and style references. Example CLI commands clarify reproducibility, such as:

```python
python omnitext.py --task removal --input input.png --mask mask.png --n_steps 20
python omnitext.py --task insertion --input blank_region.png --mask mask.png --text "HELLO"
python omnitext.py --task style_insertion --input blank_region.png --mask mask.png --text "SALE" --ref_img style_ref.png --ref_mask style_mask.png --lambda_C 5 --lambda_S 10
python omnitext.py --task editing --input old_sign.png --mask old_mask.png --text "OPEN" --lambda_C 5 --lambda_S 10
python omnitext.py --task move --input sign.png --mask old_mask.png --new_mask new_mask.png --text "ENTER"
```

Generated outputs are benchmarked against ground-truth data using the metrics and crop protocols stipulated above [2510.24093].

## 6. Research Significance and Applications

OmniText-Bench establishes a versatile and controllable 1D image-text testbed, directly enabling universal TIM research and development:

- **Broader TIM Applicability**: Encompasses removal, insertion, arbitrary editing, geometric and stylistic variants.
- **Fine-Grained Style Control**: Through referential guidance, it supports tasks involving heterogeneous font, stroke, and color transfer.
- **Unified Metrics and Protocols**: Enables rigorous comparison across methods, including both generalist and specialist architectures.

A plausible implication is that this design paradigm facilitates rapid iteration for both foundational TIM models and downstream applied research in real-world signage, document editing, and graphic design scenarios. All operational details, annotation formats, and benchmarking routines are specified for transparent reproduction [2510.24093].

Source: https://www.emergentmind.com/topics/controllable-1d-image-text-testbed