Xray-Visual: Unified Image-Video Model
- Xray-Visual is a unified vision model family that jointly processes images and videos using a shared transformer architecture and large curated social-media datasets.
- It employs a three-stage training pipeline combining self-supervision, weak supervision, and CLIP-style alignment to boost classification and retrieval performance.
- Efficiency is achieved through token reorganization and 3D tokenization, reducing computational costs while maintaining robust performance across modalities.
Xray-Visual denotes a family of large-scale unified vision models and a scaling recipe for joint image and video representation learning trained on industry-scale Facebook and Instagram data. In the paper’s presentation, it supports images, videos, and text, and is aimed at image classification, video understanding, image-text and video-text retrieval, and production retrieval and recommendation systems. The paper also alternates between the names “Xray-Visual,” “XRay,” and “XrayVisual,” and reports multiple variants, including image-only and unified image-video models built on Vision Transformers and efficient token reorganization (Mishra et al., 18 Feb 2026).
1. Definition, scope, and model family
Xray-Visual is framed as a response to a scaling asymmetry between vision and language: the paper argues that contemporary vision encoders underperform LLMs in part because vision has not yet been trained at comparable scale or with equally rich curation. Its answer is not a single fixed network but a coordinated system comprising very large curated corpora, a unified image-video encoder, and a three-stage pretraining pipeline that mixes self-supervision, weak supervision, and CLIP-style alignment. The model family is explicitly intended for image classification, video understanding and action recognition, image-text and video-text retrieval, and production retrieval and recommendation use cases (Mishra et al., 18 Feb 2026).
A central design choice is unification rather than modality-specific specialization. Instead of maintaining separate image and video towers, the best unified variants use a video ViT with 3D tokenization and adapt still images to that same frontend by repeating the image along the temporal axis and using zero-padding and unpadding for positional embeddings. This means image and video examples share encoder weights and most training objectives. The paper presents this as a key reason joint training improves video performance while keeping image performance strong (Mishra et al., 18 Feb 2026).
The family includes large EViT-based backbones, especially a roughly 2B-parameter EViT-2B, alongside smaller distilled variants such as ViT-H and ViT-L. The overall emphasis is less on a novel multimodal fusion block than on a scalable recipe: large data, unified pretraining across images and videos, token-efficiency mechanisms, and stronger text encoders for retrieval (Mishra et al., 18 Feb 2026).
2. Data curation and corpus construction
A major part of Xray-Visual is its data pipeline. The paper states that the raw corpora initially exceeded 100 billion images and videos, then were curated into task-oriented datasets. For images, the corpus is ViSE, sourced from public Facebook and Instagram posts; for video, the corpus is URU, sourced from Instagram posts. The paper is explicit that these corpora are not used in raw form: both undergo aggressive cleaning, balancing, and noise suppression (Mishra et al., 18 Feb 2026).
For images, the text is cleaned by removing URLs, emails, emojis, user tags, and non-semantic characters, while preserving hashtag words after stripping the #. WordNet synsets are used to identify visually meaningful concepts and deduplicate synonyms, and captions are segmented into sentences so one image can yield multiple image-sentence pairs. Long-tail balancing is performed by undersampling head concepts and oversampling rare ones. The paper reports that, for a ViT-B trained from scratch for 700K iterations, ImageNet linear-probe accuracy improved from 65.4% without resampling to 72.7% with resampling, a gain of 7.3 points. For additional noise suppression, image-text pairs are filtered using MetaCLIP similarity; the best reported setting uses a threshold of 0.25 plus label smoothing in contrastive training, improving ImageNet linear accuracy from 71.7% to 72.8% after 200K iterations (Mishra et al., 18 Feb 2026).
For video, the paper argues that hashtags are often better supervisory signals than raw captions because they are more visually and semantically targeted. Hashtags are canonicalized with WordNet and a manually or algorithmically built mapping of about 75k user-hashtag to canonical-hashtag pairs. After long-tail rebalancing, the paper reports about 5B video-canonical-hashtag pairs and defines two supervised label spaces: URU-Video-Noun-5B with 22,013 labels for the MAE and hashtag-classification stages, and URU-Video-Action-5B with 31,619 labels for CLIP training. It also states that synthetic captions are generated with an internal multimodal LLM and then rewritten by Llama 3B prompts to reduce repetition (Mishra et al., 18 Feb 2026).
| Corpus | Reported curated scale | Notes |
|---|---|---|
| ViSE | over 15B image-text pairs; approximately 10B in another section | Public Facebook and Instagram posts |
| URU | 10B video-hashtag pairs in the abstract; about 5B in the video section | Instagram posts; noun and action label spaces |
These counts are themselves part of the paper’s ambiguity. The abstract says Xray-Visual leverages over 15 billion curated image-text pairs and 10 billion video-hashtag pairs, while the data sections also report approximately 10 billion high-quality image-text pairs for ViSE and about 5 billion video-hashtag pairs for URU. That inconsistency is material because the scale claim is one of the paper’s central arguments, and the article treats it as an unresolved documentation issue rather than a settled statistic (Mishra et al., 18 Feb 2026).
3. Unified architecture and efficiency mechanisms
Architecturally, Xray-Visual is built on a Vision Transformer backbone, with a unified image-video encoder realized by choosing a video ViT with 3D tokenization. Videos are processed directly; images are adapted to the same interface by temporal repetition and positional-embedding adjustment. The paper presents this as a principled way to avoid the duplication of separate image and video encoders while preserving strong performance on both modalities (Mishra et al., 18 Feb 2026).
Its headline efficiency mechanism is EViT, which reduces the quadratic cost of transformer attention by pruning inattentive patch tokens. The token-importance score is derived from attention between the CLS token and patch tokens, and token reorganization is inserted at three uniformly selected transformer layers. On ViT-H14, moving from 224 resolution with keep rate 1.0 to 336 resolution with keep rate 0.5 changes throughput from 29.85 to 23.87 images/s on one V100 while improving linear classification from 86.99 to 87.38 and zero-shot performance from 79.05 to 80.03. The paper’s headline efficiency claim is that Xray reaches 89.3% ImageNet top-1 using 336 px input and only 288 active tokens, versus baselines at 448 px and 1024 tokens, corresponding to a 71.9% token reduction, a 43.75% pixel-area reduction, about an 84.2% reduction in the token×pixel-area proxy, and roughly 6.3× lower combined cost (Mishra et al., 18 Feb 2026).
The paper also reports inference-time compression techniques. With ToMe token merging on ViT-H at 280 resolution, ImageNet accuracy drops by less than 0.5% and Kinetics by only 0.04%, while video inference speed increases by 58%. Progressive-resolution training is also used to control compute: stages 1 and 2 run at 224×224, while stage 3 uses a schedule of 98→154→224→336→448 (Mishra et al., 18 Feb 2026).
On the language side, the baseline text encoder is a CLIP-style transformer, but the paper regards it as too small and constrained by a 77-token limit. Scaling the standard text tower from 12 layers, 8 heads, 512 dimension to 36 layers, 20 heads, 1280 dimension raises ImageNet zero-shot only from 71.24% to 72.13%, while linear-probe performance remains around 83.1%. The more consequential change is LLM2CLIP-style integration of LLaMA-1B: the LLM is adapted with LoRA, bidirectionalized with LLM2Vec, fine-tuned with supervised SimCSE on MS-COCO captions plus reannotated ShareCaptioner captions, then kept frozen during final CLIP training while a 4-layer adapter is learned for alignment (Mishra et al., 18 Feb 2026).
4. Three-stage training recipe
The training pipeline has three stages. Stage 1 is MAE pretraining on images and videos. The masking rates are 75% for image patches and 90% for video patches. Stage 2 is semi-supervised hashtag classification on large-scale noun-centric hashtag data, including URU-Video-Noun-5B for video and similarly hashtagged images. Stage 3 is CLIP-style contrastive learning on image-caption and video-caption pairs, augmented by several auxiliary design choices: register tokens, data augmentations, SLIP self-supervision, denoising loss, and progressive-resolution training (Mishra et al., 18 Feb 2026).
The denoising term is one of the few objectives described concretely. The visual embedding is down-projected to 0.1× its original size via an MLP encoder, Gaussian noise is added, and an MLP decoder reconstructs the original embedding. The denoising loss is an objective,
The paper describes the total objective as effectively combining the CLIP loss and the denoising loss, and reports that works best, giving about +0.3% ImageNet accuracy over CLIP-only training and improving scaling from 1B to 5B unique examples at fixed 500K iterations (Mishra et al., 18 Feb 2026).
SLIP is used as an additional self-supervised term. For video, its positive pairs are two frames from the same video rather than two crops from the same image. The training-stage ablation on ViT-H reports a cumulative pattern: MAE→URU gives 75.1 K700 / 87.1 IN, adding CLIP gives 75.2 / 87.3, adding register tokens gives 75.4 / 87.3, adding augmentations gives 76.1 / 87.6, and adding SLIP reaches 76.7 / 87.8. The augmentations in this stage include brightness jitter, saturation jitter, horizontal flip, and Gaussian blur (Mishra et al., 18 Feb 2026).
This staged design is the paper’s operational core. Rather than training one objective end-to-end from scratch, Xray-Visual first acquires scalable masked-reconstruction priors, then injects weak semantic supervision through hashtags, then aligns visual and textual representations for retrieval and zero-shot transfer. The paper presents this as a practical recipe for large-scale multimodal vision pretraining rather than as a narrow benchmark optimization (Mishra et al., 18 Feb 2026).
5. Empirical performance and robustness
Empirically, the strongest image-only result is X-Ray Image EViT-2B at 89.30% ImageNet top-1 linear probe. The unified X-Ray Visual EViT-2B reaches 88.10% ImageNet and 78.10% Kinetics top-1 linear probe, compared in the paper’s table with 76.68% Kinetics for PE-G and 70.19% for DINOv3. On video linear probing, it reports 98.24% on UCF101 and 74.69% on HMDB51, while also noting that PE-G is higher on HMDB51 at 77.50%, so Xray is not best on every benchmark (Mishra et al., 18 Feb 2026).
For zero-shot and retrieval-oriented evaluation, smaller distilled models remain competitive. X-Ray Visual ViT-H reaches MSCOCO image-to-text R@1 of 73.83, text-to-image R@1 of 55.11, and ImageNet top-1 of 83.33. In LLM2CLIP settings, retrieval improvements are clearer than classification improvements: with a ViT-H vision backbone, replacing the standard text encoder by LLM2CLIP improves COCO image-to-text R@1 from 48.21 to 49.71, COCO text-to-image R@1 from 31.12 to 37.1, MSRVTT video-to-text R@1 from 51.4 to 58.2, and MSRVTT text-to-video R@1 from 27.92 to 31.9, while ImageNet and K700 decline slightly in that table, from 79.19 to 77.05 and from 61.53 to 60.52 respectively (Mishra et al., 18 Feb 2026).
The paper also emphasizes robustness and transfer. Xray EViT-2B scores 94.4 on ImageNet-Rendition, 72.9 on ImageNet-Sketch, 88.9 on ImageNet-A, and 83.8 on ObjectNet in one table. In another MMEB comparison, XRay EViT-2B reports 79.4 on ObjectNet, 68.6 on ImageNet-A, and 95.3 on ImageNet-R. For zero-shot video MMEB, a 400M X-Ray Visual model scores 63.0 overall versus 57.6 for Qwen2-VL 7B, with 66.7 on K-700 and 54.2 on Something-Something V2 (Mishra et al., 18 Feb 2026).
The paper’s most deployment-oriented evidence comes from internal retrieval tasks. XRay-LLM2CLIP ViT-L achieves 90.21 AUC on ads-to-ads, 67.03 on Instagram reels-to-ads, and 77.7 on Facebook reels-to-ads, versus 88.54, 56.14, and 67.36 for PE-L. The authors interpret this as evidence that the model’s primary gains come not merely from benchmark tuning but from training on much more diverse social-media data and from stronger text semantics in the LLM-based text encoder (Mishra et al., 18 Feb 2026).
6. Ambiguities, limitations, and the name “Xray-Visual”
Several limitations are intrinsic to the paper’s presentation. Most conspicuously, key dataset counts are inconsistent: ViSE is described as over 15 billion image-text pairs in some places and approximately 10 billion in others, while URU appears as 10 billion video-hashtag pairs in the abstract and about 5 billion in the main data description. Synthetic-caption quality is also reported inconsistently, with about 3% repetition before rewriting in one passage and hallucinations, repetition, or verbosity affecting less than 1% of captions in another. The paper is also explicit that Xray is not best on every benchmark, citing weaker results on HMDB51 and on some academic transfer datasets such as SUN397, RESISC45, and CIFAR, which it attributes partly to domain and resolution mismatch (Mishra et al., 18 Feb 2026).
The training recipe has its own trade-offs. Synthetic captions help retrieval but can harm action recognition: the paper reports about a 4% Kinetics drop when training only with synthetic captions, mitigated with a siamese loss combining synthetic and hashtag captions at a 1:1 weighting. The work is also unusually strong on scale and engineering detail while remaining weaker on methodological precision in places: several formulas are omitted, and the vision-side story is inseparable from large proprietary social-media corpora, which raises unresolved questions about noise, bias, governance, and language skew (Mishra et al., 18 Feb 2026).
A recurrent source of confusion is the name itself. Despite “Xray-Visual,” the model family is not a radiography or medical X-ray system. It is a large-scale general-purpose image-video model trained on Facebook and Instagram data (Mishra et al., 18 Feb 2026). That distinguishes it from genuinely X-ray-domain work such as training-free open-vocabulary object detection for baggage scans in RAXO (Garcia-Fernandez et al., 21 Mar 2025), self-supervised multiview radiograph correspondence learning for fracture analysis (Dabboussi et al., 30 Jun 2025), and physically grounded sparse-view X-ray reconstruction in X-Field (Wang et al., 11 Mar 2025). A plausible implication is that the name should be read as a branding label for a vision scaling framework rather than as an indicator of literal X-ray imaging scope.
In that sense, Xray-Visual is best understood less as a single architecture than as a compound program for scalable unified visual representation learning: aggressively curated large corpora, a shared image-video transformer, efficiency through token reduction, staged pretraining, and LLM-backed text alignment. Its significance lies in showing how far that recipe can be pushed under industry-scale data and compute, while its main unresolved questions concern documentation consistency, social-data dependence, and the extent to which benchmark gains reflect generalizable vision competence rather than exceptional scale (Mishra et al., 18 Feb 2026).