Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Interfacing Foundation Models' Embeddings (2312.07532v2)

Published 12 Dec 2023 in cs.CV, cs.AI, and cs.CL

Abstract: Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in the teaser figure, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  2. Openleaf: Open-domain interleaved image-text generation and evaluation. arXiv preprint arXiv:2310.07749, 2023.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  4. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  5. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.
  6. UNITER: universal image-text representation learning. In ECCV, pages 104–120, 2020.
  7. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022.
  8. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023.
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
  10. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2982–2992, 2021.
  11. Davit: Dual attention vision transformers. In European Conference on Computer Vision, pages 74–92. Springer, 2022.
  12. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023.
  13. Leo Grady. Random walks for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 28(11):1768–1783, 2006.
  14. Levit: a vision transformer in convnet’s clothing for faster inference. In ICCV, pages 12259–12269, 2021.
  15. Rethinking spatial dimensions of vision transformers. In ICCV, pages 11936–11945, 2021.
  16. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  17. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  18. Grounding language models to images for multimodal inputs and outputs. 2023.
  19. Obelics: An open web-scale filtered dataset of interleaved image-text documents. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
  20. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 1:2, 2023a.
  21. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023b.
  22. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pages 121–137, 2020.
  23. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  24. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023a.
  25. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023b.
  26. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016.
  27. Modeling context between objects for referring expression understanding. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 792–807. Springer, 2016.
  28. OpenAI. Improving image generation with better captions. Technical report, OpenAI, 2023a.
  29. OpenAI. Gpt-4 technical report. Technical report, OpenAI, 2023b.
  30. Scaling vision with sparse mixture of experts. NeurIPS, 34, 2021.
  31. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv: Computer Vision and Pattern Recognition, 2021.
  32. Bottleneck transformers for visual recognition. In CVPR, pages 16519–16529, 2021.
  33. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  34. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  35. Attention is all you need. In NeurIPS, 2017.
  36. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
  37. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175, 2023.
  38. Rethinking and improving relative position encoding for vision transformer. In ICCV, pages 10033–10041, 2021.
  39. Florence-2: Advancing a unified representation for a variety of vision tasks. arXiv preprint arXiv:2311.06242, 2023.
  40. Unified contrastive learning in image-text-label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19163–19173, 2022.
  41. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023.
  42. FILIP: fine-grained interactive language-image pre-training. In ICLR, 2022.
  43. Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016.
  44. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
  45. Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305.18279, 2023.
  46. Scaling vision transformers. arXiv: Computer Vision and Pattern Recognition, 2021.
  47. Vinvl: Making visual representations matter in vision-language models. arXiv preprint arXiv:2101.00529, 2021.
  48. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023.
  49. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15116–15127, 2023a.
  50. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718, 2023b.
Citations (2)
View on Semantic Scholar

Summary

  • The paper presents FIND, a novel method that unifies vision and language embeddings without altering foundation model weights.
  • It leverages a lightweight transformer to create a shared embedding space, enhancing retrieval and segmentation tasks with state-of-the-art results.
  • FIND offers prototypability and extendability, enabling versatile multi-modal AI systems that adapt across a variety of tasks.

An Analysis of "Interfacing Foundation Models' Embeddings"

The paper "Interfacing Foundation Models' Embeddings" presents a novel approach named FIND, designed to unify foundation models' embeddings, specifically across vision and language modalities. This initiative addresses the growing complexity and specialization of models within these domains.

Overview and Methodology

FIND introduces a transformative concept in model interfacing by leveraging a lightweight transformer-based architecture that doesn't necessitate tuning the foundational model weights. This architecture effectively collaborates with various foundation models, such as GPT-4(V), DALLE-3, and SAM, across tasks including retrieval and segmentation, without altering their intrinsic configurations. The proposed solution offers four significant attributes:

  1. Generalizability: FIND is applicable to a multitude of tasks spanning different granularity levels and modalities.
  2. Prototypability: It allows task-specific configurations through the prototyping of attention masks and embedding types rather than architectural modifications.
  3. Extendability: The architecture is adaptive to new tasks and models, ensuring long-term applicability.
  4. Interleavability: It interleaves embedding spaces across tasks, creating a shared space that facilitates more cohesive interaction between image and LLMs.

The development of FIND-Bench complements FIND's capabilities by providing a benchmark that includes new annotations for evaluating interleaved segmentation and retrieval tasks, grounded on WELLstructured COCO dataset annotations.

Strong Numerical Results

The evaluation of FIND demonstrated superior performance on FIND-Bench in interleaved tasks and competitive outcomes across standard tasks such as retrieval and segmentation. It effectively handled both in-domain and zero-shot tasks, showing adaptability and robustness in multi-modal environments. Notably, FIND achieved state-of-the-art results in interleave segmentation and retrieval tasks, underscoring its interleaved shared embedding space's efficacy.

Theoretical and Practical Implications

Theoretically, FIND contributes a significant advancement in the quest for unified architectures capable of leveraging the strengths of disparate foundational models. By creating a shared embedding space, it sets the precedent for more cohesive multimodal integration, which could be pivotal in developing generalist models with broader applicability than the current set of specialist models.

Practically, FIND's modular design allows it to be a flexible and robust solution for a range of applications, from image data processing to complex language tasks that require contextual understanding. This adaptability is particularly relevant given the increasing investment in multi-modal AI applications in industry-specific contexts, such as personalized content delivery and advanced automated assistance systems.

Future Trajectories in AI

While FIND exhibits a level of versatility and cohesiveness uncommon in the current state of multi-modal AI systems, future work could focus on exploring additional tasks such as interleaved image/video generation and extending the interaction framework to incorporate a broader range of foundation models. Furthermore, addressing the challenges associated with long-context understanding across both visual and textual domains could redefine the capability spectrum of multi-modal transformers.

In summary, FIND sets a foundational milestone in interfacing embeddings across modalities, showcasing that innovative architectural strategies can significantly augment the collaborative potential of language and vision foundation models. This research opens avenues for further exploration into creating more unified, efficient, and contextually aware AI systems.

Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - UX-Decoder/FIND (91 stars)