RETVec: Resilient and Efficient Text Vectorizer (2302.09207v3)

Abstract: This paper describes RETVec, an efficient, resilient, and multilingual text vectorizer designed for neural-based text processing. RETVec combines a novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model is pre-trained using pair-wise metric learning to be robust against typos and character-level adversarial attacks. In this paper, we evaluate and compare RETVec to state-of-the-art vectorizers and word embeddings on popular model architectures and datasets. These comparisons demonstrate that RETVec leads to competitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVec is available under the Apache 2 license at https://github.com/google-research/retvec.

• The paper introduces RETVec, a novel text vectorization model that integrates a unique UTF-8 character encoding and a compact embedding model to yield a 256-dimensional representation.
• The paper leverages pair-wise metric learning to pre-train embeddings, enhancing typo resilience by up to 15% and reducing adversarial vulnerabilities by over 10%.
• The method achieves high efficiency with a sub-1MB memory footprint and fast processing on GPU and multi-core CPUs, making it ideal for on-device applications.

RETVec: Resilient and Efficient Text Vectorizer

The paper discusses the development and evaluation of RETVec, a novel text vectorization approach designed to offer resilience and efficiency, particularly in the face of multilingual text processing challenges and adversarial attacks. RETVec's innovative design integrates a unique character encoding method with a compact embedding model, which together enable the conversion of text into a 256-dimensional vector space. This approach circumvents common pitfalls associated with out-of-vocabulary (OOV) issues, such as susceptibility to typographical errors and character-level adversarial attacks, by utilizing pair-wise metric learning. The extensive evaluation highlights its competitive performance against state-of-the-art vectorizers and embeddings like SentencePiece, BPE, and fastText.

Key Contributions and Findings

1. Combined Character Encoding and Embedding Model: RETVec incorporates a unique UTF-8 character encoder, comprising an Integerizer layer that converts characters to UTF-8 codepoints and a Binarizer layer that encodes characters into a 24-bit binary format. This innovative encoding scheme leads to an efficient and learnable representation. An optional 230k parameter embedding model further refines the projection into a 256-dimensional space, demonstrating a marked improvement in accuracy and adversarial resilience.
2. Pre-trained Robust Embeddings: RETVec's embeddings are pre-trained using pair-wise metric learning, which ensures that similar words are embedded closer together in the vector space. This training approach enhances the model's resilience to typographical errors and adversarial manipulations, which are critical for real-world applications where human inputs often contain typos or intentionally deceptive modifications.
3. Resilience to Typos and Adversarial Attacks: Experimental results reveal that RETVec improves typo resilience by up to 15% at a 20% word typo rate compared to other vectorizers. It also reduces vulnerability to character-level adversarial attacks by more than 10%, thus strengthening the performance in anti-abuse environments such as spam detection.
4. Efficiency and Applicability: The implementation of RETVec results in a memory footprint of less than 1MB, making it resource-efficient and suitable for deployment in on-device scenarios. The evaluation outlined that RETVec is faster than other vectorizers, particularly when GPU acceleration is available, and exhibits favorable computational performance on multi-core CPUs.
5. Performance Across Diverse Architectures and Tasks: The assessment across various datasets and models (RNN, CNN, and Transformer-based architectures) establishes that RETVec consistently delivers competitive accuracy and robustness across diverse languages and text processing tasks, including multilingual classification.

Implications and Future Directions

The development of RETVec addresses significant challenges in the domain of text processing, particularly with its robustness against adversarial attacks and typographical errors. This robustness is critical in deploying models in safety-critical and adversarial environments. The space and computational efficiency of RETVec also make it a viable option for on-device applications, offering a notable reduction in computational overhead and resource demands.

In relation to the broader AI landscape, RETVec represents an important step towards developing text vectorizers and embeddings that maintain high performance across linguistic barriers and diverse input conditions. However, the adaptation of RETVec for generative tasks and integration within LLMs remains an open area for exploration. Strategies to extend RETVec to these domains could leverage its compact representation and robustness to potentially reduce the computational footprint of LLMs without sacrificing performance.

The paper has laid foundational work in embedding robustness, opening pathways for enhanced multilingual support and adversarial attack mitigation. It will be pertinent for future research to explore how RETVec can be refined and expanded to further improve its applicability, particularly under the demands of advanced AI systems.