Piccolo2: General Text Embedding with Multi-task Hybrid Loss Training (2405.06932v1)

Abstract: In this report, we introduce Piccolo2, an embedding model that surpasses other models in the comprehensive evaluation over 6 tasks on CMTEB benchmark, setting a new state-of-the-art. Piccolo2 primarily leverages an efficient multi-task hybrid loss training approach, effectively harnessing textual data and labels from diverse downstream tasks. In addition, Piccolo2 scales up the embedding dimension and uses MRL training to support more flexible vector dimensions. The latest information of piccolo models can be accessed via: https://huggingface.co/sensenova/

• The paper introduces a multi-task hybrid loss training method that effectively manages retrieval, classification, and semantic similarity tasks.
• It scales embedding dimensions from 768 to 1792 and employs Matryoshka Representation Learning for flexible, robust performance.
• The model achieves superior results on the CMTEB benchmark, demonstrating its strong capability in clustering and nuanced text similarity.

Understanding Piccolo2: Advancements in Chinese Text Embeddings

Introduction to Text Embeddings

Text embeddings are a cornerstone in the field of NLP. They convert textual content into a numerical format that machines can understand, maintaining semantic meanings in a lower-dimensional space. Utilized in various applications like sentiment analysis, document retrieval, and more, embeddings are crucial for effectively handling and processing language data.

Piccolo2 emerges as an advancement in this arena, particularly focusing on multi-task applications and optimizing performance across heterogeneous tasks. It employs a new training methodology distinct from the typical single-task-focused models.

Training Enhancements in Piccolo2

Multi-Task Hybrid Loss Training

Piccolo2 revolutionizes the standard training process by incorporating a multi-task hybrid loss training approach. This methodology is tailored to accommodate diverse requirements of different NLP tasks such as retrieval, classification, and sentence similarity—each having unique nuances and demands.

• Retrieval Tasks: Standard practices utilize InfoNCE loss, leveraging in-batch negatives to enhance learning from each query-document pair.
• Semantic Text Similarity (STS) and Classification: Here, finer distinctions in text pairs are crucial. Piccolo2 employs a cosent loss function designed specifically for fine-grained label differentiation, providing a significant boost in performance for STS tasks.
• Clustering and Category Classification: These are tackled using an innovative contrastive learning format, transforming classification labels into contrastive pairs, simplifying the learning process.

This hybrid loss approach enables Piccolo2 to perform robustly across a broad spectrum of tasks, as demonstrated by its leading results on the CMTEB benchmark.

Dimension Scaling and MRL Training

Increasing the embedding dimension was another strategy used. By scaling up from 768 to 1792 dimensions, the model essentially expands its capacity to encapsulate more information within each embedding.

Moreover, Matryoshka Representation Learning (MRL) is applied, which supports embeddings of variable lengths. This offers flexibility in deployment environments where computational resources or latency constraints might vary. The ability to maintain high performance even with reduced dimensionality underscores the efficiency and technical sophistication of Piccolo2.

Data Strategy and Benchmark Performance

Data Synthesis and Hard Negative Mining

Piccolo2's training leverages both synthetic and mined hard negatives, enhancing the model’s ability to discern subtle nuances in text similarity and relevance. Through techniques in synthetic data generation and strategic hard negative sampling, the model is exposed to a diverse array of training scenarios, further robustifying its performance.

Benchmarking Against CMTEB

When assessed on the CMTEB benchmark, which evaluates models across six different tasks, Piccolo2 demonstrated superior performance, particularly enhancing results in classification and clustering tasks. This validates the effectiveness of the multi-task hybrid training approach and the utility of hard negative mining in real-world application scenarios.

Future Directions and Conclusions

Piccolo2's success on the CMTEB benchmark is just the starting point. With its flexible, high-capacity embedding and multi-task oriented training approach, it sets a new standard for text embedding models, especially in handling Chinese language data.

Potential future work could explore the integration of unsupervised or self-supervised learning elements to further refine the embedding qualities without heavy reliance on labeled data. Additionally, extending these methodologies to other languages could see Piccolo2's applications becoming globally pertinent.

In conclusion, Piccolo2 represents a significant step forward in text embedding technology, providing a robust, scalable solution tailored for the intricate demands of multiple NLP tasks. Its development not only enhances performance metrics but also broadens the potential for real-world applications of Chinese NLP technologies.