Gemini Embedding: Graph & LLM Techniques

• Gemini Embedding is a neural embedding class that uses graph structures for binary code analysis and transformer-based models for natural language and code tasks.
• It employs contrastive learning with specialized pipelines, achieving state-of-the-art performance in binary similarity, multilingual retrieval, and code understanding.
• The approach offers rapid inference, robust benchmark results, and scalable future extensions into multimodal domains.

Gemini Embedding denotes a class of embedding models and methodologies rooted in the Gemini neural architecture, with two distinct and influential implementations underpinning its evolution: (1) the original Gemini graph embedding for cross-platform binary code similarity analysis (Xu et al., 2017), and (2) Gemini Embedding as a LLM–based, general-purpose embedding system with preeminent multilingual and code understanding capabilities (Lee et al., 10 Mar 2025). Both approaches exploit advanced neural architectures and contrastive learning, but target fundamentally different representational domains—program control-flow graphs and natural language/code at scale, respectively.

1. Model Architectures and Embedding Construction

(A) Code Similarity Graph Embedding

Gemini’s graph embedding for binaries (Xu et al., 2017) constructs numeric vector representations of binary functions via a Graph Neural Network (GNN) pipeline operating on disassembly-derived control-flow graphs (CFGs). Each function is disassembled (e.g., with IDA Pro or objdump) and represented as a directed graph $G = (V, E)$, where nodes represent basic blocks and edges encode control flow. Each node is annotated with a feature vector $x_v \in \mathbb{R}^F$ capturing instruction counts, opcode histograms, degree statistics, call sites, and other lightweight block attributes.

A two-layer GraphSAGE-style GNN propagates local block features as follows. For each layer $k=1,2$:

• Aggregate neighbor messages: $m_v^{(k)} = \sum_{u \in N(v)} h_u^{(k-1)}$
• Update hidden state with learnable weights:

$$h_v^{(k)} = \mathrm{ReLU}\left(W_\text{self}^{(k)} h_v^{(k-1)} + W_\text{nb}^{(k)} m_v^{(k)} + b^{(k)}\right)$$

• Final function embedding: mean-pool node states after $K=2$ layers,

$E(f) = \frac{1}{|V|} \sum_{v\in V} h_v^{(2)}$

(B) LLM-Based Gemini Embedding

Gemini Embedding (Lee et al., 10 Mar 2025) instantiates a bidirectional attention transformer initialized from Google Gemini’s LLM backbone (“1.5-series”), incorporating all transformer encoder layers, LayerNorm, and a shared subword tokenizer—applied identically to natural language and code. Input $x\in\mathcal{X}$ is tokenized, embedded, summed with positional encodings, and processed through $N$ transformer layers:

$$\mathbf{T}_{\text{embed}} = \mathcal{M}(\{\mathrm{Emb}(t_i) + E_{\text{pos}(i)}\}_{i=1}^L) \in \mathbb{R}^{L \times d_\mathcal{M}}$$

Global mean pooling yields a fixed-size vector:

$x_v \in \mathbb{R}^F$0

A trainable linear projection, $x_v \in \mathbb{R}^F$1, produces the final $x_v \in \mathbb{R}^F$2-dimensional embedding; $x_v \in \mathbb{R}^F$3 (with exposed sub-dimensions $x_v \in \mathbb{R}^F$4 via Matryoshka Representation Learning).

2. Training Objectives and Optimization

(A) Graph-Based Gemini (Code Similarity)

A siamese contrastive loss is used to ensure that embeddings of similar functions (e.g., differing by compiler optimizations or platform) are close in Euclidean space, while those of unrelated functions are separated by at least a margin $x_v \in \mathbb{R}^F$5:

$x_v \in \mathbb{R}^F$6

Positive pairs share source or semantics; negatives are sampled from unrelated code.

(B) Gemini Embedding (LLM-Based)

Contrastive Noise-Contrastive Estimation (NCE) loss, operationalized with cosine similarity, is used over query-positive pairs and hard negatives:

$x_v \in \mathbb{R}^F$7

Model training comprises large-scale pre-finetuning on (title, passage) pairs, with final fine-tuning over blended retrieval, classification, and code datasets, including synthetic positives/hard negatives generated by Gemini LLMs and filtered by auto-rating tools. Multiple checkpoints are “souped” (averaged) to improve generalization.

3. Benchmarking, Evaluation, and Performance

(A) Code Similarity Benchmarking

Gemini achieves state-of-the-art cross-platform and cross-optimization binary similarity on test sets of ~50,000 functions, with AUC ≈ 0.94–0.96 and Precision@10 ≈ 0.92. Embedding computation is highly efficient (~2 ms per function on GPU), representing 40–60× speedup over prior graph-matching approaches (e.g., quadratic-time graph edit distance). End-to-end retrieval query time, including nearest-neighbor search in 100-D, is under 5 ms (Xu et al., 2017).

(B) MMTEB and Multilingual Benchmarks

Gemini Embedding sets new state-of-the-art results on the Massive Multilingual Text Embedding Benchmark (MMTEB) (Lee et al., 10 Mar 2025), encompassing 164 tasks in 250+ languages and diverse task types (retrieval, classification, clustering, reranking, STS). Key aggregate scores compared to prior SOTA Gecko Embedding:

Benchmark	Gemini Embedding	Gecko Embedding	Δ
MTEB(Multilingual)	68.32	62.13	+6.19
MTEB(English v2)	73.30	69.53	+3.77
MTEB(Code)	74.66	65.40	+9.26
XOR-Retrieve	90.42	65.67	+24.75
XTREME-UP	64.33	34.97	+29.36

Substantial improvements are observed in code retrieval (+8–10 MRR on CodeSearchNet, AppsRetrieval) and low-resource language tasks, attributed to Gemini pretraining and synthetic data.

4. Specialized Capabilities and Ablation Analysis

Gemini Embedding processes code and natural language with identical subword tokenization and neural pipeline, obviating the need for code-specific adapters or vocabularies. Its multilingual strength is derived from cross-lingual pretraining on 100+ languages and robust inclusion of synthetic and hard-negative instances in the training mixture.

Ablations analyzing data composition, training recipe, synthetic data, filtering, and hard negative mining highlight:

• Training on Gemini-generated synthetic classification data improves average accuracy by +17.6 points.
• Hard negative mining increases recall@k for retrieval, with optimal performance at 1–3 negatives per query.
• Model Soup (checkpoint averaging) further boosts downstream robustness across diversified tasks.

Code retrieval datasets benefit from Gemini Embedding’s cross-modal pretraining: specialist code models are consistently outperformed (+8–10 MRR).

5. Implementation, Footprint, and Application Considerations

Gemini Embedding, in its LLM-based instantiation, deploys a ~70B parameter encoder (dependent on variant), with projection head parameters negligible by comparison. TPU v4 pods are used for training; inference is supported on GPU/TPU/CPU, with ONNX Runtime enabling broader deployment. Batch inference (batch size 64, 3072-dim output) achieves 8 ms latency per text on A100 GPUs, halved by float16 or int8 quantization without accuracy degeneration.

Supported applications include real-time semantic search (web/code/cross-lingual), multilingual clustering, nearest-neighbor and logistic regression classification, and emerging cross-modal indexing.

6. Strengths, Limitations, and Future Directions

Gemini-based graph embedding (Xu et al., 2017) demonstrates extremely rapid, structurally robust, and accurate function-level embeddings for binary similarity; however, it remains confined to structural/local block features, may under-represent long-range dependencies in complex functions, and relies on margin-based loss, which may exhibit slow convergence absent hard negative mining.

LLM-based Gemini Embedding (Lee et al., 10 Mar 2025) achieves high generalization across language, code, and classification/retrieval tasks. Its empirical strengths include improved performance in extremely low-resource and code-heavy settings, facilitated by synthetic and blended training data and robust checkpoint averaging (Model Soup). Limitations center on resource consumption (large memory and parameter footprint), inference cost at high query-per-second, and the current absence of multimodal (vision/audio) extension, which is noted as planned future work.

Potential research extensions for both paradigms, as highlighted in their respective sources, include incorporation of richer block/node features, attention-based neighbor aggregation, deeper or residual GNN architectures, fusing data-flow with control-flow representations, and expanding embedding modalities beyond language and code.

7. Historical Context and Research Trajectory

Gemini Embedding has evolved from a domain-specific solution in binary analysis (Xu et al., 2017)—pioneering efficient GNN-based representations for program semantics—to a general-purpose, high-dimensional embedding system stemming from LLM advances (Lee et al., 10 Mar 2025) that unifies multilingual, code, and retrieval tasks within a single neural architecture. Its modern LLM-based variant embodies trends in open-domain pretraining, synthetic data utilization, and contrastive embedding optimization—providing a foundation for further expansion into cross-modal and more resource-efficient embedding research.