SSEmb: Structural & Semantic Embedding

• SSEmb is a joint structural and semantic embedding framework that fuses graph-based structural embeddings via GraphCL with contextual semantic embeddings from Sentence-BERT.
• It employs a two-stage retrieval process where structural ranking filters candidates before semantic analysis refines the results.
• Ablation studies show that combining formula structure with surrounding text context significantly boosts retrieval performance compared to syntax-only methods.

SSEmb is a joint structural and semantic embedding framework for mathematical formula retrieval, introduced for contextualized mathematical formula search in the ARQMath-3 task, where a query formula from a community QA post is used to retrieve related formulas from a very large corpus (Li et al., 6 Aug 2025). The framework combines structural formula embedding from Operator Graphs through Graph Contrastive Learning with contextual semantic embedding from surrounding text through Sentence-BERT, then ranks candidates by a weighted fusion of structural and semantic similarities. Its central premise is that formula relevance cannot be judged by structure alone: formulas with nearly identical syntax may mean different things in different domains, while structurally different formulas may still be relevant because of shared surrounding concepts.

1. Retrieval objective and conceptual scope

SSEmb targets mathematical formula retrieval in ARQMath-3, where the corpus consists of formulas extracted from Math Stack Exchange posts, and the objective is to retrieve formulas that are mathematically meaningful and contextually relevant rather than merely syntactically similar (Li et al., 6 Aug 2025). The framework is therefore positioned against embedding-based systems that either focus mainly on formula structure or use limited contextual semantics.

This design choice is consequential because mathematical expressions are simultaneously symbolic objects and discourse objects. In the formulation adopted by SSEmb, the query is a formula, but the relevance signal is not exhausted by operator-level structure. The surrounding post text can reveal research domain, usage, definitions, and underlying concepts. Conversely, structural information remains indispensable because formulas often encode equivalence, transformation, or reuse patterns that may not be recoverable from text alone. SSEmb treats these two sources of evidence as complementary rather than substitutable.

2. System architecture and ranking mechanism

SSEmb is organized into three modules: StructEmb, SemEmb, and Rank and Retrieval (Li et al., 6 Aug 2025). StructEmb encodes formula structure from an Operator Graph using Graph Contrastive Learning. SemEmb encodes the formula’s surrounding text using Sentence-BERT. Rank and Retrieval computes structural and semantic similarities separately and fuses them with a weighted score.

For a query formula $q$ and candidate formula $c$, the structural and semantic scores are

$S_\text{structural}(q, c) = \cos(f_q, f_c),$

$S_\text{semantic}(q, c) = \cos(t_q, t_c),$

and the final fused score is

$$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$

with $\lambda \in [0,1]$. In the reported experiments, $\lambda = 0.5$, so structural and semantic signals are equally weighted.

The retrieval procedure is explicitly two-stage. First, all candidates are ranked by structural similarity, and the top 500,000 are retained. Second, semantic similarity is computed on those retained candidates, the two scores are fused, and the top 1,000 formulas are returned. The paper does not mention additional score normalization before fusion; cosine similarity is used directly on both sides.

3. Structural representation: Operator Graphs and Graph Contrastive Learning

On the structural side, SSEmb represents each formula as an Operator Graph (OPG), adopted from prior work by Song and Chen (Li et al., 6 Aug 2025). An OPG is described as a labeled directed acyclic graph built from the Operator Tree. Its defining property is the sharing of identical substructures. Compared with a plain operator tree, the OPG is more compact because repeated identical subtrees are merged, and more expressive and information-dense because hierarchical structure is preserved. The paper illustrates this with the formula

$\frac{1}{a-b} + (a-b)^2 = 1,$

where the repeated subexpression $a-b$ can be shared in the OPG rather than duplicated.

StructEmb follows the GraphCL paradigm. For a graph $G$, two augmented views $c$0 and $c$1 are generated as a positive pair, while other graphs in the batch act as negative samples. A multi-layer GNN encoder $c$2 maps the two views to graph embeddings,

$c$3

and a projection head $c$4 maps them into the contrastive space,

$c$5

Each GNN layer is a GIN layer, and the projection transformation is normalized and then ReLU is applied. The contrastive objective is InfoNCE.

The structural augmentation design is a distinguishing part of SSEmb. The framework uses attribute masking and substructure substitution, and explicitly argues that node dropping and edge perturbation are often unsuitable for formulas because they can break syntax or meaning. Substructure substitution operates on the OPG from leaves upward. It selects substructures at increasing depth or order, then for each selected substructure replaces the root node attribute with a randomly generated wildcard and disconnects the root node from lower-level nodes. Three levels are defined: first-order substitution on leaf nodes with probability $c$6, second-order substitution on parents of leaf nodes with probability $c$7, and third-order substitution on grandparents of leaf nodes with probability $c$8. In experiments,

$c$9

The stated complexity of the augmentation is $S_\text{structural}(q, c) = \cos(f_q, f_c),$0, where locating leaf nodes and ancestors is $S_\text{structural}(q, c) = \cos(f_q, f_c),$1 and replacing selected substructures is $S_\text{structural}(q, c) = \cos(f_q, f_c),$2. The intended effect is to preserve the overall computational skeleton for low-order substitutions while producing mathematically coherent structural variation.

4. Semantic context modeling and cross-signal fusion

SemEmb encodes the text surrounding the formula in the post where the formula occurs (Li et al., 6 Aug 2025). The paper emphasizes that this text can reveal research domain, usage, definitions, and underlying concepts. This is broader than sentence-only context: SSEmb uses long surrounding text rather than only the sentence containing the formula.

To improve efficiency and reduce redundancy, the extracted string is truncated to length 1024. The semantic encoder is explicitly Sentence-BERT all-MiniLM-L6-v2, producing a semantic vector $S_\text{structural}(q, c) = \cos(f_q, f_c),$3. Query and candidate text embeddings are then compared with cosine similarity through $S_\text{structural}(q, c) = \cos(f_q, f_c),$4.

The structural and semantic signals are not jointly encoded in a single interaction model. Instead, they are computed independently and fused by the linear rule $S_\text{structural}(q, c) = \cos(f_q, f_c),$5. This makes the architecture modular. It also means that the semantic stage operates only after structural preselection, since the first-stage ranking is entirely structural.

5. Experimental protocol and retrieval results

The evaluation is conducted on ARQMath-3 formula retrieval, with 76 queries and 28,320,920 formulas from 2,466,080 posts (Li et al., 6 Aug 2025). For StructEmb training, the reported subset contains 16,080,179 formulas, excluding formulas in comments and formulas whose OPG has fewer than 3 nodes. The hardware configuration is 2 × NVIDIA RTX 4090 GPUs (24GB each), 32 vCPUs, and 240 GB RAM.

Reported StructEmb hyperparameters include 25 epochs, Adam, embedding dimension 400, batch size 2560, learning rate $S_\text{structural}(q, c) = \cos(f_q, f_c),$6, 2 GNN layers, attribute masking rate 0.01, and $S_\text{structural}(q, c) = \cos(f_q, f_c),$7. There are 175,749 node labels in total; labels with frequency $S_\text{structural}(q, c) = \cos(f_q, f_c),$8 are discarded, 11,868 node labels are retained, and those retained node labels are initialized randomly. The paper does not provide a custom negative-mining procedure beyond the GraphCL convention that negatives are other graphs in the batch.

The main ARQMath-3 results are as follows:

System	$S_\text{structural}(q, c) = \cos(f_q, f_c),$9	$S_\text{semantic}(q, c) = \cos(t_q, t_c),$0
SSEmb	0.7343	0.6803
TangentCFT2ED	0.6868	0.6105
Approach0	0.7511	0.6882
Approach0 + SSEmb	0.7837	0.7158

SSEmb also reports $S_\text{semantic}(q, c) = \cos(t_q, t_c),$1, while TangentCFT2ED reports $S_\text{semantic}(q, c) = \cos(t_q, t_c),$2, and Approach0 reports $S_\text{semantic}(q, c) = \cos(t_q, t_c),$3 (Li et al., 6 Aug 2025). Relative to the best previous embedding-based run, TangentCFT2ED, SSEmb improves by about $S_\text{semantic}(q, c) = \cos(t_q, t_c),$4 in $S_\text{semantic}(q, c) = \cos(t_q, t_c),$5 and about $S_\text{semantic}(q, c) = \cos(t_q, t_c),$6 in $S_\text{semantic}(q, c) = \cos(t_q, t_c),$7, matching the claim of improvement by over 5 percentage points on key metrics. The paper further states that SSEmb is the best automated method reported in the table and approaches the strong manual system Approach0. When fused with other systems through Reciprocal Rank Fusion, it improves all compared methods; most notably, Approach0 + SSEmb reaches $S_\text{semantic}(q, c) = \cos(t_q, t_c),$8 and $S_\text{semantic}(q, c) = \cos(t_q, t_c),$9, exceeding both Approach0 alone and Approach0 + DPRL.

6. Ablation findings, interpretive boundaries, and terminological note

The ablation study identifies the joint use of structure and semantics as the central design choice (Li et al., 6 Aug 2025). Full SSEmb achieves $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$0, $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$1, and $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$2. Removing StructEmb, yielding a semantic-only variant, reduces performance to $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$3. Among structural variants, the full StructEmb configuration combining attribute masking and substructure substitution performs best. Substructure substitution only achieves $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$4, $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$5, and $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$6, while attribute masking + node dropping + edge perturbation yields $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$7, and attribute masking only yields $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$8. These results support the paper’s claim that formula-specific augmentation matters and that generic graph corruption is inferior in this setting.

Several interpretive boundaries follow from the reported setup. A plausible implication is that semantic quality depends on the availability and quality of surrounding text. A plausible implication is also that the fixed scalar weight $$S_\text{final}(q, c) = \lambda \cdot S_\text{structural}(q, c) + (1 - \lambda) \cdot S_\text{semantic}(q, c),$$9 does not adapt to query-dependent structure-semantic tradeoffs, since the reported fusion mechanism is a fixed linear combination. Likewise, because the first-stage ranking depends entirely on structural embeddings, a plausible implication is that some semantically relevant but structurally distant formulas may be pruned before the semantic stage. The paper also does not describe a learned semantic-structural interaction mechanism; structural and semantic similarities are computed separately and fused afterward.

The label “SSEmb” is not unique across arXiv. In an unrelated speech-processing context, “SSEmb” is used for a method for learning speech sequence embeddings through nearest-neighbors contrastive learning (Algayres et al., 2022). That usage concerns variable-length speech segments and is distinct from the mathematical formula retrieval framework described here. In current mathematical information retrieval usage, however, SSEmb denotes the joint structural-semantic framework built around Operator Graphs, graph contrastive learning, surrounding-text encoding, and weighted late fusion (Li et al., 6 Aug 2025).