---
title: 'AlgoSimBench: Benchmarking Algorithmic Similarity'
url: https://www.emergentmind.com/topics/algosimbench
type: topic
---

# AlgoSimBench: Benchmarking Algorithmic Similarity

AlgoSimBench is a benchmark for evaluating whether code-oriented large language models can recognize **algorithmically similar problems (ASPs)** in competitive programming: problems that differ in context, surface wording, or narrative framing but are solvable by similar algorithmic strategies. Introduced as a benchmark of algorithm recognition rather than end-to-end code synthesis, it operationalizes similarity through **fine-grained algorithm tags** and constructs adversarial multiple-choice questions in which the correct option is algorithmically matched yet textually dissimilar, while distractors are textually similar yet algorithmically dissimilar [2507.15378]. In this formulation, AlgoSimBench isolates a capability that standard coding benchmarks often conflate with code generation, debugging, and template recall: identifying the underlying solution method from the problem statement itself.

## 1. Conceptual focus and definition

AlgoSimBench studies a narrow but consequential subskill in program synthesis: whether a model can detect that two competitive-programming problems share the same underlying algorithmic structure. The paper’s informal definition describes ASPs as problems that, despite differences in surface wording or story, can be solved using similar algorithmic strategies; operationally, two problems are treated as algorithmically similar when they share **exactly the same set of fine-grained algorithm labels** [2507.15378].

This design responds to a limitation of existing competitive-programming benchmarks. Suites such as APPS, Codeforces-based evaluations, and USACO-based benchmarks chiefly test end-to-end performance: given a statement, can a model write code that passes tests? That setup combines several competencies—problem comprehension, algorithm selection, planning, and low-level implementation—and can obscure failures in **algorithmic transfer**. A system may achieve strong pass@$k$ by retrieving templates or exploiting shallow cues without robustly recognizing that a new problem is, for example, another instance of a known technique. AlgoSimBench therefore decouples **algorithm recognition** from code synthesis and makes it the explicit target of evaluation [2507.15378].

The benchmark is also directly relevant to retrieval-augmented code generation. If retrieval is intended to surface useful exemplars or prior solutions, then the retrieval system must privilege **solution-level similarity** rather than surface textual overlap. AlgoSimBench makes that requirement measurable by constructing a setting in which textual similarity is adversarial rather than helpful [2507.15378].

## 2. Corpus, taxonomy, and annotation scheme

The benchmark begins from a raw collection of **1,522** competitive-programming problems, each paired with one reference implementation, gathered from four curated topic and tagging communities: **The Ultimate Topic List**, **CP Algorithms**, **ProgVar**, and **USACO Guide**. These communities reference problems hosted on **Codeforces**, **AtCoder**, and **CodeChef**. After filtering broad, ambiguous, and duplicate labels, the retained core pool contains **1,317** problems [2507.15378].

Across the raw collection, the authors manually unify tags from the source communities and obtain **319** distinct fine-grained tags, of which **278** have category/subcategory metadata. In the final MCQ subset, **231** fine-grained tags appear. The tagging scheme is intentionally finer than platform-level labels such as “dp” or “graphs”; the paper emphasizes technique-level categories such as “0/1 knapsack”, “binary lifting”, or “FFT convolution” [2507.15378].

| Component | Size | Notes |
|---|---:|---|
| Raw tagged collection | 1,522 problems | One reference implementation per problem |
| Core filtered pool | 1,317 problems | Codeforces 1114, AtCoder 182, CodeChef 18 |
| Final MCQ benchmark | 402 MCQs | 903 distinct problems, 231 fine-grained tags |

The filtered pool has an average of **489.4** tokens per problem and **1.11** tags per problem. The final MCQ subset contains **402** questions built from **903** distinct problems, with an average of **2681.4** tokens per MCQ problem because the benchmark retains full statements and examples [2507.15378].

The reliance on community-curated fine-grained tags is central to the benchmark’s semantics. No new large-scale relabeling is introduced; instead, the benchmark inherits tags from existing expert-maintained communities, then standardizes nomenclature and removes overly broad labels. Human annotators additionally inspect edge cases to ensure that distractors do not share subtle algorithmic ideas with the reference problem [2507.15378].

## 3. Adversarial MCQ construction

Each multiple-choice item consists of a **reference problem** \(p\) and four options: one correct ASP \(q\) and three distractors \(d_1,d_2,d_3\). If \(Y_i\) denotes the set of fine-grained labels for problem \(i\), then the correct option is selected to satisfy

\[
Y_p = Y_q.
\]

Distractors are required to satisfy stronger dissimilarity constraints. For each distractor \(d_i\), the label sets must obey

\[
Y_p \cap Y_{d_i} = \emptyset,
\]

and the distractor’s category/subcategory metadata must also be disjoint from that of the reference problem. The benchmark further filters out lexically similar label names and applies manual inspection to avoid near-miss algorithmic overlap [2507.15378].

The benchmark’s defining feature is the inversion of the usual relationship between textual and algorithmic similarity. Dense sentence embeddings are computed for problem statements, and the **correct ASP** is chosen as the problem with the **lowest textual similarity** among those sharing the same fine-grained tag set. By contrast, distractors are chosen from algorithmically dissimilar candidates with the **highest textual similarity** to the reference; the paper specifies \(n=3\) top textually similar distractors under these constraints, together with similarity thresholds to ensure sufficient separation [2507.15378].

This adversarial construction produces what the paper characterizes as a **hard negative design**. Classical retrieval by statement similarity is intentionally pushed toward the wrong answers, because the textually nearest neighbors are distractors rather than ASPs. If a candidate reference problem does not admit one correct ASP and three distractors meeting the tagging and similarity constraints, it is removed as a reference, though it may still appear as an option in other questions [2507.15378].

A common misconception is that AlgoSimBench is simply a paraphrase-recognition or semantic-similarity benchmark. Its construction is designed to negate precisely that interpretation: statement-level similarity is not merely insufficient but frequently adversarial, and success requires access to the underlying algorithmic structure rather than narrative resemblance [2507.15378].

## 4. Tasks, representations, and evaluation protocol

AlgoSimBench defines two principal evaluation regimes. The first is **end-to-end ASP selection**. The model receives the reference problem, the four options, a definition of ASPs, and an instruction to choose the most algorithmically similar option. Performance is measured by **accuracy** over the **402** MCQs:

\[
\text{Accuracy} = \frac{\#\{\text{correctly selected options}\}}{402}.
\]

The benchmark randomizes the position of the correct answer because appendix experiments show that models tend to prefer later options such as D; without randomization, positional bias would confound results [2507.15378].

The second regime casts ASP identification as **retrieval or ranking**. Here, each problem is represented independently, and similarity between the reference and the four candidates is computed via an IR or embedding model. Several problem representations are used:

- **Statement**: the raw problem text.
- **Summary**: an LLM-generated abstraction intended to remove narrative elements.
- **ASM-NL**: an LLM-generated natural-language attempted solution.
- **ASM-PL**: an LLM-generated Python solution attempt.
- **Solution**: the human-written oracle solution code, used as an upper-bound style representation rather than a realistic input [2507.15378].

The retrieval methods include **BM25**, **TF-IDF**, **BERT-base-uncased**, **BART-base**, and several code encoders, including **CodeBERT-base**, **GraphCodeBERT-base**, **CodeSage-v2**, **CodeRankEmbed**, **SFR-Embedding-Code-400M\_R**, and **Jina-Embeddings-v2-base-code** [2507.15378].

For retrieval-style experiments, the benchmark reports both **Accuracy@1** and **Mean Reciprocal Rank (MRR)**,

\[
\text{MRR} = \frac{1}{N}\sum_{i=1}^{N}\frac{1}{\text{rank}_i},
\]

where \(\text{rank}_i\) is the position of the correct ASP in the retrieved list for query \(i\). The benchmark therefore supports both closed-set four-way selection and broader representation-learning or retrieval analyses [2507.15378].

## 5. Attempted Solution Matching

To improve similarity detection, the benchmark introduces **attempted solution matching (ASM)**. The central observation is that algorithmic similarity is more visible in a model’s attempted solution than in the original statement, and that models frequently make **similar mistakes on algorithmically similar problems**. ASM therefore removes dependence on human reference solutions and instead compares model-generated solution attempts against one another [2507.15378].

Formally, for a problem statement \(s_i\) and an LLM \(M\), ASM constructs either a natural-language attempt

\[
a_i^{\text{NL}} = M(s_i; \text{prompt}_{\text{NL}})
\]

or a program attempt

\[
a_i^{\text{PL}} = M(s_i; \text{prompt}_{\text{PL}}).
\]

Given a reference problem \(p\) and candidate set \(C=\{q,d_1,d_2,d_3\}\), the system computes similarities \(s(p,i)=\text{Sim}(a_p,a_i)\) and predicts

\[
\hat{i} = \arg\max_{i \in C} s(p,i).
\]

In retrieval-based selection, this becomes a two-stage pipeline: generate attempts for all problems, index them, and compare them via BM25 or embedding-based similarity. In end-to-end selection, the attempted solutions themselves replace the original statements in the multiple-choice prompt [2507.15378].

ASM is motivated in part by the limitations of using human solution code as the retrieval substrate. Matching model-generated code to human code introduces stylistic and language mismatches, and it assumes the model’s first attempt already captures the correct method. ASM relaxes that assumption: even incorrect attempts can still encode the same mistaken or approximate algorithmic schema and therefore cluster ASPs effectively [2507.15378].

The representation choice has a cost dimension. The paper reports that **ASM-NL** costs roughly **1.5–2.6×** more tokens than Statements, whereas **ASM-PL** costs roughly **2.6–5.3×** more. Because ASM-NL typically outperforms ASM-PL on **5/7** models in end-to-end selection, the paper presents natural-language solution descriptions as the more cost-effective ASM variant [2507.15378].

## 6. Empirical findings

On the primary end-to-end task using raw Statements, average accuracy across the seven evaluated LLMs is **50.4%**. The best-performing model, **o3-mini-medium**, achieves **65.9%** accuracy. Other reported statement-based results include **DeepSeek-R1** at **63.7%**, **DeepSeek-V3** at **55.2%**, **Gemini 2.0 Flash** at **51.2%**, **Claude-3.5-Sonnet** at **44.2%**, **GPT-4o** at **41.5%**, and **GPT-4o-mini** at **35.5%** [2507.15378].

Providing the **oracle Solution** representation raises average end-to-end accuracy to **66.5%**, confirming that algorithmic similarity is more apparent in solution space than in statement space. For some models the gain is large; for **GPT-4o**, performance rises from **41.5%** on Statements to **63.4%** on Solutions [2507.15378].

ASM yields the benchmark’s main performance improvements. In end-to-end selection, **ASM-NL** improves accuracy by **6.7** to **11.7** absolute points across models. Representative examples include:

- **GPT-4o**: **41.5% \(\rightarrow\) 53.2%** with ASM-NL.
- **o3-mini-medium**: **65.9% \(\rightarrow\) 74.4%** with ASM-NL and **75.1%** with ASM-PL.
- **DeepSeek-R1**: **63.7% \(\rightarrow\) 69.2%** with ASM-NL and **70.4%** with ASM-PL.
- **Claude-3.5-Sonnet**: **44.2% \(\rightarrow\) 54.7%** with ASM-NL.
- **Gemini 2.0 Flash**: **51.2% \(\rightarrow\) 57.9%** with ASM-NL [2507.15378].

A notable result is that for **o3-mini-medium**, both ASM variants outperform the oracle Solution baseline, which is **72.6%**. This suggests that a model’s own internally aligned representation of solution strategies can, in some cases, be more useful to that same model than human-written code [2507.15378].

Retrieval experiments highlight the severity of the adversarial design. Statement-based retrieval methods often perform **at or below random**, which is approximately **25%** in the four-way setting. The paper reports that adversarial problem selection can degrade retrieval performance to **less than random**, but that simply summarizing problems to remove narrative elements eliminates this effect [2507.15378]. On the retrieval side, the best single configuration is **DeepSeek-R1 + ASM-NL + BM25**, reaching **52.2%** accuracy. Across models, **ASM-NL + BM25** averages **43.7%**, **ASM-PL + BM25** averages **40.4%**, and **Summary + BM25** averages **29.6%** [2507.15378].

The retrieval results also show that sparse matching can outperform specialized dense code embeddings when the representation already contains algorithmic keywords. On ASM text, BM25 consistently beats dense code encoders, a result the paper attributes to explicit lexical markers such as “Fenwick tree”, “prefix sums”, or “two pointers” in the attempted solution descriptions [2507.15378].

Several analytic results refine the interpretation of benchmark difficulty. Problem-solving difficulty, as measured by Codeforces rating, is not significantly correlated with ASP-identification failure: the reported Pearson correlation between problem difficulty and probability of incorrect ASP identification is **\(-0.15\)** with **\(p=0.45\)** [2507.15378]. Tag-wise analyses indicate that failures are over-represented in **DP variants**, **FFT**, and **constructive algorithms**, whereas highly specific techniques such as **centroid decomposition** or straightforward **Dijkstra** instances are easier to categorize [2507.15378].

## 7. Benchmark significance, limitations, and relation to adjacent work

AlgoSimBench occupies a distinct position among algorithmic benchmarks. Unlike standard code-generation suites, it does not ask whether a model can produce executable code; instead, it asks whether the model can identify the latent algorithmic method from the statement. In this respect it complements benchmarks such as **AlgoBench**, which evaluates a different capability: whether models can adapt when a transformed problem invalidates the original algorithm and imposes new complexity constraints. AlgoBench introduces complexity-aware metrics such as **OptT**, **OptS**, **TRAPRATE**, **GAPT**, and **CONSENS**, whereas AlgoSimBench isolates recognition of shared algorithmic structure before synthesis begins [2607.00062].

The benchmark also reveals a methodological point about retrieval-augmented programming systems. Retrieval over raw statements is insufficient in an adversarial setting; what matters is retrieval in a representation space closer to the solution process. The success of ASM indicates that **solution-space representations** can be more informative than statement-space representations even when those solutions are model-generated and imperfect [2507.15378].

Several limitations are explicit. The domain is restricted to **competitive programming** from Codeforces, AtCoder, and CodeChef, so transfer to broader software-engineering tasks is not guaranteed. The benchmark size—**402 MCQs** and **1,317 problems**—is substantial but not exhaustive. The fine-grained tags come from community sources and may contain occasional noise or borderline cases. Each problem has only **one human solution**, so alternative valid algorithms are not represented. The current ASM pipeline also relies on surface similarity over natural-language descriptions or code, rather than deeper structural program analysis [2507.15378].

The released resources include the full **1,317-problem** corpus with problem IDs, platforms, statements, human solution code, and fine-grained tags, together with the **402** MCQs and scripts for similarity computation, BM25 and embedding-based retrieval, and LLM prompting. The repository is available at `https://github.com/lijierui/AlgoSimBench` [2507.15378].

Taken together, AlgoSimBench makes a previously implicit capability explicit and testable: whether a model can recognize that two programming problems are algorithmically the same even when their statements are not. Its principal contribution is not merely another leaderboard, but a formalization of **algorithmic similarity as an adversarial identification problem**, together with evidence that current LLMs remain substantially less reliable at this task than their performance on conventional coding benchmarks might suggest [2507.15378].

Source: https://www.emergentmind.com/topics/algosimbench