AlgoSimBench: Benchmarking Algorithmic Similarity

• AlgoSimBench is a benchmark for assessing algorithm recognition by identifying problems that share fine-grained algorithm tags despite differing textual narratives.
• It employs an adversarial MCQ setup where the correct option has low textual similarity yet shares the same algorithmic structure, while distractors mimic surface-level cues.
• Experimental findings show that model-generated solution attempts (ASM-NL/ASM-PL) improve accuracy, highlighting the benefit of leveraging solution-space representations.

AlgoSimBench is a benchmark for evaluating whether code-oriented LLMs can recognize algorithmically similar problems (ASPs) in competitive programming: problems that differ in context, surface wording, or narrative framing but are solvable by similar algorithmic strategies. Introduced as a benchmark of algorithm recognition rather than end-to-end code synthesis, it operationalizes similarity through fine-grained algorithm tags and constructs adversarial multiple-choice questions in which the correct option is algorithmically matched yet textually dissimilar, while distractors are textually similar yet algorithmically dissimilar (Li et al., 21 Jul 2025). In this formulation, AlgoSimBench isolates a capability that standard coding benchmarks often conflate with code generation, debugging, and template recall: identifying the underlying solution method from the problem statement itself.

1. Conceptual focus and definition

AlgoSimBench studies a narrow but consequential subskill in program synthesis: whether a model can detect that two competitive-programming problems share the same underlying algorithmic structure. The paper’s informal definition describes ASPs as problems that, despite differences in surface wording or story, can be solved using similar algorithmic strategies; operationally, two problems are treated as algorithmically similar when they share exactly the same set of fine-grained algorithm labels (Li et al., 21 Jul 2025).

This design responds to a limitation of existing competitive-programming benchmarks. Suites such as APPS, Codeforces-based evaluations, and USACO-based benchmarks chiefly test end-to-end performance: given a statement, can a model write code that passes tests? That setup combines several competencies—problem comprehension, algorithm selection, planning, and low-level implementation—and can obscure failures in algorithmic transfer. A system may achieve strong pass@$k$ by retrieving templates or exploiting shallow cues without robustly recognizing that a new problem is, for example, another instance of a known technique. AlgoSimBench therefore decouples algorithm recognition from code synthesis and makes it the explicit target of evaluation (Li et al., 21 Jul 2025).

The benchmark is also directly relevant to retrieval-augmented code generation. If retrieval is intended to surface useful exemplars or prior solutions, then the retrieval system must privilege solution-level similarity rather than surface textual overlap. AlgoSimBench makes that requirement measurable by constructing a setting in which textual similarity is adversarial rather than helpful (Li et al., 21 Jul 2025).

2. Corpus, taxonomy, and annotation scheme

The benchmark begins from a raw collection of 1,522 competitive-programming problems, each paired with one reference implementation, gathered from four curated topic and tagging communities: The Ultimate Topic List, CP Algorithms, ProgVar, and USACO Guide. These communities reference problems hosted on Codeforces, AtCoder, and CodeChef. After filtering broad, ambiguous, and duplicate labels, the retained core pool contains 1,317 problems (Li et al., 21 Jul 2025).

Across the raw collection, the authors manually unify tags from the source communities and obtain 319 distinct fine-grained tags, of which 278 have category/subcategory metadata. In the final MCQ subset, 231 fine-grained tags appear. The tagging scheme is intentionally finer than platform-level labels such as “dp” or “graphs”; the paper emphasizes technique-level categories such as “0/1 knapsack”, “binary lifting”, or “FFT convolution” (Li et al., 21 Jul 2025).

Component	Size	Notes
Raw tagged collection	1,522 problems	One reference implementation per problem
Core filtered pool	1,317 problems	Codeforces 1114, AtCoder 182, CodeChef 18
Final MCQ benchmark	402 MCQs	903 distinct problems, 231 fine-grained tags

The filtered pool has an average of 489.4 tokens per problem and 1.11 tags per problem. The final MCQ subset contains 402 questions built from 903 distinct problems, with an average of 2681.4 tokens per MCQ problem because the benchmark retains full statements and examples (Li et al., 21 Jul 2025).

The reliance on community-curated fine-grained tags is central to the benchmark’s semantics. No new large-scale relabeling is introduced; instead, the benchmark inherits tags from existing expert-maintained communities, then standardizes nomenclature and removes overly broad labels. Human annotators additionally inspect edge cases to ensure that distractors do not share subtle algorithmic ideas with the reference problem (Li et al., 21 Jul 2025).

3. Adversarial MCQ construction

Each multiple-choice item consists of a reference problem $p$ and four options: one correct ASP $q$ and three distractors $d_1,d_2,d_3$. If $Y_i$ denotes the set of fine-grained labels for problem $i$, then the correct option is selected to satisfy

$Y_p = Y_q.$

Distractors are required to satisfy stronger dissimilarity constraints. For each distractor $d_i$, the label sets must obey

$Y_p \cap Y_{d_i} = \emptyset,$

and the distractor’s category/subcategory metadata must also be disjoint from that of the reference problem. The benchmark further filters out lexically similar label names and applies manual inspection to avoid near-miss algorithmic overlap (Li et al., 21 Jul 2025).

The benchmark’s defining feature is the inversion of the usual relationship between textual and algorithmic similarity. Dense sentence embeddings are computed for problem statements, and the correct ASP is chosen as the problem with the lowest textual similarity among those sharing the same fine-grained tag set. By contrast, distractors are chosen from algorithmically dissimilar candidates with the highest textual similarity to the reference; the paper specifies $n=3$ top textually similar distractors under these constraints, together with similarity thresholds to ensure sufficient separation (Li et al., 21 Jul 2025).

This adversarial construction produces what the paper characterizes as a hard negative design. Classical retrieval by statement similarity is intentionally pushed toward the wrong answers, because the textually nearest neighbors are distractors rather than ASPs. If a candidate reference problem does not admit one correct ASP and three distractors meeting the tagging and similarity constraints, it is removed as a reference, though it may still appear as an option in other questions (Li et al., 21 Jul 2025).

A common misconception is that AlgoSimBench is simply a paraphrase-recognition or semantic-similarity benchmark. Its construction is designed to negate precisely that interpretation: statement-level similarity is not merely insufficient but frequently adversarial, and success requires access to the underlying algorithmic structure rather than narrative resemblance (Li et al., 21 Jul 2025).

4. Tasks, representations, and evaluation protocol

AlgoSimBench defines two principal evaluation regimes. The first is end-to-end ASP selection. The model receives the reference problem, the four options, a definition of ASPs, and an instruction to choose the most algorithmically similar option. Performance is measured by accuracy over the 402 MCQs:

$p$0

The benchmark randomizes the position of the correct answer because appendix experiments show that models tend to prefer later options such as D; without randomization, positional bias would confound results (Li et al., 21 Jul 2025).

The second regime casts ASP identification as retrieval or ranking. Here, each problem is represented independently, and similarity between the reference and the four candidates is computed via an IR or embedding model. Several problem representations are used:

• Statement: the raw problem text.
• Summary: an LLM-generated abstraction intended to remove narrative elements.
• ASM-NL: an LLM-generated natural-language attempted solution.
• ASM-PL: an LLM-generated Python solution attempt.
• Solution: the human-written oracle solution code, used as an upper-bound style representation rather than a realistic input (Li et al., 21 Jul 2025).

The retrieval methods include BM25, TF-IDF, BERT-base-uncased, BART-base, and several code encoders, including CodeBERT-base, GraphCodeBERT-base, CodeSage-v2, CodeRankEmbed, SFR-Embedding-Code-400M_R, and Jina-Embeddings-v2-base-code (Li et al., 21 Jul 2025).

For retrieval-style experiments, the benchmark reports both Accuracy@1 and Mean Reciprocal Rank (MRR),

$p$1

where $p$2 is the position of the correct ASP in the retrieved list for query $p$3. The benchmark therefore supports both closed-set four-way selection and broader representation-learning or retrieval analyses (Li et al., 21 Jul 2025).

5. Attempted Solution Matching

To improve similarity detection, the benchmark introduces attempted solution matching (ASM). The central observation is that algorithmic similarity is more visible in a model’s attempted solution than in the original statement, and that models frequently make similar mistakes on algorithmically similar problems. ASM therefore removes dependence on human reference solutions and instead compares model-generated solution attempts against one another (Li et al., 21 Jul 2025).

Formally, for a problem statement $p$4 and an LLM $p$5, ASM constructs either a natural-language attempt

$p$6

or a program attempt

$p$7

Given a reference problem $p$8 and candidate set $p$9, the system computes similarities $q$0 and predicts

$q$1

In retrieval-based selection, this becomes a two-stage pipeline: generate attempts for all problems, index them, and compare them via BM25 or embedding-based similarity. In end-to-end selection, the attempted solutions themselves replace the original statements in the multiple-choice prompt (Li et al., 21 Jul 2025).

ASM is motivated in part by the limitations of using human solution code as the retrieval substrate. Matching model-generated code to human code introduces stylistic and language mismatches, and it assumes the model’s first attempt already captures the correct method. ASM relaxes that assumption: even incorrect attempts can still encode the same mistaken or approximate algorithmic schema and therefore cluster ASPs effectively (Li et al., 21 Jul 2025).

The representation choice has a cost dimension. The paper reports that ASM-NL costs roughly 1.5–2.6× more tokens than Statements, whereas ASM-PL costs roughly 2.6–5.3× more. Because ASM-NL typically outperforms ASM-PL on 5/7 models in end-to-end selection, the paper presents natural-language solution descriptions as the more cost-effective ASM variant (Li et al., 21 Jul 2025).

6. Empirical findings

On the primary end-to-end task using raw Statements, average accuracy across the seven evaluated LLMs is 50.4%. The best-performing model, o3-mini-medium, achieves 65.9% accuracy. Other reported statement-based results include DeepSeek-R1 at 63.7%, DeepSeek-V3 at 55.2%, Gemini 2.0 Flash at 51.2%, Claude-3.5-Sonnet at 44.2%, GPT-4o at 41.5%, and GPT-4o-mini at 35.5% (Li et al., 21 Jul 2025).

Providing the oracle Solution representation raises average end-to-end accuracy to 66.5%, confirming that algorithmic similarity is more apparent in solution space than in statement space. For some models the gain is large; for GPT-4o, performance rises from 41.5% on Statements to 63.4% on Solutions (Li et al., 21 Jul 2025).

ASM yields the benchmark’s main performance improvements. In end-to-end selection, ASM-NL improves accuracy by 6.7 to 11.7 absolute points across models. Representative examples include:

• GPT-4o: 41.5% $q$2 53.2% with ASM-NL.
• o3-mini-medium: 65.9% $q$3 74.4% with ASM-NL and 75.1% with ASM-PL.
• DeepSeek-R1: 63.7% $q$4 69.2% with ASM-NL and 70.4% with ASM-PL.
• Claude-3.5-Sonnet: 44.2% $q$5 54.7% with ASM-NL.
• Gemini 2.0 Flash: 51.2% $q$6 57.9% with ASM-NL (Li et al., 21 Jul 2025).

A notable result is that for o3-mini-medium, both ASM variants outperform the oracle Solution baseline, which is 72.6%. This suggests that a model’s own internally aligned representation of solution strategies can, in some cases, be more useful to that same model than human-written code (Li et al., 21 Jul 2025).

Retrieval experiments highlight the severity of the adversarial design. Statement-based retrieval methods often perform at or below random, which is approximately 25% in the four-way setting. The paper reports that adversarial problem selection can degrade retrieval performance to less than random, but that simply summarizing problems to remove narrative elements eliminates this effect (Li et al., 21 Jul 2025). On the retrieval side, the best single configuration is DeepSeek-R1 + ASM-NL + BM25, reaching 52.2% accuracy. Across models, ASM-NL + BM25 averages 43.7%, ASM-PL + BM25 averages 40.4%, and Summary + BM25 averages 29.6% (Li et al., 21 Jul 2025).

The retrieval results also show that sparse matching can outperform specialized dense code embeddings when the representation already contains algorithmic keywords. On ASM text, BM25 consistently beats dense code encoders, a result the paper attributes to explicit lexical markers such as “Fenwick tree”, “prefix sums”, or “two pointers” in the attempted solution descriptions (Li et al., 21 Jul 2025).

Several analytic results refine the interpretation of benchmark difficulty. Problem-solving difficulty, as measured by Codeforces rating, is not significantly correlated with ASP-identification failure: the reported Pearson correlation between problem difficulty and probability of incorrect ASP identification is $q$7 with $q$8 (Li et al., 21 Jul 2025). Tag-wise analyses indicate that failures are over-represented in DP variants, FFT, and constructive algorithms, whereas highly specific techniques such as centroid decomposition or straightforward Dijkstra instances are easier to categorize (Li et al., 21 Jul 2025).

7. Benchmark significance, limitations, and relation to adjacent work

AlgoSimBench occupies a distinct position among algorithmic benchmarks. Unlike standard code-generation suites, it does not ask whether a model can produce executable code; instead, it asks whether the model can identify the latent algorithmic method from the statement. In this respect it complements benchmarks such as AlgoBench, which evaluates a different capability: whether models can adapt when a transformed problem invalidates the original algorithm and imposes new complexity constraints. AlgoBench introduces complexity-aware metrics such as OptT, OptS, TRAPRATE, GAPT, and CONSENS, whereas AlgoSimBench isolates recognition of shared algorithmic structure before synthesis begins (Song et al., 30 Jun 2026).

The benchmark also reveals a methodological point about retrieval-augmented programming systems. Retrieval over raw statements is insufficient in an adversarial setting; what matters is retrieval in a representation space closer to the solution process. The success of ASM indicates that solution-space representations can be more informative than statement-space representations even when those solutions are model-generated and imperfect (Li et al., 21 Jul 2025).

Several limitations are explicit. The domain is restricted to competitive programming from Codeforces, AtCoder, and CodeChef, so transfer to broader software-engineering tasks is not guaranteed. The benchmark size—402 MCQs and 1,317 problems—is substantial but not exhaustive. The fine-grained tags come from community sources and may contain occasional noise or borderline cases. Each problem has only one human solution, so alternative valid algorithms are not represented. The current ASM pipeline also relies on surface similarity over natural-language descriptions or code, rather than deeper structural program analysis (Li et al., 21 Jul 2025).

The released resources include the full 1,317-problem corpus with problem IDs, platforms, statements, human solution code, and fine-grained tags, together with the 402 MCQs and scripts for similarity computation, BM25 and embedding-based retrieval, and LLM prompting. The repository is available at https://github.com/lijierui/AlgoSimBench (Li et al., 21 Jul 2025).

Taken together, AlgoSimBench makes a previously implicit capability explicit and testable: whether a model can recognize that two programming problems are algorithmically the same even when their statements are not. Its principal contribution is not merely another leaderboard, but a formalization of algorithmic similarity as an adversarial identification problem, together with evidence that current LLMs remain substantially less reliable at this task than their performance on conventional coding benchmarks might suggest (Li et al., 21 Jul 2025).