Attempted Solution Matching (ASM)

• Attempted Solution Matching (ASM) is a method that uses LLM-generated first-attempt solutions to represent and compare the underlying algorithmic structure of competitive programming problems.
• ASM improves retrieval accuracy by replacing text-based comparisons with solution-based representations, yielding gains of up to 11.7% in benchmark evaluations.
• The approach is simple and modular, relying on off-the-shelf similarity measures, though it faces challenges with novel algorithms and higher inference costs.

Searching arXiv for the primary paper and closely related retrieval-for-programming work. arxiv_search(query="AlgoSimBench Attempted Solution Matching competitive programming", max_results=5) Attempted Solution Matching (ASM) is a method for identifying algorithmically similar problems by comparing LLM generated first-attempt solutions rather than the original problem statements. It was introduced in the context of competitive programming, where the target task is to select, for a query problem, the one candidate that is solvable by essentially the same fine-grained algorithmic strategy while rejecting distractors that are textually similar but algorithmically different. In this formulation, the provisional solution written by the model—either as natural-language reasoning or code—is treated as a representation of algorithmic structure, and similarity is computed over those representations instead of over surface text (Li et al., 21 Jul 2025).

1. Task definition and benchmark setting

ASM was proposed for a deliberately difficult retrieval regime. Given a reference competitive-programming problem $p$ and a candidate set $O=\{o_1,\dots,o_4\}$, the objective is to identify the single candidate $q \in O$ that is algorithmically similar to $p$. In the underlying benchmark design, algorithmically similar problems (ASPs) are defined as pairs with exactly the same set of fine-grained algorithmic tags, $Y_p = Y_q$. By construction, the three distractors satisfy $Y_p \cap Y_d = \varnothing$, share no subcategories, have low lexical tag similarity, and are manually vetted. The resulting multiple-choice setting inverts ordinary retrieval assumptions: the correct answer is intentionally textually dissimilar, whereas the distractors are textually close but algorithmically distinct (Li et al., 21 Jul 2025).

This design is motivated by a specific empirical observation. Statement-based retrieval methods such as BM25, TF-IDF, and dense embeddings on problem statements are poor proxies for algorithmic similarity under adversarially arranged narratives. The paper reports that textual retrieval can perform below random on AlgoSimBench, whereas oracle-solution retrieval performs much better, approximately $40\%$ versus approximately $12\%$ accuracy. The central premise of ASM is therefore that solution representations encode the algorithmic essence of a problem more faithfully than the statement alone, but that in realistic deployment the true solutions are unavailable. ASM substitutes those missing oracle solutions with LLM-generated first attempts (Li et al., 21 Jul 2025).

The benchmark introduced alongside ASM, AlgoSimBench, contains 1,317 competitive-programming problems and 402 four-choice MCQs. The paper further reports that the best-performing model on the MCQ task, o3-mini, reaches only $65.9\%$ accuracy, indicating that algorithmic-similarity detection remains substantially unsolved even for strong reasoning models (Li et al., 21 Jul 2025).

2. Formal representation and decision rule

The formal setup models each problem $i$ as a triple

$O=\{o_1,\dots,o_4\}$0

where $O=\{o_1,\dots,o_4\}$1 is the natural-language statement, $O=\{o_1,\dots,o_4\}$2 is a known oracle solution program, and $O=\{o_1,\dots,o_4\}$3 is the set of fine-grained algorithmic labels. For a query problem $O=\{o_1,\dots,o_4\}$4 and option set $O=\{o_1,\dots,o_4\}$5, ASM defines a representation map $O=\{o_1,\dots,o_4\}$6 whose form depends on the method under study (Li et al., 21 Jul 2025).

The paper distinguishes five representation families. In the Statement baseline, $O=\{o_1,\dots,o_4\}$7. In Summary, $O=\{o_1,\dots,o_4\}$8 is an LLM-generated abstract summary of the statement. In ASM-NL, $O=\{o_1,\dots,o_4\}$9 is an LLM-generated natural-language first-attempt solution. In ASM-PL, $q \in O$0 is an LLM-generated code solution attempt, specifically a Python program in the reported implementation. In the Solution upper bound, $q \in O$1, that is, the true oracle code. Given $q \in O$2 and a similarity function $q \in O$3, the predicted answer is

$q \in O$4

Accuracy is defined as the fraction of MCQs for which $q \in O$5 is the true ASP, and Mean Reciprocal Rank (MRR) is also reported in retrieval-oriented experiments (Li et al., 21 Jul 2025).

A notable feature of the method is its simplicity. The paper explicitly states that there are no additional weighting schemes or loss functions; ASM relies on off-the-shelf similarity measures applied to alternative representations. This makes the contribution primarily representational rather than architectural: the key change is what text is compared, not how the comparison function is learned (Li et al., 21 Jul 2025).

3. Generation and matching pipeline

In the retrieval-based formulation, ASM operates in four stages. First, the system constructs a representation for each element of $q \in O$6 according to the chosen method. For ASM-NL, the LLM is prompted to provide a first-attempt natural-language solution; for ASM-PL, it is prompted to write a first-attempt Python program. Second, if the similarity backend requires embeddings, the representations are encoded by an embedding model; otherwise they remain raw text for lexical retrieval. Third, the system computes a score for each candidate option against the query representation. Fourth, it returns the candidate with maximal similarity score (Li et al., 21 Jul 2025).

The same paper also studies an End-to-End Selection variant. In that setting, steps one and two are effectively collapsed: the representations are packed into a single prompt, and the LLM is asked directly to choose the best option. The method therefore exists in two forms. One is explicitly retrieval-based and modular with respect to representation and similarity metric; the other uses the generated attempted solutions as context for direct model selection (Li et al., 21 Jul 2025).

The implementation details are concrete. BM25 uses $q \in O$7 and $q \in O$8. Embedding models evaluated include BART-base, GraphCodeBert-base, CodeBert, CodeSage-v2, SFR-Embedding, Jina-Code, and CodeRankEmbed. The retrieval stack uses Python’s rank-bm25.BM25Okapi for BM25, HuggingFace Transformers for BERT-, BART-, and code-oriented encoders, and cosine similarity for embedding retrieval. The LLMs listed in the evaluation include GPT-4o-mini, GPT-4o, o3-mini-medium, Deepseek-R1, Deepseek-V3, Claude-3.5-sonnet, and Gemini 2.0 Flash. The prompt templates for summaries and attempted solutions instruct the model to “show your thought process, then produce the solution in the specified format” (Li et al., 21 Jul 2025).

Two conceptual distinctions are central here. First, ASM is not solution verification; the first-attempt output need not be correct. Its function is to expose latent algorithmic structure. Second, ASM is not equivalent to oracle-solution retrieval, because the compared objects are model-generated attempts for both query and candidates. The paper argues that this “model-to-model” matching can be beneficial because it reduces style mismatch between LLM-produced representations and human-written reference code (Li et al., 21 Jul 2025).

4. Empirical results

The reported experiments cover multiple evaluation regimes: four-way MCQ selection on AlgoSimBench, retrieval ranking with MRR, and in-context exemplar selection for code generation on USACO. The quantitative pattern is consistent across these settings: replacing statement representations with attempted-solution representations improves the ability to recover algorithmic similarity (Li et al., 21 Jul 2025).

Setting	Method	Score
End-to-End	Statement	50.4% accuracy
End-to-End	Summary	48.8% accuracy
End-to-End	ASM-NL	59.5% accuracy
End-to-End	ASM-PL	58.5% accuracy
End-to-End	Solution*	66.5% accuracy
Retrieval + BM25	Summary+BM25	29.6% accuracy
Retrieval + BM25	ASM-NL+BM25	43.7% accuracy
Retrieval + BM25	ASM-PL+BM25	40.4% accuracy

In the end-to-end setting, ASM-NL improves average accuracy by $q \in O$9 percentage points over the statement baseline, and ASM-PL improves it by $p$0 points. Across seven LLMs, the paper summarizes the absolute gain from ASM over statement-based selection as ranging from $p$1 to $p$2. The oracle Solution representation still performs best at $p$3, which serves as an upper-bound indication that algorithmic information is indeed more recoverable from solutions than from statements (Li et al., 21 Jul 2025).

The retrieval-based results show the same directionality. With BM25, Summary+BM25 reaches $p$4 accuracy, whereas ASM-NL+BM25 reaches $p$5 and ASM-PL+BM25 reaches $p$6. The abstract further reports that combining ASM with a keyword-prioritized method, BM25, can yield up to $p$7 accuracy, and that summarizing the problem to remove narrative elements eliminates the effect of adversarial textual selection. These findings reinforce the claim that narrative surface form is a confounder, whereas attempted solutions preserve more of the computational core (Li et al., 21 Jul 2025).

On the USACO in-context learning task, the paper measures Pass@1 improvement from exemplar retrieval. Episodic Retrieval achieves $p$8 for GPT-4o-mini and $p$9 for GPT-4o. ASM-NL retrieval yields $Y_p = Y_q$0 and $Y_p = Y_q$1, respectively, while ASM-PL retrieval yields $Y_p = Y_q$2 and $Y_p = Y_q$3. The gains are modest but consistent with the benchmark result: exemplars chosen by attempted-solution similarity are slightly more useful for downstream code generation than exemplars chosen by the compared retrieval baseline (Li et al., 21 Jul 2025).

5. Mechanisms, strengths, and limitations

The mechanism posited for ASM is that algorithmic signal often appears in an LLM’s own intermediate problem-solving behavior even when the solution attempt is incomplete or partially incorrect. Two dynamic-programming problems may induce similar recurrence-oriented prose; two graph shortest-path problems may induce similar code structure or reasoning steps. A qualitative example in the paper contrasts a textually similar distractor whose solution requires a segment tree with an algorithmically similar candidate presented through a different narrative but solved by Dijkstra’s algorithm. Statement retrieval is misled by overlapping phrasing, whereas ASM aligns problems through analogous attempted solutions (Li et al., 21 Jul 2025).

A common misconception is to treat ASM as merely a stronger lexical retriever. The paper’s results argue against that interpretation. The decisive shift is from matching descriptions to matching procedural hypotheses about how the task might be solved. In this sense ASM is closer to representation transduction than to conventional information retrieval: it asks the model to rewrite each problem into a solution-shaped latent space and only then applies similarity (Li et al., 21 Jul 2025).

The method has clear failure modes. If the LLM’s first attempt is entirely irrelevant, ASM can match problems on the basis of the same wrong paradigm and thereby pair unrelated items. The paper also notes that very novel or tricky algorithms, including FFT and centroid decomposition, may not be well captured in a single first attempt. Cost is another limitation. ASM-NL incurs an inference cost of approximately $Y_p = Y_q$4–$Y_p = Y_q$5 the statement baseline, and ASM-PL incurs approximately $Y_p = Y_q$6–$Y_p = Y_q$7. The paper therefore presents ASM as a simple, self-contained method with measurable benefit, but not as a complete solution to algorithmic similarity detection (Li et al., 21 Jul 2025).

The proposed future directions follow directly from these limitations. The paper suggests iterative ASM, in which multiple solution drafts are generated and aggregated; hybrid retrieval, in which ASM representations are combined with keyword signals such as BM25 in a learned reranker; and tag-aware prompting, which nudges generation toward specific algorithm classes. Each proposal preserves the basic premise that attempted solutions are valuable algorithmic representations, while acknowledging that a single draft is often an unstable proxy (Li et al., 21 Jul 2025).

6. Scope of the term and relation to other “ASM” usages

Within the competitive-programming literature, “ASM” refers specifically to Attempted Solution Matching as introduced in AlgoSimBench (Li et al., 21 Jul 2025). The acronym is, however, overloaded in other arXiv domains. In hardware for genome sequence analysis, “ASM” denotes Approximate String Matching, and “ASMCap” names a capacitive ML-CAM accelerator that targets edit-distance-based genome search with hardware–algorithm co-optimization (Zhong et al., 2023). In interactive theorem proving, related language appears in Matita’s “smart application” tactic, which uses a superposition-based engine for matching goals and lemmas modulo an equational background (Asperti et al., 2010).

These usages are conceptually unrelated. Attempted Solution Matching is a representation strategy for LLM-based retrieval over programming problems; approximate string matching in genomics concerns edit distance and specialized memory circuits; superposition-based smart matching in theorem proving concerns equality reasoning and proof reconstruction. The shared acronym can therefore obscure substantial differences in objective, representation, and computational substrate (Zhong et al., 2023).

Within its own domain, ASM is best understood as a method for recovering algorithmic similarity when lexical form is adversarial or misleading. Its significance lies less in a new retrieval objective than in the observation that first-attempt solutions—natural-language or code—can serve as a practically useful intermediate representation of algorithmic structure. In the reported experiments, that representational shift consistently improves multiple-choice similarity identification and modestly improves exemplar selection for code generation, while still leaving substantial headroom relative to oracle-solution access and to the broader problem of robust algorithmic abstraction (Li et al., 21 Jul 2025).