Papers
Topics
Authors
Recent
Search
2000 character limit reached

The Gold Rush in AI4Math, II: What are signatures of AI-assisted Proofs?

Published 5 Nov 2026 in cs.AI and stat.AP | (2611.08412v1)

Abstract: The growing use of AI in mathematics raises a question: can we detect AI involvement in proofs, separately from AI assistance with writing? Historical arXiv papers, papers disclosing substantive AI use, and manuscripts released by OpenAI provide a rich empirical basis for studying this question. However, several challenges remain: lack of matched case-control pairs; potential incompleteness of disclosure; and lack of ready-to-use features designed to capture signals of AI involvement in proofs. To address these challenges, we construct five cohorts comprising 8,281 papers by combining existing manuscripts with those we produced using generative AI. These cohorts span different levels of AI involvement and include matched pairs of papers addressing the same research problems. We then introduce ProofTrace, a framework that represents each paper as a graph of immediate proof support (GIPS) and extracts interpretable features of mathematical organization. Our study yields three discoveries. First, principal-component projections of our features reveal a qualitative gradient of AI involvement, where AI-reworded papers remain close to human papers, the two AI-generated cohorts depart in broadly shared directions, and papers disclosing substantive AI use occupy intermediate positions which shifts toward AI-generated cohorts over time. Second, we identify recurring signatures of AI-generated proofs, such as fewer externally cited results, greater reliance on intermediate conclusions, and altered graph connectivity; many of these signatures persist under controls for research field, manuscript length, or graph size. Finally, in a separate detection experiment, ProofTrace distinguishes human-written papers from those disclosing substantive AI use. Its performance remains stable under AI rewording, whereas the performance of state-of-the-art text-based detectors declines sharply.

Summary

  • The paper introduces ProofTrace, a graph-based framework identifying 342 structural features in math papers to detect AI involvement in proofs, finding 265 significant differences between Human and AI-rebuilt papers.
  • By comparing cohorts of Human, AI-reworded, Hybrid (AI-assisted), AI-rebuilt, and AI papers, the analysis reveals that AI-assisted proofs exhibit reduced external attribution and increased dependency on intermediate conclusions, differentiating themselves from Human-written proofs.

The paper studies whether AI involvement in mathematical proof development can be detected independently of AI assistance with prose. Its central methodological claim is that proof-level detection requires representations of mathematical organization rather than conventional linguistic detectors. To support this claim, the authors construct five cohorts totaling 8,281 mathematical papers and introduce ProofTrace, a graph-based feature-extraction framework designed to characterize dependencies among stated results and intermediate conclusions. The empirical analysis combines matched Human–AI-rebuilt pairs, Human–AI-reworded controls, disclosed Hybrid papers, and a collection of fully AI-generated manuscripts.

Research design and cohort construction

The cohort design is intended to separate three confounded variables: research topic, mathematical contribution, and linguistic presentation. The Human cohort consists of mathematics papers posted to arXiv in January 2022, serving as a historical reference. The Hybrid cohort contains papers whose authors disclose substantive AI assistance with mathematical ideas, proof development, checking, or formalization. The AI cohort consists of 722 manuscripts released by OpenAI and produced with minimal human involvement. The authors explicitly caution that disclosure is incomplete and heterogeneous: absence of an AI-use statement is not evidence of purely human mathematical development, while a disclosure does not quantify the fraction of mathematical content attributable to AI.

Two experimentally constructed cohorts provide the principal controls. For 700 Human papers, the authors create research briefs containing the mathematical setting and target conclusion but omitting the original arguments, intermediate results, proof strategies, and formulas. An AI system then generates a complete manuscript from each brief, producing the AI-rebuilt cohort. These pairs align the research problem and intended conclusion while allowing the proof organization to vary. The design is therefore closer to a topic-matched counterfactual than to an ordinary comparison of unrelated human and machine papers.

For 625 separate Human papers, an LLM rewrites most of the prose while preserving equations, formal statements, citations, and reasoning steps. These outputs form the AI-reworded cohort. Because the mathematical content is intended to remain human-originated, Human and AI-reworded papers constitute controls for writing-level changes, whereas AI-rebuilt and AI papers constitute cases involving AI-generated mathematical organization.

Figure 1

Figure 1: Five cohorts and the two paired designs used to separate mathematical generation from AI-mediated rewriting.

The separation of the paired source sets is important. The Human papers used for AI-rewording are disjoint from those used to generate AI-rebuilt papers, reducing direct leakage between the two principal experiments. Nevertheless, the AI-rebuilt cohort remains a generated counterfactual rather than an observation of naturally occurring AI-assisted research. Its results therefore identify structural differences associated with this particular prompting and generation pipeline, not necessarily all possible forms of AI-assisted proof development.

ProofTrace and the GIPS representation

ProofTrace represents each paper as a directed, attributed graph of immediate proof support, or GIPS. Nodes correspond to explicitly stated results—such as theorems, lemmas, propositions, and corollaries—or to substantive intermediate conclusions asserted within proofs. A directed edge A→BA \to B records that the written argument establishing BB directly uses the conclusion of AA. The graph retains isolated nodes and disconnected components rather than imposing a single proof chain.

Each node includes its result type, statement and proof locations, external attribution, a textual summary, and supporting evidence. Each edge records the location and quotation showing where the source conclusion is used. The extraction pipeline first anchors explicitly stated results, then uses an LLM in separate node and edge stages to identify intermediate conclusions and direct-use relations. This decomposition is intended to make extraction more stable and auditable than asking a model to infer an entire dependency graph in one operation.

The decision to include intermediate conclusions is central to the paper. A graph containing only explicitly labeled theorems and lemmas would represent the paper’s formal scaffolding but omit much of the reasoning actually performed inside proofs. ProofTrace instead treats important displayed equations, asserted subclaims, and other substantive conclusions as graph nodes. The resulting representation is substantially richer than theorem-only dependency graphs, although it also introduces dependence on LLM judgments about what constitutes a substantive conclusion.

Figure 2

Figure 2: Example GIPS subgraph showing stated results, intermediate conclusions, and direct proof-support edges.

The authors construct 342 structural features in six families:

Feature family Main object of measurement Number of features
References External attribution and cited results 38
Organization Ordering and textual placement of results and dependencies 82
Node Statistics Composition of stated and intermediate conclusions 28
Edge Statistics Direct-use relationships and premise structure 86
Graph Motifs Local dependency configurations 34
Global Topology Connectivity, reachability, and graph components 74

The features are designed to exclude the wording of recorded quotations and summaries. Examples include the fraction of externally cited-result nodes, the proportion of intermediate conclusions, the number of premises per proved result, feed-forward triangle density, the number of weakly connected components per node, and incoming harmonic reachability. The authors also construct 37 linguistic features for a separate ProofTrace++ detector.

This feature construction is interpretable but not fully independent of language modeling. The graph is extracted by an LLM, and several features depend on extraction decisions concerning intermediate conclusions, proof spans, statement locations, and direct use. Thus, “structural” refers to the downstream feature definitions, not to a completely model-independent measurement process.

The structural landscape of AI involvement

The 342-dimensional feature space exhibits the qualitative ordering anticipated by the cohort design. Human and AI-reworded papers overlap broadly, Hybrid papers occupy a shifted but nearby region, and AI-rebuilt and AI papers are more distant from Human papers. The AI cohort lies farther from Human papers than the AI-rebuilt cohort along the first principal component, while the AI-rebuilt cohort separates from Human papers along additional components. The authors correctly note that this contrast cannot be attributed to model identity alone because the AI and AI-rebuilt cohorts differ in research topics, generation conditions, and data provenance.

Figure 3

Figure 3: Structural projections and feature-wise relative positions of the five cohorts.

The paper quantifies each cohort’s feature-wise position between Human and AI-rebuilt means. The median relative positions are:

Cohort Median relative position
AI-reworded −0.03-0.03
Hybrid $0.26$
AI $0.94$

The AI-reworded cohort remains close to Human papers, supporting the claim that the structural feature space is comparatively insensitive to prose rewriting. Hybrid papers lie approximately one-quarter of the way from Human to AI-rebuilt papers, whereas the AI cohort is close to the AI-rebuilt reference overall. The AI cohort moves in the AI-rebuilt direction on 243 of the selected features and extends beyond the AI-rebuilt position on 132.

This result supports a graded rather than binary interpretation of AI involvement. The disclosed Hybrid papers are not structurally equivalent to fully AI-generated manuscripts, but they are not indistinguishable from the historical Human reference either. The interpretation is necessarily cohort-level: the relative position does not estimate the proportion of mathematical work performed by AI in an individual paper.

The Hybrid cohort also changes over time. After adjustment for research field, reported AI-use group, and revised-version status, the median structural position increases from $0.01$ in September–October 2025 to $0.31$ in August 2026. The trajectory initially falls to −0.22-0.22 in November 2025–January 2026. Of the 265 structural features with significant paired AI-rebuilt–Human differences, 187 have higher adjusted positions at the end than at the beginning of the observation period. Linguistic features show no comparable aggregate movement: their adjusted median changes from $0.28$ to $0.30$.

Figure 4

Figure 4: Temporal positions of Hybrid papers relative to Human and AI-rebuilt structural and linguistic references.

The temporal pattern is compatible with increasing similarity between disclosed Hybrid papers and AI-rebuilt papers in proof organization. It does not identify the mechanism. Changes in model capability, AI-use practices, disclosure behavior, manuscript selection, or the composition of submitting researchers could all contribute. In particular, the absence of a contemporaneous Human control series prevents a clean separation between AI-related change and broader temporal change in mathematical writing.

Recurring structural signatures

The principal paired comparison contains 700 Human–AI-rebuilt pairs addressing aligned mathematical settings and target conclusions. Among 342 structural features, 265 show significant mean differences after Benjamini–Hochberg correction, with a median absolute standardized mean difference of $0.38$. An additional 35 features differ significantly in distribution without a significant mean difference. Significant differences occur in all six feature families, rather than being concentrated in a narrow subset of graph statistics.

Figure 5

Figure 5: Significance and distribution of structural differences across the six ProofTrace feature families.

Several signatures are especially strong. The externally cited-result fraction is $0.061$ in Human papers and $0.002$ in AI-rebuilt papers, with standardized mean difference d=−1.39d=-1.39 and adjusted q=4.1×10−103q=4.1\times10^{-103}. AI-rebuilt graphs contain a much larger fraction of uncited-to-uncited dependencies: $0.994$ versus $0.899$. Human papers also contain more stated results without recorded proof spans, $0.211$ versus $0.002$. These findings indicate that the AI-rebuilt manuscripts rely less on explicitly attributed external results and more heavily on internally generated or internally represented proof material.

The organization statistics show shorter recorded statements and proof spans in AI-rebuilt papers. The 25th percentile of statement length is $3.21$ in AI-rebuilt papers versus $3.99$ in Human papers, with d=−2.02d=-2.02 and q=3.7×10−181q=3.7\times10^{-181}. Multiple nodes share a statement location at rates of $0.084$ and $0.018$, respectively, suggesting that AI-rebuilt arguments more often pack several extracted conclusions into the same textual passage. These results must be interpreted with care because statement locations and lengths are partly determined by the extraction protocol, including its segmentation of asserting sentences and displayed mathematics.

AI-rebuilt papers also contain more intermediate conclusions: intermediate nodes account for $0.904$ of all nodes, compared with $0.841$ in Human papers. Intermediate conclusions serve as direct premises of stated results in $0.872$ of cases versus $0.583$ for Human papers. The AI-rebuilt graphs therefore represent a larger proportion of reasoning as within-proof intermediate structure, and those intermediate nodes are more directly integrated into the proof of named results.

The edge and motif statistics point to greater local reuse and upstream connectivity. AI-rebuilt papers have more premises per proved result, $3.57$ versus $2.58$, higher incoming harmonic reachability, $3.17$ versus $2.21$, and a higher transitive-to-cyclic triad ratio, $15.14$ versus $8.05$. Human papers, by contrast, have more terminal stated results and more weakly connected components per node: $0.038$ versus $0.016$ for terminal stated results, and $0.242$ versus $0.143$ for weak components per node. The paper interprets this as a difference between more interconnected, internally developed AI-rebuilt proofs and more branched or separately organized Human arguments.

Figure 6

Figure 6: Representative structural features with the largest positive and negative standardized differences within each feature family.

The paper does not claim a single invariant AI proof style. In comparisons with the Hybrid cohort, 146 of the 265 AI-rebuilt–Human significant features recur with the same sign, while 30 reverse sign and 89 become insignificant. In the AI–Human comparison, 216 recur with the same sign, 30 reverse, and 19 become insignificant. The recurrence supports common structural tendencies across independently generated and disclosed-AI settings, but the reversals are equally informative: they show that the AI-related profile depends on the model, task, generation process, and form of human involvement.

Figure 7

Figure 7: Recurrence and robustness of AI-rebuilt signatures in Hybrid and AI cohorts.

The findings remain broadly stable under several controls. Of 166 significant Hybrid–Human structural differences in the six-field analysis, 155 remain significant in the same direction after field adjustment, 144 after length-only matching, and 134 after matching on both field and manuscript length. No significant reversals appear in these adjusted comparisons. In the paired AI-rebuilt–Human analysis, adjustment for node and edge counts leaves 192 of the original 265 differences significant in the same direction; the median ratio of adjusted to unadjusted effect sizes is $0.71$. Thus, graph size explains a meaningful portion of the signal but not most of it.

The field-specific results qualify the aggregate conclusion. Significant structural differences appear in combinatorics, number theory, probability, and algebraic geometry, but none reaches significance in machine learning or optimization and control under the corresponding tests. The null results in those fields do not establish the absence of signatures; they may reflect sample size, field-specific proof conventions, heterogeneity in AI use, or weaker extraction reliability.

Detection of substantive AI involvement

The detection experiment evaluates whether ProofTrace features can distinguish Human papers from papers disclosing substantive AI assistance. The training set contains 2,491 Human and 2,489 Hybrid papers; the principal test set contains 625 Human and 629 Hybrid papers. ProofTrace uses all 342 structural features, while ProofTrace++ combines selected structural features with linguistic features.

The first test uses original Human and Hybrid papers. The second replaces the Human test papers with their AI-reworded versions, preserving the Human mathematical content while changing the prose. This is the critical intervention for testing whether a detector responds to proof organization rather than merely to writing style.

Method Original AUROC Original AP Reworded AUROC Reworded AP
ProofTrace 0.8306 0.8383 0.8365 0.8424
ProofTrace++ 0.8903 0.9048 0.6246 0.6488
ModernBERT-MAGE, LR 0.9554 0.9622 0.5529 0.5397
RADAR 0.9576 0.9620 0.5952 0.6916
MELD 0.7904 0.7602 0.1618 0.3369
RAIDAR 0.6477 0.6937 0.1305 0.3280

ProofTrace is nearly invariant under AI rewording: AUROC changes from $0.8306$ to $0.8365$, and average precision from $0.8383$ to $0.8424$. By contrast, the fitted text-based baselines deteriorate sharply. ModernBERT-MAGE with logistic regression falls from AUROC $0.9554$ to $0.5529$, and RADAR falls from $0.9576$ to $0.5952$. The result provides direct evidence that ProofTrace captures a signal not eliminated by substantial prose rewriting.

The comparison also exposes a trade-off between structural robustness and peak performance. On original papers, ProofTrace++ achieves AUROC $0.8903$ and AP $0.9048$, outperforming structural-only ProofTrace and approaching the fitted transformer baselines. Its reworded performance declines to AUROC $0.6246$ and AP $0.6488$, demonstrating that the linguistic component contributes materially to its original-paper accuracy. ProofTrace alone is less accurate on the original test set but substantially more stable under the controlled rewriting intervention.

These results should not be interpreted as proof that the classifier detects AI assistance in an individual proof. The Hybrid labels are disclosure-derived and potentially noisy; the training and evaluation distributions are drawn from the same broader source environment; and the test manipulation preserves mathematical content according to an LLM-mediated rewriting procedure rather than a formal equivalence guarantee. The experiment establishes robustness to this particular rewording intervention, not adversarial robustness or transportability across unseen fields, models, and author populations.

Limitations and open questions

The paper’s most consequential limitation is that ProofTrace measures organization and presentation of mathematical reasoning, not proof techniques or mathematical validity. A graph can record that a conclusion is used without assessing whether the inference is correct, whether the conclusion is necessary, or whether the proof is mathematically insightful. The observed signatures may therefore reflect exposition conventions, extraction artifacts, or model-generated verbosity rather than intrinsic properties of AI-derived mathematical reasoning.

The GIPS itself is produced by an LLM. Although the authors use anchored results, staged extraction, fixed schemas, evidence quotations, and a common protocol across cohorts, the model may process human and AI-generated manuscripts differently. Differences in formatting, explicitness, notation, and proof style could affect node and edge extraction. Features based on statement location, proof length, and intermediate-conclusion identification are especially vulnerable to this issue.

The cohort labels also impose assumptions. The Human cohort is historical, not a contemporaneous randomized control. The Hybrid cohort is identified through disclosure statements whose completeness and interpretation may vary. The AI cohort is generated under conditions unlike ordinary mathematical collaboration, while the AI-rebuilt cohort is conditioned on briefs produced from Human papers. Consequently, the estimated signatures describe contrasts among these operational cohorts rather than a universal taxonomy of AI-assisted proofs.

Finally, the study does not test adversarial adaptation. An AI system could potentially generate proofs with more explicit citations, fewer intermediate nodes, or a Human-like component structure while preserving its mathematical origin. Whether ProofTrace remains informative under such strategic manipulation is open. The paper also leaves unresolved whether the recurring graph features correlate with independently assessed proof quality, correctness, readability, or reusability.

Conclusion

The paper provides an empirical framework for studying AI involvement in mathematical proof development without reducing the problem to textual style detection. Its principal contribution is the combination of matched cohorts and GIPS-based structural features. Across 8,281 papers, the analysis identifies a graded structural separation among Human, AI-reworded, Hybrid, AI-rebuilt, and AI cohorts. The strongest recurring signatures involve reduced external attribution, increased reliance on intermediate conclusions, altered dependency connectivity, and fewer separate proof strands.

The detection experiment reinforces the methodological distinction between proof organization and prose. ProofTrace remains stable under AI rewording, whereas conventional text detectors degrade substantially; ProofTrace++ obtains higher accuracy on original papers but sacrifices much of that robustness. The main unresolved issue is whether these signatures measure general properties of AI-assisted mathematical reasoning or properties of the particular models, prompts, extraction system, and disclosed-paper population studied here.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

Explain it Like I'm 14

1. What is the paper about?

This paper asks a difficult question:

Can we tell whether artificial intelligence helped create the mathematical ideas and proofs in a paper, rather than only helping improve the writing?

For example, AI might only fix grammar and make sentences clearer. That is different from AI helping invent the steps of a proof. The researchers wanted to find clues that reveal this deeper kind of AI involvement.

To do this, they created a system called ProofTrace. It studies the structure of a mathematical proof—similar to making a map that shows which ideas depend on which other ideas.

2. What questions did the researchers investigate?

The paper mainly studies these questions:

  1. Do AI-assisted proofs have recognizable patterns?
  2. Are these patterns different from patterns caused only by AI rewriting the wording?
  3. Do papers written mostly by AI look different from human-written papers?
  4. Can ProofTrace help detect papers where AI played a major role in developing the mathematics?
  5. Do these patterns remain visible even when papers differ in topic, length, or number of mathematical steps?

The researchers were especially interested in the organization of proofs, not just the words used to explain them.

3. How did the researchers do the study?

The five groups of papers

The researchers examined 8,281 mathematical papers divided into five groups:

Group What it means
Human Papers written before widespread AI use in mathematical research
AI-reworded Human papers whose wording was rewritten by AI, while the mathematics was meant to stay the same
Hybrid Papers whose authors reported that AI helped with mathematical work
AI-rebuilt New papers generated by AI from a short description of a human paper’s topic and conclusion
AI Mathematical papers produced by an AI system with very little human help

The most useful comparison was between a human paper and an AI-rebuilt paper about the same mathematical topic. This is like comparing two students who receive the same assignment but solve it in different ways.

Turning proofs into maps

The main tool, ProofTrace, turns each paper into a directed graph called a graph of immediate proof support, or GIPS.

A graph here is not a chart of numbers. It is more like a map:

  • Each node represents a mathematical result, such as a theorem, lemma, or important step inside a proof.
  • Each arrow shows that one result was directly used to prove another result.

For example:

1

Lemma A  →  Intermediate Step B  →  Theorem C

The researchers also included important ideas that appeared inside proofs, even when those ideas were not given names such as “Lemma 1.”

From these maps, ProofTrace measured 342 structural features. These included:

  • How often the paper used results from other researchers
  • How many intermediate steps appeared inside proofs
  • How results were connected
  • Whether the proof had one main chain of ideas or several separate branches
  • How often the same ideas were reused
  • Whether the proof’s written order matched the logical order of its steps

The researchers also used statistical tests. In simple terms, these tests helped them decide whether differences between human and AI papers were probably real patterns rather than random accidents.

4. What did the researchers find?

AI rewriting alone usually did not change the proof structure

Human papers and their AI-reworded versions were usually very similar when examined by ProofTrace.

This is important because it suggests that ProofTrace is not simply noticing writing style. If AI changes the sentences but does not change the mathematics, the proof map usually stays close to the original.

Different levels of AI involvement formed a kind of scale

The papers appeared to form a rough pattern:

1
2
3
4
5
6
7

Human
  ↓
AI-reworded
  ↓
Hybrid
  ↓
AI-rebuilt and fully AI-generated

Human and AI-reworded papers were close together. Papers with reported mathematical AI assistance were somewhere in the middle. Papers created mainly by AI were farther away from human papers.

The researchers also found that hybrid papers became more similar to AI-generated papers over the period they studied. However, this does not prove that AI systems themselves became better. It could also be caused by changes in how people used AI or which researchers chose to report AI use.

AI-generated proofs used different kinds of organization

The researchers found many differences between human and AI-rebuilt papers. Some important patterns were:

1. AI papers used fewer outside mathematical results

Human papers were more likely to clearly use and credit results from earlier research.

AI-rebuilt papers relied more heavily on conclusions developed inside the paper itself. This does not necessarily mean that AI papers were wrong, but it suggests that they organized their reasoning differently.

2. AI papers contained more intermediate conclusions

AI-rebuilt papers often included many small steps inside their proofs. These steps acted like temporary building blocks.

An analogy would be solving a puzzle:

  • A human solution might reuse a few important ideas in several places.
  • An AI solution might create many smaller temporary pieces and connect them in a longer chain.

3. AI papers had different connection patterns

The proof maps showed that AI-generated papers often had more strongly connected chains of intermediate steps. Human papers more often had separate branches or multiple lines of reasoning.

This is similar to comparing:

1
2

One long connected path:
A → B → C → D → E

with:

1
2
3
4
5

Several branches:
A → B → D
A → C → D

E → F

These patterns were found in many different measurements, not just one. Many differences remained even after the researchers accounted for:

  • The subject area of the mathematics
  • The length of the paper
  • The number of nodes and arrows in the proof map

ProofTrace could detect substantial AI involvement

The researchers trained computer programs to distinguish human papers from hybrid papers.

They tested the programs in two situations:

  1. Using the original papers
  2. Comparing hybrid papers with human papers whose wording had been rewritten by AI

ProofTrace’s performance stayed fairly similar after the human papers were reworded by AI. Many ordinary AI-writing detectors performed much worse after rewording.

This suggests that ProofTrace was detecting differences in the organization of mathematical reasoning, rather than simply detecting unusual vocabulary or writing style.

A version called ProofTrace++, which combined proof structure with language features, performed better on the original papers. However, its performance decreased more when the wording was changed.

5. Why are these findings important?

The study suggests that AI involvement in mathematical research may leave clues in the way proofs are organized.

This could be useful for several reasons:

  • Reviewers and journals could have another tool for checking whether AI may have contributed to a proof.
  • Researchers could better understand how AI develops mathematical arguments.
  • AI developers could design systems that produce clearer and easier-to-check proofs.
  • Universities and publishers could create more informed rules about disclosing AI assistance.
  • The same idea might work in areas such as physics, statistics, economics, and computer science, where mathematical reasoning is important.

The method could also help improve AI systems. For example, if AI creates too many unnecessary intermediate steps, a future system might learn to combine or simplify them.

Important limitations

The researchers also point out that their system is not perfect.

First, an AI system could eventually learn to imitate the features that ProofTrace looks for. This would make detection more difficult.

Second, ProofTrace studies the organization of proofs, not necessarily whether the proofs are correct, creative, or mathematically valuable. A well-organized proof can still contain mistakes, and a messy-looking proof can still contain an important idea.

Third, the papers were collected and labeled in different ways. Some authors may not fully report their AI use, so the categories may not always be completely accurate.

Conclusion

The paper introduces a new way to study AI involvement in mathematics. Instead of focusing only on writing style, ProofTrace examines the structure of proofs by turning them into maps of connected ideas.

The main message is that papers created or strongly developed with AI often show different patterns: they may use fewer cited results, include more intermediate steps, and connect proof ideas in different ways. These patterns can remain visible even when the wording is changed.

This research could help people understand, evaluate, and improve AI-generated mathematics. However, ProofTrace should be treated as a helpful clue rather than a final judge of whether AI was used or whether a proof is correct.

View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The paper’s findings support several applications that can be implemented with existing document-processing, graph-analysis, and machine-learning infrastructure, although deployment should treat the output as a screening signal rather than proof of AI use.

  • Journal and conference screening for possible AI-assisted proofs — scholarly publishing and peer review
    • Integrate ProofTrace into manuscript-ingestion systems to convert mathematical papers into graphs of immediate proof support (GIPS) and calculate structural indicators such as:
    • the proportion of externally cited results;
    • the ratio of intermediate conclusions to stated theorems and lemmas;
    • graph connectivity and the number of separate proof strands;
    • reuse of intermediate conclusions;
    • local motifs such as feed-forward triangles and shared-premise structures.
    • Use the resulting score to flag manuscripts for additional disclosure checks or expert review, not to reject papers automatically.
    • Dependency: The system requires reliable extraction of theorem statements, intermediate conclusions, proof spans, and dependency edges. The paper’s extraction pipeline relies on LLM judgments, so false positives and false negatives remain possible.
  • AI-use disclosure auditing — publishers, universities, and research funders
    • Compare a manuscript’s ProofTrace profile with reference distributions for human-written, AI-reworded, hybrid, and AI-generated papers.
    • Use the method to identify papers whose mathematical organization is unusually close to known AI-assisted profiles even when disclosure statements are absent or vague.
    • Provide authors with an opportunity to clarify whether AI was used for idea generation, proof development, checking, or formalization.
    • Dependency: The method estimates similarity to cohort-level patterns; it cannot establish who actually wrote or developed a proof. Institutional policies should therefore prohibit treating the score as conclusive evidence of misconduct.
  • Review prioritization and reviewer assistance — mathematics and mathematically intensive disciplines
    • Attach a structural summary to a submitted paper that highlights:
    • heavily AI-like proof organization;
    • unusually dense chains of within-proof intermediate conclusions;
    • weak external attribution;
    • disconnected or unexpectedly connected argument components;
    • dependencies whose textual order is difficult to follow.
    • Reviewers can use the GIPS as a navigation map to inspect central premises, unsupported transitions, and intermediate claims requiring verification.
    • Potential tool: A reviewer dashboard displaying the paper’s proof graph, node quotations, cited sources, and high-centrality conclusions.
    • Dependency: The graph must preserve source quotations and locations accurately; otherwise, visualization may create a misleading impression of logical structure.
  • Proof-audit and explanation workflows — research groups and mathematical software
    • Use GIPS graphs to locate intermediate results that are reused across multiple later claims, have unusually high in-degree, or form critical dependency bottlenecks.
    • Researchers can manually verify these nodes first, improving the efficiency of proof checking and collaborative reading.
    • The same workflow can support literature reviews by showing which external theorems or lemmas are central to a manuscript.
    • Dependency: ProofTrace records dependencies expressed in prose; it does not formally verify validity. Human or formal-methods review is still required.
  • Robust benchmarking of AI-proof detectors — academia and AI laboratories
    • Use the paper’s paired design as a template for evaluating detection systems:
    • human paper versus AI-rebuilt paper for matched mathematical content;
    • human paper versus AI-reworded paper for writing-only changes;
    • disclosed hybrid papers for real-world evaluation.
    • Evaluate detectors using AUROC, average precision, and performance under controlled rewording, rather than relying only on accuracy on unrelated datasets.
    • Dependency: The reported cohorts and generation pipelines may not represent other models, mathematical fields, languages, or future prompting strategies. External validation is necessary.
  • Assessment of AI-generated mathematical manuscripts — AI developers
    • Apply structural diagnostics during model evaluation to determine whether a model tends to:
    • introduce too many undocumented intermediate conclusions;
    • underuse or undercite external results;
    • produce overly connected proof graphs;
    • generate fewer independent proof strands than human mathematicians.
    • These diagnostics can complement correctness benchmarks and formal proof-checking rates.
    • Dependency: Structural similarity is not equivalent to mathematical quality. A model may have an AI-like organization while producing a correct and valuable proof, or a human-like organization while producing an invalid one.
  • Teaching proof organization — education
    • Use simplified GIPS visualizations in graduate mathematics courses to teach students how lemmas, intermediate claims, and theorems support one another.
    • Compare alternative proof organizations for the same result, helping students identify:
    • missing dependencies;
    • unnecessary intermediate claims;
    • poorly integrated citations;
    • excessive fragmentation of an argument.
    • Dependency: Educational use should emphasize reasoning and revision rather than labeling student work as AI-generated.
  • Personal reading and writing assistant for mathematicians — daily academic practice
    • A local or browser-based tool could process a draft and produce:
    • a proof-dependency map;
    • a list of intermediate claims lacking explicit labels;
    • possible citation gaps;
    • repeated or unused conclusions;
    • disconnected sections that may need explanatory transitions.
    • This would help authors improve exposition without requiring AI to generate new mathematics.
    • Dependency: The tool must clearly distinguish suggestions from verified logical dependencies and should protect confidential or unpublished manuscripts.
  • Policy monitoring of AI adoption in mathematical research — universities, funders, and professional societies
    • Aggregate structural indicators across journals, repositories, or research fields to estimate trends in substantive AI assistance.
    • Such monitoring could inform disclosure standards, reviewer guidance, and responsible-use policies.
    • Dependency: Aggregate estimates are sensitive to cohort construction, changes in models, disclosure behavior, field composition, and manuscript versions. Results should be reported with uncertainty and should not be interpreted as exact measures of AI usage.

Long-Term Applications

The following applications require broader validation, improved extraction, formal verification, or development of production-scale systems.

  • Cross-disciplinary detection of AI-assisted reasoning — theoretical physics, statistics, econometrics, probability, and machine learning
    • Adapt GIPS and ProofTrace to papers in fields where arguments combine equations, simulations, empirical claims, and domain-specific notation.
    • A generalized framework could model dependencies among propositions, assumptions, model specifications, estimators, and derived results.
    • Dependency: Different disciplines use substantially different proof and argument conventions. Features that distinguish AI-generated mathematics may reflect field-specific writing norms rather than AI involvement, as suggested by the paper’s weaker within-field evidence for machine learning and optimization/control.
  • Formal proof reconstruction and verification — theorem proving and formal mathematics
    • Convert extracted GIPS nodes into candidate formal lemmas for systems such as Lean, Isabelle, or Coq.
    • A future workflow could:
    • 1. identify central intermediate conclusions;
    • 2. propose explicit lemma boundaries;
    • 3. map dependencies into a formal proof plan;
    • 4. use a proof assistant to verify each step.
    • Potential product: An “AI proof compiler” that transforms informal manuscripts into a formally checkable dependency structure.
    • Dependency: Natural-language intermediate conclusions are often underspecified. Formalization requires resolving notation, hidden assumptions, quantifier scope, and domain-specific definitions.
  • AI systems that generate more interpretable proofs — mathematical AI and theorem-proving research
    • Train models with structural objectives encouraging:
    • explicit and reusable lemmas;
    • transparent attribution to prior results;
    • modular dependency graphs;
    • clear separation between independent proof strands;
    • fewer redundant or weakly motivated intermediate claims.
    • ProofTrace features could become reward signals or evaluation criteria in reinforcement learning and model selection.
    • Dependency: Optimizing for detectable structural signatures could lead to superficial gaming. Structural objectives should be combined with formal correctness, expert quality ratings, novelty, and citation accuracy.
  • Adversarially robust AI-proof detection — cybersecurity and trustworthy AI
    • Develop detectors that remain useful when models deliberately imitate human-like proof organization, insert citations, or rewrite graphs to avoid known signatures.
    • Robust evaluation should include adversarially generated papers, paraphrased proofs, model mixtures, human-edited AI drafts, and multiple generations from the same prompt.
    • Dependency: Any fixed feature set may become obsolete once it is publicly known. Detection should therefore be probabilistic, periodically recalibrated, and supplemented by provenance and process evidence.
  • Provenance-aware research infrastructure — repositories, publishers, and laboratory platforms
    • Combine ProofTrace-style structural analysis with secure records of:
    • model versions;
    • prompts and intermediate drafts;
    • formal-checking logs;
    • citation retrieval history;
    • human edits and approval stages.
    • This could produce auditable “AI contribution statements” that distinguish writing assistance from mathematical contribution.
    • Dependency: Adoption requires privacy-preserving data governance, standardized disclosure schemas, author consent, and agreement on what constitutes substantive AI involvement.
  • Automated proof refactoring and manuscript improvement — mathematical software
    • Build systems that propose alternative graph organizations while preserving the proof’s mathematical content, such as:
    • extracting a repeated intermediate conclusion into a named lemma;
    • regrouping dependent claims into modules;
    • reducing unnecessary graph crossings;
    • separating auxiliary arguments from the main proof;
    • making external dependencies explicit.
    • Potential product: A proof-aware editor that offers “refactor proof structure” suggestions alongside conventional grammar and formatting tools.
    • Dependency: Refactoring must preserve semantics and proof validity. It will require equivalence checking, domain-aware mathematical parsing, and careful treatment of arguments whose logic is implicit.
  • Human–AI collaboration studies in mathematical discovery — academia and science-of-science research
    • Track how ProofTrace features change across successive drafts as mathematicians use AI for brainstorming, proof search, checking, exposition, or formalization.
    • This could distinguish productive collaboration from unverified delegation and identify which forms of assistance improve clarity or correctness.
    • Dependency: The current paper is largely observational and cohort-based. Longitudinal studies with process logs and independent assessments of mathematical quality are needed.
  • Quality-sensitive evaluation of proof organization — peer review and research assessment
    • Link structural signatures to independent expert judgments of correctness, readability, novelty, robustness, and pedagogical value.
    • This would determine whether features associated with AI-generated proofs are merely stylistic or correlate with substantive weaknesses.
    • Dependency: The paper explicitly does not measure proof techniques or mathematical quality directly. Without expert and formal verification, structural features should not be used as quality scores.
  • Field-level forecasting and policy design — national research agencies and academic institutions
    • Combine temporal ProofTrace profiles with disclosure data to study how AI-assisted mathematical research evolves and whether practices converge toward particular organizational patterns.
    • Findings could support adaptive policies on disclosure, attribution, reviewer training, and acceptable uses of AI in research.
    • Dependency: Temporal shifts may reflect changes in models, prompting, author selection, disclosure norms, or research topics rather than increasing AI capability alone. Policy conclusions therefore require contemporaneous human controls and repeated sampling.
  • Public-facing tools for evaluating mathematical claims — advanced education and science communication
    • A future system could show readers how a proof’s conclusions depend on one another, which claims rely on external literature, and which steps are not formally verified.
    • This may improve transparency for students, interdisciplinary researchers, and non-specialist reviewers.
    • Dependency: Automated explanations can appear authoritative even when extraction is wrong. Any public interface should expose uncertainty, source passages, and verification status rather than presenting the graph as a definitive proof certificate.
View Paper Prompt  View All Prompts 

Glossary

  • Adversarial attacks: Deliberate attempts to manipulate a system so that it produces incorrect or misleading results. “The first limitation is that we do not take adversarial attacks into account.”
  • Area under the receiver operating characteristic curve (AUROC): A metric measuring a classifier’s ability to distinguish between two classes across decision thresholds. “We use the area under the receiver operating characteristic curve (AUROC) and average precision (AP) as evaluation metrics”
  • Attribution: Explicit acknowledgment that a mathematical result originates from an external source. “Node attributes record result type, statement and proof locations, external attribution, a content summary, and a supporting quotation.”
  • Benjamini–Hochberg procedure: A multiple-testing correction method that controls the expected proportion of false discoveries. “To account for testing many features, we apply the Benjamini--Hochberg procedure jointly across the structural features”
  • Bipartite graph: A graph whose vertices are divided into two sets, with edges connecting vertices from different sets. “Human graphs have a larger local bi-fan-to-fork ratio”
  • Case-control comparison: A comparison between instances possessing a target characteristic and matched reference instances that do not. “The Human--AI-rebuilt pairs provide matched case-control comparisons.”
  • Counterfactual: A hypothetical alternative scenario used to estimate what might have happened under different conditions. “For a given human-written paper, our goal is to approximate a counterfactual”
  • Directed acyclic graph: A directed graph containing no directed cycles. “This representation captures which mathematical results appear in a paper and how they are connected through proof dependencies.”
  • Directed edge: An edge with an orientation from one node to another. “An edge A→BA\to B records that the written argument establishing BB directly uses the conclusion of AA.”
  • Feed-forward triangle: A three-node directed pattern in which one node reaches a third node both directly and through an intermediate node. “Corollary~2.2 contributes to Theorem~4.1 both directly and through I40, forming a feed-forward triangle”
  • Feature bank: A collection of computed variables available for statistical or machine-learning analysis. “providing a feature bank for downstream statistical analysis.”
  • Feature space: A multidimensional space whose coordinates represent measured properties of data instances. “We compare the five cohorts as a whole in the ProofTrace feature space.”
  • Fine-tuning: Adapting a pretrained model to a particular dataset or task through additional training. “The other five are fitted and adapted baselines, and are fine-tuned on our training set.”
  • Generative AI: Artificial-intelligence systems that produce new content such as text, code, or mathematical proofs. “we construct five cohorts by combining existing papers with manuscripts produced using generative AI tools”
  • Graph connectivity: The degree to which nodes in a graph are linked to one another. “altered graph connectivity”
  • Graph motif: A recurring local subgraph pattern used to characterize network structure. “Graph Motifs measures local dependency patterns”
  • Graph of immediate proof support (GIPS): A directed graph representing mathematical results and the direct proof dependencies among them. “its graph of immediate proof support (GIPS) is a directed graph whose nodes represent mathematical results.”
  • Harmonic reachability: A graph measure based on the reciprocal distances from a node to other reachable nodes. “higher incoming harmonic reachability (3.17 versus 2.21; panel k)”
  • Hybrid cohort: The group of papers whose authors disclosed substantive AI assistance with mathematical work. “The Hybrid cohort contains papers whose authors disclose substantive AI assistance with mathematics”
  • In-degree: The number of incoming edges incident to a node in a directed graph. “Maximum in-degree / mean in-degree”
  • Intermediate conclusion: A substantive result developed within a proof rather than stated as a separate formal lemma or theorem. “Intermediate conclusions make up 90\% of AI-rebuilt nodes”
  • Interquartile range: The range between the first and third quartiles of a distribution. “Dots and bars show medians and interquartile ranges.”
  • Kolmogorov–Smirnov test: A nonparametric test comparing an empirical distribution with another distribution or with a second empirical distribution. “A supplementary two-sample Kolmogorov--Smirnov test compares the marginal distributions”
  • LLM: A neural LLM trained on very large text datasets to generate or analyze language. “we then use a LLM, guided by carefully designed prompts”
  • Least-squares model: A statistical model whose parameters minimize the sum of squared prediction errors. “we use a least-squares model to estimate the Hybrid mean mjpm_{jp}”
  • Linguistic feature: A measurable property derived from language or writing rather than mathematical structure. “we separately constructed 37 linguistic features from the text recorded in the node and edge attributes”
  • Matched pair: Two observations deliberately selected to correspond closely on relevant characteristics. “The dataset includes two complementary paired subsets.”
  • Mean difference: The average difference between corresponding values in two groups or paired observations. “call a mean difference significant when its adjusted pp-value q<0.05q<0.05.”
  • Node attribute: Metadata associated with a graph node, such as its result type, location, or source. “Node attributes record result type, statement and proof locations, external attribution”
  • Principal-component analysis (PCA): A dimensionality-reduction method that transforms correlated variables into orthogonal components explaining decreasing amounts of variance. “First, we examine principal-component projections of all 8,281 documents”
  • Proof dependency: A relationship in which one mathematical result is used to establish another. “This representation captures which mathematical results appear in a paper and how they are connected through proof dependencies.”
  • ProofTrace: The paper’s framework for extracting structural features from graphs of proof support. “we introduce ProofTrace, a feature extraction framework”
  • Qualitative gradient: An ordered pattern indicating relative levels or degrees without necessarily assigning a precise numerical scale. “The five cohorts described in Section~\ref{sec:cohort-design} span different levels and forms of AI involvement.”
  • Reachability: The property that one graph node can be reached from another through a sequence of edges. “Global Topology describes overall connectivity, reachability, and separate argument strands.”
  • Structural feature: A feature describing mathematical organization and dependencies rather than wording. “We call these features structural because they use dependency relations, result types, external attribution, and source locations”
  • Transitive triangle: A directed three-node pattern in which edges represent a transitive relation among the nodes. “AI-rebuilt also has a larger transitive-to-cyclic triad ratio”
  • Zero-shot: The use of a model on a task without task-specific training examples. “The first five in Table~\ref{tab:detection} are pretrained and zero-shot baselines”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 7 tweets with 574 likes about this paper.

Sign Up for Free