OPENXRD: QA Benchmark for XRD Crystallography

• OPENXRD is a benchmark-and-inference framework for crystallography QA focused on X-ray diffraction, featuring both closed-book and open-book modes.
• It employs GPT-4.5 generated support text refined by experts to provide concise, domain-specific explanations without directly revealing answers.
• Empirical results reveal that mid-sized models benefit most from targeted support, showing significant accuracy gains on concept-based crystallography questions.

Searching arXiv for OPENXRD and closely related XRD automation/QA papers to ground the article. OPENXRD is an open-book benchmark-and-inference framework for question answering in crystallography, specifically focused on X-ray diffraction. It evaluates LLMs and multimodal LLMs on expert-level XRD questions under both closed-book and open-book conditions, where the open-book setting supplies concise domain-specific support text generated by GPT-4.5 and, in a stronger variant, refined by crystallography experts. In its reported form, OPENXRD is a textual question-answering system rather than a diffraction-analysis engine, and its central claim is that carefully targeted supporting material can materially improve crystallography performance for models with incomplete domain knowledge (Vosoughi et al., 12 Jul 2025).

1. Definition and scope

OPENXRD is defined as a benchmark-plus-inference framework for crystallography question answering. Its domain is X-ray diffraction, and its task formulation is multiple-choice QA rather than peak fitting, phase identification, or structure refinement. The framework was introduced to address a specific mismatch: many LLMs and MLLMs exhibit strong general scientific language ability but remain unreliable on crystallography questions involving diffraction, symmetry, structure factors, extinction rules, unit cells, and scattering physics (Vosoughi et al., 12 Jul 2025).

The system operates in two modes. In the closed-book setting, a model receives only the question and answer options. In the open-book setting, the model also receives a short supporting passage relevant to the question. The support text is designed to summarize the governing crystallographic concept without directly leaking the correct answer. This makes OPENXRD both a benchmark for crystallographic competence and a method for augmenting weaker models at inference time.

A recurring source of confusion is the name itself. OPENXRD does not denote, in its current form, an automated powder-diffraction solver, a Rietveld platform, or a beamline data-processing stack. It is an NLP framework for XRD-centric scientific reasoning. The paper explicitly states that the current implementation is “exclusively textual,” even when evaluated on multimodal model architectures (Vosoughi et al., 12 Jul 2025).

2. Benchmark construction and knowledge support

The benchmark contains 217 domain-specific multiple-choice questions, each with three or four options and exactly one correct answer. Each question also includes a brief rationale or explanation and a subtask label. The subtasks span crystal structure fundamentals, diffraction geometry, unit-cell analysis, symmetry-related topics, structure factors, atomic spacing, coordination numbers, Laue patterns, powder diffraction, Bragg’s law, wave scattering, calculation methods, and diffraction limitations (Vosoughi et al., 12 Jul 2025).

A central design decision is the replacement of scanned textbooks with synthetic support text. The framework does not rely on textbook-image retrieval because systematically scanning textbooks is described as impractical and because scanned copyrighted books raise copyright and data-access concerns. Instead, GPT-4.5 generates compact, domain-specific passages that function as short textbook-like references. These passages are deliberately concise, relevant to the question, and intended not to reveal the answer directly.

A stronger support condition introduces expert review. Three PhD students in crystallography refine the GPT-4.5-generated passages by correcting technical inaccuracies, sharpening terminology, ensuring concept coverage, improving clarity, removing misleading material, and adding missing contextual details. The paper’s example on the atomic scattering factor shows the difference clearly: the raw AI text gives a basic definition, while the reviewed version adds the low-angle electron-count limit, high-angle damping from electron-cloud interference, the heavier-atom trend at low angles, and the connection to diffraction intensities (Vosoughi et al., 12 Jul 2025).

This design implies a specific view of domain knowledge. OPENXRD does not attempt to replace crystallographic literature with a general retrieval system. Instead, it uses short, question-specific explanatory context as an inference-time scaffold.

3. Pipeline architecture and evaluation protocol

The pipeline is structurally simple. In open-book mode, the model input is a formatted combination of support text, question, and options: $x = \mathrm{Format}(x_s, x_q, x_o),$ where $x_s$ is the supporting text, $x_q$ is the question, and $x_o$ is the answer-choice set. In closed-book mode, $x_s = \emptyset$, but the remainder of the formatting is preserved for fairness (Vosoughi et al., 12 Jul 2025).

Evaluation is zero-shot throughout. The models are not fine-tuned on the 217-question benchmark, and no in-context examples are used. Accuracy is defined as

$$\mathrm{Accuracy} = \frac{|\{\mathrm{correctly\ answered\ questions}\}|}{|\{\mathrm{all\ questions}\}|}\times 100\%.$$

For subtask $t$, the paper defines

$$\mathrm{Accuracy}_{\mathbf{M},t} = \frac{|\{q \in Q_t : \mathbf{M}(q)=y_q\}|}{|Q_t|}\times 100\%,$$

where $Q_t$ is the set of questions in that subtask, $y_q$ is the correct answer, and $x_s$0 is the model prediction. Performance gain $x_s$1 is defined as open-book accuracy minus closed-book accuracy (Vosoughi et al., 12 Jul 2025).

The evaluated model set spans both text-only frontier models and multimodal architectures used in text-only prompting. The list includes GPT-4.5-preview, O3-mini, O1, GPT-4-turbo-preview, GPT-4, and multiple LLaVA-family systems with Mistral, LLaMA, and QWEN backbones. Although some of these models are MLLMs by architecture, the reported experiments do not provide real images, diffraction patterns, or crystal diagrams as model inputs.

The architecture is therefore best understood as retrieve-then-read only in a loose sense. The “retrieval” layer is per-question support generation or prepared support text rather than corpus search over a large indexed database.

4. Empirical performance and the open-book effect

Closed-book results establish a strong ceiling for current general-purpose models. GPT-4.5-preview attains 93.09% accuracy, answering 202 of 217 questions correctly. O3-mini reaches 88.94%, O1 87.56%, GPT-4-turbo-preview 83.41%, and GPT-4 81.11%. The best-performing open-weight or smaller VLM-style systems are substantially lower, with LLaVA-v1.6-34B at 66.80% and several 7B-class or smaller models in the $x_s$2–$x_s$3 range, while LLaVA-v1.5-7B reaches only 18.00% (Vosoughi et al., 12 Jul 2025).

The raw AI-generated support condition produces mixed results. Smaller and mid-capacity models usually improve, but the strongest models improve little or can degrade. Expert-reviewed support produces a clearer pattern: large gains for mid-sized systems, modest gains for some frontier text models, and degradation for GPT-4.5-preview itself. The paper describes this as a “sweet spot” or inverted-$x_s$4 pattern, in which the largest benefits accrue to models that have enough reasoning capacity to exploit support but insufficient crystallographic knowledge to answer reliably without it (Vosoughi et al., 12 Jul 2025).

The following table summarizes representative closed-book and expert-reviewed open-book results.

Model	Closed-book	Open-book, expert-reviewed
GPT-4.5-preview	93.09%	89.40%
O3-mini	88.94%	89.90%
GPT-4	81.11%	82.50%
LLaVA-v1.6-34B	66.80%	78.30%
LLaVA-v1.6-mistral-7B	53.00%	64.10%
LLaVA-v1.5-7B	18.00%	28.60%

The largest reported improvements under expert-reviewed support include +11.50 percentage points for LLaVA-v1.6-34B, +11.10 for LLaVA-v1.6-mistral-7B, +10.10 for LLaVA-onevision-QWEN2-7B-ov-chat, and +10.60 for LLaVA-v1.5-7B. By contrast, GPT-4.5-preview declines by 3.69 points under the same condition. Aggregated by model-size category, AI-generated support yields average gains of $x_s$5 for large models, +5.71 for mid-sized models, and +3.11 for small models; expert-reviewed support changes these to +0.08, +9.64, and +6.55, respectively (Vosoughi et al., 12 Jul 2025).

A plausible implication is that OPENXRD functions less as a universal improvement mechanism than as a knowledge-transfer layer from a stronger model and domain experts to models below the frontier.

5. Failure modes, misconceptions, and limitations

OPENXRD’s strongest gains occur on conceptual clarification, not on every kind of crystallographic reasoning. The subtask analysis is explicit on this point. For LLaVA-onevision-QWEN2-7B-ov under expert-reviewed support, Atomic Spacing rises from 66.7% to 100.0%, Coordination Numbers from 20.0% to 60.0%, Crystal Structure from 0.0% to 100.0%, Laue Patterns from 0.0% to 100.0%, Metal Structures from 0.0% to 100.0%, Powder Diffraction from 0.0% to 100.0%, and Structure Factors from 60.0% to 85.0%. By contrast, Bragg’s Law remains at 0.0%, Calculation Methods remains at 0.0%, and Diffraction Limitations remains at 0.0%; Complex Mathematics worsens from 100.0% to 50.0%, and Wave Scattering from 100.0% to 0.0% (Vosoughi et al., 12 Jul 2025).

The most important failure mode is not irrelevance but partial relevance. A support passage can mention a related concept that acts as a distractor. The paper’s example asks which scattering mechanism causes damping of the atomic scattering factor at higher angles. The correct answer is electron-cloud interference. LLaVA-v1.6-34B answers correctly in closed-book mode but changes to the wrong answer under open-book support because the support text also mentions thermal vibration through the Debye–Waller factor. The model is therefore misled by concept adjacency rather than by total ignorance (Vosoughi et al., 12 Jul 2025).

Several further limitations are stated directly. The benchmark contains only 217 questions. The task is multiple-choice QA rather than open-ended scientific argument or derivation. The strongest current version is text-only; no real diffraction patterns, crystal diagrams, or figure parsing enter the reported evaluation. The framework depends on GPT-4.5 both as a strong baseline and as a support generator, and expert review improves support materially but may be difficult to scale. The largest models may not benefit from extra context at all. These limitations make OPENXRD a domain-specific augmentation framework rather than a complete crystallographic reasoning environment.

A second common misconception is that support text alone solves advanced crystallographic mathematics. The results do not support that interpretation. The paper explicitly suggests future integration with symbolic math modules or domain-specific solvers for structure-factor calculations and multistep interference derivations (Vosoughi et al., 12 Jul 2025).

6. Position within open diffraction research

OPENXRD occupies a distinct layer within the broader open XRD ecosystem. Its task is scientific language reasoning about diffraction, not direct inversion of diffractograms. A plausible implication is that it complements, rather than replaces, experimental-pattern databases and phase-analysis toolchains.

That distinction is clear when compared with experimental and analytical infrastructures. The opXRD database provides 92,552 experimental powder diffractograms, of which 2,179 are labeled and 90,373 are unlabeled, specifically to address the simulated-to-experimental gap in ML for powder diffraction (Hollarek et al., 7 Mar 2025). XERUS, by contrast, is an open-source Python framework for semi-automatic and automatic powder XRD phase identification and refinement automation that retrieves structures from open databases and couples similarity screening to GSAS-II-based refinement (Castro et al., 2021). OPENXRD does neither of these. It does not supply a large experimental diffractogram corpus, nor does it perform structure solution or phase decomposition.

The distinction is also methodological. Much of the recent open XRD literature targets pattern-centric tasks such as phase identification, space-group classification, lattice estimation, or refinement initialization. OPENXRD addresses the linguistic and conceptual interface to crystallography: answering technical questions, clarifying concepts, and measuring whether a model can reason over diffraction-related knowledge in natural language. This suggests an architectural separation inside future open systems: pattern-processing modules, databases, and refinement engines on one side; domain-specific scientific QA and decision support on the other.

The paper’s future directions are consistent with that interpretation. It proposes adding real crystal diagrams or diffraction patterns, improving multimodal grounding, and extending beyond text-only support. If realized, such extensions would move OPENXRD closer to a broader scientific-assistance layer for crystallography. In its reported form, however, it remains a specialized open-book QA framework whose main contribution is to show that concise, question-specific, expert-vetted support can substantially improve XRD question answering for mid-capacity models, while also revealing the limits of support-only approaches in mathematically intensive crystallographic reasoning (Vosoughi et al., 12 Jul 2025).