SciEx: Dual Approaches in LLM Research

• SciEx is a term that encompasses both a composable framework for structured extraction from scientific publications and a benchmark for evaluating LLMs on exam tasks.
• The extraction framework employs a modular pipeline addressing long documents, multimodal data, and schema variability through iterative verification and aggregation.
• The exam benchmark tests LLM capabilities on freeform university questions using human expert grading and LLM-as-judge protocols to assess performance.

SciEx is an overloaded designation in contemporary large-language-model research. In one usage, it denotes a modular and composable framework for scientific information extraction from scientific publications, with explicit support for long-context documents, multi-modal evidence, rapidly changing schemas, and multi-paper aggregation (Li et al., 10 Dec 2025). In another, it denotes a benchmark of university computer-science exams for evaluating LLM performance on scientific tasks and for studying automatic grading with LLM-as-a-judge protocols (Dinh et al., 2024). The two uses share a name but address different technical problems: structured extraction from literature and freeform scientific task solving.

1. Terminological Scope and Disambiguation

The name SciEx currently refers to two distinct artifacts in the literature.

SciEx usage	Domain	Defining characteristics
SciEx framework	Scientific information extraction	Modular pipeline for PDF parsing, multi-modal retrieval, extraction, verification, and aggregation
SciEx benchmark	Scientific exam evaluation	Multilingual, multi-modal, freeform university CS exams with expert grading and LLM-as-a-judge evaluation

The first SciEx is centered on transforming raw scientific publications into structured records under user-defined schemas. Its stated purpose is to automate fine-grained information extraction from text, tables, and figures in support of meta-analysis, knowledge-base construction, and systematic reviews (Li et al., 10 Dec 2025). The second SciEx is centered on capability evaluation: it uses real university exams to assess whether LLMs can solve scientific tasks such as writing algorithms, querying databases, and giving mathematical proofs, and whether they can serve as graders (Dinh et al., 2024).

A common source of confusion is the assumption that the two projects are variants of a single platform. The published descriptions instead indicate separate research programs with different datasets, evaluation protocols, and end goals.

2. SciEx as a Framework for Scientific Information Extraction

The information-extraction SciEx is designed around three stated challenges in scientific literature: long, multi-section documents that exceed LLM context windows, multi-modality because key data reside in text, tables, and figures, and rapidly changing schemas or ontologies that make repeated retraining undesirable (Li et al., 10 Dec 2025). Its core design principles are modularity and composability, prompt-driven and retrieval-augmented operation, iterative verification, and schema-guided aggregation.

The architecture is organized into four core modules. The PDF pre-processing module converts raw PDFs using Docling, described as layout-aware PDF-to-HTML/XML/Markdown conversion. It also uses a vision-LLM for binary classification of scientific figures versus decorative images and for OCR-free parsing of axis labels, legends, and data points into structured JSON. The outputs include text chunks, table images with extracted table structure, figure images with structured figure JSON, and full-page images for joint visual and textual cues. These artifacts are embedded and ingested into a contextualized database consisting of a vector database plus a metadata index (Li et al., 10 Dec 2025).

The schema processing module supports two modes of schema ingestion: an explicit schema specified by the user, including field names and data types, or a natural-language description that is converted by an LLM into a schema with fields and types. The stated canonical representation is exemplified by JSON Schema. This design makes schema definition prompt-driven rather than tied to a fixed ontology.

The retrieval-extraction-verification module implements an iterative closed loop. Retrieval uses vector-based semantic search over text-chunk embeddings and figure or table metadata, with top-$k$ set to $5$ passages or figures per field query. Extraction uses GPT-4o or Gemini-2.5-Flash with prompts that condition on a schema field, its type, and retrieved evidence, and the decoding is schema-constrained so that outputs conform exactly to prescribed JSON keys and types. Verification is performed by a self-verifier prompt that checks whether schema fields are missing or uncertain; missing fields trigger targeted retrieval and extraction rounds until fields are filled or a confidence threshold or maximum number of rounds is reached (Li et al., 10 Dec 2025).

The final aggregation and resolution module groups per-paper records by a shared key, merges field values into composite records, performs automatic unit normalization using schema rules, maps lexical variants to schema synonyms, and applies a hierarchical conflict-resolution strategy based on numeric range checks, consistency voting across prompts or model variants, and null assignment when evidence is absent. The output is a unified JSON file conforming to the user schema.

3. Extraction Workflow, Formalization, and Empirical Performance

SciEx formalizes extraction around schema-constrained records. The provided schema example includes fields such as virus_strain, temperature_C, relative_humidity_%, and survival_time_h, with required-field declarations and explicit units (Li et al., 10 Dec 2025). This framing makes extraction a structured information-extraction problem rather than an open-ended generation task.

For row-level evaluation, SciEx aligns extracted rows with ground-truth rows using bipartite matching based on field-level similarity scores. It then reports classical information-extraction metrics. The definitions given for precision and recall are

$$\text{Precision} = \frac{|\text{Correct Extractions}|}{|\text{Total Extractions}|}, \qquad \text{Recall} = \frac{|\text{Correct Extractions}|}{|\text{Total Ground-Truth}|}.$$

The reported metrics also include $F_1$ and field-level accuracy (Li et al., 10 Dec 2025).

The experimental evaluation spans three datasets: Virus Decay with 143 papers, Ultraviolet with 105 papers, and Coagulation–Flocculation–Sedimentation with 43 papers and 98 variables on water quality and virus reduction. Across all datasets, GPT-4o outperforms Gemini-2.5-Flash. The reported $F_1$ values are 0.33 versus 0.24 on Ultraviolet, 0.38 versus 0.30 on Virus Decay, and 0.25 versus 0.18 on Coagulation–Flocculation–Sedimentation, the most complex setting (Li et al., 10 Dec 2025).

The qualitative error analysis identifies several recurring failure modes: low-quality or old PDFs leading to illegible axes and numeric errors, table variability such as nested headers and merged cells causing schema mismatches, figure complexity such as overlapping curves and shared legends causing missed datapoints, and cross-reference errors when text and figures refer to each other indirectly. The paper also reports that recall exceeds precision in all cases, indicating that extractions are often relevant but include spurious points.

The framework’s stated strengths are end-to-end multi-modal operation without in-house model fine-tuning, high modularity for rapid experimentation with prompts, models, and schemas, and improved completeness from iterative retrieval-extraction-verification relative to one-shot extraction. Its stated limitations are equally explicit: precision remains modest at 0.24–0.33 on complex tasks, the system is sensitive to PDF quality and table-layout heterogeneity, LLM hallucinations during canonicalization can introduce subtle semantic shifts, and current aggregation assumes relatively simple conflict patterns rather than full statistical synthesis or meta-analysis (Li et al., 10 Dec 2025).

4. SciEx as a Benchmark for Scientific Exams

The benchmark SciEx is constructed from ten distinct exams drawn from Karlsruhe Institute of Technology courses. Five exams are in English, seven are in German, and two are bilingual. The covered areas include NLP, AI, deep learning for vision and neural networks, HCI, databases, graphics, theory, and algorithms (Dinh et al., 2024).

Its defining characteristic is the use of freeform university exam questions rather than multiple-choice items. The benchmark contains 154 questions, each annotated with a maximum point value and a difficulty level. The reported distribution is Easy 51, Medium 71, and Hard 32. Most questions also have gold reference answers and average student scores. The benchmark is also multi-modal: images appear in 33 of 154 questions, including UML diagrams in HCI, ray-tracing figures in computer graphics, and automaton drawings in theory (Dinh et al., 2024).

The treatment of images depends on model class. For text-only LLMs, images are omitted. Llava receives a single concatenated image with padding. Claude-3 and GPT-4V are given both the image and an explicit “Figure 1: …” caption in the prompt. When questions instruct students to draw on an image, the LLMs instead provide verbal descriptions of the intended drawing. These implementation details are important because the benchmark evaluates genuinely freeform multimodal reasoning rather than a simplified text-only proxy.

The benchmark is motivated by the observation that scientific tasks in practice include writing algorithms, querying databases, and giving mathematical proofs. Its design therefore measures performance on open-ended problem solving under authentic university assessment conditions, rather than on curated short-form question answering.

5. Grading Protocols and Measured Outcomes

SciEx uses two grading methodologies. First, human expert grading is performed by the original exam lecturers who authored the questions. Answers are anonymized so graders do not know which model produced them. A dedicated web interface records a numeric score between 0 and the question’s maximum points, and graders may also leave qualitative comments (Dinh et al., 2024).

Second, SciEx studies LLM-as-a-judge grading. The judge prompt takes a tuple of Question, Answer, Max Points, and Reference Answer, and the judge model outputs a single numeric score in $[0,\text{Max Points}]$, optionally with chain-of-thought reasoning. The evaluated judge models are Mixtral-8×7B, Llama 3 70B, and GPT-4V. The prompt can include 0, 1, or 2 shots drawn from the same question, the same exam, or a different exam, and reference answers can be included or omitted (Dinh et al., 2024).

Agreement between manual and automatic grades is quantified by the Pearson correlation coefficient,

$$r = \frac{\sum_k (x_k-\bar{x})(y_k-\bar{y})}{\sqrt{\sum_k (x_k-\bar{x})^2 \cdot \sum_k (y_k-\bar{y})^2}}.$$

At the exam level, GPT-4V as judge achieves the best reported correlation, $r = 0.948$, followed by Llama 3 70B at $r = 0.883$, while Mixtral-8×7B reaches approximately $0.56$–$5$0 depending on shot setting. At the question level, under the best prompt setting of 2-shot with reference, GPT-4V reaches approximately $5$1 on text questions and approximately $5$2 on image questions, Llama 3 reaches approximately $5$3, and Mixtral reaches approximately $5$4 (Dinh et al., 2024).

The benchmark also reports direct task performance by answer-generating LLMs. The average exam-level scores are 59.4% for Claude-3, 58.2% for GPT-4V, 41.1% for Mixtral, 35.4% for Qwen, 32.8% for GPT-3.5, 25.9% for Mistral, and 21.5% for Llava, with the student average at 45.3%. Only Claude-3 and GPT-4V exceed the student average, and even the best model remains under 60% (Dinh et al., 2024).

The results vary systematically by difficulty, modality, and language. Weaker LLMs follow the human pattern of performing best on Easy questions and worst on Hard questions, but Claude and GPT-4V perform relatively better on Hard questions, suggesting that LLM difficulty does not align cleanly with human difficulty labels. Text-only LLMs perform poorly on image-related questions. Among multimodal models, Claude and GPT-4V outperform students on both text and image questions, whereas Llava remains well below students on images. On English questions, all LLMs except GPT-3.5 outperform students; on German questions, the gains shrink or reverse, indicating a persistent English bias (Dinh et al., 2024).

6. Limitations, Failure Modes, and Interpretive Context

The two SciEx projects converge on a similar empirical conclusion: current LLM pipelines can be productive, but they remain unreliable in ways that matter operationally. In the information-extraction framework, precision remains too low for fully automated deployment on complex tasks, and the paper explicitly states that human oversight is still required for high-stakes scientific extraction (Li et al., 10 Dec 2025). In the exam benchmark, LLMs are described as decent graders rather than perfect ones, and strong answer-generating models still leave substantial headroom relative to complete mastery of the underlying tasks (Dinh et al., 2024).

The failure modes are also complementary. For literature extraction, the dominant issues are document-conditioned: PDF quality, heterogeneous table layouts, complex figures, and indirect cross-references. For exam solving and grading, the dominant issues are reasoning- and rubric-conditioned: mathematical and proof-heavy items, strict brevity constraints, image manipulation prompts, German-language performance, and judge-model pathologies such as Mixtral’s tendency to over-assign full points or GPT-4V’s tendency to copy example grades when examples are too similar (Li et al., 10 Dec 2025, Dinh et al., 2024).

Several common misconceptions are therefore not supported by the published evidence. One is that multimodality by itself resolves scientific extraction or scientific problem solving. The extraction framework still reports missed datapoints in overlapping or multi-subplot figures, and the benchmark reports that Llava remains well below students on image questions. Another is that prompt-driven flexibility eliminates the need for careful validation. The framework’s modular schema handling reduces retraining costs, but its aggregation module can still produce semantic shifts through hallucinated canonicalization; similarly, automatic grading can scale evaluation, but robustness depends on prompt design, shot selection, and reference-answer conditioning.

Taken together, the two SciEx lines of work delineate adjacent parts of an emerging methodology for LLMs in scientific practice. One line addresses the transformation of heterogeneous scientific artifacts into structured knowledge under user-defined schemas; the other addresses how to measure LLM competence and grading fidelity on authentic scientific tasks. A plausible implication is that the shared name unintentionally highlights a broader research agenda: scientific extraction and scientific evaluation are increasingly being treated as coupled problems in the design, deployment, and assessment of LLM systems.