MyScholarQA: Personalized Scholarly QA System

• MyScholarQA is a modular, open‐source personalized deep-research question answering system that integrates hybrid retrieval, LLM generation, and user profiling for scholarly literature analysis.
• It employs a cascaded pipeline combining SPARQL queries, divide-and-conquer decomposition, and extractive BERT-based QA to ensure accurate, multi-section synthesis with citation attribution.
• The system achieves high answer coverage and superior citation recall while exposing limitations in domain adaptation and classifier robustness, prompting further enhancements.

MyScholarQA (MySQA) is a modular, open-source personalized deep-research question answering system designed for scholarly domains. It integrates user profiling, personalized action recommendation, and multi-section synthesis of scientific literature, combining retrieval and LLM generation with fine-grained user control and attribution. MySQA draws on hybrid retrieval architectures, divide-and-conquer algorithmic decomposition, and multi-stage evaluation protocols. Its development is situated at the intersection of semantic QA over knowledge graphs, retrieval-augmented generation for literature analysis, and personalized research tools (Balepur et al., 17 Mar 2026, Fondi et al., 2024, Hicke et al., 2023, Singh et al., 15 Apr 2025).

1. System Overview and Problem Definition

MySQA addresses the problem of scholarly question answering (QA) in the space of natural-language queries $Q$ over multiple heterogeneous sources $S = \{S_1, S_2, S_3\}$, where $S_1$ is a knowledge graph (KG) derived from DBLP, $S_2$ is SemOpenAlex KG, and $S_3$ comprises Wikipedia-based scholarly texts. Each query $q \in Q$ targets specific factual items (e.g., author h-index, affiliations) and demands extraction or synthesis of answer sets $A(q) \subseteq \Sigma^*$. The system is formalized by a mapping $f: Q \times S \to A$ to produce correct answers $a \in A$ for each $(q, S_q)$ tuple (Fondi et al., 2024).

Distinct from generic QA, MySQA implements explicit user profiling, infers research intent, proposes query-specific personalized actions, and generates multi-section literature reports with attribution. Personalization is achieved by extracting a structured profile $S = \{S_1, S_2, S_3\}$0—covering knowledge, style, audience, positions—from a user's paper corpus $S = \{S_1, S_2, S_3\}$1, and by letting users refine and control persistent profiles and action lists for subsequent queries (Balepur et al., 17 Mar 2026).

2. Architecture and Algorithmic Workflow

MySQA is architected as a cascaded pipeline, typically comprising:

• SPARQL Query Module: Instantiates pre-written query templates conditioned on classified question types, operating against public KGs (DBLP, SemOpenAlex). Outputs variable bindings including URIs, property values, and entity attributes.
• Divide-and-Conquer Module: Partitions questions based on entity type and attribute via keyword matching and splits each into atomic subquestions $S = \{S_1, S_2, S_3\}$2, routing them to the appropriate engine (SPARQL or extractive QA).
• Extractive QA Module: Employs a pre-trained BERT-base-cased-SQuAD2 model to extract answer spans from context paragraphs assembled from triple data.
• Answer Merger & Formatter: Aggregates, deduplicates, and formats final answers $S = \{S_1, S_2, S_3\}$3 using rule-based merging (Fondi et al., 2024).

In personalized deep-research mode, MySQA adds:

• User Profiling Engine: Extracts a profile $S = \{S_1, S_2, S_3\}$4 from user-uploaded or selected papers via LLM prompting (e.g., Claude-4), citing supporting document spans. Profiles are editable by the user to obtain the final $S = \{S_1, S_2, S_3\}$5.
• Personalized Action Proposal: For a new query $S = \{S_1, S_2, S_3\}$6, generates candidate action sets $S = \{S_1, S_2, S_3\}$7 (generic) and $S = \{S_1, S_2, S_3\}$8 (personalized, conditioned on $S = \{S_1, S_2, S_3\}$9), spanning content, style, specificity, and research ideas.
• Documentation Synthesis: Modifies classical retrieval-cluster-generation pipelines (e.g., those of ScholarQA) by conditioning each generation phase (retrieval, section planning, and generation) on user-approved actions $S_1$0 (Balepur et al., 17 Mar 2026).

The offline inference and action selection loop can be rendered as:

$S_1$9 (Hicke et al., 2023)

3. Retrieval, Synthesis, and Attribution

MySQA extends standard retrieval-augmented generation (RAG) pipelines with both hybrid (dense + sparse) retrieval and structured prompt engineering:

• Retrieval: Combines sparse BM25 keyword search with dense embedding search (e.g., FAISS using all-MiniLM-L6-v2 or mxbai-embed-large-v1), with ranking by hybrid scores $S_1$1, $S_1$2. Retrieval is pre-filtered on user metadata (e.g., year, venue, field) and further re-ranked using cross-encoder models (e.g., mxbai-rerank-large-v1), improving nDCG@10 and mean reciprocal rank (Singh et al., 15 Apr 2025).
• Generation and Sectioning: Utilizes multi-step LLM prompting:
  1. Quote Extraction: “Return verbatim quotes from these passages relevant to the question.”
  2. Outline & Clustering: “Generate an ordered list of themes/sections (intro + N topics).”
  3. Section Synthesis: “Write a {paragraph|bulleted list}, include TLDR, cite each claim as [PaperID] or [LLM Memory].”
  4. Table Generation: For section title + abstracts, output comparison table (rows: papers, columns: shared aspects).

• Attribution: All factual claims are linked to source papers or LLM memory, with popups showing precise supporting excerpts. Citation scoring uses entailment-based precision and recall metrics evaluated via GPT-4o (Singh et al., 15 Apr 2025).

4. Personalization: Profiling, Actions, and User Control

Personalization in MySQA is achieved by:

• User Profiling ($S_1$3): Extracted from user-supplied paper collections; each inference covers one of five aspects and is citation-grounded; profilings are editable (allowing removal, refinement, or addition of entries).
• Personalized Action Proposal ($S_1$4): Conditioned on both the input query $S_1$5 and the user profile $S_1$6, the LLM recommends actions covering content, style, specificity, and research ideas. Action relevance is scored via $S_1$7, and only the top-ranked actions (after user curation) are executed (Balepur et al., 17 Mar 2026).
• Execution of Actions in Report Synthesis: Action instructions are passed to all downstream synthesis stages, with explicit calls (“execute action $S_1$8”) enforced per section. Generated spans are tracked via highlights for traceability.

User studies reveal that persistent, transparent profiles and togglable action lists are valued; however, risks include cognitive overload from highlights and subtle failures in relevance or authenticity of personalizations—many undetectable by automated metrics.

5. Evaluation Protocols and Experimental Results

MySQA has been evaluated via both synthetic benchmarks (using LLM “judges” and synthetic users) and real-user studies:

• Synthetic Benchmarks: Compare MySQA to open-source (e.g., STORM, OpenScholar) and commercial (e.g., Perplexity, o3 DR) baselines on metrics such as answer coverage, citation recall, action adherence, and rubric scores. MySQA achieves the highest answer coverage (91.4%), improved citation recall (81.4% vs. 76.9% for ScholarQA), and superior action adherence (83.2% vs. 81.3%) (Balepur et al., 17 Mar 2026). In literature-centric QA benchmarks, end-to-end pipelines leveraging hybrid retrieval and multi-step generation (as in Ai2 Scholar QA and MySQA) deliver higher rubric scores and citation precision (Singh et al., 15 Apr 2025).
• Real-User Studies: 21 CS researchers participated in extended interviews, editing profiles and action lists, and rating personalized reports. Nuanced personalization errors were observed, such as misaligned subfield terminology, overgeneralization, action irrelevance, and style mismatches. 73% overall satisfaction was achieved, with emphasis on user control, modality diversity, and persistent customization (Balepur et al., 17 Mar 2026).
• Metrics: Include Exact Match (EM), F-score, BertScore-F1, and human/GPT-4 rubric evaluations. Prototype hybrid systems combining SPARQL + BERT achieve EM ≈ 0.64, F ≈ 0.71, outperforming both SPARQL-only and LLM-only baselines (Fondi et al., 2024).

6. Error Modes, Limitations, and Proposed Enhancements

Failure analysis uncovers algorithmic and modeling challenges:

• KG Coverage: Incomplete or ambiguous SPARQL returns (e.g., multiple URIs per author or missing institutions) can lead to empty or noisy contexts for extractive QA models.
• Classifier Robustness: Divide-and-conquer keyword classifiers may misroute subquestions, causing degraded routing performance.
• Span Extraction: Extractive QA errors arise when contexts are either too terse or polluted by irrelevant data.
• Personalization Gaps: Real-user evaluations uncover nuanced errors (e.g., domain, contrast, or presentation mismatches) invisible to LLM-judged benchmarks (Balepur et al., 17 Mar 2026).

Limitations include the absence of domain-adaptive LLM fine-tuning, rule-based merging logic for multi-value answers, no end-to-end retriever training, and lack of curriculum awareness for multi-turn queries (Fondi et al., 2024, Hicke et al., 2023).

Proposed extensions involve fine-tuning BERT or larger LLMs on QALD datasets, replacing keyword-based split/classify with supervised neural tagging (e.g., BiLSTM), incorporating graph-driven merging, batch reranking for latency reduction, and expansion to multi-modal or code/document signals.

7. Open-Source Implementation, APIs, and Future Directions

MySQA builds upon the blueprints laid out by Ai2 Scholar QA and related systems, offering:

• Modular Python packages (core retriever, reranker, generator) and FastAPI-based backend interfaces.
• Public APIs supporting streaming, sectioned QA, and filtered search over scholarly corpora.
• Downloadable passage indices, dev/test queries, and evaluation rubrics.
• Guidelines for extensibility along the axes of modularity, open APIs, domain-adaptive retrieval, and streaming user experience (Singh et al., 15 Apr 2025).

Future work is focused on tighter integration of user feedback, curriculum-aware retrieval, domain-adapted embeddings and LLMs, uncertainty estimation, improved multi-turn dialogue, and comprehensive mixed-method evaluation (combining synthetic, LLM, and real-user protocols) (Hicke et al., 2023, Balepur et al., 17 Mar 2026).

---

Collectively, MyScholarQA exemplifies a new paradigm in scholarly QA and literature synthesis: modular hybrid retrieval, user-driven personalization, fine-grained attribution, and mixed-method evaluation, informed by both algorithmic and real-user requirements.