Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research

Abstract: LLMs are increasingly used for qualitative data analysis, but many workflows obscure how analytic conclusions are produced. We present QualAnalyzer, an open-source Chrome extension for Google Workspace that supports atomistic LLM analysis by processing each data segment independently and preserving the prompt, input, and output for every unit. Through two case studies -- holistic essay scoring and deductive thematic coding of interview transcripts -- we show that this approach creates a legible audit trail and helps researchers investigate systematic differences between LLM and human judgments. We argue that process auditability is essential for making LLM-assisted qualitative research more transparent and methodologically robust.

• The paper's main contribution is the development of QualAnalyzer, a Chrome extension that enables atomistic LLM analysis for transparent qualitative research.
• It introduces a methodology that processes each segment independently to mitigate hallucination risks while ensuring reproducibility through stored prompts and outputs.
• Case studies on essay scoring and thematic coding validate its effectiveness in improving methodological rigor and facilitating prompt revision in AI-assisted research.

Affording Process Auditability with QualAnalyzer: An Atomistic LLM Analysis Tool for Qualitative Research

Introduction

This paper proposes QualAnalyzer, an open-source Chrome extension designed to facilitate atomistic LLM analysis within qualitative research. The motivation is to address methodological opacity prevalent in current LLM-assisted qualitative workflows, which often lack traceability regarding analytic procedures, prompt application, and segment-level evidence. QualAnalyzer introduces a paradigm that emphasizes processing each segment independently and retaining a granular audit trail, enabling researchers to discern and diagnose model outputs with heightened rigor.

Paradigms for LLM-Assisted Qualitative Analysis

The paper delineates three paradigms for qualitative analysis: traditional human researcher, holistic LLM, and atomistic LLM workflows. The traditional approach features sequential reading, interpretive accountability, and iterative memoing, while holistic LLM workflows process documents in a single context window, risking position biases and long-context failures. Atomistic LLM analysis—embodied by QualAnalyzer—processes each segment uniformly and stores input, prompt, and output for every unit, mitigating hallucination risk and offering reproducibility and transparency.

Figure 1: Three paradigms for qualitative analysis highlighting sequential, parallel, and atomistic processing strategies for document analysis.

Through formalization, the atomistic workflow avoids batch position effects and supports reproducibility, side-by-side model comparisons, and downstream validation. The methodological framing extends prior literature by offering a practical system for segment-level processing accessible to research teams without programming expertise.

Workflow and Methodological Design

QualAnalyzer integrates within Google Workspace, utilizing Sheets and Docs as primary interfaces. Each unit of analysis is a sheet row, with context columns preserved and output columns systematically populated. Prompt iteration and auditability are structured around stored prompts and recorded outputs at the segment level.

Figure 2: The atomistic workflow in QualAnalyzer, supporting reproducibility, model comparison, and downstream validation via structured tabular processing.

Key design principles include segment-level visibility, practical prompt iteration, diagnosis via stored outputs and agreement metrics, and accessibility without programming. The architecture comprises a Chrome side panel, LLM analysis modules (Doc Parser, Prompt Builder, Sheet Processor, IRR Calculator), a core orchestrator for execution tracking, and an infrastructure layer for Workspace and LLM API access.

Figure 3: QualAnalyzer system architecture showing task configuration, modular analysis, batch jobs, and real-time progress reporting within Google Workspace.

The interface design supports traceability with minimal disruption to familiar collaborative environments, leveraging spreadsheet logic for visible, auditable output alignment and prompt revision.

Case Studies: Application and Auditability

Essay Scoring using ASAP Dataset

QualAnalyzer was validated on holistic essay scoring with the Automated Student Assessment Prize (ASAP) dataset. The prompt was constructed from rubric definitions and scored anchor papers as few-shot examples, ensuring construct traceability. Dual-pass atomistic analysis using GPT-5.2 yielded highly consistent numeric scores across runs, reinforcing prompt stability and instrument reliability. Segment-level justification, including quoted evidence, facilitates granular audit trails for each judgment.

Deductive Thematic Coding of Interview Transcripts

The tool was further applied to deductive coding of qualitative data curation interview transcripts. Codebook operationalization required supplementing published labels with formal definitions. Atomistic runs with binary Present/Absent classifications and mention counts produced perfect IRR ($k=1.00$, $n=23$) for binary codes and high agreement for mention counts (83–87%), but systematic excesses in mention counts versus published human annotations. Preservation of quoted passages alongside mention counts enables targeted review and iterative prompt revision, embodying process auditability beyond mere documentation.

Implications and Future Directions

QualAnalyzer reframes segment-level LLM analysis as a methodological paradigm, supporting transparent auditability and iterative refinement. The audit trail's dual role—confirmatory and diagnostic—is evident across case studies. The tool underscores the importance of codebook specificity and exposes alignment issues between human and LLM judgments, facilitating targeted revision and reliability checking.

Hybrid workflows integrating holistic synthesis with atomistic rigor are envisioned, but the primary analytical agency remains with the researcher. QualAnalyzer operationalizes transparent model-assisted interpretation by assembling existing capabilities around practical audit needs, without expanding the automation scope beyond auditable trace facilitation.

Practically, QualAnalyzer democratizes accessible LLM-enhanced qualitative workflows. Theoretically, it advances discourse on process transparency, reproducibility, and interpretive accountability in AI-assisted research. Future development should address structured cross-segment synthesis without undermining auditability and encourage active engagement with audit trails to mitigate risks of passive transparency.

Limitations

The atomistic paradigm constrains analyses that require holistic synthesis, as QualAnalyzer treats segments separately. Passive transparency remains a risk; active researcher engagement is essential to interpret and revise model outputs, ensuring methodological rigor. High internal consistency does not guarantee alignment with substantive research intent, necessitating human oversight in prompt construction, output validation, and analytic interpretation.

Conclusion

QualAnalyzer offers a practical infrastructure for transparent, auditable segment-level LLM-assisted qualitative analysis, accessible to non-programmers and adaptable to diverse research contexts. Atomistic workflows preserve traceability and facilitate iterative diagnostic review, positioning auditability as foundational for methodologically robust AI-enhanced qualitative inquiry. Future AI developments should further integrate auditability and active researcher engagement, advancing transparent and defensible qualitative methodologies.