Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

Abstract: Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of LLMs to access information outside their parameters. We evaluate if cluster-based semantic chunking improves retrieval and answer quality compared to fixed-size and recursive chunking evaluating on long, structured academic theses using the Retrieval Augmented Generation Assessment (RAGAs) framework. RAGAs based faithfulness shows limited reliability in this setup. Performance on fixed versus document specific questions varied substantially, likely related to the formatting of documents and preprocessing. Under the tested configuration, cluster-based chunking did not outperform simpler strategies.

• The paper evaluates fixed-size, recursive, and cluster-based chunking methods in RAG, revealing significant differences in retrieval performance.
• Recursive chunking achieves higher context F1 for thesis-specific queries, while cluster-based methods incur computational overhead without quality gains.
• Experimental results highlight that resource constraints and document artifacts can undermine retrieval faithfulness and overall answer quality in RAG systems.

Evaluation of Chunking Strategies in Retrieval-Augmented Generation for Academic Theses

Retrieval-Augmented Generation (RAG) systems augment LLM capabilities via dynamic access to external documents, but such systems remain fundamentally constrained by the document segmentation, or chunking, process. This paper systematically benchmarks fixed-size, recursive, and cluster-based semantic chunking on long-form, highly structured academic theses, leveraging the RAGAs framework to measure retrieval and answer quality. Critical insights emerge regarding chunker efficacy, faithfulness evaluation, and computational trade-offs.

Background: RAG, Chunking, and Evaluation

RAG architectures rely on vector databases to store and retrieve document chunks, which are then presented to LLMs for answer synthesis. As LLM context windows and embedding models impose strict input length limits, document chunking becomes indispensable. Conventional chunking approaches include:

• Fixed-sized chunking: Uniformly splits documents by length, often with overlapping windows.
• Recursive (format-driven) chunking: Splits along structural delimiters, recursively segmenting large blocks.
• Cluster-based (semantic) chunking: Clusters semantically similar sentences into chunks using sentence embedding similarity.

While fixed and recursive chunking are computationally light, semantic chunking is more expensive, theoretically offering improved coherence at the cost of operational complexity [quSemanticChunkingWorth2025]. However, empirical support in RAG setups, especially for domain-specific, lengthy documents, remains limited.

RAG pipeline evaluation is itself an open problem. Manual labeling is infeasible at scale; thus, this work employs RAGAs, which automates measurement via LLM-based assessments of retrieval faithfulness (claims traceable to cited context), answer relevance, and a composite Answer Quality Score (AQS):

$AQS = \frac{2 \cdot F \cdot AR}{F + AR}$

where $F$ is faithfulness and $AR$ is answer relevancy. Context F1, measuring semantic overlap in retrieval, and TF-IDF bigram cosine similarity are also computed.

(Figure 1)

Figure 1: Pipeline concerning the evaluation of chunking methods using RAGAS.

Experimental Setup

The evaluation corpus consists of thirteen academic theses (10k–27k words), with five fixed-format and five thesis-specific QA queries. The LLM pipeline is strictly resource-constrained (16GiB VRAM): all-MiniLM-L6-v2 for embedding, llama3.2:3b for generation, and deepseek-r1:8b for evaluation.

Cluster-based chunking employs single-linkage clustering with a distance threshold of 0.5 and a weighted positional-semantic factor ($\lambda=0.25$), producing semantically coherent 3-sentence chunks, adapted for small-model constraints.

Results: Breakdown by Chunking Strategy and Metric

Substantial rates of metric failure (notably LLM-based faithfulness) are observed, with 44% of faithfulness computations failing due to resource constraints, affecting all chunkers similarly. Only context F1 and answer relevancy remain robust.

Context F1:

• For fixed-format (generic) questions, F1 medians are zero for all chunkers, indicating irrelevant retrieval due to structural artifacts in document preliminaries (e.g., dot leaders, headings).
• For thesis-specific questions, recursive chunking yields the highest F1 (median ≈ 0.5), outperforming fixed-size (median ≈ 0.3); cluster-based performs poorly, with wide IQR but low central tendency.

AQS (Answer Quality Score):

• AQS remains low for all chunkers on fixed questions.
• For free questions, fixed-size and recursive chunking achieve medians near 0.65, with recursive showing less variance. Cluster-based lags with a median of 0.40.
• Notably, the disjunction between context and answer metrics for fixed questions suggests LLMs sometimes answer correctly using parametric knowledge despite irrelevant retrieved context—highlighting limitations of recall-based retrieval proxies in highly template-driven text.

Comparative Interpretation

The results confirm recent claims that computationally intensive semantic or cluster-based chunking does not generically outperform fixed-size or formatting-driven chunking in end-to-end RAG [quSemanticChunkingWorth2025, wangDocumentSegmentationMatters2025]. In this study, cluster-based chunking does not yield superior retrieval or answer quality, even with careful adaptation, and introduces notable computational and system complexity. Recursive chunking offers marginal gains in retrieval for content-dense questions but struggles with noisy or artifact-heavy preliminaries.

Performance for all chunkers is fundamentally suppressed by factors including:

• Small embedding models with inferior semantic discrimination.
• Heuristic cleaning yielding suboptimal input for retrieval.
• Lack of strong metric reliability for faithfulness via constrained LLM judges.

Results reinforce the domain-, document-, and hardware-dependent nature of optimal chunking granularity [bhatRethinkingChunkSize2025]. The pipeline's overall answer quality is moderate only on document-specific queries and unreliable for fixed-format ones.

Implications for RAG and Future Directions

Practically, the study provides evidence that for long-form, structured academic texts and mid-range compute, simple chunking paradigms are preferable unless more powerful embedding and evaluation resources are available. Faithfulness evaluation via LLMs is brittle under resource limits and document artifact density.

Theoretically, RAG evaluation must move toward:

• ID-based recall metrics: To decouple answer manipulation from retrieval grounding.
• Granularity adaptation: Mix-of-granularity and graph-based segmentation approaches may offer more robust query-adaptive retrieval [zhongMixofGranularityOptimizeChunking2024, edgeLocalGlobalGraph2025].
• Benchmarks with human annotation: To validate or correct LLM-based proxies for chunk and answer quality.

Additionally, while recent literature suggests semantic chunking may help for semantically complex, cross-domain corpora, that benefit is contextually contingent and cannot be assumed for structurally uniform, formulaic text [quSemanticChunkingWorth2025, yepesFinancialReportChunking2024].

Conclusion

This paper systematically evaluates chunking strategies on academic theses in a resource-constrained RAG setup, finding that cluster-based semantic chunking does not outperform simpler methods and can be counterproductive in practical settings. The evaluation exposes limitations in popular automated metrics and highlights the complexity of robustly benchmarking RAG systems.

Future directions include scaling up embedding and evaluation models, developing more discriminative metrics backed by human annotation, adopting adaptive chunking granularity, and investigating advanced chunking (e.g., graph- or element-type-based) on heterogeneous corpora.

References:

• "Is Semantic Chunking Worth the Computational Cost?" [2025.findings-naacl.114]
• "Document Segmentation Matters for Retrieval-Augmented Generation" [2025.findings-acl.422]
• "Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation" (Zhong et al., 2024)
• "Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems" (Caspari et al., 2024)
• "RAGAs: Automated Evaluation of Retrieval Augmented Generation" [2024.eacl-demo.16]
• "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models" (Günther et al., 2024)