RAGDoll: VR Avatar & RAG Serving System

• RAGDoll is a dual-purpose framework offering VR avatars for digital heritage exploration and an offloading-oriented RAG serving system for efficient question answering.
• It employs modular pipeline design with adaptive batch scheduling, joint memory management, and metadata tagging to enhance retrieval accuracy and generation performance.
• The system significantly reduces latency and resource usage, streamlining immersive digital research and scalable deployment on single-GPU setups.

RAGDoll denotes two distinct but related advances in the application of Retrieval-Augmented Generation (RAG)—one as a class of VR-embedded, conversational avatars for immersive digital heritage, and the other as an offloading-oriented, efficiency-maximizing RAG serving system for resource-constrained hardware. Each approach illustrates how modular pipeline design, memory management, and human–computer interfaces can drive advances in large-scale, context-rich question answering.

1. Definitions and Motivations

RAGDoll as a VR Avatar (Editor’s term for “RAG-based avatar”):

A RAGDoll is a Retrieval-Augmented-Generation–driven interactive avatar integrated in a virtual reality (VR) environment. Formally, it comprises:

• a domain document store $\mathcal{D}$,
• an embedding function $f_{\text{emb}}$,
• a retrieval mechanism ranking document chunks $d$ via normalized similarity $$s(q,d) = \langle e_q, e_d \rangle / (\|e_q\| \|e_d\|)$$ with $e_q = f_{\text{emb}}(q)$,
• and an LLM-based generator conditioned on $\text{top-}k$ retrieved contexts.

This interface enables users—via speech—to query a domain-specific digital expert during immersive exploration of 3D reconstructions, circumventing static knowledge bases and anchoring outputs in curated textual fragments. This supports dynamic, up-to-date factuality and curatorial control without frequent full-model retraining (Kerle-Malcharek et al., 24 Mar 2026).

RAGDoll as a RAG Serving System:

RAGDoll is also a resource-efficient, online RAG serving framework optimized for single-GPU, memory-bound deployments. It separates retrieval (CPU-bound) and generation (GPU-bound) into parallel pipelines, orchestrates joint memory placement across device tiers, and employs adaptive dynamic batch scheduling. This yields substantial improvements in latency and throughput over serial RAG designs, supporting efficient, large-scale QA workloads on commodity hardware (Yu et al., 17 Apr 2025).

2. System Architectures

RAGDoll Avatar Pipeline (Kerle-Malcharek et al., 24 Mar 2026)

• Preprocessing: Domain texts are OCR’d (Tesseract) to plain text, chunked (1000-char, 200 overlap), and annotated with metadata (author, title, publication_type, relevance).
• Retrieval: Spoken queries transcribed (Whisper STT), embedded ($$f_{\text{emb}} \in \{\mathrm{MLE5}, \mathrm{Qwen3-embedding}\}$$), retrieved by cosine similarity, and filtered by metadata.
• Generation: LLMs (Teuken-7B, Llama 3.1/8B, Llama 3.3/70B) yield outputs $P(y|x) = \prod_{t=1}^T P(y_t | y_{<t}, x)$; generation temperature is set low ($T=0.3$) to enforce consistency.

RAGDoll Serving System Pipeline (Yu et al., 17 Apr 2025)

• Parallel Pipelines:
  • Retrieval Worker (CPU): Batches queries, retrieves top-$k$ vectors (Milvus), builds context batches.
  • Context Queue: In-memory, FIFO-transfers (query + retrieved context) to GPU.
  • Generation Worker (GPU): Offloads weights and KV cache (FlexGen-style), streams tensors as needed, emits tokens.
• Memory Placement: Mapping

$f_{\text{emb}}$0

where $f_{\text{emb}}$1 are transformer layers, $f_{\text{emb}}$2 weights, $f_{\text{emb}}$3 cache, $f_{\text{emb}}$4 vector partitions.

• Scheduling: Retrieval/generation run asynchronous, backlog-aware batch schedulers—minimizing average per-request latency for varying influx and model/load sizes.

3. Design and Requirement Space

Axes for RAGDoll Avatars (Kerle-Malcharek et al., 24 Mar 2026)

• Application Realm: From collection management (museum use), conservation, research (requiring multi-hop and citation Q&A), to public presentation/teaching.
• Avatar Personality:
  • Epistemic authority: Personal, non-personal, or collective (e.g., “the Academy”).
  • Perspective: First-, third-person, or omniscient/authorial.
  • Embodiment: Human, robotic, or bodiless; embodiment modulates gesture/UI presence and user expectations.
• I/O Modalities:
  • Output: Visual (UI overlays, avatar animation), Audio (speech), Haptic, proprioceptive.
  • Input: Voice (STT), gaze, pointing, or controller input.
  • Example: Prototype relied on audio-only (Whisper STT, Piper TTS) with optional visual overlays on in-VR wrist panels. This streamlined interaction but limited real-time citation display, mitigated by UI augmentations.

4. Memory Management and Scheduling Strategies

Joint Memory Placement (Yu et al., 17 Apr 2025)

RAGDoll formalizes memory placement as: $f_{\text{emb}}$5 subject to device memory bounds, where $f_{\text{emb}}$6 parameterizes fractions for $f_{\text{emb}}$7, $f_{\text{emb}}$8, and disk placement of weights, KV cache, and the number of vector DB partitions in RAM. This enables:

• Preloading critical partitions in RAM,
• Offloading model weights to balance GPU and CPU usage,
• Dynamic adjustment based on observed workload intensity.

Adaptive Batch Scheduling

Each (retrieval, generation) pipeline operates a batch scheduler, using empirically estimated $f_{\text{emb}}$9. Batches are split adaptively to minimize latency, with consideration for the super-linear scaling of GPU memory demands (KV cache) at large $d$0. Retrieval batched in bulk due to flat cost; generation batch size finely tuned to avoid memory overcommitment and head-of-line blocking.

5. Evaluation and Empirical Results

Avatar System Evaluation (Kerle-Malcharek et al., 24 Mar 2026)

• Answer Quality Metrics (QA on VR Archaeology):
  • Automatic: METEOR, BERTScore.
  • LLM-as-a-Judge (Prometheus 2): Relevance, coherence, factuality, depth (1–5 scale).
  • Findings:
  • Metadata tagging (“main/relevant/adjacent”) increased scores by ~0.3.
  • Larger models plus relevance metadata reached mean rating $d$1.
  • GraphRAG (CIDOC-CRM) offered moderate performance gains but higher setup cost.
• User Workload (NASA-TLX, $d$2 participants):
  • Mean overall workload: $d$3 (below average).
  • Mental demand: 11.9/21; Physical: 2.7; Temporal: 8.5; Performance: 9.0; Effort: 8.7; Frustration: 9.5.
  • No significant workload–accuracy correlation.

Serving System Performance (Yu et al., 17 Apr 2025)

• Test Platforms:
  • PF-High: Xeon 4314, 256 GB RAM, NVIDIA A30 (24GB), 8TB NVMe.
  • PF-Low: Xeon 6238R, 176 GB RAM, NVIDIA A5000 (12GB), 2TB SSD.
  • Workload: TriviaQA with 650k QA pairs, 256GB Milvus, LLaMA 3.1 models.
• Latency and Throughput:
  • Up to 3.6$d$4 lower latency vs serial vLLMRAG (555s vs 1990s, 8B model).
  • Up to 11.7$d$5 speedup vs HuggingFace Accelerate under load.
  • For 70B models on PF-Low, 99th percentile latency 50% lower vs vLLMRAG.

Method	Waiting (s)	Retrieval (s)	Generation (s)
RAGDoll	170	320	66
vLLMRAG	1640	293	57
Accelerate	3421	288	176

On PF-Low, 8B model; waiting dominates non-pipelined baselines.

6. Practical Guidelines, Limitations, and Future Directions

Key Insights

• Separation and Pipelining: Async CPU-bound retrieval & GPU-bound generation reduces idle time from 80% to 30% in typical serial RAG (Yu et al., 17 Apr 2025).
• Memory Placement: Coordinated offload enables larger effective batch sizes and prevents device contention/thrashing.
• Metadata Tagging: Simple, curator-applied metadata on corpus chunks is an effective, low-effort method to improve factuality and retrieval performance (Kerle-Malcharek et al., 24 Mar 2026).

Limitations

• Avatar QA Evaluation: Small evaluation set (10 items); requires scaling for robust statistical assessment.
• Relevance Annotation: Curator bias in tagging may impact retrieval quality (Kerle-Malcharek et al., 24 Mar 2026).
• Profiling Assumptions: Serving system assumes flat retrieval cost in batch size; for disk-based ANN systems, this may not hold (Yu et al., 17 Apr 2025).
• Single-GPU Limitation: No multi-GPU or hybrid edge–cloud support yet. Future work may explore distributed context management and advanced retrieval sparsification.

Open Directions

• Visual Uncertainty and Hypothesis Surfacing: Explicit display of evidential support and alternative hypotheses in VR.
• Multi-Modal and Multi-Speaker I/O: Support for gesture, haptic feedback, and ensemble avatar personalities.
• No-Code Configurator: Tools for curators to dynamically upload/annotate domain sources, select persona traits, and configure retrieval/generation behaviors.
• Approximate Retrieval: Potential integration with TurboRAG, CacheBlend for further efficiency.

7. Significance and Broader Impact

RAGDoll exemplifies the interplay between modular retrieval-augmented architectures and application-driven interface integration. As a VR avatar, it enables immersive digital experts supporting research, conservation, and education in digital heritage. As a serving system, it demonstrates that cross-device pipeline orchestration, adaptive memory allocation, and batch scheduling can deliver order-of-magnitude efficiency gains, making high-quality RAG deployments feasible even on commodity hardware. These contributions lower the barrier for scalable, context-grounded QA both in specialized domains (e.g., archaeology) and in general-purpose, large-knowledge deployments (Kerle-Malcharek et al., 24 Mar 2026, Yu et al., 17 Apr 2025).