EGAgent: Extraction & Video Understanding

• EGAgent is a dual-framework system combining automated extraction of scholarly papers using RPA with multimodal analysis of long egocentric videos via structured scene graphs.
• It employs LLM-driven planning and modular tools for task decomposition, cross-modal retrieval, and precise geographic or entity classification in diverse workflows.
• Empirical results demonstrate high precision and robustness, while highlighting challenges like ambiguous affiliations and perceptual errors in live video data.

EGAgent refers to two distinct agentic frameworks in recent academic literature: one, a system for automated extraction and geographic identification of scholarly papers with robotic process automation (RPA) (Vishesh et al., 11 Sep 2025); the other, a multimodal agentic framework for persistent, long-horizon understanding of egocentric video streams via entity scene graphs and hybrid reasoning tools (Rege et al., 26 Jan 2026). Both instantiations exemplify the convergence of LLMs, structured knowledge representations, cross-modal retrieval, and agentic planning in complex academic and real-world workflows.

1. Extraction and Geographic Identification: EGAgent/AEGIS

EGAgent (also referred to as AEGIS) is a fully automated pipeline designed for accurate extraction and geographic classification of papers from scholarly proceedings, with direct integration of RPA to drive actions such as automated nomination submissions (Vishesh et al., 11 Sep 2025). The architecture comprises four modules:

• Document Ingestion: A headless browser (e.g., Selenium with Chrome) loads proceedings URLs, rendering dynamic content and storing static HTML snapshots.
• Metadata Extraction: HTML parsing and DOM traversal identify candidate paper URLs, employing layout-aware strategies for flat or multi-track listings, normalization, and deduplication. Each URL is processed via a publisher-specific prompt sent to an LLM-based REST API (“Agent-E”) to extract titles, authors, and affiliations. Regex and JSON parsing yield a normalized record.
• Geographic Region Classification: Affiliation strings are tokenized and compared with a curated set of region identifiers. Boolean logic determines target-region membership via string normalization and substring matching; LLM-based disambiguation is used for ambiguous cases.
• RPA Submission: The RPA module, implemented with Selenium, auto-fills web forms for nominations by mapping structured metadata to dynamic HTML fields, handling interaction logic (e.g., adding input fields, scrolling), and confirming submission by DOM inspection.

Formal Definitions and Classification Algorithm

Let $N$ be the number of papers, with ground-truth labels $y_i \in \{0,1\}$ and predictions $\hat{y}_i$. Performance metrics:

• Recall: $\frac{TP}{TP + FN}$
• Precision: $\frac{TP}{TP + FP}$
• Accuracy: $\frac{TP + TN}{TP + TN + FP + FN}$
• F1-score: $$2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$$

Rule-based region classification (core pseudocode, exact from (Vishesh et al., 11 Sep 2025)):

$y_i \in \{0,1\}$5

Experimental Protocol and Quantitative Results

EGAgent/AEGIS was validated on 586 papers from five conference sources. With ground truth assigned via manual PDF inspection, the system achieved:

• Recall = 100% (CI: 0.90–1.00)
• Precision ≈ 92.5%
• Accuracy ≈ 99.5% (rounded to 99.4%)

Performance was robust across all conference subdatasets, with only 3 false positives and 0 false negatives. The rigid recall-focused thresholding ensures completeness in target identification for downstream RPA actions (Vishesh et al., 11 Sep 2025).

RPA Integration

EGAgent’s RPA agent operates in both headless and debug modes, executes DOM stabilization, injects necessary HTML fields, and matches extracted JSON keys to form elements. Submissions are logged with retry logic for transient failures. Success feedback is deterministic via detection of confirmation banners.

Limitations and Extensions

Notable limitations include potential false positives from ambiguous or nonstandard affiliation entries, inability to handle heavily paginated or CAPTCHA-protected sites, and manual maintenance of region keyword lists. Possible extensions involve learned classifiers for affiliation embeddings, multi-region or multi-class support, application to other domains (e.g., grant applications, patent mining), and incorporation of an active-learning loop for incremental refinement (Vishesh et al., 11 Sep 2025).

2. Agentic Very Long Video Understanding: EGAgent

A separate instantiation of EGAgent is an agentic framework for persistent, cross-modal, entity-centric video understanding over longitudinal egocentric recordings (Rege et al., 26 Jan 2026). The system is designed for continuous query answering—even over weeks of personal video—by integrating scene graph construction, hybrid retrieval, and planning agent orchestration.

Core Architecture

EGAgent consists of five principal modules:

• Entity Scene Graph Construction: At 1 FPS, visual frames (captioned/located) and audio transcripts (with speaker diarization) are parsed by LLM extractors, producing entities and relations incrementally aggregated into a persistent, temporally annotated scene graph. Nodes are typed (Person, Object, Location), and edges encode relations (talks_to, interacts_with, mentions, uses) with interval attributes.
• Planning Agent: Given a user query $Q$, an LLM decomposes $Q$ into subtasks $S_i$, selects appropriate tools (Visual, Audio, Graph), and advances working memory $y_i \in \{0,1\}$0. The agent plans task/tool/subquery tuples to maximize evidence acquisition.
• Hybrid Search Tools:
  • Visual Search: Dense retrieval from frame embeddings via cosine similarity, with optional semantic filters, returns time-stamped thumbnails.
  • Audio Transcript Search: LLM-based extraction or BM25 retrieval of relevant transcript spans.
  • Entity Graph Search: SQL-based queries over the graph database; a “strict → relaxed” cascade gradually loosens search constraints for challenging cases.
• Analyzer Tool: LLM(s) ingest retrieved context, distill evidence, and update working memory.
• VQA Agent: A multimodal LLM produces the final answer conditioned on both $y_i \in \{0,1\}$1 and $y_i \in \{0,1\}$2.

Algorithms and Formulation

Graph construction employs temporal merging: overlapping edges (with the same source/target/type) are merged by interval union. Planning is formalized as searching for subtask sequences $y_i \in \{0,1\}$3 maximizing expected evidence utility:

$y_i \in \{0,1\}$4

Visual / audio similarity and BM25 ranking are directly integrated.

Datasets and Empirical Results

EGAgent’s performance was established on the EgoLifeQA (50 hours, 500 MCQs) and Video-MME (Long) (300 videos, 900 MCQs) datasets:

Dataset	Accuracy (EGAgent+Gemini 2.5 Pro)	Baselines	Notes
EgoLifeQA	57.5%	Gemini Uniform: 46.8%	+20.8% on RelationMap, +22.2% on TaskMaster
Video-MME (Long)	74.1%	Gemini: 82.0%	5× fewer frames processed

Ablations show that removing entity scene graph search degrades relation-type question accuracy by ~36%, demonstrating the significance of structured entity memory for multi-hop reasoning (Rege et al., 26 Jan 2026).

Critical Analysis

EGAgent’s design enables compositional, multi-hop inference, integrating cross-modal evidence. Limitations include upstream dependency on perception and ASR accuracy, latency dominated by LLM analyzer calls, and the lack of end-to-end learning (static graphs cannot recover from hallucinated or missing relations). Extensions proposed include graph neural refinement, efficient dense retrieval, dynamic memory pruning, and analyzer fine-tuning on multimodal retrieval benchmarks (Rege et al., 26 Jan 2026).

3. Underlying Agentic, Modular, and Tool-Augmented Paradigm

Both EGAgent variants exemplify an agentic paradigm wherein an LLM-based planner orchestrates modules implementing specialized “tools”—from HTML parsing and entity-graph traversal (scholarly or video), to cross-modal retrieval and RPA. Task decomposition, tool selection, memory management, and context updating are orchestrated in iterative loops, facilitating end-to-end automation and complex reasoning over heterogeneous data.

This approach benefits from interpretability via persistent state (structured metadata, scene graphs, logs); modularity and extensibility (swapping or adding tools); and robust chaining of evidence over extended horizons—whether temporal (as in video) or procedural (as in scholarly pipelines). A plausible implication is that such agentic frameworks will generalize to diverse information extraction and reasoning domains.

4. Comparative Perspective with Adjacent Systems

EGAgent’s design contrasts with single-task or narrow-context LLM-based methods in both extraction/RPA (Vishesh et al., 11 Sep 2025) and video QA (Rege et al., 26 Jan 2026). Related agentic systems—e.g., Egent for equivalent width measurement in spectroscopy (Ting et al., 1 Dec 2025), and EEGAgent for LLM-driven, multi-tool EEG signal analysis (Zhao et al., 13 Nov 2025)—share architectural themes: central LLM planners, principled toolboxes, structured memory, and iterative chain-of-thought orchestration. The field is converging on hybrid agentic architectures enabling rapid, reproducible, expert-level automation in data-rich scientific domains.

5. Limitations, Future Directions, and Generalization

Despite high precision, recall, and extensibility, both EGAgent systems rely on brittle components—rule-based classifiers for region identification, or post hoc graph construction subject to noisy upstream perception. Maintenance of metadata templates and region keyword lists, as well as latency due to LLM analyzer/reasoner steps, remain open engineering challenges. Major extensions include neural module replacement for handcrafted classifiers, scaling to richer or more global domains (multinational, multimodal), and active learning or uncertainty-aware human-in-the-loop frameworks. Given persistent advances in multimodal LLMs, these paradigms are poised for widespread adoption in longitudinal, cross-domain extraction–reasoning workflows.

References:

• EGAgent for scholarly proceedings: (Vishesh et al., 11 Sep 2025)
• EGAgent for very long video understanding: (Rege et al., 26 Jan 2026)
• Egent for spectral line analysis: (Ting et al., 1 Dec 2025)
• EEGAgent for EEG analysis: (Zhao et al., 13 Nov 2025)