GitTaskBench: Evaluating Repository-Aware Agents

• GitTaskBench is a benchmark that systematically evaluates code agents' ability to understand, navigate, and execute end-to-end tasks within authentic GitHub repositories.
• It introduces 54 real-world tasks across seven domains, emphasizing practical challenges like repository navigation, dependency resolution, and workflow management.
• The evaluation framework uses dual metrics—Execution Completion Rate and Task Pass Rate—along with an economic alpha-value to rigorously measure performance and cost-effectiveness.

GitTaskBench is a benchmark specifically designed to evaluate the capacity of code agents to solve end-to-end, real-world tasks by leveraging open-source code repositories. Unlike benchmarks that focus primarily on code generation from scratch, GitTaskBench systematically assesses an agent's ability to navigate, comprehend, and utilize existing repositories across a spectrum of practical, workflow-driven domains. This expands the evaluation paradigm from isolated code synthesis to the broader, more authentic context of repository-aware reasoning, environment setup, automated execution, and criterion-driven task fulfiLLMent (Ni et al., 26 Aug 2025).

1. Design Principles and Scope

GitTaskBench aims to measure the capabilities of agents in performing complex tasks that are anchored in authentic software engineering workflows. The benchmark introduces 54 tasks, each grounded in a real-world GitHub repository. Agents are evaluated not only on their understanding of the codebase but also on their ability to execute or modify repository code to achieve specific, pre-defined end goals. Each task is accompanied by an automated, human-curated evaluation harness that enforces both technical correctness and pragmatic utility, targeting scenarios that require agents to handle workflows from setup through deployment.

The central design principle is to model "user-centric, daily-life tasks" that require code agents to perform dependency resolution, environment configuration, repository navigation, and data processing. This approach compels agents to reason about repository structures, overcome setup challenges, and satisfy multifaceted success criteria reflecting genuine developer needs.

2. Task Modalities and Domains

To ensure comprehensive coverage of real-world challenges, GitTaskBench encompasses tasks across seven major modalities and domains:

Modality/Domain	Example Task Types	Data Engagement
Image Processing	Style transfer, coloring, restoration, enhancement	Process images, handle model assets
Video Processing	Action analysis, style transfer, video coloring	Manipulate video files, multi-step flows
Speech Processing	Recognition, enhancement, separation	Audio input/output, require models
Physiological Signal Processing	EDA/ECG/EEG analysis	Bio signal time-series, specialized libs
Security and Privacy	Data simulation, watermark embedding/extraction	Bitstream and metadata handling
Web Scraping	Information extraction from HTML/webpages	Automated network interaction
Office Document Processing	Multi-sheet Excel parsing, PDF content extraction	Spreadsheet and document parsing

This multi-domain, multi-modal design introduces heterogeneity in file formats, API usage, data pre-processing, and domain-specific evaluation. A key implication is that agentic success requires not only language competency but also accurate environmental orchestration and cross-modal data manipulation.

3. Evaluation Metrics and Automated Harness

GitTaskBench employs a two-layer, automated evaluation harness with human-verified ground truths and custom test scripts. Two primary metrics are used:

• Execution Completion Rate (ECR): The fraction of tasks for which the agent executes the repository code successfully and produces a non-empty output file of the expected format. This metric isolates basic viability of repository navigation, environment setup, and workflow completion.
• Task Pass Rate (TPR): The proportion of executed tasks whose outputs meet stringent, domain-specific quality criteria. These checks utilize quantitative measures (e.g., image similarity such as CIEDE2000, perceptual audio quality such as PESQ, or information extraction accuracy) specific to each task type.

Additionally, GitTaskBench introduces the alpha-value metric, designed to quantify the net economic benefit of agent performance in comparison to human labor. Defined as:

$$\alpha = \frac{1}{n} \sum_{i=1}^n [ (T \times MV \times Q) - C ]$$

where $T$ is a binary task success indicator, $MV$ the market value for human-completed tasks, $Q$ a normalized human-rated output quality, and $C$ the agent's operational cost. The alpha-value provides a cost-benefit analysis at the granularity of per-task agent deployment.

4. Experimental Results and Agent Performance

Empirical benchmarking was conducted across three major agent frameworks—Aider, SWE-Agent, and OpenHands—and diverse LLM backends, including GPT-4o, GPT-4.1, Anthropic Claude 3.5/3.7, and several open-source models (DeepSeek-V3, Qwen, Llama variants). The principal findings include:

• The highest observed task pass rate was 48.15% (OpenHands with Claude 3.7). This indicates that less than half of the repository-driven, end-to-end tasks could be solved by the most advanced models tested.
• Repository-centric, multimodal benchmarks remain substantially more challenging than code synthesis. Agents achieved higher performance on text-only tasks and lower when required to manage environment setup, cross-modal data flows, or complex dependency graphs.
• There is a nontrivial trade-off between performance and computational efficiency. Some configurations delivered moderate success rates with low token consumption, while leading success rates required correspondingly higher operational costs.

These results highlight significant gaps in current agents’ capabilities for generalizing to practical environments where workflow robustness and programmatic integration are essential.

5. Error Taxonomy and Identified Bottlenecks

Detailed error analysis partitioned agent failures into five categories:

• E1 – Environment Setup Errors (∼65%): Failures stemmed from unsatisfied dependencies, incompatible package versions, and absent system libraries.
• E2 – Workflow Planning Errors: Agents failed to execute sequential, multi-step instructions or halted after partial repository analysis.
• E3 – Repository Comprehension Errors: Mistakes in identifying main entrypoints, incorrect API utilization, and misreading repository organization.
• E4 – Runtime Execution Errors: Incomplete runs due to timeouts, crashes, or excessive resource usage (e.g., memory overruns).
• E5 – Instruction Non-compliance: Incorrect output file naming, incomplete results, or neglect of required repository usage.

This distribution underscores the centrality of robust environment setup and comprehensive repository analysis—facets not captured by traditional code-generation tasks. Improvements in these domains are critical to increasing real-world agent utility.

6. Access, Implementation, and Reproducibility

GitTaskBench is provided as a fully open-source benchmark, including all task definitions, automated harness scripts, and comprehensive documentation. Resources are made available at https://github.com/QuantaAlpha/GitTaskBench, incorporating:

• Configuration templates for popular agent frameworks (OpenHands, SWE-Agent, Aider)
• Example outputs, failure logs, and detailed criteria for both ECR and TPR metrics
• Public leaderboards and avenues for community-driven benchmark expansion

The open, modular structure is designed to facilitate both rigorous scientific benchmarking and practical adoption by research groups targeting real-world deployment of agentic code frameworks.

7. Research Implications and Future Directions

GitTaskBench marks a shift from code-level evaluation toward holistic, workflow-driven benchmarks that align more closely with real-world developer practices. The benchmark exposes persistent weaknesses in repository comprehension, generalized environment orchestration, sequential reasoning, and multi-modal integration. Key recommended directions include:

• Advancing workflow management mechanisms in agent frameworks
• Improving automated dependency and environment configuration strategies
• Expanding task and repository coverage, including technical ML and multi-agent scenarios, to increase benchmark representativeness
• Refining the alpha-value economic assessment, particularly in the context of hybrid human-in-the-loop workflows

A plausible implication is that as models mature in these areas, measurable improvements on GitTaskBench will serve as a leading indicator of readiness for general adoption in enterprise and open-source automation scenarios (Ni et al., 26 Aug 2025).