Hephaestus-Forge Corpus Overview

• Hephaestus-Forge Corpus is a large-scale agent-oriented dataset with 103 billion tokens designed to enhance API function calling and sequential planning.
• It employs a tripartite approach—API documentation, action trajectories, and code-to-text synthesis—combined with empirical scaling laws for continual pre-training.
• Benchmarking shows that models pre-trained with Hephaestus-Forge achieve superior multi-step reasoning and environmental adaptation compared to standard counterparts.

The Hephaestus-Forge Corpus is a large-scale, agent-oriented dataset devised to substantially enhance the core capabilities of LLM agents. Motivated by the limited availability of dedicated datasets for autonomous agent pre-training, Hephaestus-Forge provides foundational resources for API-centric function call reasoning, sequence planning, and adaptive interaction with environmental feedback. The corpus comprises 103 billion meticulously curated tokens, with documentation for 76,537 APIs, action trajectories, and synthesized code-to-text examples. This infrastructure is a pivotal resource supporting continual agent pre-training, empirically validated by comprehensive benchmark results and explicit scaling laws, and designed to maximize both specialized agentic skill and generalization.

1. Corpus Composition and Curation

Hephaestus-Forge is distinguished by the scale and specificity of its agent-focused content. The corpus contains approximately 103 billion tokens, assembled from three primary modalities:

• API Documentation and Contextual Metadata: The corpus indexes 76,537 APIs sourced from over 1,400 public endpoint documentation sites, official repositories (e.g., Huggingface, TorchHub, Python Modules), and systematically expanded by multi-level URL crawling. For each API, the dataset incorporates function specifications, input/output parameters, and behavioral descriptions.
• Function Calling Trajectories: Action sequences (i.e., function call trajectories) are extracted both from public repositories and simulated interactive environments. These trajectories demonstrate stepwise API use in solving tasks, capturing both syntactic invocation and semantic reasoning.
• Code-to-Text Synthesis: Using automated synthesis tools such as StarCoder-API, code snippets are transformed into documentation-style natural language, augmenting the corpus with explanations anchored to executable examples.

This tripartite approach ensures agentic breadth—from knowledge acquisition to multistep operational planning—with explicit documentation serving both the identification and detailed reasoning underpinning function calls.

2. Training Protocols and Scaling Laws

Hephaestus-Forge underpins a continual pre-training protocol, emphasizing empirically justified data mixture ratios. Experiments conducted across model sizes (45M–0.65B parameters) reveal that the benchmark loss, $\mathcal{L}$, exhibits power law dependence on the fraction $x$ of agent data:

$\mathcal{L} = c + k \cdot x^{\alpha}$

where $c$, $k$, and $\alpha$ are empirically fit. The scaling law enables the prediction of the optimal agent-to-general-to-code data ratio when pre-training larger models. A mixture close to 36% agent-specific data and 32% each for code and general text (effectively a 1:1:1 ratio) is shown to deliver maximal gains in both agent-specific and general-purpose capabilities. This protocol ensures the corpus does not induce task overfit or compromise language understanding.

3. Enhanced Agentic Capabilities

Utilization of the Hephaestus-Forge Corpus confers several critical capabilities to downstream LLM agents:

• API Function Calling: Pre-training with tool documentation and trajectories enables precise mapping from language-based goals to executable API calls, including multi-parameter and ordered invocation.
• Intrinsic Reasoning and Planning: Exposure to multistep function call trajectories trains agents to decompose complex objectives into actionable plans, evidenced by chain-of-thought solutions and dynamic task adaptation.
• Adaptation to Environmental Feedback: Segments of the corpus encode simulated agent-environment interactions, where observations ($o_1, o_2, \ldots$) inform subsequent function calls. This iterative agent-environment loop imparts robust adaptability to interactive and uncertain scenarios.

The integration of environmental feedback via agent data further distinguishes Hephaestus-Forge, enabling models to adjust plans algorithmically in response to observed outcomes.

4. Benchmarking and Comparative Performance

Hephaestus-Forge supports rigorous benchmarking of agentic skills. Evaluation is performed across three major benchmarks:

• AgentBench: Environments span OS, Database, Household, Web Browsing, Knowledge Graphs, and Web Shopping, testing real-world planning and adaptability.
• Berkeley Function Calling Leaderboard (BFCL): Both BFCL-v2 and BFCL-v3 grade function calling accuracy and the ability to generalize over complex, multi-step API usage.
• Additional: Systems such as NexusRaven and API-Bank/API-Bench are included to broaden the spectrum of evaluated function utilization.

Empirical results show Hephaestus-based models (e.g., Hephaestus-8B-IFT) outperform LLaMA-3-8B-based instruction-tuned models, with up to 9.6% relative gains in certain metrics. Furthermore, Hephaestus rivals commercial models such as GPT-3.5-turbo and Claude-3-Haiku, achieving comparable scores on agentic tasks while preserving general language mastery. Continual pre-training with Hephaestus-Forge demonstrably delivers models proficient in both specialized agentic capabilities and broad language generalization.

Benchmark	Capabilities Tested	Hephaestus Relative Performance
AgentBench	Reasoning, adaptability	Superior to open-source, close to commercial
BFCL-v2/v3	Function calling sequences	Outperforms LLaMA-3-8B, rivals GPT-3.5-turbo
NexusRaven/API-Bank	Multi-domain utilization	Consistent top-tier results

5. Generalization to Novel Tasks and Environments

The diversity of the Hephaestus-Forge Corpus fortifies model robustness across new tasks and domains. Agentic reasoning and planning capabilities, trained across varied function call trajectories, are not tightly coupled to specific task templates. Instead, models generalize planning strategies, composing appropriate function sequences for tasks not explicitly present during training.

Continual pre-training is executed in a dual-stage process: initial focus on broad agent knowledge followed by agent-specific refinement. This architectural separation guards fundamental reasoning and planning capacities against erosion by over-specialization, enabling effective transfer to novel environments and domains.

6. Role in Cross-Domain Benchmarks and Future Directions

Hephaestus-Forge provides a foundational resource for efforts across agent-centric LLM research, knowledge graph question answering (e.g., as envisioned in Q²Forge (Taghzouti et al., 19 May 2025)), and combinatorial optimization via graph-embedding approaches (see Forge (Shafi et al., 28 Aug 2025)). A plausible implication is that integrating Hephaestus-Forge with these corpora may lead to improved agentic reasoning and generalization in multi-modal and multi-domain agent architectures.

Future considerations include refining the balance of modalities, expanding environmental feedback scenarios, and developing protocols for cross-system interoperability (e.g., Model Context Protocol, agent-to-agent communication). Efforts directed at automated corpus expansion via modular pipelines—incorporating human and LLM feedback—are anticipated to further enhance reference sets for benchmarking and research.

7. Impact and Significance

Hephaestus-Forge represents a comprehensive solution to the longstanding challenge of agent-oriented LLM pre-training, replacing ad hoc prompting and brittle fine-tuning with a principled, scalable approach. Through robust scaling law justification and empirical benchmarking, it delivers models that execute accurate API calls, plan multi-step sequences, and adapt interactively—all while maintaining generalization across tasks and domains. The corpus sets a new standard for foundation models in agentic and autonomous settings, with broad applicability spanning tool use, knowledge graph QA, and optimization.

The Hephaestus-Forge Corpus is thus integral not only for training agentic LLMs but also as a resource for evaluation, replication, and extension in academic and applied research communities.