---
title: 'xLAM-7B-FC-R: Function-Calling 7B Transformer'
url: https://www.emergentmind.com/topics/xlam-7b-fc
type: topic
---

# xLAM-7B-FC-R: Function-Calling 7B Transformer

xLAM-7B-FC-R is a function-calling–specialized 7B-parameter large action model within the xLAM family, developed to advance open-source AI agent capabilities. Built atop the DeepSeek-Coder-7B-instruct backbone, it integrates end-to-end fine-tuning for high-fidelity JSON-style function calls interleaved with self-consistent “thought” traces. The model is distinguished by a unified, execution-verified dataset, architectural optimizations for API tool-use, and empirical performance rivaling significantly larger proprietary baselines [2409.03215].

## 1. Model Architecture

xLAM-7B-FC-R inherits the dense Transformer architecture of DeepSeek-Coder-7B, comprising $L=32$ decoder-only Transformer layers, each with hidden dimension $H=4096$, feed-forward inner size $D_{ff}=16,384$, and $A=16$ self-attention heads. No mixture-of-experts (MoE) are present in FC variants. The multi-head attention at each layer ℓ receives input representations $H^{(\ell-1)} \in \mathbb{R}^{T \times H}$ and computes:
\[
Q = H^{(\ell-1)} W_Q,\quad K = H^{(\ell-1)} W_K,\quad V = H^{(\ell-1)} W_V
\]
\[
\text{head}_i = \text{softmax}(Q_i K_i^\top / \sqrt{d_k}) V_i
\]
\[
\text{MHA}(H^{(\ell-1)}) = [\text{head}_1; \ldots; \text{head}_A] W_O
\]
This is followed by a gated GeLU feed-forward layer,
\[
\text{FFN}(x) = W_2 (\text{GeLU}(W_1 x)) + x
\]
Layer normalization precedes each sublayer, and residual connections are incorporated post-sublayer.

## 2. Training Data and Pipeline

The model employs a unified training corpus composed of three primary sources:
- **Cleaned and Augmented Agent Data:** Tool-usage and web-agent trajectories standardized to a JSON schema (task, tools, format, few-shot setups, stepwise traces).
- **Synthetic Function-Calling Data:** 60,000 high-quality, execution-verified samples generated by APIGen across 3,673 real-world APIs spanning 21 categories.
- **General Instruction Tuning:** Approximately 20–30% sourced from DialogStudio and Data Provenance, filtered for non-commercial licensing and rated by Mixtral-8x22B and DeepSeek-V2 models to remove repetitious or low-quality dialogue.

During fine-tuning for the FC-R variant, each minibatch is composed of 50% execution-verified function-calling data and 50% agent/general instruction data sampled evenly. Optimization is conducted using full-model PyTorch FSDP on NVIDIA H100s, with a batch size of 128 sequences per GPU (effective 1,024 for 8 GPUs), sequence length up to 4,000 tokens, cosine-decay learning rate peaking at $2 \times 10^{-5}$ (100-step warmup), total training exposure of approximately 150 billion tokens over three epochs, weight decay 0.1, and dropout 0.1 in FFN.

Loss function includes both standard cross-entropy,
\[
L_{CE}(\theta) = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, x; \theta)
\]
and pairwise ranking loss for DPO alignment,
\[
L_{DPO} = \sum \log\left(1 + \exp\big(-[s(x,y^+)-s(x,y^-)]\big)\right)
\]
where $s(\cdot)$ denotes pre-softmax sequence score.

## 3. Function-Calling Integration

xLAM-7B-FC-R employs a unified JSON-based schema for function-calling, interleaving "thought" reasoning strings and “tool_calls” in the generated output. Instead of allocating a separate API-head, the model learns during supervised fine-tuning to emit target sequences combining natural language and JSON special tokens. A minimal output template is:
```json
{
  "thought": "...",
  "tool_calls": [
    {"name": "get_weather", "arguments": {"loc": "Palo Alto"}}
  ]
}
```
At inference, standard token-by-token autoregressive decoding proceeds over the merged vocabulary, with no additional special parameters for API function calls. Decoding is operationalized by:
```python
for t in 1...T:
    logits = ModelDecoder(prev_tokens)
    next_token = argmax(softmax(logits/τ))
    append next_token to prev_tokens
```
This method ensures high-fidelity, schema-conforming function call outputs interleaved with structured reasoning.

## 4. Empirical Performance

On the Berkeley Function-Calling Leaderboard v2 (as of 2024-09-03), xLAM-7B-FC-R attains an Overall Accuracy of 80.18%, outpacing many larger open-source models. The breakdown is as follows:

| Evaluation Metric              | xLAM-7B-FC-R (%) |
|-------------------------------|------------------|
| Overall Accuracy               | 80.18            |
| AST-only, simple               | 70.52            |
| AST-only, multiple             | 78.22            |
| AST-only, parallel             | 73.88            |
| AST-only, parallel-multiple    | 68.50            |
| Executable, simple             | 95.21            |
| Executable, multiple           | 90.00            |
| Executable, parallel           | 88.00            |
| Executable, parallel-multiple  | 77.50            |
| Relevance: ignore-irrelevance  | 79.54            |
| Relevance: detect-relevance    | 80.49            |

Comparisons: GPT-4-0125 (function-call prompt) yields 81.78% overall; Gorilla-OpenFunctions-v2 achieves 79.10%; GPT-3.5 Turbo (FC) records 75.41%. Notably, xLAM-7B-FC-R, at 7B parameters, demonstrates competitive or superior performance relative to open-source baselines of substantially greater scale [2409.03215].

## 5. Key Design Optimizations

Critical design decisions contributing to model effectiveness include:
- **Unified JSON Schema:** Standardizing all trajectories into a ["thought","tool_calls"] format with explicit module boundaries ensures persistent API schema adherence.
- **Prompt-format and Paraphrase Augmentation:** Employing stochastic shuffling of tool lists and paraphrased formatting instructions mitigates overfitting to prompt structure, improving generalization.
- **Multi-stage Quality Verification:** Pre-training data undergoes rigorous filtering—rule-based checks (undefined tools/arguments), LLM-driven hallucination detection, and human-in-the-loop trajectory rating—to excise approximately 15% of low-quality samples.
- **APIGen Synthesis:** The synthetic dataset, with all API calls execution-verified, enhances parallel-call accuracy by +4%.
- **Balanced Mini-batch Composition:** Equal sampling of function-calling and agent/general instruction data in minibatches yields substantial gains on both AST and executable function metrics, while maintaining language understanding capacity.

## 6. Context and Significance

xLAM-7B-FC-R exemplifies the impact of a systematic data engineering and training pipeline on function-calling performance for autonomous AI agents. By enforcing schema-unification, incorporating execution-verified synthetic data (50% of fine-tuning), and applying multi-stage quality control, the model achieves accuracy levels near that of state-of-the-art proprietary models within a compact 7B-parameter dense Transformer. This outcome evidences that scalable, unified methodologies can democratize high-fidelity function-calling even at modest parameter counts, advancing open-source alternatives for AI agent systems [2409.03215].

Source: https://www.emergentmind.com/topics/xlam-7b-fc