---
title: 'Falcon-180B: 180B-Param Transformer LLM'
url: https://www.emergentmind.com/topics/falcon-180b
type: topic
---

# Falcon-180B: 180B-Param Transformer LLM

Falcon-180B is a 180 billion-parameter causal decoder-only Transformer language model constituting the largest member of the Falcon series of open large language models (LLMs). Trained on approximately 3.5 trillion tokens of predominantly web-sourced text, Falcon-180B represents the largest openly documented pretraining run to date and nears the performance of leading closed models such as GPT-4 and PaLM-2-Large while being available under a permissive “Responsible-Use” license. The model substantially outperforms prior open LLMs (e.g., Chinchilla, Gopher, LLaMA-2) and demonstrates competitive results on a broad suite of language understanding, commonsense reasoning, and coding benchmarks [2311.16867].

## 1. Model Architecture

Falcon-180B implements a causal decoder-only Transformer architecture with a total parameter count $P = 180 \times 10^9$, distributed across $L = 80$ layers (Transformer blocks) and a hidden size $d_{\text{model}} = 14\,848$. The model utilizes a multigroup attention mechanism (an extension of multiquery attention) featuring $n_{q} = 232$ query heads and $n_{kv} = 8$ key/value heads, serving to optimize memory efficiency and performance at scale. Rotary positional embeddings (RoPE) are adopted for encoding position information.

The feed-forward sublayers consist of a two-layer MLP with GeLU activation, no SwiGLU variant, and biases are removed from all linear layers. Attention and MLP blocks are parallelized to minimize synchronization overhead within tensor-parallel execution. Notably, the memory footprint for the key-value cache scales as $O(\log P)$, improving over prior $O(\sqrt{P} \cdot \log n)$ approaches, and empirical depth scaling for optimal model performance is observed to be approximately proportional to $\log P$.

## 2. Pretraining Dataset and Data Refinement

Falcon-180B is pretrained on $T \approx 3.5 \times 10^{12}$ tokens drawn from a high-quality, diverse corpus. The primary data source is the RefinedWeb English web crawl, accounting for 76% of tokens (approximately 2.7T tokens), with additional coverage from RefinedWeb in European languages (8%), Project Gutenberg books (6%), conversation data such as Reddit and StackOverflow (5%), permissively licensed code from GitHub (3%), and technical domains including arXiv, PubMed, USPTO, and Wikipedia (2%).

The RefinedWeb macrodata refinement pipeline consists of three stages: (1) URL-based blocklisting and language identification (resulting in 48% English content); (2) heuristic filtering on text quality, length, and symbol ratios, which halves the dataset; and (3) two-stage deduplication (using MinHash followed by suffix-array methods) that further halves the corpora, retaining only 12% of the original material. Microdata mixes in curated subsets of the Pile and Reddit data, with conversation trees encoded via custom attention masks to cover all branches without repetition.

A 600B-token extract of RefinedWeb is released for reproducibility and community research.

## 3. Training Methodology and Compute Infrastructure

Training was conducted on a GPU cluster of up to 4,096 NVIDIA A100 40GB GPUs hosted on AWS, interconnected with 50 Gbps bandwidth. Parallelism is implemented in a 3D configuration: tensor parallelism (TP=8), pipeline parallelism (PP=8), and data parallelism (DP=64), augmented with ZeRO-1 optimizer sharding. The model processes input sequences of length 2,048 tokens.

The total compute budget comprises approximately 43,500 PF-days ($\approx 3 \times 10^{23}$ FLOPs), computed as 
$$\mathrm{FLOPs} \simeq 2 \cdot P \cdot T \cdot L \cdot O(1).$$
Batch and learning rate schedules involve micro-batches of 2,048 tokens per GPU, warm-up over 4B tokens to a peak learning rate ($1.25 \times 10^{-4}$), batch size ramp-up to 100B tokens for data-parallel efficiency, cosine decay schedule ($\times 10$ reduction), weight decay of 0.1, z-loss of $1 \times 10^{-4}$, and gradient clipping at 0.4.

Memory optimizations include the use of FlashAttention (Triton) for exact $O(N \cdot d)$ attention, selective (“monolayer”) recomputation of layer norm and activations (halves activation memory), and bfloat16 training (with no stochastic rounding).

## 4. Distributed Training Tooling and Software Engineering

The custom distributed codebase “Gigatron” is based on PyTorch with 3D parallelism and ZeRO-1 optimizer integration. Communication optimizations encompass tensor-parallel splitting for attention/MLP columns and rows with minimal all-reduce operations, pipeline parallelism managed via 1F1B scheduling with scatter/gather improvements, and a workflow where gradient all-reduce is followed by reduce_scatter, the optimizer step, and all_gather for parameters.

Kernels for FlashAttention and fused RoPE/LayerNorm are implemented with Triton, and run management supports topology-agnostic checkpoints, automated node health testing, and bandwidth-based placement using Gromov-Wasserstein optimal transport. AdamW is employed as the optimizer, using bfloat16 for weights and gradients and fp32 for moments.

## 5. Empirical Performance and Evaluation

Falcon-180B achieves competitive performance on a range of established NLP benchmarks. On PaLM-style 1-shot GPT-3 tasks, it attains 77.1% average accuracy, nearly matching PaLM-2-Large (77.5%, or 99.5% of their score). On GPT-4 reported few-shot tasks, the model scores 89.0 on HellaSwag (cf. GPT-3.5: 85.5; GPT-4: 95.3), 87.1 on Winogrande (cf. 81.6/87.5), 87.8 on ARC-Challenge (cf. 85.2/96.3), and 70.6 on 5-shot MMLU (cf. 70.0/86.5).

Falcon-180B outperforms Chinchilla, Gopher, and LLaMA-2 on commonsense and QA (e.g., HellaSwag 89.0, ARC-Ch 63.7), and matches leading models in code generation (HumanEval pass@1: 35.4%, comparable to Inflection-1 and PaLM-Coder at 35.9%). On the EleutherAI Harness aggregate (HellaSwag, LAMBADA, Winogrande, PIQA, ARC, OpenBookQA), Falcon models lead within each size class.

## 6. Licensing, Dataset Release, and Open-Science Commitment

Falcon-180B is distributed under a “Responsible-Use” license, which carries downstream restrictions, while Falcon-7B and Falcon-40B are released under Apache-2.0. The 600B-token extract of RefinedWeb is available under a permissive license. A principal aim is to democratize access to frontier LLM research, foster reproducibility, and enable community-driven model improvements.

## 7. Key Formulas and Scaling Properties

Critical formulas employed in Falcon-180B’s system design include the total parameter count:
$$
P = L \cdot (4 d_{\mathrm{model}}^2 + \ldots ) \simeq 180 \times 10^9
$$
and the total compute (FLOPs):
$$
\mathrm{FLOPs} \simeq 2 \cdot P \cdot T \cdot (d_{\mathrm{model}})
$$
Memory per parameter for AdamW and moment storage is calculated as $m = 2~(\mathrm{bf16}) + 4 \times 3~(\mathrm{fp32}) = 14$ bytes, with sharding by DP yielding $4 + 16/\mathrm{DP}$ bytes per parameter.

These scaling and efficiency properties drive Falcon-180B’s ability to provide high-resource LLM capabilities with reduced pretraining and inference cost relative to comparable proprietary models [2311.16867].

Source: https://www.emergentmind.com/topics/falcon-180b