---
title: 'BERT4beam: Transformer for Beamforming'
url: https://www.emergentmind.com/topics/bert4beam
type: topic
---

# BERT4beam: Transformer for Beamforming

Searching arXiv for the specified paper and closely related beamforming optimization work.
arxiv_search.query({"search_query":"id:2509.11056","start":0,"max_results":5})
arxiv_search.query({"search_query":"all:\"BERT4beam\"","start":0,"max_results":10})
arxiv_search.query({"search_query":"all:\"beamforming optimization transformer wireless\"","start":0,"max_results":10})
BERT4beam is a framework for generalized beamforming optimization that reformulates downlink MU–MISO beamforming as a token-level sequence learning problem and implements that reformulation with a BERT-based Transformer architecture [2509.11056]. It is presented as a large AI model for wireless communications rather than as a fine-tuned large language model in the usual text-processing sense: the source sequence is channel state information (CSI), tokenized into real-valued vectors, and the target sequence is the corresponding beamforming vectors. The framework contains two variants. The first is a single-task BERT model for one utility at a time; the second is a multi-task model, termed UBERT, that uses a finer-grained tokenization strategy and is designed to generalize directly across utilities, user counts, and antenna configurations [2509.11056].

## 1. Beamforming problem and utility-driven formulation

BERT4beam is developed for a downlink MU–MISO system in which a base station with \(N_{\rm T}\) antennas serves \(K\) single-antenna users. The channel from the base station to user \(k\) is \(\mathbf{h}_k\in\mathbb{C}^{N_{\rm T}}\), the beamforming vector is \(\mathbf{w}_k\in\mathbb{C}^{N_{\rm T}}\), and the received signal is
$$
y_k = \mathbf{h}_k^H\mathbf{w}_k s_k + \sum_{i\neq k}\mathbf{h}_k^H\mathbf{w}_i s_i + n_k,\quad n_k\sim\mathcal{CN}(0,\sigma_k^2).
$$
With \(\mathbb E\{|s_k|^2\}=1\), the achievable rate of user \(k\) is
$$
R_k(\{\mathbf{w}_i\})=\log_2\!\Bigl(1+\frac{|\mathbf{h}_k^H\mathbf{w}_k|^2}{\sum_{i\neq k}|\mathbf{h}_k^H\mathbf{w}_i|^2+\sigma_k^2}\Bigr).
$$
The optimization target is utility-driven beamforming under a total transmit-power constraint:
$$
\max_{\{\mathbf{w}_k\}}\;U(\{\mathbf{w}_k\})\quad\text{s.t.}\;\sum_{k=1}^K\|\mathbf{w}_k\|_2^2\le P_{\max}.
$$
The utility \(U(\cdot)\) may be the sum rate (SR), minimum rate (MR), or energy efficiency (EE), specifically
$$
U=\sum_k R_k,\qquad
U=\min_k R_k,\qquad
U=\sum_k R_k\big/\bigl(\sum_k\|\mathbf{w}_k\|^2+P_{\rm C}\bigr),
$$
respectively [2509.11056].

The central modeling decision is to interpret beamforming as sequence-to-sequence learning. In that viewpoint, CSI is not treated as a raw matrix to be processed by a fixed architecture tied to one system dimension; instead, it is converted into tokens, embedded, and passed through Transformer encoder blocks. This directly targets the paper’s stated goal of adapting and generalizing across diverse tasks defined by system utilities and scales [2509.11056].

## 2. Tokenization of CSI and sequence representation

BERT4beam defines two tokenization strategies. In the single-task BERT formulation, each user’s CSI vector \(\mathbf{h}_k\in\mathbb C^{N_{\rm T}}\) is split into real and imaginary parts and concatenated into one real token,
$$
\mathbf{t}_k=\bigl[\Re(\mathbf{h}_k)^T,\;\Im(\mathbf{h}_k)^T\bigr]^T\in\mathbb R^{2N_{\rm T}}.
$$
The model input is then
$$
\mathbf{T}=[\mathbf{t}_1;\dots;\mathbf{t}_K]\in\mathbb R^{K\times 2N_{\rm T}}.
$$

In the multi-task UBERT formulation, tokenization is element-wise. Each scalar channel coefficient \(\mathbf{h}_k[j]\) becomes its own token,
$$
\mathbf{t}_{k,j}=[\Re(\mathbf{h}_k[j]),\,\Im(\mathbf{h}_k[j])]\in\mathbb R^2,
$$
yielding a \(K\times N_{\rm T}\) tensor of 2-dimensional tokens. The paper states that this finer granularity decouples model input size from \((K,N_{\rm T})\) [2509.11056].

A notable point is that the framework does not use quantization per se. Each token is linearly projected by a fully connected layer into an \(F\)-dimensional embedding space, then normalized and activated. This clarifies a potential misconception: although the framework borrows the BERT paradigm, its “tokens” are numerical CSI-derived vectors rather than text symbols or codebook indices [2509.11056].

## 3. Architecture: single-task BERT and multi-task UBERT

The single-task BERT model begins with an embedding block that maps \(\mathbf{T}\in\mathbb R^{K\times 2N_{\rm T}}\) through an FC layer \((2N_{\rm T}\to F)\), LayerNorm, and ELU to obtain \(\mathbf{T}_{\rm emb}\in\mathbb R^{K\times F}\). The embedded sequence is then processed by \(L\) identical Transformer encoder blocks. Each block contains multi-head self-attention with \(C\) heads and head dimension \(d=F/C\),
$$
\mathbf{O}^{(c)}=\mathrm{Softmax}\bigl(\tfrac{\mathbf{Q}^{(c)}(\mathbf{K}^{(c)})^T}{\sqrt{d}}\bigr)\mathbf{V}^{(c)},
$$
followed by concatenation and projection,
$$
\mathrm{MHA}=\mathrm{Concat}(\mathbf{O}^{(1)},\dots,\mathbf{O}^{(C)})\mathbf{W}_O,
$$
with residual connection and LayerNorm. A position-wise FFN maps \(F\to d'\to F\) with GELU, again with residual connection and LayerNorm. The output layer uses two real FCs to produce complex weights \(\mathbf{W}_{\rm out}\in\mathbb C^{K\times N_{\rm T}}\), and a parameter-free Generalizable Power Adapter (GPA) rescales \(\mathbf{W}_{\rm out}\) to satisfy \(\sum\|\mathbf{w}_k\|^2\le P_{\max}\). The model size used in simulation is \(F=1024\), \(L=12\), \(C=16\) [2509.11056].

UBERT introduces an Antenna Encoding Block (AEB) before the shared Transformer encoder. The element-wise tokens first receive cosine positional encoding and linear embedding to produce an \(N_{\rm T}\times F\) representation. UBERT then applies additive attention across antennas:
$$
\alpha_{ij}=\mathrm{ReLU}(\mathbf{a}^T\mathbf{W}_s\mathbf{t}_i+\mathbf{a}^T\mathbf{W}_t\mathbf{t}_j),
$$
$$
\mathbf{t}_i\leftarrow\sum_j\mathrm{softmax}(\alpha_{i:})\,\mathbf{t}_j.
$$
After summing over \(j=1,\dots,N_{\rm T}\) and applying a linear transform \(\mathbf{W}_{\rm ext}\), UBERT yields one user token in \(\mathbb R^F\) per user, stacks the \(K\) user tokens, and prepends or appends a learnable task token \(\mathbf{t}_{\rm task}\in\mathbb R^F\), producing a \((K+1)\times F\) sequence. The shared Transformer encoder blocks are then applied as in the single-task model. In the output stage, the updated task token is added back to each user token; the resulting features are redistributed to per-antenna tokens and linearly projected, via global average pooling and complex mapping, to \(\mathbf{W}_{\rm Uout}\in\mathbb C^{K\times N_{\rm T}}\) [2509.11056].

| Component | Single-task BERT | UBERT |
|---|---|---|
| CSI tokenization | User-wise token \(\mathbf{t}_k\in\mathbb R^{2N_{\rm T}}\) | Element-wise token \(\mathbf{t}_{k,j}\in\mathbb R^2\) |
| Scale handling | Reconfigure input/output modules | Decouples input size from \((K,N_{\rm T})\) |
| Task handling | One utility at a time | Learnable task token for multi-task learning |

This architectural split is central to the paper’s notion of generalized beamforming optimization. The single-task model is adapted by reconfiguring the input and output modules while retaining the Transformer encoder blocks, whereas UBERT is designed so that the unified backbone and output head remain unchanged across fine-tuning [2509.11056].

## 4. Pre-training and fine-tuning strategies

For the single-task BERT model, supervised pre-training uses a hybrid loss for one utility \(U\):
$$
\mathcal{L}_{\rm pre}=\lambda_1\bigl[1-\tfrac{|\mathbf{W}_{\rm cvx}^H\mathbf{W}_{\rm out}|^2}{\|\mathbf{W}_{\rm cvx}\|^2\|\mathbf{W}_{\rm out}\|^2}\bigr]-\lambda_2\,U(\mathbf{W}_{\rm out}).
$$
Here \(\mathbf{W}_{\rm cvx}\) is the label produced by a convex solver such as SCA. Unsupervised fine-tuning omits solver labels and directly optimizes the utility,
$$
\mathcal{L}_{\rm fin}=-\,U(\mathbf{W}_{\rm out}).
$$
When \((K,N_{\rm T})\) or the utility changes, only the embedding and output layers are reconfigured; the Transformer encoder block weights are retained [2509.11056].

For UBERT, supervised pre-training is explicitly multi-task and spans EE, SR, and MR:
$$
\mathcal{L}_{\rm u\text{-}pre}=\mathcal{L}_{\rm EE}+\mathcal{L}_{\rm SR}+\mathcal{L}_{\rm MR},
$$
where each component is a cosine-similarity-based regression loss to the corresponding \(\mathbf{W}_{\rm cvx}\). Training uses uniform-task sampling in each mini-batch, sampling equally from each utility’s dataset to avoid gradient conflicts and imbalance. Fine-tuning preserves the same architecture and loss as pre-training; no module swapping is needed [2509.11056].

The distinction between the two regimes is methodologically important. In the single-task case, generalization is mediated by a reusable backbone plus reconfigured I/O layers. In UBERT, the finer-grained tokenization, the antenna encoding, and the task token are presented as sufficient to support direct generalization to diverse tasks without changing the backbone or output head. This suggests a progression from task-adaptive transfer to task-unified transfer within the same beamforming optimization family [2509.11056].

## 5. Training setup, complexity, and convergence behavior

The reported datasets are generated from Rayleigh fading channels. Two pre-training settings are used, \((6,12)\) and \((8,16)\), with 80k labeled samples per power budget \(P_{\max}\in\{1,2,3\}\). Fine-tuning uses \((5,11)\), \((7,13)\), and \((9,15)\), with 10k unlabeled samples each. The train/validation/test split is 80/10/10. Optimization uses Adam with batch size 32. Pre-training runs for 100 epochs with initial learning rate \(2\times10^{-4}\) and cosine decay; fine-tuning runs for 10 epochs with learning rate \(2\times10^{-5}\) [2509.11056].

The paper reports that BERT and UBERT converge in \(\mathcal O(10^2)\) epochs during pre-training and \(\mathcal O(10)\) epochs during fine-tuning. The per-pass computational scaling is stated as \(O(K^2F + KN_{\rm T}F)\) for BERT and \(O(KN_{\rm T}^2F+K^2F)\) for UBERT attention blocks [2509.11056].

These details frame BERT4beam as a large AI model in the sense of model scale and transfer behavior rather than in the sense of generic web-scale pre-training. The training process remains tightly coupled to a wireless-system simulator, convex-solver labels, and direct utility optimization, which is consistent with the paper’s emphasis on task-specific pre-training and fine-tuning rather than language-model adaptation [2509.11056].

## 6. Simulation results, ablations, and reported design guidelines

The simulation setup covers MU–MISO systems with \(N_{\rm T}\in\{11,12,13,15,16,17\}\), \(K\in\{5,6,7,8,9\}\), \(P_{\max}\in\{1,2,3\}\) W, and \(\mathrm{SNR}=10\) dB. Baselines are SCA, MLP, CNN, GCN, GAT, and GPT. The reported performance metric is average utility relative to SCA [2509.11056].

The principal numerical findings are as follows. Both BERT and UBERT achieve within 1–3% of SCA on SR and MR, and can exceed SCA on EE because small CVX inexactness can be outperformed. BERT generalizes across \(K\) and \(P_{\max}\) with less than 3% loss without fine-tuning, while UBERT shows stronger power generalization. When fine-tuning to larger or smaller \((K,N_{\rm T})\), both models require only at most 10 epochs to recover more than 90% performance. Cross-task fine-tuning with frozen Transformer encoder blocks still yields more than 95% performance, which the paper interprets as evidence that the multi-head attention layers learn utility-agnostic beamforming structure. In few-shot fine-tuning with fewer than 600 samples, UBERT reaches more than 80% performance on all utilities, whereas BERT remains near 65–70%. Ablation studies identify the cosine positional embedding and task token in UBERT, and the uniform sampling scheme in multi-task training, as necessary components [2509.11056].

The paper closes with explicit design principles for wireless large AI models. These are: tokenize numerical CSI and variables into real-valued vectors rather than raw text; embed relative structure among antennas or users with positional or graph-attention modules; separate a small utility-specific input/output head from a large shared Transformer backbone; combine supervised pre-training on solver outputs with direct utility maximization and then fine-tune unsupervised or multi-task with uniform sampling; and decouple model input/output dimensions from system scale via element-wise tokenization [2509.11056].

Within that framing, BERT4beam is best understood as a Transformer-based beamforming optimizer whose main contribution is not merely replacing one neural architecture with another, but establishing a tokenization, encoder, and transfer-learning scheme intended to preserve performance across utilities and system scales. A plausible implication is that the framework’s most consequential claim lies in this generalization behavior rather than in any single benchmark number.

Source: https://www.emergentmind.com/topics/bert4beam