CloudFormer: Transformer VM Degradation Predictor

• CloudFormer is a dual-branch Transformer model that predicts VM performance degradation by jointly modeling temporal dynamics and system-level metric interactions.
• It leverages 206 system metrics sampled every second to separate workload evolution from interference effects in public cloud settings.
• Empirical evaluations on the CloudPerfTrace dataset show that CloudFormer outperforms baselines, making it a practical tool for interference-aware cloud management.

CloudFormer is an attention-based regression model for predicting virtual-machine performance degradation in multi-tenant public clouds under black-box conditions, where providers observe only host-side system traces and not application internals. It is introduced as a dual-branch Transformer architecture that jointly models temporal dynamics and system-level metric interactions from 206 system metrics sampled at 1-second resolution, and it is evaluated on CloudPerfTrace, a dataset spanning 11 cloud applications across static and dynamic workload scenarios. The prediction target is a normalized performance level $\mathcal{P} \in [0,1]$ defined relative to isolated execution, so the task is degradation-aware scalar prediction rather than classification (Shahbazinia et al., 3 Sep 2025).

1. Problem formulation and target quantity

CloudFormer is situated in the problem of performance degradation prediction for co-located VMs in public-cloud environments. The underlying premise is that virtualization can enforce resource allocation for CPU, memory, and storage, yet cannot guarantee performance isolation. VMs still contend for shared resources such as last-level cache, memory bandwidth, I/O subsystems, and network interfaces, and this contention induces performance interference. Under nominally identical VM configurations, the same application may therefore experience substantially different execution time, throughput, or latency depending on co-location conditions (Shahbazinia et al., 3 Sep 2025).

The paper frames this as a particularly difficult inference problem in public-cloud black-box settings. Providers do not have access to source code, internal runtime state, or application-specific telemetry, so prediction must rely on host-observable system-level traces such as hardware counters and hypervisor statistics. These signals are indirect and noisy. A further complication is that workload variation is not attributable solely to interference from neighboring VMs: intrinsic workload evolution over time also changes application behavior. Distinguishing external interference from internal workload dynamics is therefore central to the task.

The target variable is a normalized performance level $\mathcal{P}$, defined as degradation relative to isolated execution. For an execution $i$, the observed trace is represented as

$x_i \in \mathbb{R}^{F \times T_i},$

where $F$ is the number of system features and $T_i$ is the execution duration in time steps. The performance level is defined as

$$\mathcal{P} = \frac{PM_{\text{ideal}}}{PM_{\text{actual}}}, \qquad 0 < \mathcal{P} \le 1.$$

Here, $PM_{\text{ideal}}$ denotes the performance metric without interference and $PM_{\text{actual}}$ the observed metric under the current execution condition. A value of $\mathcal{P}=1$ denotes no degradation, while smaller values indicate stronger degradation. Depending on the application, $\mathcal{P}$0 may be operations/s, requests/s, throughput, execution time (s), or latency (s or ms). Accordingly, CloudFormer predicts a scalar degradation-aware score rather than an application-specific raw metric.

This formulation is motivated by cloud-management use cases identified in the paper, including VM scheduling, server consolidation, and resource provisioning. These functions require not merely assigned resource quantities, but predictions of actual performance loss under co-location. A plausible implication is that the model is intended as a decision-support component for interference-aware control loops rather than as a generic multivariate forecasting model.

2. Architectural design

CloudFormer is a dual-branch Transformer-based regression model whose two branches specialize in complementary structures: temporal evolution and cross-metric interaction. The model input is a multivariate time series

$\mathcal{P}$1

where $\mathcal{P}$2 is the number of system metrics and $\mathcal{P}$3 the number of time steps; in experiments, $\mathcal{P}$4. The output is a scalar estimate $\mathcal{P}$5 (Shahbazinia et al., 3 Sep 2025).

The temporal branch models how host-visible behavior evolves over time. Each time step is projected through a dense layer with ReLU activation into an embedding space of dimension $\mathcal{P}$6. A learnable class token is prepended, sinusoidal positional encoding is added, and the resulting sequence of shape $\mathcal{P}$7 is processed by $\mathcal{P}$8 Transformer encoder blocks. Each block contains masked multi-head self-attention, dropout, residual connections, layer normalization, and a position-wise feedforward network. The masking is used for variable-length executions with padding masks rather than for causal autoregression. The final temporal representation is the class-token output.

The positional encoding is given as

$\mathcal{P}$9

The system branch models system-level interactions among metrics. It first applies mean pooling over time, conceptually reducing each metric time series to

$i$0

These pooled features are then projected to dimension $i$1 using a 1D convolution. A learnable class token is prepended, but no positional encoding is used, because the metrics are treated as unordered. This sequence is processed by $i$2 Transformer encoder blocks with unmasked multi-head self-attention. The system summary is again taken from the class-token representation.

In both branches, self-attention is defined as

$i$3

with 4 attention heads and head size 16, consistent with an embedding width of 64. The feedforward subnetwork dimension is 256.

The two class-token outputs are concatenated:

$i$4

since $i$5. The fused vector is passed through an MLP head comprising dense layers, layer normalization, dropout, Swish activation, and a final scalar output layer. The implementation section reports a dropout rate of 0.4 and a sigmoid final activation, yielding

$i$6

The paper also notes an internal discrepancy: an earlier architectural description refers to a softmax for scalar prediction, whereas the implementation section specifies sigmoid. For a scalar output bounded in $i$7, sigmoid is presented as the coherent interpretation, but the discrepancy remains part of the paper’s technical record. The model differs from a standard Transformer through its dual-branch structure, branch-specific handling of temporal order, mean pooling plus convolution in the system pathway, and scalar regression objective.

3. Input signals and CloudPerfTrace

CloudFormer is trained on CloudPerfTrace, a public dataset collected over about two months from a dedicated cloud testbed, comprising approximately 317 days of recorded traces across all application runs (Shahbazinia et al., 3 Sep 2025). The test server consists of two Intel Xeon Gold 6240 processors, each with 36 virtual CPUs, 384 GB ECC memory, and 6 TB NVMe SSD. The software stack includes Ubuntu 20.04, libvirt, QEMU, Open vSwitch, and KVM. Each VM is configured with 4 vCPUs and 8 GB memory. Each CPU socket can host up to 9 VMs concurrently, so a server VM may suffer interference from the other 8 VMs on the same socket. To emulate client-server applications, the second socket is dedicated to 9 client VMs, while server VMs run on the first socket.

The model uses 206 system-level metrics assembled from three sources: 53 VM-level metrics via libvirt, 38 low-level hardware metrics via Linux perf, and 12 Intel Top-Down Analysis metrics. Representative examples include CPU utilization, unused memory, cycles, retired instructions, LLC misses, frontend bound, and backend bound. To capture inter-VM effects, the metrics of neighboring VMs running concurrently with the target VM are aggregated by averaging. This neighborhood aggregation contributes to the final set of 206 distinct metrics. The paper does not enumerate all metrics individually.

The dataset’s temporal resolution is a central feature. Metrics are sampled at 1-second resolution, which is described as substantially finer than many prior cloud traces with minute- or hour-level granularity. Each execution therefore forms a multivariate sequence of variable duration. This granularity is intended to preserve transient interference effects, including sudden bursts and short-lived contention patterns that would be obscured by coarse sampling.

CloudPerfTrace includes both static and dynamic workload scenarios. In static scenarios, workload level does not vary over time. Dynamic scenarios include monotonic variation, periodic variation with three frequencies, and random variation intended to emulate real-world request patterns. The paper’s argument is that static traces are easier and more controlled, whereas dynamic traces more closely reflect public-cloud conditions; this motivates the temporal branch of CloudFormer.

The dataset contains 11 cloud applications: Data Serving, Redis, Web Search, Graph Analytics, Data Analytics, MLPerf, HBase, Alluxio, Minio, TPCC, and Flink. Their performance metrics vary by application: Operations/s for Data Serving and Web Search, Requests/s for Redis and MLPerf, execution time (s) for Graph Analytics and Data Analytics, latency (s) for HBase, throughput for Alluxio and Minio, latency (ms) for TPCC, and Operations/ms for Flink. The reported trace counts are 4209 for Data Serving, 4862 for Redis, 2849 for Web Search, 3195 for Graph Analytics, 3085 for Data Analytics, 3355 for MLPerf, 4759 for HBase, 3085 for Alluxio, 3139 for Minio, 2985 for TPCC, and 3181 for Flink.

A plausible implication of this dataset design is that CloudFormer is evaluated under a broader combination of workload dynamics and metric diversity than would be available from coarse-grained cloud telemetry corpora, though the paper confines its claims to improvements in temporal granularity, metric diversity, workload diversity, and the inclusion of dynamic scenarios.

4. Learning objective, training protocol, and baselines

CloudFormer is trained with logarithmic cosh loss, stated as log-cosh in the paper. The loss is described functionally as

$i$8

or equivalently averaged over samples. The paper characterizes this loss as behaving similarly to MSE for small errors and more like MAE for larger errors, thereby smoothing the influence of outliers (Shahbazinia et al., 3 Sep 2025).

The evaluation protocol is centered on cross-application generalization. The dataset is partitioned by application rather than by random samples: 7 applications are used for training and 4 entirely unseen applications for testing. Within this scheme, the training set includes 4 static-only applications and 3 applications with static and dynamic scenarios, while the test set includes 2 static-only applications and 2 applications with mixed static/dynamic scenarios. To reduce sensitivity to a particular application split, the authors randomly choose applications according to this composition and repeat the process six times. Results are therefore reported as mean $i$9 standard deviation across six random seeds or splits. In the paper’s operational sense, an “unknown workload” is an application unseen during training, predicted solely from black-box system metrics without access to application identity or internals.

Optimization uses Adam with an initial learning rate of $x_i \in \mathbb{R}^{F \times T_i},$0 and a scheduler combining linear warm-up with cosine decay. Regularization includes dropout at 0.4 and normalization layers in the architecture. The data are normalized before training, but the paper does not specify the exact scaling method, missing-value handling, sequence padding or truncation policy, batch size, epoch count, early stopping criterion, training hardware, or runtime.

Five baselines are used for comparison: Linear Regression (LR), Gamma Regression with inverse power link (GLR), Decision Tree (DT), Random Forest (RF), and LSTM. These are framed as covering both feature-domain methods and a time-domain deep-learning method. For DT and RF, hyperparameters are tuned using Bayesian search with cross-validation, though final selected values are not reported.

Evaluation metrics are MSE and MAE, and the results analysis also includes an error-band distribution over thresholds $x_i \in \mathbb{R}^{F \times T_i},$1, $x_i \in \mathbb{R}^{F \times T_i},$2, $x_i \in \mathbb{R}^{F \times T_i},$3, $x_i \in \mathbb{R}^{F \times T_i},$4, $x_i \in \mathbb{R}^{F \times T_i},$5, and $x_i \in \mathbb{R}^{F \times T_i},$6. The experiments cover overall comparison against baselines, static-versus-dynamic scenario analysis, generalization to unseen workloads, and an ablation study with CF-Temporal, CF-System, and the full fused model.

5. Empirical results and ablation findings

The main quantitative comparison is summarized below (Shahbazinia et al., 3 Sep 2025).

Method	MSE	MAE
LR	$x_i \in \mathbb{R}^{F \times T_i},$7	$x_i \in \mathbb{R}^{F \times T_i},$8
GLR	$x_i \in \mathbb{R}^{F \times T_i},$9	$F$0
DT	$F$1	$F$2
RF	$F$3	$F$4
LSTM	$F$5	$F$6
CF	$F$7	$F$8

CloudFormer attains an MAE of 7.8% and is reported to outperform existing methods by at least 28%. The strongest baseline is Random Forest, with MAE $F$9, so the reported margin is aligned with the paper’s discussion. The baseline comparison is interpreted as follows in the paper: LR and GLR perform poorly, indicating that the degradation mapping is strongly nonlinear; DT improves substantially; RF is the strongest classical baseline; and LSTM does not match RF and performs markedly worse than CloudFormer. The paper uses this pattern to argue that temporal modeling alone is insufficient and that cross-metric interaction modeling is also necessary.

The error-band distribution analysis shows that CloudFormer has the largest share of predictions in the lowest error bands, specifically $T_i$0 and $T_i$1, and the smallest share above $T_i$2. Its advantage is therefore described not only in terms of average error but also in terms of consistency.

The ablation study compares the two branches individually against the full model:

Method	MSE	MAE
CF-Temporal	$T_i$3	$T_i$4
CF-System	$T_i$5	$T_i$6
CF	$T_i$7	$T_i$8

This ablation supports three claims made in the paper: both branches are individually strong, the temporal branch is slightly better than the system branch when used alone, and the fused dual-branch model is best. The stated conclusion is that temporal information and system-level interaction modeling are complementary. The paper does not provide direct interpretability artifacts such as attention visualizations, saliency maps, or ranked feature-importance plots, although it argues conceptually that the temporal branch captures transient interference effects and the system branch captures relations such as those between LLC occupation and cache misses.

6. Generalization behavior, practical role, and limitations

The evaluation explicitly targets unseen workloads by reserving four applications that are entirely absent from training. Across six random train/test splits, the paper reports that CloudFormer is the most consistent method overall on per-application heatmaps. It also notes isolated cases in which another baseline outperforms CloudFormer on a single static application in a particular seed, but characterizes these as unstable and accompanied by worse errors elsewhere (Shahbazinia et al., 3 Sep 2025).

Static scenarios are reported to be generally easier to predict than dynamic ones. The paper attributes the increased difficulty of dynamic settings to changing workload intensity, changing contention, and stronger transient behavior. Even so, CloudFormer is described as the strongest model across both static and dynamic cases. Some static workloads, notably TPCC and Flink, can still exhibit higher error due to distinctive or irregular behavior.

The practical significance assigned to CloudFormer lies in interference-aware cloud management. The paper identifies or motivates VM scheduling and placement, VM consolidation, resource provisioning or autoscaling, SLA protection, and capacity planning as use cases. A key deployment property is that the model uses only host-observable black-box metrics and does not require source code, application-specific instrumentation, in-VM internals, or prior manual categorization of applications into classes such as CPU-bound or network-bound. Its deployment assumptions are access to hypervisor and host-side monitoring, 1-second sampled system metrics, and aggregated neighbor-VM statistics. However, the paper does not measure runtime inference overhead, memory footprint, or deployment latency.

Several limitations are acknowledged or implied. CloudPerfTrace is collected on a single testbed server platform with a specific hardware and virtualization stack, so cross-platform generalization is motivated rather than directly evaluated. The model depends on a large set of 206 host-level metrics, which may limit applicability where observability is restricted. The experiments focus on co-located VMs within a single server or socket setting. The model is trained offline rather than with online adaptation. The authors identify three future directions explicitly: extension to multi-node settings, online adaptation under changing workloads, and incorporation of energy-efficiency objectives alongside performance prediction.

Taken together, these elements position CloudFormer as a black-box VM performance degradation predictor whose central technical claim is that accurate prediction in public clouds requires explicit modeling of both how host-visible metrics evolve over time and how those metrics interact at the system level. The reported gains under cross-application evaluation suggest that this dual-structure inductive bias is effective for previously unseen workloads, while the paper’s own scope conditions indicate that broader deployment claims depend on future validation beyond the single-node, single-platform regime.