AIM-Bench: A Family of Benchmarking Frameworks

• AIM-Bench is a family of benchmarking frameworks that span synthetic DNN workload synthesis, production data-center evaluation, affective image manipulation, and agentic inventory management.
• Each variant employs tailored methodologies—from clustering and genetic algorithms to hierarchical filtering and POMGs—to capture task-specific characteristics and performance metrics.
• The evaluation protocols integrate multiple metrics such as MAC matching, aesthetic fidelity, and decision bias analysis, demonstrating a shift from static benchmark suites to dynamic, application-specific workload synthesis.

Searching arXiv for papers titled or referring to "AIM-Bench" / "AI Matrix" to ground the article and disambiguate the term. I’ll look up the relevant arXiv records for “AIM-Bench” and “AI Matrix”. The available arXiv records show that the label “AIM-Bench” and the closely related “AI Matrix” name have been used for several unrelated benchmark programs spanning synthetic DNN workload modeling, production data-center benchmarking, affective image manipulation, and agentic inventory control. In that sense, AIM-Bench is not a single benchmark lineage but an overloaded designation attached to distinct evaluation frameworks with different task definitions, data-construction procedures, and metrics (Wei et al., 2018, Zhang et al., 2019, Chen et al., 12 Apr 2026, Zhao et al., 15 Aug 2025).

1. Name usage and disambiguation

Two early arXiv records use AI Matrix for deep-learning benchmarking. "AI Matrix - Synthetic Benchmarks for DNN" defines a synthetic benchmarks framework that generates one or a few benchmarks matching the statistical workload characteristics of a combination of applications of interests (Wei et al., 2018). "AI Matrix: A Deep Learning Benchmark for Alibaba Data Centers" defines a benchmark suite of real or “production-close” DL models reflecting Alibaba’s major services and covering computer vision, recommendation, and language processing (Zhang et al., 2019).

Two later records use AIM-Bench directly. "AIM-Bench: Benchmarking and Improving Affective Image Manipulation via Fine-Grained Hierarchical Control" introduces the first benchmark designed for affective image manipulation and couples it with the AIM-40k instruction-tuning dataset (Chen et al., 12 Apr 2026). "AIM-Bench: Evaluating Decision-making Biases of Agentic LLM as Inventory Manager" introduces a benchmark for LLM agents in uncertain supply chain management scenarios through inventory replenishment experiments (Zhao et al., 15 Aug 2025).

Name	Domain	Defining focus
AI Matrix	DNN hardware benchmarking	Synthetic CNNs matching profiled workloads
AI Matrix	Alibaba data centers	Real or “production-close” DL workloads
AIM-Bench	Affective image manipulation	Fine-grained hierarchical control and evaluation
AIM-Bench	Inventory-manager LLM agents	Decision-making biases under uncertainty

This suggests that any technical discussion of AIM-Bench requires immediate domain qualification. Without that qualification, the term is ambiguous.

2. AI Matrix as synthetic benchmarking for DNN workloads

In "AI Matrix - Synthetic Benchmarks for DNN," the benchmark is generated from profiled workload characteristics data rather than from a fixed suite of open-sourced applications (Wei et al., 2018). A lightweight “application monitoring” backend hooks into each DNN’s inference or training graph and logs, for every layer-or-operation, the parameter tuple $H$, $W$, $C_{in}$, $C_{out}$, $k$, and $s$. From these parameters, the framework computes the per-layer number of multiply–accumulate operations,

$$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$

and launched GPU warps, $WP$, measured directly by the GPU profiler. Each convolutional layer is represented by a feature vector in $\mathbb{R}^6$,

$x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$

and the monitored workload is summarized through empirical distributions of the coordinates of $W$0, together with

$W$1

Synthetic-benchmark generation proceeds in three stages. First, the convolutional layers are clustered into $W$2 groups using a standard k-means or nearest-neighbor clustering objective,

$W$3

After convergence, each group $W$4 has centroid

$W$5

and count $W$6. Second, for each group, a genetic algorithm such as NSGA-II evolves candidate sub-networks so that synthetic MACs and synthetic warp counts match the real group totals. The paper gives the single-objective fitness

$W$7

with $W$8, and termination when $W$9 is below a small threshold of approximately $C_{in}$0. Third, the synthetic nodes are assembled into a single CNN by ordering groups in descending order of $C_{in}$1, sorting nodes within a group by decreasing filter size, inserting pooling when downsampling is needed, following each convolution with BN + ReLU, optionally inserting concat, and appending a single FC layer + Softmax.

By construction, the synthetic network inherits $C_{in}$2 and $C_{in}$3 from centroid values rounded to integers, sets $C_{in}$4, uses $C_{in}$5, sets $C_{in}$6, and treats $C_{in}$7 as the genetic-search decision variable. The resulting AI Matrix network is described as a compact synthetic CNN whose per-group parameter statistics and global $C_{in}$8 totals are within at most $C_{in}$9 of the real mix.

The evaluation protocol measures total inference time, throughput, and GPU utilization metrics including SM occupancy, L2 bandwidth, and DRAM bandwidth. The paper reports two experiments: a mix of AlexNet + VGG16 + GoogleNet, totaling approximately $C_{out}$0 convolution layers, and a $C_{out}$1-layer industrial-platform mix from Alibaba. In both cases, the synthetic benchmark uses only approximately $C_{out}$2 groups, or about $C_{out}$3–$C_{out}$4 convolution nodes, runs in a few seconds rather than minutes or hours, matches per-group MACs and warps within $C_{out}$5–$C_{out}$6, and yields inference-time and hardware-utilization curves that are statistically indistinguishable from the real mix. One of its explicit design claims is adaptability: newly emerging DNNs can be incorporated by re-running profiling, recomputing centroids, and feeding the updated statistics back into the GA and network-assembly pipeline, with an optional exponential centroid update

$C_{out}$7

The methodological significance of this version of AIM-Bench lies in its shift from benchmark-suite enumeration to statistical workload synthesis. A plausible implication is that it targets hardware selection and performance prediction in settings where proprietary workloads dominate and direct benchmark release is impractical.

3. AI Matrix as a production benchmark for Alibaba data centers

"AI Matrix: A Deep Learning Benchmark for Alibaba Data Centers" takes the opposite approach: instead of synthesizing a compact representative workload, it defines a suite of real or production-close workloads chosen to reflect Alibaba’s deployment mix (Zhang et al., 2019). The paper states that none of the public offerings covers the full spectrum of DL models and data workflows that together account for over $C_{out}$8 of Alibaba’s GPU usage, and positions AI Matrix as a benchmark suite for understanding the computational needs of those applications.

The suite contains twenty workloads in three categories. The computer-vision group includes image classification with GoogLeNet, ResNet50, and DenseNet121; object detection with SSD, DSSD, Faster R-CNN, Mask R-CNN, and Cascaded Pyramid Network; and text recognition with SegLink and CRNN. The recommendation group includes DIN, DIEN, Wide & Deep, GCN, and NCF. The language-processing group includes Google NMT and BERT. The workloads are mapped to Alibaba services such as smart-city traffic analysis, human-pose estimation on Taobao, click-through-rate prediction in ad-ranking pipelines, high-order behavior modeling in Taobao feeds, and translation and question-answer features across Alibaba apps.

The paper characterizes each model using FLOPs per batch, DRAM-to-L2 data transfer, arithmetic intensity, time per batch, and achieved FLOPs/s. The principal formulas are

$C_{out}$9

the roofline prediction

$k$0

and FLOPs utilization

$k$1

The public setup uses a single NVIDIA V100 GPU with FP32 peak $k$2 TFLOPS and $k$3 GB/s DRAM bandwidth, without TensorCores enabled for baseline, and employs TensorFlow or Caffe according to in-house usage.

An excerpted overview table in the paper shows substantial variation across workloads. GoogLeNet at batch size $k$4 is listed with $k$5 FLOPs per batch, $k$6 bytes of memory read, arithmetic intensity $k$7, time per batch $k$8 s, and achieved throughput $k$9 FLOPs/s. ResNet50 at batch size $s$0 is listed with $s$1 FLOPs per batch and $s$2 achieved FLOPs/s. DenseNet121 at the same batch size shows arithmetic intensity $s$3 and $s$4 achieved FLOPs/s. On the recommendation side, DIN at batch size $s$5 reaches $s$6 FLOPs/s with arithmetic intensity $s$7. In language processing, NMT at batch size $s$8 is reported at $s$9 FLOPs/s, whereas BERT at batch size $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$0 reaches $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$1 FLOPs/s with arithmetic intensity $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$2.

The reported bottleneck analysis distinguishes low-AI workloads such as Wide & Deep and DIN, which are memory-bandwidth bound, from high-AI workloads such as CPN and BERT, which still achieve well under $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$3 of peak throughput, indicating suboptimal kernel fusion, launch overhead, or memory stalls. The recommendations therefore span both hardware and software: higher sustained memory bandwidth for low-AI workloads, low-precision and tensor-core-style support for high-AI CV and NLP models, optimized embedding lookup and fully connected kernels for sparse recommendation workloads, and specialized kernels for RNN and transformer blocks.

Relative to the synthetic AI Matrix, this Alibaba variant is a production-representative benchmark suite rather than a statistical surrogate. The contrast between the two is central: one compresses a workload mix into a small synthetic CNN, while the other exposes the heterogeneity of a large industrial DL portfolio.

4. AIM-Bench for affective image manipulation

"AIM-Bench: Benchmarking and Improving Affective Image Manipulation via Fine-Grained Hierarchical Control" defines AIM-Bench in a vision-editing setting, where Affective Image Manipulation (AIM) aims to evoke specific emotions through targeted editing (Chen et al., 12 Apr 2026). The benchmark is built on a dual-path affective modeling scheme. The high-level semantic path uses Mikels’ eight-category taxonomy—Amusement, Awe, Contentment, Excitement, Disgust, Fear, Sadness, and Anger—to define discrete editing goals. The fine-grained attribute path embeds those categories in a continuous Valence–Arousal–Dominance space using an auxiliary dataset $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$4 of $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$5 real $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$6 pairs constructed with GPT-4o. For a source image $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$7 with original label $$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$8, the target is sampled as

$$MAC_i = H_i \cdot W_i \cdot C_{in,i} \cdot C_{out,i} \cdot k_i^2,$$9

and evaluation in the continuous space uses

$WP$0

The benchmark curation process is explicitly hierarchical and human in the loop. It begins from EmoSet-118k, extracts CLIP-ViT-L/14 features, performs MiniBatchKMeans within each Mikels category, chooses the number of clusters by maximizing the silhouette score, and retains images only when three annotators all agree. This yields $WP$1 representative source images. Editing then proceeds through target generation, instruction generation, and candidate generation. The five editing types are Attribute Change, Action Change, Replace Object, Delete Object, and Add Object. For each source image, Gemini-2.5-Pro produces eight distinct natural-language instructions aligned to the target emotion and VAD coordinate, and Gemini-2.5-flash-image generates eight candidate edited images.

Filtering occurs in three stages. Stage 1 uses CLIP-T score via LongCLIP, an aesthetic predictor $WP$2, and VIEScore

$WP$3

which is fused into

$WP$4

The top five by $WP$5 are kept. Stage 2 performs a discrete affective check with Gemini-2.5-Pro and then ranks the correctly classified candidates by $WP$6, discarding the two with largest deviation and keeping the top three. Stage 3 uses three trained experts to select the best candidate according to affective alignment, visual realism, semantic coherence, and naturalness; failed cases are flagged for regeneration. The result is $WP$7 final tuples covering all eight emotions and five editing types.

The evaluation suite is composite and three-axis. Semantic Fidelity uses CLIP-T, VIEScore-SC, and Ground-Truth Consistency defined as

$WP$8

where $WP$9 is semantic match and $\mathbb{R}^6$0 is visual similarity. Aesthetic Quality uses the improved aesthetic predictor and VIEScore-PQ. Emotional Expressiveness uses discrete Accuracy and F1-score and continuous mean $\mathbb{R}^6$1. Each metric is normalized to $\mathbb{R}^6$2 using empirical bounds, and the normalized metrics are averaged into a single Overall AIM score.

The benchmark is used to evaluate 13 state-of-the-art instruction-based editing models. A principal empirical claim is a prevalent positivity bias: nearly every model edits positive targets more accurately and with smaller $\mathbb{R}^6$3 than negative targets. The paper gives Qwen-Image-Edit-2509 as an example, with $\mathbb{R}^6$4 accuracy on positive emotions and $\mathbb{R}^6$5 on negatives. To address this, the work introduces the inverse repainting data engine and the balanced instruction-tuning dataset AIM-40k. For each image $\mathbb{R}^6$6 EmoSet, a caption $\mathbb{R}^6$7 is generated by Qwen3-VL-8B, a repainted target $\mathbb{R}^6$8-Image-Turbo$\mathbb{R}^6$9 is created, and CLIP-I$x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$0 is enforced. A prompt from DeepSeek-V3.2 induces an emotion shift, FLUX.1-Kontext-dev synthesizes the input image, Gemini-3.0-Flash.analyze generates the instruction, and the triplet $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$1 is added to AIM-40k. The resulting dataset contains $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$2 balanced triplets, or $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$3k per emotion category. Fine-tuning Qwen-Image-Edit-2509 on AIM-40k for one epoch with LoRA rank $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$4 increases the Overall score from $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$5 to $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$6, a $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$7 relative improvement, while negative-emotion accuracy rises from $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$8 to $x_i = [H_i, W_i, C_{in,i}, C_{out,i}, k_i, s_i]^T,$9.

This version of AIM-Bench is therefore both a benchmark and a data-centric intervention. Its significance lies not only in evaluating emotion-aware editing, but in operationalizing failure analysis—especially positivity bias—into a benchmark-linked dataset construction pipeline.

5. AIM-Bench for agentic LLM inventory management

"AIM-Bench: Evaluating Decision-making Biases of Agentic LLM as Inventory Manager" defines AIM-Bench in operations management, where LLM agents make replenishment decisions under uncertainty (Zhao et al., 15 Aug 2025). The benchmark is motivated by the absence of a unified, multi-environment framework that spans single-period to multi-echelon settings, measures classical human decision biases such as anchoring, framing, and bullwhip, and evaluates both outcome and process metrics.

All environments are cast as a Partially Observable Markov Game

$W$00

The five textual supply-chain environments are the Newsvendor Problem, Multi-Period Replenishment, Beer Game, Two-Level Warehouse Network, and Supply Chain Network. In the Newsvendor Problem, with demand $W$01, purchase cost $W$02, revenue $W$03, unit underage cost $W$04, and overage cost $W$05, the optimal order quantity is

$W$06

with critical ratio

$W$07

and payoff

$W$08

The benchmark uses a linear anchoring model

$W$09

where higher $W$10 indicates stronger pull to the mean $W$11, and a demand-chasing correlation $W$12.

The Multi-Period Replenishment setting introduces stochastic demand, stochastic vendor lead time, and review dates. Inventory evolves as

$W$13

with period cost

$W$14

and ex-post optimal review order

$W$15

The process metric is

$W$16

The Beer Game uses four echelons and fixed lead time, with inventory update

$W$17

total cost

$W$18

and Bullwhip Effect Index

$W$19

with $W$20 indicating variance amplification.

The evaluation metrics divide into decision bias measures, outcome metrics, and process metrics. Decision bias measures include mean anchoring factor $W$21, demand-chasing correlation, framing effect defined as the difference in average $W$22 under profit versus cost framing, and bullwhip index $W$23. Outcome metrics include Average Cost,

$W$24

Turnover Rate,

$W$25

and Stockout Rate,

$W$26

The evaluated LLM agents are GPT-4.1, GPT-4o, Gemini-2.5-flash-lite, DeepSeek-V3, and Qwen-2.5-72B.

The reported findings emphasize that different LLMs typically exhibit varying degrees of decision bias that are similar to those observed in human beings. In the Newsvendor Problem, all LLMs exhibit pull-to-centre bias except Gemini-2.5, which has $W$27; GPT-4o is reported at approximately $W$28 under profit framing and $W$29 under negative framing, GPT-4.1 at approximately $W$30, and DeepSeek-V3 at approximately $W$31. Demand-chasing correlation is negligible for LLMs and close to zero. The paper states that there is no evidence of risk reversal across positive versus negative frames, describing LLM risk preferences as context-dependent. In Multi-Period Replenishment, GPT-4.1 and Qwen-2.5 have similar stockout rates, $W$32 versus $W$33, but distances $W$34 versus $W$35, which is used to argue that process metrics discriminate model quality better than aggregate stockout or turnover measures. In the Beer Game, all agents amplify upstream variance; Gemini-2.5 has $W$36–$W$37, and Qwen-2.5 has $W$38.

The mitigation strategies are prompt-level and information-level. Cognitive Reflection Prompts inject explicit reasoning steps and reduce anchoring: for Qwen-2.5, $W$39 decreases from $W$40 to $W$41 under profit framing and from $W$42 to $W$43 under negative framing. Information sharing expands the observation space to include partner orders and inventories. Under information sharing, Qwen-2.5 reduces $W$44 from $W$45 to $W$46 on the first link and from $W$47 to $W$48 on the second link, whereas GPT-4o exhibits action-chasing, driving $W$49 toward $W$50 at the cost of exploration.

This AIM-Bench instance is notable for recasting benchmark evaluation from static prediction quality to sequential decision behavior under uncertainty. A plausible implication is that it belongs as much to behavioral operations and mechanism design as to LLM evaluation.

6. Comparative interpretation and nearby benchmark ecosystems

Across these uses, benchmark construction follows markedly different logics. The synthetic AI Matrix work profiles convolutional layers and reconstructs their statistical workload signature in a compact CNN (Wei et al., 2018). The Alibaba AI Matrix work selects models that together consume over $W$51 of Alibaba GPU cycles and characterizes them with throughput, arithmetic intensity, memory traffic, and latency (Zhang et al., 2019). The affective-image AIM-Bench curates $W$52 editing triplets through hierarchical controlled filtering and evaluates them with semantic, aesthetic, and emotional metrics (Chen et al., 12 Apr 2026). The inventory AIM-Bench defines five textual supply-chain environments as POMGs and measures both biases and operational outcomes (Zhao et al., 15 Aug 2025).

This suggests a shared benchmark-design pattern despite the domain divergence: each framework rejects a single scalar notion of performance and instead combines task-faithful data construction with multi-axis evaluation. In the synthetic DNN setting, fidelity is defined by matching MACs and warps. In the Alibaba data-center setting, it is defined by production representativeness and systems-level profiling. In affective image manipulation, it is defined jointly by instruction consistency, aesthetics, and emotional expressiveness. In inventory control, it is defined jointly by decision bias, cost, stockouts, turnover, and process distance.

A nearby but distinct development is AnalogNAS-Bench, which is not titled AIM-Bench but addresses Analog In-Memory Computing (AIMC) through a dedicated NAS benchmark (Bessalah et al., 23 Jun 2025). It fully evaluates CIFAR-10 architectures under Baseline Accuracy, PTQ Accuracy, QAT Accuracy, Noisy Accuracy, Analog Accuracy, drift metrics at $W$53 s, $W$54 h, $W$55 h, and $W$56 d, one-day accuracy, AVM, parameter count, and search time. Its central claim is that standard quantization techniques fail to capture AIMC-specific noises, and its architectural analysis identifies wider and branched blocks, skip connections, and reduced dependence on $W$57 convolutions as signatures of robustness. The relevance here is comparative rather than terminological: it illustrates the same broader movement toward domain-specific benchmarks whose evaluation protocol explicitly models the failure modes of the target deployment substrate.

Taken together, the AIM-Bench and AI Matrix records indicate that the name now denotes a family of benchmarking practices rather than a coherent benchmark family. The common denominator is methodological: explicit task formalization, curated or profiled workload construction, and evaluation suites designed around the operational failure modes of the system under study.