LH-Bench: Disambiguation & Benchmark Insights

• LH-Bench is a term used ambiguously to refer to both LwHBench, a low-level hardware benchmark for SBCs, and a misremembered reference to LLF-Bench for language-based interactive learning.
• LwHBench measures CPU, GPU, memory, and storage performance on devices like Raspberry Pi using component cross-measurement to reduce bias and enable precise device fingerprinting.
• LLF-Bench, distinct from LwHBench, focuses on simulation benchmarks for sequential decision-making with natural-language instructions and feedback, emphasizing randomized environments and multiple task types.

“LH-Bench” is not a single unambiguous benchmark name in the supplied arXiv literature. In one usage, it is a shorthand for LwHBench, a low-level hardware component benchmark and dataset for Single-Board Computers (SBCs), with a primary implementation on Raspberry Pi devices and an additional RockPro64 case study (Sánchez et al., 2022). In a separate and explicitly distinct usage, the LLF-Bench paper states that it does not describe any benchmark named “LH-Bench,” and treats that string as a likely misremembering of LLF-Bench, the “Learning from Language Feedback Benchmark” for interactive learning from natural-language instructions and feedback (Cheng et al., 2023). This suggests that any technical discussion of “LH-Bench” requires disambiguation at the outset.

1. Nomenclature and disambiguation

In the supplied sources, the term “LH-Bench” is associated with two different interpretive paths: a shorthand for LwHBench, and a possible confusion with LLF-Bench. The first is a hardware benchmarking framework for SBCs; the second is a simulation benchmark for sequential decision-making with language feedback (Sánchez et al., 2022).

Benchmark name	Domain	Relation to “LH-Bench”
LwHBench	Low-level hardware component benchmark and dataset for SBCs	Sometimes referred to in shorthand as “LH-Bench”
LLF-Bench	Interactive learning from language feedback	Its paper states that no “LH-Bench” benchmark is described

Because the two systems address unrelated problem classes, conflating them obscures both methodology and intended use. LwHBench centers low-level performance characterization of CPU, GPU, memory, and storage, whereas LLF-Bench centers natural-language instruction and feedback in sequential decision-making. A plausible implication is that “LH-Bench” should be treated as a context-dependent label rather than a canonical benchmark title.

2. LwHBench as a low-level SBC benchmark

LwHBench was introduced to address benchmarking needs in IoT and Edge computing, where SBCs such as Raspberry Pi are widely deployed because they are cheap, flexible, and powerful enough to run AI and data processing workloads at the network edge (Sánchez et al., 2022). The motivating claim is that high-level benchmarks specialized in particular application scenarios do not expose low-level behavior of individual hardware components, including internal cycle-level timing, component-to-component skew, and manufacturing imperfections.

The benchmark was designed to fill several gaps identified in the SBC benchmarking literature: lack of extensive datasets suitable for ML/DL, predominant focus on high-level applications, self-measuring bias, limited component coverage, and the absence of low-level SBC performance datasets. In response, LwHBench measures CPU, GPU, memory, and storage at a low level, avoids self-measurement bias by using one component to measure another, and produces a large dataset explicitly intended for AI/ML use cases such as device identification, context impact analysis, and performance modeling.

Architecturally, LwHBench is a Python-based benchmark that runs as a system service on SBCs. Its core elements are a measurement engine for microbenchmarks, low-level access libraries, and a data collection service that starts at boot as data_collection.service. The service repeatedly executes a fixed sequence of microbenchmarks to produce a feature vector, stores vectors in CSV files named by MAC address, and reboots the device after 800 samples to limit runtime-related noise. A separate MAC-Model.txt file links MAC addresses to Raspberry Pi models.

3. Measurement model, microbenchmarks, and noise control

The central measurement idea in LwHBench is component-to-component timing. Because Raspberry Pis use a single base crystal oscillator and PLLs to derive component frequencies, many logical clocks are not independent. To avoid self-timing a component with its own clock, CPU execution is measured using GPU cycle counts, while GPU execution, memory operations, and storage I/O are measured using CPU cycle or time sources (Sánchez et al., 2022). The benchmark therefore records raw counters and times rather than normalized GFLOPS or MB/s, reflecting a focus on low-level fingerprinting and internal characterization.

Component	Measurement reference	Operations
CPU	GPU cycles	Sleep system calls of 1, 2, 5, 10, and 120 seconds; fixed string hash; pseudo-random number generation; /dev/urandom on 100 MB; Fibonacci calculation such as $F_{20}$
GPU	CPU time via perf_counter_ns()	matrix mul, matrix sum, scopy
Memory	CPU time	list creation, mem reserve, csv read
Storage	CPU time	100 reads of 100 kB and 100 writes of 100 kB

Each sample also includes generic features such as timestamp and temperature, and each row carries the device MAC address as label. Storage is treated differently from the other components: the 100 reads and 100 writes are recorded as separate features, which preserves intra-device variability and noise instead of collapsing them immediately into aggregates.

To obtain stable and comparable measurements, the benchmark applies multiple system-level controls. These include fixed frequencies through turbo_mode=1 and CPU governor performance, high scheduling priority using chrt --rr or nice -n -20, temporary disabling of ASLR for memory tests with sysctl kernel.randomize_va_space=0, Python compilation with Profile-Guided Optimization via --enable-optimizations, disabling the Python garbage collector during runs, and fixing PYTHONHASHSEED=0. On Raspberry Pi 3 and Raspberry Pi 4, one CPU core is reserved for the benchmark using isolcpus, nohz_full, and rcu_nocbs. The paper presents these measures as a way to reduce measurement noise and emphasize intrinsic hardware behavior rather than OS-level jitter.

4. Dataset structure and empirical findings

For the main dataset, LwHBench was implemented on 45 Raspberry Pi devices and executed continuously for 100 days from Dec 6, 2021 to Mar 17, 2022, producing 2,386,126 vectors with a total size of approximately 4 GB (Sánchez et al., 2022). The platform mix comprised 15 Raspberry Pi 4 Model B units, 10 Raspberry Pi 3 Model B+ units, 10 Raspberry Pi Model B+ units, and 10 Raspberry Pi Zero units. All devices were configured with Raspbian 10 (Buster), 32-bit, Linux kernel 5.4.83, matching 16 GB SanDisk Ultra A1 Class 10 SD cards, and a common physical environment including the same lab room, identical cases, and aluminum heat sinks.

Per-model sample counts were reported as 505,584 for RPi 1B+, 547,800 for RPi 3B+, 548,647 for RPi Zero, and 784,095 for RPi 4B. The differences were attributed to device speed and to some outages or late additions. Most devices contributed more than 50,000 samples, while five devices had fewer samples due to interruptions.

The paper reports several statistical observations. For temperature–performance correlation, RPi3 devices showed high sensitivity to temperature, with many features exhibiting strong positive or negative correlation. Other models, including RPi1, RPi4, and RPi Zero, showed low correlation, with most values near 0 and small variations of approximately $\pm 0.05$. For inter-device variability within a model, gpu_matrixmul and storage_read_1 had nearly identical distributions across devices, whereas cpu_sleep_120s showed roughly Gaussian distributions centered at slightly different values for each device, and storage_write_1 showed bimodal behavior with one outlier device. These patterns are presented as the empirical basis for device fingerprinting.

The dataset was also used in AI-enabled use cases. In unsupervised model identification, all CPU, GPU, memory, and storage features were used except timestamp and temperature. PCA, t-SNE, and UMAP were tested for dimensionality reduction, and k-means and DBSCAN for clustering. For PCA plus k-means with $k=4$, four clearly separated clusters were obtained, and the cluster sizes matched exactly the sample counts per model: cluster 0 with 784,095 samples for RPi4, cluster 1 with 505,584 for RPi1B+, cluster 2 with 547,800 for RPi3B+, and cluster 3 with 548,647 for RPi Zero.

For supervised individual device identification, all component features were used, temperature was included as an input feature, and storage timings were summarized per sample by average, median, minimum, and maximum for reads and for writes. The split was 80% for training and cross-validation and 20% for testing.

Classifier	Macro F1	Notes
Decision Tree	0.88	Precision 0.88, Recall 0.88
Random Forest	0.93	Precision 0.93, Recall 0.93
XGBoost	0.97	$lr=0.1$, $\gamma=0.01$, max_depth=20
k-NN	$\approx 0.31$	$k=7$
Naive Bayes	$\approx 0.12$	—
SVM	$\approx 0.50$	Linear kernel, $\gamma=0.01$
MLP	$\pm 0.05$0	1 hidden layer, 50 neurons, ReLU

The reported confusion matrix indicated per-device identification accuracy of at least 0.88 for every device, with most devices reaching at least 0.93. The paper interprets these results as evidence that low-level performance vectors encode distinctive hardware behavior fingerprints even when SBC models and software images are nominally identical.

5. Adaptation to RockPro64, comparison to prior benchmarks, and reproducibility

LwHBench was adapted to an agriculture-focused deployment involving three PINE64 RockPro64 SBCs in an agriculture sensor network with requirements for runtime performance monitoring and device identification or fingerprinting (Sánchez et al., 2022). The RockPro64 platform differs substantially from Raspberry Pi in its accelerator stack: it uses a Mali T860 GPU rather than VideoCore IV or VI, and the adaptation therefore required GPU-specific changes. The paper states that ARM’s HWCPipe library was used to access GPU performance counters, GPU_ACTIVE was selected as the measurement source, and ARM Compute Library was used to implement GPU compute kernels. CPU timing remained based on perf_counter_ns(), and the same logical microbenchmarks were retained.

The RockPro64 case study ran for one week, producing approximately 35.9 MB of data and approximately 12,800 vectors, or roughly 4,000 per device. The distribution of the cpu_sleep_120s feature showed three distinct distributions, one for each device. Using the same classification pipeline as in the Raspberry Pi experiments, all models achieved 100% F1 and accuracy in per-device identification.

Relative to prior benchmarks such as HERMIT, DeFog, EdgeBench, DeepEdgeBench, and pCAMP, LwHBench is distinguished by its low-level focus, component cross-measurement, broad component coverage, long-term and large-scale dataset, and open-source and open-data release. The benchmark code is available on GitHub at sxz0/LwHBench, and the dataset is available through IEEE DataPort with DOI 10.21227/8zmy-7z87. The released materials include benchmark scripts, the systemd service, support code for Raspberry Pi and RockPro64, CSV files per device, and the MAC-Model.txt mapping file.

The authors also note several limitations. Implementation is hardware-dependent because access to CPU and GPU counters can require deep knowledge of the SoC and drivers. The resulting metrics are non-standard for traditional application benchmarking: CPU “performance” expressed in GPU cycles is not directly interpretable as GFLOPS or standard throughput. Core isolation and kernel-noise reduction improve measurement quality but may reduce capacity for application workloads on resource-limited SBCs.

6. LLF-Bench as a separate benchmark family

LLF-Bench, introduced as the Learning from Language Feedback Benchmark, is a separate benchmark with a different research target and should not be identified with LwHBench or its shorthand (Cheng et al., 2023). It is a simulation benchmark for evaluating whether AI agents can interactively learn in sequential decision-making tasks using natural-language instructions and feedback as the learning signal. The benchmark wraps diverse environments including bandits, recommendation, text generation, navigation, and robotic control, and exposes them through a unified Gym-style API in which observations are dictionaries containing task observation, instruction, and feedback.

Its task families are llf-bandit, llf-poem, llf-reco-movie, llf-optimization, llf-parking, llf-gridworld, llf-alfworld, and llf-metaworld. The benchmark distinguishes instruction from feedback and allows configuration of instruction_type as Basic (b), Complete (c), or Practical (p). It also defines atomic feedback types: reward textualization (r), hindsight positive (hp), hindsight negative (hn), future positive (fp), and future negative (fn), together with composite configurations All (a), Mix (m), and None (n). A TextWrapper converts non-text environments into a text interface for LLM-based agents.

A defining design feature is randomization. LLF-Bench uses paraphrased templates for instructions and feedback, with about 4–20 paraphrases per type, and it randomizes latent environment parameters such as arm ordering in llf-bandit, graph structure and treasure location in llf-gridworld, sampled preference profiles in llf-reco-movie, houses and task instances in llf-alfworld, and tasks and initial configurations in llf-metaworld. The paper explicitly states that it does not describe any benchmark named “LH-Bench,” and presents “LH-Bench” only as a likely misremembering of “LLF-Bench.” Accordingly, “LH-Bench” in the supplied literature properly names LwHBench only in the shorthand sense, while LLF-Bench remains a distinct benchmark family for interactive learning from language feedback.