---
title: 'LongPiBench: LLM Bias & Edge Deployment'
url: https://www.emergentmind.com/topics/longpibench
type: topic
---

# LongPiBench: LLM Bias & Edge Deployment

LongPiBench is a benchmarking framework designed to systematically assess two complementary frontiers in large language model (LLM) research: (1) the positional biases encountered by transformer LLMs when aggregating multiple relevant information pieces in lengthy contexts [2410.14641], and (2) the deployment performance of quantized LLMs on single-board computers (SBCs), with a focus on low-cost, resource-constrained edge inference [2511.07425]. By providing rigorously controlled tasks and unified evaluation pipelines, LongPiBench enables in-depth analysis of both model-level capabilities and practical on-device deployment limitations.

## 1. Motivation and Scope

LongPiBench was conceived to fill critical gaps left by previous LLM benchmarks:

- **Positional Aggregation Bias**: While the "lost in the middle" bias—LLMs ignoring single relevant facts placed at context center—has been substantially mitigated in modern long-context models, tasks requiring simultaneous retrieval and integration of several relevant items, with variable pairwise spacing, remain a prominent source of failure. Existing benchmarks such as Needle‐in‐a‐Haystack and LooGLE do not probe this regime, as their focus remains single-item, absolute position, or search-localization tasks [2410.14641].

- **Edge Deployment on SBCs**: The surging need for privacy-preserving, affordable on-device inference requires rigorous evaluation of LLM throughput, memory, and energy under SBC resource constraints. Prior systematic evaluations on platforms like Raspberry Pi and Orange Pi were lacking [2511.07425].

LongPiBench thus constitutes a dual-purpose suite: one targeting systematic measurement of context-length-related biases in LLM architecture and another establishing empirical constraints for lightweight inference on commodity hardware.

## 2. Benchmark Construction and Experimental Design

### 2.1 Positional Bias Assessment

Each LongPiBench instance for positional bias consists of $n=10$ relevant elements within a context of configurable length $L$ (up to 256K tokens), embedded among distractor "noise." Two orthogonal experimental axes are controlled:

- **Absolute Position**: The context is divided into $N=16$ equal segments. All $n$ relevant elements are placed within segment $l$ to set their mean absolute location
  $$
  \bar{x}_{\rm abs} = \left(\frac{l-1}{N-1}\right)L
  $$
  Varying $l$ realizes absolute-position profiles.

- **Relative Position**: The initial location of the first relevant element is fixed; the inter-element distance is controlled by
  $$
  d = \left(\frac{L}{n-1}\right)\left(\frac{l-1}{N-1}\right)
  $$
  where $l=1$ yields all relevant elements adjacent ($d \approx 0$), while $l=N$ distributes them uniformly through the context. This explicitly probes the model's capacity to aggregate distant facts, disentangled from absolute context location.

Multiple tasks of varying complexity are instantiated: Table SQL retrieval, Timeline Reordering, and Equation Solving. To prevent spurious performance from memorization, data undergoes knowledge masking and manual correction.

### 2.2 On-Device Inference Evaluation

The hardware benchmarking arm assesses 25 q4_k_m-quantized open-source LLMs (up to 7B parameters) across three SBCs—Raspberry Pi 4, Raspberry Pi 5, and Orange Pi 5 Pro—using the Ollama and Llamafile inference runtimes. Experiments employ three prompt lengths, four execution trials per setting, and metric logging of:

- Tokens per second (TPS) throughput
- Peak RAM usage (MB)
- Mean power consumption (W)

CPU scaling is explored (varying cores from 4–8), and model/board/routine configurations span practical deployment scenarios.

## 3. Metrics and Evaluation Protocols

### 3.1 Positional Bias: Task Recall

The primary metric for positional bias is recall:
$$
M_{\rm SQL}(R, L) = \frac{1}{|L|} \sum_{i=1}^{|L|} \mathbf{1}[\ell_i \in R]
$$
where $L$ is the ground-truth set and $R$ the model output. Performance is analyzed over absolute and relative placement levels, isolating the drop in average recall:
- Absolute: $\Delta_{\text{abs}} = A(\bar{x}_{\rm abs,\,1}) - A(\bar{x}_{\rm abs,\,N})$
- Relative: $\Delta_{\text{rel}} = A(d_{1}) - A(d_{N})$

### 3.2 Device Performance: Throughput, RAM, and Power

Throughput is defined as:
$$
\text{Throughput} = \frac{\text{Tokens generated}}{\text{Inference time (s)}}
$$
Power is reported as average consumption during inference, and RAM as peak usage.

Performance is reported both aggregated and per-board/runtime/model size for reproducibility. 

| Model        | Parameters | Board            | Runtime    | Throughput (TPS) | Peak RAM (MB) | Power (W) |
|--------------|------------|------------------|------------|------------------|---------------|-----------|
| TinyLlama    | 1.1B       | RPi 5            | Ollama     | 12.7             | 3,200         | 10.0      |
| TinyLlama    | 1.1B       | RPi 5            | Llamafile  | 24.5             | 2,400         | 6.5       |
| LLaMA2-3B    | 3B         | Orange Pi 5 Pro  | Ollama     | 3.8              | 4,800         | 14.0      |
| LLaMA2-3B    | 3B         | Orange Pi 5 Pro  | Llamafile  | 12.1             | 3,300         | 8.4       |

## 4. Key Observations and Analysis

### 4.1 Absolute Versus Relative Positional Effects

- Absolute position sensitivity ("lost in the middle") is largely absent in state-of-the-art and scaled-up LLMs (recall ≳ 95% across segments 1–16), but persists in smaller open models such as Qwen-2.5-7B and WizardLM-2 (manifesting as U-shaped performance dips) [2410.14641].
- Relative position bias is universal: as mean inter-element distance $d$ increases—from adjacent to spread across half the context—recall drops 20–35 percentage points, even in leading commercial models (e.g., GPT-4o-mini, Claude-3-Haiku). This effect does not diminish as rapidly with parameter scaling, indicating that span-coattention robustness does not emerge solely through scale.

### 4.2 Impact of Query Placement and Model Scaling

Placing the query tokens at the head of context, rather than the tail, reduces both absolute and relative positional biases for decoder-only architectures. Scaling Qwen models from 7B to 72B dramatically mitigates $\Delta_{\text{abs}}$, but only marginally improves $\Delta_{\text{rel}}$ severity.

### 4.3 SBC Inference Trade-Offs and Bottlenecks

- Llamafile delivers up to 4x higher throughput and 30–40% lower power draw compared to Ollama but scales poorly past 4 cores.
- Ollama scales throughput linearly with core count but at the expense of higher RAM and energy.
- On-device practical upper limits: Raspberry Pi 4 is restricted to ≤360M parameter models; Raspberry Pi 5 supports up to 1.5B parameters; Orange Pi 5 Pro can accommodate models up to 7B with reduced speed.
- Observed bottlenecks include memory bandwidth saturation and cache contention on high-core SMP SBCs, as well as RAM capacity limiting feasible model sizes.

## 5. Practical Recommendations

- **Bias Mitigation**: Architectural or algorithmic innovations (e.g., calibrated positional attention, explicit co-attention objectives, or memory-based retrieval layers) are likely required to address relative positional bias. Prompt engineering—such as placing queries at both ends, or using chunked context windows—offers partial amelioration.
- **On-device Deployment**: 
   - Use q4_k_m quantization for maximal efficiency.
   - Restrict Llamafile to four high-performance cores; use all available CPU with Ollama if throughput is paramount.
   - Monitor thermal and RAM constraints; employ active cooling for long or batch prompt sessions.
   - Select board-model-runtime combinations keyed to operational goals (e.g., Raspberry Pi 5 for small/medium LLMs, Orange Pi 5 Pro for larger models).

## 6. Implications and Future Work

LongPiBench identifies that, although large-context LLMs have remedied "lost in the middle" failures for isolated fact retrieval, a more consequential bias persists when aggregating multiple, widely spaced facts—a key requirement in information synthesis and multi-step reasoning tasks. The lack of emergent mitigation solely through scale, and the pronounced drop in recall with increasing inter-element distance, indicate a need for new long-range attention mechanisms and optimization objectives tailored to span aggregation.

On the hardware front, LongPiBench demonstrates that cost-effective edge inference is viable for a large subset of use cases, contingent on appropriate quantization, runtime, and hardware choices. This suggests a rising practical role for local LLM deployment where throughput, privacy, or latency are crucial and massive models are not mandatory.

Open avenues include broader benchmarking of memory-augmented or retrieval-augmented transformer architectures, analysis of more diverse real-world corpora and prompt types, and further integration of bias-calibrated metrics into model selection and evaluation pipelines.

## 7. Public Accessibility and Community Impact

LongPiBench provides publicly released code, data, and detailed results, facilitating reproducible research and enabling the broader community to probe, compare, and mitigate context-related biases in LLMs. By decoupling and quantifying absolute from relative positional effects and mapping deployment realism on affordable hardware, it establishes a critical yardstick for future LLM development and evaluation [2410.14641, 2511.07425].

Source: https://www.emergentmind.com/topics/longpibench