---
title: Frequency Framed Prompting (REFER)
url: https://www.emergentmind.com/topics/frequency-framed-prompting-refer
type: topic
---

# Frequency Framed Prompting (REFER)

Frequency Framed Prompting, abbreviated **REFER**, is a prompting framework that elicits large language model reasoning using **frequency-based information**—counts of opinions over an explicit reference class—rather than abstract probability language or generic instructions to be “fair” or “balanced.” In its canonical formulation, the model is asked first to determine how many input documents express each social value or stance, and only then to generate a summary that reflects those frequencies [2509.15723]. Within opinion summarisation, REFER is intended to mitigate systematic bias in how summaries represent majority and minority viewpoints; more broadly, it belongs to a family of prompting methods that treat prompt structure, explicit counting, and staged reasoning as first-class control variables rather than incidental phrasing choices [2509.15723].

## 1. Definition and conceptual basis

REFER is defined as **F**re**re**quency Framed P**r**ompting, a framework that replaces abstract fairness instructions with explicit reasoning over counts such as “how many reviews out of \(N\) are positive versus negative,” where \(N\) is the actual number of source documents in the input collection [2509.15723]. The core representation is frequency-based rather than probability-based. If social values are denoted \(v_1, \ldots, v_k\), REFER operates over counts \(n_i\) in a reference class of size \(N\), with the paper giving the conceptual mapping
\[
p_i \approx \frac{n_i}{N}, \qquad \sum_{i=1}^k n_i = N.
\]
The prompting emphasis is not on supplying the model with ground-truth \(p_i\), but on instructing it to infer and use \(n_i\) directly from the input [2509.15723].

The framework is grounded in cognitive science results on **natural frequencies**. The motivating claim is that frequency formats make reference classes explicit, reduce cognitive load, and encourage sequential analysis of counts before qualitative judgment. In the REFER formulation, these effects are translated into prompt design through three mechanisms named explicitly in the source: **Reference Class Specification**, **Numerical Anchoring**, and **Sequential Deliberation** [2509.15723]. The first fixes the denominator; the second introduces concrete count tokens; the third enforces a two-stage reasoning path in which counting precedes summarisation.

REFER also presupposes a specific notion of fairness in opinion summarisation. A fair summary should represent both majority and minority stances and preserve the distribution of opinions in the source rather than amplifying one side. The motivating problem is not merely omission, but **systematic second-order bias**: deviations from the source distribution can be patterned rather than random, such as consistently down-weighting one political side or one sentiment polarity [2509.15723].

## 2. Prompting framework and formal variants

The baseline prompt in the REFER study is a standard direct summarisation instruction with no distributional guidance: reviews or tweets are presented, separated by delimiters, and the model is asked to produce a short summary [2509.15723]. REFER modifies this by inserting a frequency-analysis stage before summarisation. Its simplest form is:

> “Let's first determine how many reviews out of {number of input in the source documents} are {social value 1, social value 2}. Then, let's generate a balanced summary that accurately reflects the frequency of these opinions. {direct prompt}” [2509.15723]

This formulation introduces three structural constraints. First, it specifies the **reference class size** using the actual input cardinality. Second, it names the target **opinion categories**. Third, it imposes an explicit **temporal order**: first count, then summarise [2509.15723].

The paper evaluates five prompt families that integrate this counting stage with different reasoning or control signals.

| Variant | Characteristic instruction |
|---|---|
| Plain REFER | Count opinions out of \(N\), then summarise |
| Prefix-Instruct-R | Add “Be fair with your output” before counting |
| Prefix-Role-R | Add “You are an unbiased person” before counting |
| CoT-R | Add “Let’s think step by step” before counting |
| Agent-R | Use summarisation, frequency, judge, and senior editor agents |

In **Prefix-Instruct-R**, fairness language is explicitly prepended to the count-then-summarise structure. In **Prefix-Role-R**, the same structure is combined with persona prompting. In **CoT-R**, the instruction “Let’s think step by step” is used to reinforce sequential deliberation. In **Agent-R**, the framework is expanded into a four-role workflow consisting of a Summarisation Agent-REFER, a Frequency Agent, a Judge Agent, and a Senior Editor Agent, with the frequency agent producing counts in a fixed format such as “{social value 1 #number, social value 2 #number}” [2509.15723].

An **oracle** variant is also included. In that condition, the prompt supplies the actual ground-truth frequencies and asks for a summary reflecting them. This serves as a theoretical upper bound on what happens when exact counts are available in the prompt [2509.15723].

A broader methodological interpretation follows from work on prompt reframing. Reframing studies show that itemized instructions, explicit sequential steps, and constrained output formats can materially improve instruction following across models and tasks, including cases where reframed prompts outperform raw prompts despite far fewer examples [2109.07830]. This suggests that REFER’s effectiveness is not only semantic—reasoning about counts—but also structural: it reframes the summarisation task into a pattern of explicit substeps that better matches the instruction-following preferences of current language models.

## 3. Experimental setting and fairness metrics

The REFER study evaluates two summarisation tasks. The first is **review summarisation**, based on Amazon Reviews 2023, where each test collection contains 8 reviews of 30–120 words. The second is **political tweet summarisation**, based on FairSumm and a curated political tweets dataset, where each test collection contains 30 tweets labelled by political stance [2509.15723].

For both tasks, the experimental design uses **900 input collections per dataset**: 300 balanced, and 300 skewed with 75/25 proportions in each direction. This setup is intended to test whether summaries preserve the input stance distribution under both balanced and asymmetric source mixtures [2509.15723]. The evaluated models are GPT-3.5-Turbo, GPT-4o-mini, GPT-4o, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct, all in zero-shot prompting mode with fixed hyperparameters: max new tokens \(=256\), temperature \(=0.001\), repetition penalty \(=1.1\) [2509.15723].

Fairness is operationalised distributionally. The central question is how closely the stance distribution of the summary matches the stance distribution of the source. The paper distinguishes **first-order** fairness—whether groups are represented at all and whether minority opinions appear—from **second-order** fairness, which concerns whether the relative differences between groups are preserved [2509.15723].

Four metrics are used:

| Metric | Intended measurement |
|---|---|
| SPD | Second-order fairness mismatch |
| BUR | Proportion of summaries judged unfair |
| UER | Under-representation error for minority opinions |
| SOF | Variance of unfairness across social values |

The paper states that **Second-Order SPD (SPD)** is taken from Huang et al. 2024 and compares summary and source distributions over social values, with lower values indicating better preservation of second-order fairness. **BUR**, **UER**, and **SOF** are taken from Zhang et al. 2023, with lower values again indicating better fairness [2509.15723]. Because summaries can contain compound sentences expressing multiple viewpoints, they are first decomposed into simple propositions using GPT-4o-mini before classification and metric computation [2509.15723].

This metric design is significant for understanding REFER. The framework is not evaluated primarily as a style intervention or a coherence intervention, but as a mechanism for controlling **distributional faithfulness** in generated summaries. That emphasis distinguishes it from prompt designs that optimize summary informativeness without explicit regard to opinion proportions.

## 4. Empirical findings and model dependence

The central empirical result is that REFER improves fairness metrics in opinion summarisation, with especially strong gains for larger models and when combined with stronger reasoning instructions [2509.15723]. In the review summarisation setting, the paper reports, for GPT-3.5-Turbo under direct prompting, \( \mathrm{SPD}=35.07\), \( \mathrm{BUR}=57.59\), \( \mathrm{UER}=8.07\), and \( \mathrm{SOF}=6.66\). Relative to this baseline, plain REFER improves SPD by \(-4.07\) and SOF by \(-0.10\), while BUR and UER slightly worsen by \(+1.22\) and \(+0.17\), respectively. By contrast, **Prefix-Instruct-R** improves all four metrics relative to Prefix-Instruct, and **CoT-R** yields across-the-board improvements relative to CoT, including SPD \(-4.07\), BUR \(-0.77\), UER \(-0.14\), and SOF \(-0.34\) [2509.15723].

The broader pattern is that **CoT-R**, **Prefix-Instruct-R**, and **Prefix-Role-R** generally reduce SPD, BUR, UER, and SOF for GPT-3.5-Turbo, GPT-4o-mini, and GPT-4o in review summarisation [2509.15723]. In political tweet summarisation, the task is explicitly described as harder: absolute SPD, BUR, UER, and SOF values are larger, meaning bias is more severe overall. REFER still yields notable improvements, especially in the second-order metrics SPD and SOF, but the gains are smaller and can be accompanied by slight regressions in BUR or UER [2509.15723].

The paper further reports that the oracle condition with actual frequencies produces the lowest SPD and SOF, confirming that exact count information is a strong upper bound [2509.15723]. Figures and appendix tables show SPD reductions often in the **5–20% range** and SOF reductions of **5–10%** relative to base prompts for larger models in review summarisation; improvements also occur for political tweets, though less uniformly [2509.15723].

Model size and instruction-following ability are decisive. Larger models such as GPT-4o and Llama-3.1-70B-Instruct are described as better able to follow complex instructions and therefore benefit more from REFER. Smaller models such as Llama-3.1-8B-Instruct often ignore the counting step and show high variability in summary length and structure, which reduces REFER’s effectiveness [2509.15723]. This dependence on instruction-following strength aligns with a separate line of work showing that prompt topology matters independently of content: when identical content is reordered into non-sequential “jumping” structures, instruction-following accuracy in multistep prompts can collapse by up to 72% relative to baseline, with current architectures behaving as though instruction following were a sequential pattern rather than a reasoning skill [2601.18924]. A plausible implication is that REFER works best when its count-then-summarise stages remain simple, linear, and explicitly ordered.

## 5. Mechanisms, interpretation, and scope conditions

The REFER paper attributes its improvements to the same three cognitive mechanisms that motivate the framework. **Reference class specification** directs the model to think in terms of a concrete denominator such as 8 reviews or 30 tweets rather than an abstract population. **Numerical anchoring** provides count tokens that can guide generation. **Sequential deliberation** enforces the order “first determine the counts, then summarise,” which is especially pronounced in CoT-R [2509.15723].

The mechanism is presented as an analogy rather than a literal cognitive equivalence. The paper explicitly notes that large language model internal mechanisms differ from human brains and that the observed improvements may arise from computational reasons other than human-like cognition [2509.15723]. Nonetheless, the empirical pattern supports the claim that frequency-style prompts can activate learned text patterns that encourage more faithful handling of opinion distributions.

There are also clear limits. REFER is tested only on English product reviews and political tweets, and only for **binary social values** consisting of two opposing stances [2509.15723]. The prompts are manually crafted rather than automatically optimized. The method addresses **model fairness in summarisation outputs**, not bias in the underlying training data. The evaluation itself depends on labelled datasets and classification pipelines, so flawed labels could distort fairness measurement [2509.15723].

The results also indicate trade-offs. In some settings, SPD and SOF improve while BUR or UER slightly worsen. The paper interprets this as evidence that improving second-order fairness—preserving proportional relationships between groups—need not improve every first-order fairness criterion simultaneously [2509.15723]. This is an important corrective to a common misconception that “fairness” in summarisation is a single scalar property. In the REFER formulation, fairness is multidimensional and metric-dependent.

A second misconception is that frequency framing merely restates probabilities in another notation. The REFER formulation instead treats counts and denominators as computationally salient prompt tokens. This difference matters because the intervention does not rely on the model being given gold percentages; it relies on the model being asked to **construct a count-based intermediate representation** from the source documents and then condition the summary on that representation [2509.15723].

## 6. Related formulations and methodological extensions

The term “frequency” appears in several adjacent prompt-design literatures, but with different semantics. In **cross-domain few-shot classification**, “Frequency-Aware Prompting” denotes a training-time mechanism based on Discrete Wavelet Transform and Inverse DWT, where high-frequency image components are zeroed or randomized and mutual attention is used to align features across original and frequency-modified views [2406.16422]. That work is not about opinion summarisation, yet it establishes a general design principle: robustness can improve when models are forced to rely less on brittle signals and more on stable structure-preserving cues [2406.16422]. A plausible analogy is that REFER treats explicit counts as stable anchors for summarisation in the same way that low-frequency visual structure is treated as a stable anchor under domain shift.

A different but complementary connection comes from **Reflective Prompt Tuning**, which optimizes prompts by evaluating a target model over an optimization set, clustering recurring failure modes, and revising prompts using structured diagnostic reports and memory across iterations [2605.21781]. Because that framework explicitly aggregates recurring failures into topics and uses their prevalence to guide edits, it provides a natural route for extending REFER beyond hand-written templates. This suggests a future REFER variant in which error frequencies such as under-representation of minority propositions or systematic miscounting are tracked over a development set and used to revise prompt structure automatically [2605.21781].

RIFT adds a structural warning to such extensions. It shows that even when semantic content is held constant, non-sequential prompt topologies can sharply degrade multistep instruction following, with failures often attributable to instruction-order violations and semantic drift [2601.18924]. For REFER, the implication is that count-based reasoning should be scaffolded in a **linear prompt regime** rather than embedded in elaborate non-linear control flow. The empirical success of CoT-R over more complex Agent-R variants in many settings is consistent with that interpretation, though the paper does not frame it in those terms [2509.15723].

Taken together, these adjacent results place REFER within a broader research trend: prompt performance depends not only on semantic content, but on how instructions expose intermediate structure, constrain outputs, and align with the sequential processing biases of present-day models [2109.07830]. In that sense, Frequency Framed Prompting is both a specific fairness intervention for opinion summarisation and a case study in how **explicit counting, reference-class binding, and staged reasoning** can reshape model behavior without retraining [2509.15723].

Source: https://www.emergentmind.com/topics/frequency-framed-prompting-refer