In-Place Double Stimulus Quality Scale (IDSQS)

• IDSQS is a double stimulus method that presents pristine and distorted images in the same spatial location to reveal subtle perceptual differences.
• It employs a press-and-hold toggle control along with a continuous 0–100 impairment scale to sensitively capture minor quality variations.
• Large-scale crowdsourcing and advanced statistical modeling, including Beta distribution fitting, validate IDSQS as a practical alternative to more complex protocols.

The In-place Double Stimulus Quality Scale (IDSQS) is a double-stimulus subjective assessment methodology for the evaluation of high quality images, particularly in the regime of high-quality to nearly visually lossless image compression where perceptual differences are very small and standard protocols often lose sensitivity. In IDSQS, a pristine reference image and its distorted counterpart are presented at exactly the same spatial location, and the observer alternates between them by pressing and holding a toggle control while reporting the degree of impairment of the distorted image on a continuous 0–100 scale. The method was introduced as a practical alternative to existing side-by-side, sequential, categorical, and ranking-based procedures, with the aim of preserving direct perceptual comparison while remaining intuitive to score and practical for large-scale crowdsourcing (Mohammadi et al., 13 Aug 2025).

1. Concept and design objective

IDSQS was proposed for subjective image-quality assessment in the specific regime where distortions are subtle, and where methods such as side-by-side double stimulus, sequential presentation, or coarse category rating can fail to reveal small perceptual differences. The central design choice is that the reference and distorted images are not shown simultaneously side by side; instead, both are displayed in the same screen position, and only one image is visible at a time. The observer alternates between them using a toggle interface.

This arrangement was intended to preserve a direct perceptual comparison while reducing the spatial bias and eye-movement burden associated with side-by-side viewing. It also avoids requiring the observer to remember a reference across a gray interval, which the source describes as a weakness of sequential protocols. Because only one image is rendered at a time, the method is also described as practical for high-resolution images.

The paper positions IDSQS as a simpler and more intuitive alternative to more involved ranking-based or boosted methods used in JPEG AIC-3, while still aiming to capture fine perceptual differences better than standard direct-rating methods. A plausible implication is that IDSQS occupies an intermediate methodological position: more comparative than single-stimulus quality rating, but less operationally complex than boosted triplet workflows.

2. In-place double-stimulus procedure and scoring semantics

The procedure is defined as a sequence of six steps. First, a pair is prepared, consisting of one pristine reference image and one compressed or otherwise distorted test image. Second, only one image is visible at a time on the interface. Third, the subject views the image in a fixed spatial location and can toggle between the reference and distorted versions by pressing and holding a button. Fourth, the interface enforces a maximum of two toggles per second in order to prevent rapid flickering; the stated goal is comparison rather than flicker-based boosting. Fifth, the subject judges the degree of impairment of the distorted image relative to the reference. Sixth, the subject reports a score on a continuous 0–100 scale (Mohammadi et al., 13 Aug 2025).

The scoring semantics are explicit. A score of 0 denotes the highest quality, essentially indistinguishable from the reference, whereas 100 denotes the lowest quality, corresponding to severe distortion. The scale is therefore an impairment scale rather than an abstract goodness scale: the observer is asked to report how much the distorted image differs from the pristine reference.

For high-quality image compression, the use of a continuous scale is significant because the target differences are often small. The paper emphasizes that this continuous scale is more fine-grained than category-based methods. This suggests that IDSQS is designed not merely as a presentation change, but as a combined presentation-and-response framework in which in-place toggling and continuous impairment scoring are intended to work together.

3. Relation to established subjective assessment methodologies

The paper contrasts IDSQS with several established protocols. The distinctions are methodological rather than merely terminological.

Method	Presentation or scoring characteristic	Contrast with IDSQS
DSCQS	Images are often shown side by side; the reference may be hidden/randomized; observers rate both stimuli on a continuous scale	IDSQS shows one image at a time in the same location
DSIS	Uses a categorical impairment scale	IDSQS uses a continuous 0–100 scale
ACR and similar single-stimulus methods	Each image is rated independently without direct comparison to reference	IDSQS keeps the reference available through toggling
JPEG AIC-2 / AIC-3 ranking or triplet methods	Often more sensitive for high-fidelity content, especially when boosted or flickered	IDSQS is a simpler direct-rating alternative

In this comparison, IDSQS is described as attempting to retain the comparative sensitivity of double-stimulus testing while preserving the intuitiveness and interpretability of a rating scale. The paper therefore frames it neither as a replacement for all prior methods nor as a variant of single-stimulus scoring, but as a distinct double-stimulus procedure tailored to the high-fidelity regime.

A common misconception would be to treat IDSQS as equivalent to conventional double-stimulus side-by-side viewing. The source explicitly distinguishes the two. Another plausible misconception is that IDSQS relies on flicker-based boosting; however, the interface limits toggling to two per second precisely to avoid that usage. The method is based on alternating comparison at the same spatial location, not on rapid temporal enhancement.

4. Crowdsourced experimental protocol

The evaluation of IDSQS was conducted as a large-scale crowdsourcing study on Amazon Mechanical Turk (MTurk). The test material consisted of 5 source images selected from the JPEG AIC-3 dataset. For each source, distortions were generated with 5 codecs or compression methods: JPEG, JPEG 2000, AVIF, VVC Intra, and JPEG XL. Each codec was represented at 10 bitrate or distortion levels, indexed from 1 to 10, where 10 is the lowest bitrate and worst quality. These levels were spaced at about 2.5 JND units each and were intended to cover high to nearly visually lossless quality. For the subjective test, images were cropped to 620 × 800 pixels (Mohammadi et al., 13 Aug 2025).

The study used a batch-based organization. Each batch contained 79 study questions, each formed by a reference and a compressed image, and 10 trap questions used to assess reliability. Trap questions were of two types. Type I used an extremely distorted image at distortion level 10, for which subjects should give a score near 100. Type II used identical reference and “distorted” images at distortion level 0, for which subjects should give a score near 0. The full study was split into 4 batches, and questions were randomized within each batch.

Participants were recruited on MTurk. Each participant could complete one assignment, and each assignment included two randomly drawn batches separated by a mandatory 3-minute break. Before testing, participants completed instructions, a visual acuity / color vision screening using Ishihara plates 3 and 4, consent, and a training phase with easy and difficult examples. Hardware requirements were also enforced: a minimum screen resolution of 1920 × 1080, with only PCs and laptops accepted.

The reported sample comprised 132 subjects, 179 collected batch instances, and 45 ratings per question. The demographic summary states that the largest age group was 31–35 years, with 87 male and 45 female participants. The paper argues that such a protocol is suitable for large-scale collection while still incorporating screening, training, trap questions, and reliability filtering.

5. Statistical processing, outlier handling, and Beta-distribution modeling

The paper defines a multi-stage analysis pipeline for the raw subjective scores. The first stage is trap-question accuracy. For a trap-question response $v \in [0,100]$, accuracy is defined as

$\text{accuracy} = \frac{v}{100}$

for Type I traps, and

$\text{accuracy} = 1 - \frac{v}{100}$

for Type II traps. Batch accuracy is then taken as the mean over trap questions. An Otsu threshold was applied to separate reliable and unreliable batches; the threshold found was 0.67, and 104 of 179 batches were discarded at this stage.

The second stage is batch outlier removal by correlation, using a method from the 2023 BT.500 recommendation update. For each batch $b$, batch-level consistency is defined through the minimum of PLCC and SROCC relative to the aggregate MOS values. A batch is removed if

$CR_b < \min(\mu(CR)-\sigma(CR), 0.85).$

Using this rule, 12 batch instances were removed.

The third stage is soft quality reconstruction or DMOS estimation based on the ITU-T P.910 soft rejection / quality reconstruction procedure. For subject $i$ and question $q$, subject bias is estimated as

$$B_i^n = \frac{1}{Q_i}\sum_{q=1}^{Q_i} [S_i(q) - MOS^{n-1}(q)],$$

where $Q_i$ is the number of questions answered by subject $i$, $\text{accuracy} = \frac{v}{100}$0 is subject $\text{accuracy} = \frac{v}{100}$1’s score for question $\text{accuracy} = \frac{v}{100}$2, and $\text{accuracy} = \frac{v}{100}$3 is the previous iteration’s MOS for question $\text{accuracy} = \frac{v}{100}$4. Residuals are

$\text{accuracy} = \frac{v}{100}$5

and subject consistency is defined as the inverse residual variance,

$\text{accuracy} = \frac{v}{100}$6

MOS is updated by weighted averaging:

$\text{accuracy} = \frac{v}{100}$7

Iteration stops when

$\text{accuracy} = \frac{v}{100}$8

Finally, the source-image MOS is subtracted to form DMOS (Mohammadi et al., 13 Aug 2025).

A distinct modeling contribution is the use of a Beta distribution to model per-stimulus score distributions after outlier removal. Scores are normalized to $\text{accuracy} = \frac{v}{100}$9 by dividing by 100. The Beta distribution has parameters $\text{accuracy} = 1 - \frac{v}{100}$0, estimated by Maximum Likelihood Estimation (MLE), or by the method of moments if MLE fails. The paper notes that $\text{accuracy} = 1 - \frac{v}{100}$1 gives a symmetric distribution, while $\text{accuracy} = 1 - \frac{v}{100}$2 gives a U-shaped distribution. A Chi-square goodness-of-fit test at significance level 0.05 was used to assess fit, and 93% of the per-stimulus distributions passed.

The interpretation assigned to the Beta parameters is practical. Large $\text{accuracy} = 1 - \frac{v}{100}$3 indicates concentrated responses and higher agreement among raters; small $\text{accuracy} = 1 - \frac{v}{100}$4 indicates greater disagreement; and $\text{accuracy} = 1 - \frac{v}{100}$5 indicates symmetric uncertainty, often observed at intermediate distortion levels. The paper also argues that batch-based cleansing is more appropriate than subject-only filtering because a subject’s behavior may change from one batch to another.

6. Empirical findings, limitations, and practical significance

The reported findings are explicitly characterized as mixed but informative. On the positive side, IDSQS produced a quality scale that broadly tracks distortion level, and it sometimes achieved good agreement with the stronger JPEG AIC-3 BTC-PTC benchmark. For some sources, especially source #2 and source #10, the correlation with the benchmark JND scale was high, with PLCC up to 0.95 and SROCC up to 0.94. The paper interprets this as evidence that IDSQS can detect subtle differences in high-quality images when the content is favorable and the distortions are perceptible enough (Mohammadi et al., 13 Aug 2025).

The aggregated results were weaker. On the full pooled dataset, the reported correlations were PLCC = 0.89, SROCC = 0.88, and Kendall $\text{accuracy} = 1 - \frac{v}{100}$6. Some sources, especially #6 and #9, showed weaker alignment with the benchmark, which the paper attributes to textured regions where artifacts were harder to detect unless boosting was used. The reconstructed DMOS curves were not always perfectly monotonic and included occasional negative scores, meaning that some compressed images were judged better than the source in a few cases.

These results define the practical limits of the methodology. The paper concludes that IDSQS is effective as a practical, lower-cost subjective method for high-quality image assessment, especially when a fully boosted triplet workflow is unnecessary. At the same time, it states that IDSQS does not always match the precision of JPEG AIC-3’s BTC-PTC methods for the most demanding fine-grained JND estimation. This is the main methodological caution associated with IDSQS: it improves sensitivity relative to ordinary direct-rating approaches in the high-quality regime, but it is not presented as uniformly equivalent to boosted triplet methods.

The study materials were made publicly available, including the dataset, subjective data, and graphical user interface, at the repository specified by the paper. The implementation details highlighted for reproducibility include one-image-at-a-time presentation at a fixed position, press-and-hold toggling, a toggle-rate limit of 2 per second, a 1920 × 1080 display requirement, desktop or laptop usage, a continuous 0–100 scale, and the inclusion of training, screening, trap questions, batch-level reliability filtering, and soft reconstruction. Within that design space, IDSQS is positioned as a middle ground between ordinary rating scales and more elaborate boosted comparison procedures.