Soft Satisfaction Rate (SSR) Overview

• Soft Satisfaction Rate (SSR) is a metric that quantifies partial fulfillment of soft constraints using weighted necessity measures in both CSPs and user satisfaction models.
• In SERP evaluations, SSR predicts user satisfaction via a logistic transformation of cumulative utility from attention and relevance signals, showing improved correlation over traditional metrics.
• SSR integrates with algorithms like branch-and-bound and forward checking, optimizing decision-making by efficiently pruning based on utility-derived upper bounds.

The Soft Satisfaction Rate (SSR) is a real-valued metric central to multiple domains where constraint satisfaction or user experience is inherently “soft”—that is, not all constraints or objectives are imperative, and partial satisfaction must be quantified. SSR explicitly models and aggregates such partial fulfillment, yielding a scalar in [0,1], and is formally defined both in the context of possibilistic constraint satisfaction problems (possibilistic CSPs) and user-centric evaluation of ranked output such as search engine result pages (SERPs). In both domains, SSR encodes and rationalizes incomplete or uncertain satisfaction, with configurable importance weights, and can incorporate learned or subjective evidence. The metric is analytically tractable and is compatible with classical optimization and filtering methods (Schiex, 2013, Chuklin et al., 2016).

1. Formal Definition Across Domains

In possibilistic CSPs, SSR is defined for a complete labeling $\ell$ over variables $X=\{x_1,\ldots,x_n\}$, subject to hard and soft constraints. Each soft constraint $c_i$ is associated with a necessity lower bound $\alpha_i\in(0,1)$, encoding its importance or certainty requirement (Schiex, 2013). For any labeling,

$$SSR(\ell) = \frac{ \sum_{i=1}^m \alpha_i \; \sigma_{c_i}(\ell)}{ \sum_{i=1}^m \alpha_i }$$

where $\sigma_{c_i}(\ell)=1$ iff $\ell$ satisfies $c_i$, otherwise 0. The SSR expresses the fraction (importance-weighted) of soft constraints satisfied.

In the context of SERP evaluation, SSR is the predicted probability of user satisfaction $P(S=1)$ under a click–attention–satisfaction (CAS) model (Chuklin et al., 2016). The cumulative expected utility is computed via

$$U = \sum_{i=1}^n \varepsilon(\vec{\varphi}_i) u_d(D_i) + \varepsilon(\vec{\varphi}_i)\alpha(R_i)u_r(R_i)$$

where $\varepsilon(\vec{\varphi}_i)$ is the examination probability, $u_d(D_i)$ is utility from snippet relevance, $u_r(R_i)$ is utility from clicked documents, and $\alpha(R_i)$ encodes attractiveness. The SSR is the output of a logistic transformation:

$$SSR = P(S=1) = \sigma(\tau_0 + U) = \frac{1}{1+\exp(-(\tau_0 + U))}$$

In both formulations, SSR lies in $[0,1]$ and quantifies the overall “soft” fulfillment of the modeled constraints or objectives.

2. Possibilistic CSPs: SSR Construction and Interpretation

Within possibilistic CSPs (Schiex, 2013), variables range over finite domains, and the constraint set $C = C_{hard} \cup C_{soft}$ partitions strict requirements from those where partial, uncertain, or varying significance is allowed. Each soft constraint $c$ is annotated with a necessity value $\alpha_c$, interpreted as a lower bound on the necessity measure $N(c)$ associated with a possibility distribution $\pi$. The SSR for labeling $\ell$ reflects the weighted count of soft constraints it satisfies, normalized to $[0,1]$:

$$SSR(\ell) = \frac{\sum_{i:\ \ell \models c_i} \alpha_i}{\sum_{i=1}^m \alpha_i}$$

Thus, SSR quantifies not only how many soft constraints are satisfied, but does so with respect to their necessity weights, allowing a graded assessment that distinguishes failures in critical versus peripheral aims.

3. SSR in User-Centered Retrieval Evaluation

SSR also appears in modern user-centric offline metrics for information retrieval, most notably the CAS model for SERP evaluation (Chuklin et al., 2016). There, SSR is the predicted probability that a real user would self-report satisfaction after interacting with a search result page. This prediction uses a latent utility model that combines:

• Attention: Items receive attention weight via a logistic function of display features ($\varepsilon(\vec{\varphi}_i)$).
• Snippet/Document Utility: Utility is accrued both from direct snippet relevance ($u_d(D_i)$) and full-document relevance via clicks ($u_r(R_i)$).
• Item Attractiveness: Probability of clicking conditional on examination ($\alpha(R_i)$).

The total utility $U$ is transformed via a sigmoid to yield SSR. This probabilistic SSR reflects partial satisfaction, accommodates non-linear page layouts, and captures phenomena like “good abandonment”—where, for example, a user finds their answer in a snippet without clicking, yet is “satisfied.”

4. Algorithmic Integration and Optimization

SSR is directly embedded into feasible algorithms adapted from the CSP framework (Schiex, 2013):

• Branch-and-bound backtracking exploits upper bounds on SSR to prune search subtrees. For a partial labeling $\ell_p$, its upper bound:

$$UB(\ell_p) = \frac{ \sum_{\text{already satisfied } c_i} \alpha_i + \sum_{\text{not yet decided } c_i} \alpha_i }{ \sum_{i=1}^m \alpha_i }$$

enables cutting off any extension that cannot exceed the globally best SSR found so far.

• Forward checking and arc-consistency filtering are adjusted to propagate soft-constraint weights, pruning assignments or values that cannot improve approximate or global SSR thresholds. For variable value $v \in D_i$:

$$ub_i(v) = \frac{ \sum_{c \in C_{soft}:\ c \text{ is already satisfied by } (x_i=v)} \alpha_c + \sum_{c \ni x_i,\ c\text{ not yet decided}} \alpha_c }{ \sum_{c\in C_{soft}} \alpha_c }$$

Practical computation of SSR in the CAS model is based on fitted model parameters and labeled rater data, with parameter estimation performed via joint log-likelihood maximization with $L_2$ regularization (Chuklin et al., 2016).

5. Properties, Comparative Metrics, and Empirical Performance

SSR as formalized in (Schiex, 2013) and (Chuklin et al., 2016) offers the following properties:

• Range: Bounded in [0,1], enabling straightforward interpretation and comparability.
• Weighting: Reflects heterogeneous importance via necessity weights or utility functions.
• Handles Partiality: Accommodates the reality that not all constraints or user intents can or need to be fully accomplished.

Empirical results in SERP evaluation show that SSR (CAS-metric) correlates with held-out self-reported satisfaction significantly better (Pearson $\rho\sim 0.45$–$0.60$) than baseline click or DCG-based metrics (which yield $\rho\sim 0.2$–$0.3$) (Chuklin et al., 2016). The improvement is especially marked on pages with complex layouts or high rates of answer-panel fulfillment (“good abandonment”).

6. Illustrative Examples

Possibilistic CSP Example

Consider $X=\{x,y\}$, $D_x=D_y=\{0,1\}$; soft constraints $c_1:x=0$ with $\alpha_1=0.8$, $c_2:y=1$ with $\alpha_2=0.6$, hard constraint $c_3:x=y$. For labelings satisfying $c_3$:

• $\ell_2 = (0,0)$: $\ell_2\models c_1$, $\ell_2 \not\models c_2$; $SSR(\ell_2) = 0.8/1.4 \approx 0.571$.
• $\ell_3 = (1,1)$: $\ell_3 \not\models c_1$, $\ell_3 \models c_2$; $SSR(\ell_3) = 0.6/1.4 \approx 0.429$.

The best SSR is $0.571$, attained by $\ell_2$ (Schiex, 2013).

SERP Evaluation Example

Consider three items: two blue-links ($i=1,2$), one answer-panel ($i=3$), with $D_1=1,\ R_1=2$; $D_2=0,\ R_2=1$; $D_3=2,\ R_3=0$, $\varepsilon(\vec{\varphi}_1)=0.6$, $\varepsilon(\vec{\varphi}_2)=0.3$, $\varepsilon(\vec{\varphi}_3)=0.8$. Using $u_d(D)=D\times 0.5$, $u_r(R)=R\times 0.3$, $\alpha(R)=\sigma(-1+0.5R)$, and bias $\tau_0=-1$:

$$U = 1.314 \quad \rightarrow \quad SSR = \sigma(-1+1.314) \approx 0.578$$

indicating a 57.8% predicted probability of user satisfaction for that SERP (Chuklin et al., 2016).

7. Context, Extensions, and Related Metrics

SSR generalizes traditional Boolean feasibility notions, enabling nuanced assessment of solution quality or user experience in both theoretical and operational settings. In constraint reasoning, it allows integration of uncertainty via possibility distributions and necessity-bounded soft constraints (Schiex, 2013). In user-facing evaluation, SSR yields interpretable performance measures, overcoming the limitations of click-based or purely ordinal metrics, especially in environments where non-click utility is significant (“good abandonments”) and layouts violate linear or sequential assumptions (Chuklin et al., 2016).

A plausible implication is that similar SSR constructions can be adapted to other domains where satisfaction is aggregative, weights are heterogeneous, and not all objectives are strictly enforced. Careful calibration of necessity weights or utility functions is critical, and empirical validation against ground-truth or user-reported outcomes remains essential.

---

References:

• "Possibilistic Constraint Satisfaction Problems or 'How to handle soft constraints?'" (Schiex, 2013)
• "Incorporating Clicks, Attention and Satisfaction into a Search Engine Result Page Evaluation Model" (Chuklin et al., 2016)