Pseudo-Semantic Loss (PSL)

• Pseudo-Semantic Loss (PSL) is a family of loss functions that uses pseudo-labels or local surrogates to enforce semantic structure in model outputs, enabling learning under weak supervision or limited labels.
• In neuro-symbolic settings, PSL replaces intractable global likelihoods with efficient local pseudolikelihood approximations, using techniques like circuit compilation to handle logical constraints.
• For semi-supervised segmentation, PSL employs a local contrastive loss that clusters pixel embeddings via pseudo-labels, leading to improved performance with minimal annotated data.

Pseudo-Semantic Loss (PSL) is a class of loss functions developed to facilitate learning under weak supervision, logical constraints, or limited labeled data by leveraging pseudo-labels or local surrogates for intractable likelihoods. Two prominent manifestations of PSL appear in the contexts of neuro-symbolic learning for autoregressive models with logical constraints (Ahmed et al., 2023), and in local pixel-level contrastive learning for semi-supervised medical image segmentation (Chaitanya et al., 2021). Both exploit pseudo-labels or local approximations to impose semantic structure on neural network outputs in a computationally tractable manner.

1. Conceptual Foundations

Pseudo-Semantic Loss arises when imposing semantic or logical structure directly on a model's output distribution is computationally prohibitive. In the neuro-symbolic setting, the probability under an autoregressive model that a generated sequence satisfies a constraint $\alpha$ is #P-hard to compute, even for simple $\alpha$. PSL replaces this intractable global likelihood with a surrogate, typically based on local approximations or pseudo-labels, that is amenable to efficient optimization. In semi/self-supervised learning, PSL denotes objectives where pseudolabels—labels imputed via network predictions—enable contrastive or cluster-forming objectives at a pixel level, circumventing the need for exhaustive manual annotation.

2. Methodology: Neuro-Symbolic Learning with Logical Constraints

In the context of autoregressive sequence models, PSL provides a rigorous procedure to optimize for logical correctness under hard constraints:

1. Model and Constraint The autoregressive model defines $$p_\theta(y) = \prod_{i=1}^{n} p_\theta(y_i | y_{<i})$$. The logical constraint $\alpha$ represents the desired property (e.g., Sudoku validity, no toxic word usage).
2. Pseudolikelihood Surrogate PSL leverages the classic pseudolikelihood approximation:

$$p_\theta(y) \approx \tilde{p}_\theta(y) \equiv \prod_{i=1}^n p_\theta(y_i | y_{-i}),$$

with $y_{-i}$ denoting all coordinates except $i$. The target then becomes maximizing $$\tilde{p}_\theta(\alpha) = \sum_{y \models \alpha} \tilde{p}_\theta(y)$$.

1. Local Pseudolikelihood Around Samples To further relax computation, PSL centers the approximation at a single sample $y \sim p_\theta$, considering only single-site perturbations. The local pseudolikelihood is defined by

$$\tilde{p}_{\theta, y}(\hat{y}) \equiv \prod_{i=1}^n p_\theta(\hat{y}_i | y_{-i})$$

where $\hat{y}$ differs from $y$ only at one position.

2. Loss Definition The pseudo-semantic loss is:

$$PSL(\alpha; p_\theta) \equiv -\log \mathbb{E}_{y \sim p_\theta} \sum_{\hat{y} \models \alpha} \prod_{i=1}^n p_\theta(\hat{y}_i | y_{-i}).$$

1. Efficient Computation The sum over valid $\hat{y}$ is efficiently computed by compiling $\alpha$ into a deterministic, decomposable, smooth logical circuit $c_\alpha$. The evaluation is a single bottom-up pass through $c_\alpha$, with computation scaling linearly in the circuit's size.
2. Optimization Algorithm Each training step samples $y$, generates all Hamming-one perturbations, computes joint log-probabilities, evaluates the circuit, applies the loss, and backpropagates through local conditionals.

Such a procedure enables semantic steering of models away from locally inconsistent or logically invalid outputs, especially impactful where global likelihood computation is infeasible (Ahmed et al., 2023).

3. Methodology: Pseudo-Semantic Loss via Local Contrastive Objectives

In semi-supervised segmentation, PSL is realized as a local, pixel-wise contrastive loss utilizing both ground-truth and pseudo-labels:

1. Pixel Embedding Formation A segmentation backbone $c_\theta(x)$ and projection head $h_\phi$ yield embedded pixels $z_i(x)\in\mathbb{R}^D$ for each spatial location.
2. Positive and Negative Set Construction Pixels with the same (pseudo-)label $c$ in either labeled ($y$) or pseudo-labeled ($\hat{y}$) images are grouped. For computational tractability, random subsampling per class occurs.
3. Contrastive Loss For randomly paired image embeddings, the pixel-level loss encourages high similarity between anchors and prototypes of the same class, and low similarity with those of different classes:

$$\ell_{i,c}(a,b) = - \log \frac{\exp[\text{sim}(a,b) / \tau]}{\exp[\text{sim}(a,b) / \tau] + \sum_{k\ne c} \exp[\text{sim}(a, \mu_k(x')) / \tau]}$$

where $\text{sim}(u,v) = u^\top v / (\|u\|\|v\|)$.

1. Self-Training and Joint Optimization The network alternates warm-up training on labeled data and joint self-training stages where pseudo-labels are used for the contrastive loss but not for direct segmentation loss. Pseudo-labels are periodically updated as the model improves.
2. Hyperparameter Robustness and Ablations The method is robust to embedding dimensionality, sampling rates, and contrastive loss weight, with only minor sensitivity in ablation studies (Chaitanya et al., 2021).

This instantiation of PSL ensures that pixel embeddings form tight, well-separated clusters even when only a tiny proportion of the training data is labeled.

4. Empirical Outcomes and Utility

PSL has demonstrated practical impact across distinct domains:

Application	Baseline	+Semantic Loss	+PSL
Sudoku (RNN, 9x9, 10 missing)	22.4% (exact)	22.1% (exact)	28.2% (exact)
Shortest Path (ResNet-18/LSTM)	55.0% (exact)	59.4% (exact)	66.0% (exact)
LLM Detox (GPT-2, ToxicProb)	34.1%	14.1% (SGEAT)	9.8%
Semi-Supervised Seg. (DSC, 1 vol)	0.69	—	0.76

• In Sudoku, PSL yielded a substantial increase in exact-match accuracy compared to both unconstrained and factorized semantic-loss baselines.
• For shortest-path prediction with combined ResNet-18 and LSTM architectures, PSL further lifted the percentage of exact and consistent outputs.
• In LLM detoxification, GPT-2 fine-tuned with PSL achieved the lowest ExpectedMaxToxicity and ToxicityProbability, surpassing the state-of-the-art SGEAT with minimal language modeling penalty (Ahmed et al., 2023).
• In medical image segmentation, adding the local pseudo-semantic contrastive loss to standard self-training raised mean DSC from approximately 0.69 to 0.76 with only one labeled 3D volume, illustrating strong gains in the low-label regime (Chaitanya et al., 2021).

5. Computational Complexity and Practical Considerations

In neuro-symbolic PSL, per-step complexity is dominated by $O(nk)$ model evaluations—where $n$ is sequence length, $k$ is vocabulary size—plus $O(|c_\alpha|)$ circuit evaluation. For large $n$ or $k$, computations may require top-$k$ filtering or subsampling. The hard constraint must be amenable to compilation into a practical circuit $c_\alpha$; extremely large, unstructured constraints may become intractable.

For the local contrastive PSL in segmentation, batch size, anchor subsampling per class, and embedding dimension are all tunable for memory and speed tradeoffs. The explicit avoidance of pseudo-labels in the segmentation branch minimizes overfitting to noisy labels in early training.

6. Limitations and Directions for Extension

PSL, as instantiated in the neuro-symbolic paradigm, is currently restricted to hard logical constraints that can be compiled into circuits. Extensions to soft or distributional constraints remain an open avenue. The reliance on a single-sample Monte Carlo introduces variance; variance reduction techniques or multi-sample estimators could enhance stability. Scalability for very large constraints or vocabulary sizes may benefit from advances in circuit compilation and sampling.

For the contrastive PSL, the approach presumes pseudo-label quality sufficient to enforce meaningful clustering, but early stages may reflect high noise. Nonetheless, robustness to contrastive weighting and pseudo-label update frequency has been empirically shown (Chaitanya et al., 2021).

A plausible implication is that hierarchical or relational constraints, if compiled into lifted circuits, could further broaden the applicability of PSL in neuro-symbolic generative modeling (Ahmed et al., 2023).

7. Relationship to Broader Research Themes

PSL occupies a key position at the intersection of neuro-symbolic reasoning, self-supervised learning, and constrained generative modeling. Unlike traditional maximum-likelihood-based logical regularization, PSL leverages local, factorized approximations and pseudo-label dynamics to propagate semantic coherence efficiently in high-dimensional output spaces. Its effectiveness across sequence generation, structured prediction, and semi-supervised segmentation demonstrates adaptability to a broad class of models and tasks. The method's reliance on knowledge compilation and circuit evaluation links it closely to developments in satisfiability solving and probabilistic inference.

References:

• "A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints" (Ahmed et al., 2023)
• "Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation" (Chaitanya et al., 2021)