Independence Is Not an Issue in Neurosymbolic AI

Abstract: A popular approach to neurosymbolic AI is to take the output of the last layer of a neural network, e.g. a softmax activation, and pass it through a sparse computation graph encoding certain logical constraints one wishes to enforce. This induces a probability distribution over a set of random variables, which happen to be conditionally independent of each other in many commonly used neurosymbolic AI models. Such conditionally independent random variables have been deemed harmful as their presence has been observed to co-occur with a phenomenon dubbed deterministic bias, where systems learn to deterministically prefer one of the valid solutions from the solution space over the others. We provide evidence contesting this conclusion and show that the phenomenon of deterministic bias is an artifact of improperly applying neurosymbolic AI.

• The paper demonstrates that using full semantic loss—including both positive and negative examples—eliminates the deterministic bias observed with truncated loss settings.
• Empirical analysis on a traffic light constraint dataset reveals that excluding negative supervision leads to a winner-take-all phenomenon in model predictions.
• The study unifies neurosymbolic AI semantic loss with disjunctive supervision, providing insights that guide the design of robust, constraint-based neurosymbolic systems.

Independence Assumptions and Deterministic Bias in Neurosymbolic AI: An Analysis

Theoretical Grounding and Problem Statement

The integration of symbolic reasoning with neural architectures, commonly termed neurosymbolic (NeSy) AI, has facilitated interpretable and data-efficient learning via logical constraints imposed on neural predictions. A central technique in this domain is the semantic loss, which encodes logical requirements as differentiable constraints [manhaeve2018deepproblog, xu2018semantic]. Recent critiques, notably van Krieken et al., have argued that an independence assumption in variable modeling produces an undesirable deterministic bias—systems converge to selecting one valid solution while ignoring other equally valid alternatives.

This paper rigorously deconstructs that critique. It formalizes the relationship between the semantic loss underpinning NeSy AI and the more general disjunctive supervision framework [zombori2024towards], highlighting key differences in supervision and the consequences for inductive biases and solution diversity.

Unification of Neurosymbolic AI and Disjunctive Supervision

The paper’s first result formally demonstrates that the neurosymbolic semantic loss is a special case of disjunctive supervision. Classical disjunctive supervision allows training data to be labeled with sets of valid outputs, rather than a unique label. The loss is computed as the negative log-likelihood over all acceptable outputs. In contrast, neurosymbolic approaches create logical formulas for each class, which are usually mutually exclusive and exhaustive, imposing stricter constraints and effectively converting the disjunctive supervision loss into a sparse, structured variant.

However, significant distinctions exist. Standard disjunctive supervision typically models the final output probabilities using softmax-based parameterizations. NeSy methods, in contrast, induce a structure via the "deterministic systems and independent choices" (DSIC) principle. Each random variable (logical atom) is modeled independently, and world probabilities are computed as products over these variables, integrating logic and probability more explicitly.

Winner-Take-All and Deterministic Bias Phenomena

The paper empirically investigates two key phenomena:

1. Winner-Take-All (WTA) in Disjunctive Supervision: As established by Zombori et al., optimizing a softmax-parameterized network in a disjunctive supervision setting can yield a WTA outcome: one "acceptable" output dominates the predictive distribution, capturing nearly all the probability mass, regardless of the actual semantic indeterminacy of the task.
2. Deterministic Bias in NeSy AI: van Krieken et al. posited that the DSIC-based conditional independence in NeSy systems similarly induces a deterministic bias, particularly when only positive (constraint-satisfying) examples are provided during training and the semantic loss is truncated to exclude negatives.

The paper clarifies the source of the deterministic bias. It shows through detailed experimentation that standard application of the semantic loss, which incorporates both positive and negative examples, does not exhibit the pathologic WTA or deterministic bias effects. However, when negatives are excluded (as in van Krieken et al.’s "truncated semantic loss"), deterministic bias emerges as predicted.

Empirical Analysis

The authors employ a canonical "traffic light" constraint dataset: two binary indicators (red, green) mapped to MNIST digits, where at most one light can be on at once.

• Disjunctive Supervision: Trained with softmax outputs over possible worlds, the network exhibits sharp convergence to a single world across all positive cases, consistent with the WTA theorem.

Figure 1: Experimental evaluation with disjunctive supervision, showing probability collapse to a single world in each valid configuration, demonstrating the WTA effect.

• Semantic Loss (with Negatives): Two neural networks independently predict red and green, and the semantic loss (incorporating both constraint-satisfying and -violating cases) is used. Here, the system consistently maintains appropriate uncertainty and distributes probability across all valid worlds per input, without biasing toward a single solution.

Figure 2: Evaluation for the traffic lights example using the semantic loss, with correct worlds maintained across configurations and no deterministic collapse.

• Semantic Loss (Truncated): Repeating the experiment with only positive examples (the setup critiqued in van Krieken et al.), the deterministic bias reappears: the system deterministically favors a single world, failing to maintain solution diversity.

Figure 3: Training with the truncated semantic loss (positives only) causes collapse to one world, failing to capture the full solution set, indicative of deterministic bias.

Technical and Theoretical Implications

The work overturns prior claims that conditional independence in neurosymbolic AI inherently induces pathologies. Instead, improper application of learning objectives—particularly omitting negative supervision—explains deterministic bias and solution collapse. The conditional independence assumption, when paired with semantic loss incorporating negatives, does not preclude correct probabilistic reasoning over multiple valid solutions.

Further, for Turing-complete probabilistic programming frameworks (such as DeepProbLog), the DSIC factorization does not limit probabilistic expressivity; arbitrary distributions can be represented. For systems with finite vocabularies, there are limitations, but these are not germane to the typical neurosymbolic applications discussed.

Finally, the empirical results also align with the theoretical diagnosis of reasoning shortcuts: underspecified constraints or losses give rise to degenerate learning dynamics—an observation with practical import for designing neurosymbolic datasets, constraints, and loss functions.

Practical Implications and Future Research Directions

The results directly inform the construction and training of neurosymbolic models for structured prediction and knowledge-guided learning. The necessity of including negative instances when enforcing logical constraints via loss function is clear; omitting them fundamentally alters learning dynamics and undermines solution diversity.

Future avenues include:

• Examining richer constraint classes and their effect on solution space diversity.
• Investigating the impact of more expressive probabilistic dependencies (going beyond DSIC) in domains where representational adequacy is provably necessary.
• Exploring reasoning shortcuts in larger, more complex neurosymbolic systems to develop automated diagnostics for constraint sufficiency.

Conclusion

The paper establishes that conditional independence is not pathological in neurosymbolic AI when semantic loss is applied fully (including both positives and negatives), as opposed to prior claims based on truncated loss formulations. The Winner-Take-All and deterministic bias effects arise from underspecified learning setups, not from the independence assumption itself. This insight has significant ramifications for the theoretical understanding and practical design of neurosymbolic learning systems.