Reason-Based Artificial Moral Agents

• RBAMAs are artificial agents that leverage explicit normative theories and rule-based systems instead of traditional scalar rewards.
• They combine symbolic and neuro-symbolic reasoning with optimization modules to create dynamic shields that ensure actions are ethically justified.
• Their design incorporates iterative human feedback, enabling robust refinement and transparent logging of moral decision chains.

Reason-Based Artificial Moral Agents (RBAMAs) are artificial agents whose moral actions are governed not by scalarized reward or hard-coded rules, but by explicit representations and computations over normative reasons—structured as formal reason-theories, prioritized default rules, or value-stratified impact matrices—ensuring actions are taken for transparent, internally justifiable reasons. RBAMAs address foundational shortcomings of standard RL and reward-centric “alignment” by making explicit, philosophically grounded reasons the operational core of moral agency. They do so by integrating symbolic or neuro-symbolic reasoning with traditional learning and optimization modules, and by allowing iterative refinement of their moral domain via structured feedback from human supervisors or judges. This architecture not only provides increased robustness and explainability but also offers experimental frameworks for evaluating the genuine “rightness” and justifiability of an agent’s actions.

1. Formal Foundations and Core Architectural Components

At the core of RBAMA design is the transition from scalar utility/reward functions to explicit, rule-based or value-based normative architectures. In foundational RBAMA formulations, the environment is modeled as a labeled Markov Decision Process (MDP) $(S, A, P, R, \gamma, L, l)$, where $L$ is a finite set of atomic, morally relevant labels and $l: S \rightarrow \mathcal{P}(L)$ encodes which facts obtain in each state (Baum et al., 2024). Agents are equipped with:

• Normative Reasons and Rule Structure: A finite set of abstract action types $\Phi$, and a reason theory $\mathcal{T} = (\mathcal{D},<)$ with default rules $\delta \in \mathcal{D}$, each mapping combinations of morally salient facts $X$ to obligations concerning abstract actions $\phi$.
• Reason-Based Shield Generator: At each decision step, a shield generator takes as input the set of current facts $l(s)$ and grounds the current reason theory to output a set of permitted actions $\mathbb{S} \subseteq A$. Only those actions which realize the non-defeated, normatively-best reasons are available for execution.
• Feedback and Iterative Refinement: A moral judge module provides case-based feedback, enabling iterative expansion and reordering of the reason-theory by inserting new defaults and updating the priority order $L$0 among them.

This structure is preserved and extended in neuro-symbolic architectures such as GRACE, which segregate a Moral Module (MM) dedicated to reason-based reasoning and a Decision-Making Module (DMM) for instrumental optimization, with a formal Guard enforcing compliance (Jahn et al., 15 Jan 2026). In value-stratification approaches (e.g., MARS), the agent reasons over a value set $L$1, stratifies values by qualitative importance, and aggregates action impacts into a lexicographically ordered preference profile, selecting actions justified by top-stratum reasons (Badea, 2021).

2. Mathematical Characterizations of Reason-Based Filtering

The mathematical instantiation of the “shield” in RBAMAs proceeds as follows (Baum et al., 2024, Dargasz, 20 Jul 2025):

• For each abstract action $L$2 and state $L$3, the agent computes $L$4: all primitive actions that are first steps on trajectories realizing $L$5.
• Conflict sets $L$6 identify groups of rules whose triggered conclusions are mutually impossible in $L$7.
• The factual background $L$8 is enriched with negations of impossible conclusion sets and combined with the default theory $L$9.
• All proper scenarios $l: S \rightarrow \mathcal{P}(L)$0 (maximal, non-conflicting, non-defeated triggerings) are computed. Each scenario $l: S \rightarrow \mathcal{P}(L)$1 generates a filter $l: S \rightarrow \mathcal{P}(L)$2.
• One proper scenario $l: S \rightarrow \mathcal{P}(L)$3 is selected (e.g., at random in the disjunctive account), and the shield becomes $l: S \rightarrow \mathcal{P}(L)$4.

Agents may only act within $l: S \rightarrow \mathcal{P}(L)$5 at each step. Learning proceeds by prompting the judge module to provide feedback (e.g., $l: S \rightarrow \mathcal{P}(L)$6) when an action fails moral evaluation, expanding the default set and updating priorities so that future scenarios prioritize the corrected interpretation.

3. Comparative Analysis with Alternative Moral Architectures

RBAMAs contrast sharply with reward-shaping and scalarized approaches:

• Standard RL and Ethical Reward Blending: Moral desiderata are encoded as components of a scalar reward; this conflates instrumental and moral value, precludes explicit justification, and typically fails to diagnose when an agent acts “for the right reason” (Baum et al., 2024, Mao et al., 2023).
• Safe-RL and Hard-Coded Shields: These use temporal-logic specifications or static rule-sets, with little learning or explicit normative prioritization (Baum et al., 2024).
• Normative Supervisor and Statistically Aligned AMAs: Supervisory paradigms impose moral verdicts top-down, lacking RBAMA’s capacity for grounding, modular justification, and dynamically learnable prioritizations (Galatolo et al., 18 Aug 2025).
• Value-Based Reasoning and Stratification: Approaches such as MARS implement a stratified, multi-valued evaluation, making possible modeling of utilitarian, deontological, and hybrid paradigms in a single formalism (Badea, 2021).

In neuro-symbolic designs such as GRACE, reason-based containment enables full decoupling of normative reasoning from instrumental agency, enhancing modularity and furnishing hard or statistical guarantees of compliance with normative boundaries (Jahn et al., 15 Jan 2026).

4. Evaluation Protocols and Behavioral Metrics

RBAMAs necessitate new evaluation regimes beyond simple task or alignment success:

• Cost-Insensitivity and “Right Reasons” Behavioral Testing: To distinguish between agents acting out of genuine moral motivation versus mere compliance, evaluation probes the agent’s cost-sensitivity in morally relevant behaviors (e.g., helping at expense) compared to neutral actions. The key metric is the slope or area-under-curve of $l: S \rightarrow \mathcal{P}(L)$7 with respect to cost $l: S \rightarrow \mathcal{P}(L)$8; selective cost insensitivity in moral actions is evidence of principled motivation (Mao et al., 2023). This method is compatible with both black-box and explicitly reason-based agents.
• Deductive and Abductive Moral Reasoning Benchmarks: Static and dynamic benchmarks (e.g., AMAeval) evaluate both explicit chain-of-reasoning verification (deductive) and the model’s capacity to generate context-sensitive precepts from values (abductive). Composite scores aggregate static F1, dynamic accuracy, and penalty for mean absolute error (Galatolo et al., 18 Aug 2025).
• Normative Robustness and Transfer: Empirical studies report that RBAMAs maintain constraint adherence and learned prioritizations across environment variations, demonstrating robustness and modular transferability (Dargasz, 20 Jul 2025).
• Explainability and Justification Logging: Symbolic traces of which reasons—defaults, prioritizations, value-strata—governed an action can be logged as justification objects for later audit and stakeholder challenge (Jahn et al., 15 Jan 2026).

5. Limitations and Open Challenges

Several limitations pertain to current RBAMA implementations and theories:

• Hand-Coded Abstraction and Trajectory Mapping: Most settings require manual specification of mappings from abstract actions $l: S \rightarrow \mathcal{P}(L)$9 to concrete trajectories/actions, limiting scalability and domain transfer (Baum et al., 2024).
• Conflict in Human Feedback and Normative Uncertainty: Inconsistencies among multiple human judges or value sets can yield intractable or unstable prioritizations; mechanisms for multi-judge resolution or aggregation are underdeveloped (Baum et al., 2024, Jahn et al., 15 Jan 2026).
• Computational Tractability: As rule-sets $\Phi$0 grow, enumerating all proper scenarios and updating priorities can become computationally intensive (Baum et al., 2024, Dargasz, 20 Jul 2025).
• Outcome-Insensitivity and Behavioral Masking: RBAMAs may enforce internal justifiability but do not natively guarantee optimality with respect to external, lived ethical consequences; outcome-based checks or feedback loops may need to be integrated (Dargasz, 20 Jul 2025).
• Interpretation Problem and Value Learning: The inherent underdetermination of rule-following by symbolic rules—articulated via the Interpretation Problem—remains a threat to the integrity of reason-based moral reasoning. Explicit value-representation, demonstration learning (“show not tell”), and relational value learning constitute important but incomplete mitigations (Badea et al., 2021).

6. Empirical Instantiations and Research Directions

RBAMA implementation has matured through several experimental paradigms:

• Bridge Environments and Dilemma Scenarios: Classic grid-world dilemmas demonstrate policy convergence, prioritization learning (e.g., rescue over non-interference), and rapid adaptation to altered constraints (Baum et al., 2024, Dargasz, 20 Jul 2025).
• Meta-RL for Behavioral Inference: Cost-insensitivity benchmarks are deployable in both meta-RL and explicit reasoning agents, offering practical metrics for “doing the right thing for the right reason” (Mao et al., 2023).
• Neuro-Symbolic and Modular Architectures: GRACE-layered systems illustrate the practical decoupling of normative and instrumental logic, with symbolic guards and formal verification applied to real-world scenarios such as LLM-based therapy (Jahn et al., 15 Jan 2026).
• Value-Stratified Decision Systems: MARS exemplifies flexible paradigm-instantiation (utilitarian, Kantian, hybrid) via its explicit stratification-and-aggregation mechanism, ensuring explainability and value-traceability (Badea, 2021).

Research continues along axes of automated abstraction learning, multi-judge consensus, scalable scenario reasoning, relational norm synthesis, and integration of abductive/deductive benchmarks for LLM-enabled AMAs (Galatolo et al., 18 Aug 2025).

7. Philosophical and Methodological Significance

The RBAMA program operationalizes long-standing philosophical demands for transparent justification and intrinsic motivation in artificial moral agency. By anchoring conduct in explicit, defeasible reason-sets and exposing the full chain from context to decision, RBAMAs not only improve explainability and contestability but also render themselves amenable to formal verification and safety analysis (Jahn et al., 15 Jan 2026). They address central theoretical risks—such as the Interpretation Problem and the modularity of “mistakes of intention” versus “instrumental mistakes”—by separating normative modules from instrumental ones and incorporating ongoing “character” formation through feedback and demonstration (Badea et al., 2021).

A plausible implication is that future RBAMA systems will require ever-richer models of human value, norm negotiation, and reasoning pattern acquisition, bridging symbolic rigor with empirical robustness and adapting to evolving normative landscapes.