Multi-Objective Affective Learning (MAL)

• Multi-Objective Affective Learning (MAL) is a framework that jointly optimizes affective prediction or generation alongside auxiliary objectives such as fairness, robustness, and user preference alignment.
• It leverages methodologies like meta-learning, bilevel optimization, and Pareto-based techniques to balance primary affective tasks with additional criteria in diverse applications.
• MAL is applied in tasks ranging from emotion classification and facial action unit detection to affective text generation, achieving improved trade-offs and calibrated outputs.

Multi-Objective Affective Learning (MAL) can be understood as a broad umbrella for affective computing problems in which learning is driven by more than one objective simultaneously rather than by a single affective loss. In the current literature, this includes affective prediction with explicit fairness constraints, affective multi-task learning with auxiliary supervision, multimodal and multi-corpus representation learning, Pareto-based affective text generation, and alignment-style formulations in which affect-relevant qualities are decomposed into separate evaluators and recombined during training or inference (Sen et al., 2020, Li et al., 2021, Resendiz et al., 2024, Ye et al., 2021). MAL is therefore less a single standardized algorithm than a family of formulations in which affective competence is optimized jointly with additional desiderata such as bias reduction, robustness, calibration, controllability, user preference alignment, or domain appropriateness.

1. Conceptual scope and terminology

The literature associated with MAL is terminologically heterogeneous. Some papers are directly affective but do not use the term “Multi-Objective Affective Learning,” while one paper uses the acronym “MAL” to denote “Meta Auxiliary Learning,” a specific method for facial action unit detection rather than a field-level concept (Li et al., 2021). A useful synthesis is that MAL denotes affective learning settings in which the objective is vector-valued, structurally composite, or explicitly trade-off driven.

Representative formulation	Affective setting	Objective configuration
"Towards Socially Responsible AI: Cognitive Bias-Aware Multi-Objective Learning" (Sen et al., 2020)	Emotion classification from text	correctness + reduction of socially harmful bias in predicted emotions
"Meta Auxiliary Learning for Facial Action Unit Detection" (Li et al., 2021)	AU detection with FER auxiliary task	primary AU detection + meta-weighted auxiliary FE supervision
"Detection of Real-world Driving-induced Affective State Using Physiological Signals and Multi-view Multi-task Machine Learning" (Lopez-Martinez et al., 2019)	Binary high/low stress from EDA and HR	affect classification + drive-profile sharing + modality weighting
"MOPO: Multi-Objective Prompt Optimization for Affective Text Generation" (Resendiz et al., 2024)	Affective text generation	multiple domain-specific emotion objectives
"Assessing Affective Objectives for Communicative Visualizations" (Lee-Robbins et al., 1 Apr 2026)	Communicative visualization design	multiple cognitive and affective learning objectives

Within this broad framing, MAL spans both prediction and generation. On the predictive side, affective tasks include single-label emotion classification, AU detection, facial expression recognition, driving stress detection, and binary arousal classification from wearable signals (Sen et al., 2020, Li et al., 2021, Lopez-Martinez et al., 2019, Ross et al., 2020). On the generative and alignment side, affective objectives appear as domain-specific emotion-expression criteria, principle-specific preference models, or objective-specific heads for subjective and interactive qualities such as engagement, helpfulness, honesty, and truthfulness (Resendiz et al., 2024, Williams, 2024, Shen et al., 1 Oct 2025).

A second conceptual distinction concerns what the additional objectives represent. In some formulations, the auxiliary objective is normative, as in fairness or social responsibility; in others it is structural, as in multi-view fusion, domain generalization, or policy-set coverage; in still others it is user-facing, as in emotional appropriateness, empathy, or engagement. This suggests that MAL is best characterized by the coexistence of an affective objective with at least one additional criterion that cannot be reduced to ordinary affect-label fit without loss of information.

2. Mathematical patterns and optimization regimes

A recurring mathematical pattern in MAL is the separation between a primary affective predictor and one or more auxiliary objectives defined over the same representation. In bias-aware emotion classification, the primary classifier predicts one of five emotions—fear, anger, joy, sadness, or neutral—while auxiliary bias heads are trained on pseudo-labels that mark sensitive emotion-identity associations such as women → fear or men → anger (Sen et al., 2020). The shared neural model is written as

$$\hat{y} = \mathrm{softmax}(\Theta_p(\Theta_s(\vec{x}))), \qquad \hat{y^B_i} = \mathrm{sigmoid}(\Theta^B_i(\Theta_s(\vec{x}))),$$

with joint objective

$$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$

The negative sign on the bias terms is the decisive multi-objective mechanism: the model is encouraged to do well on emotion prediction and poorly on the bias pseudo-tasks simultaneously (Sen et al., 2020).

A second pattern is bilevel or meta-weighted affective learning. In Meta Auxiliary Learning for AU detection, the total loss is

$$\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$$

where the sample weights are predicted by a meta network and updated so that the resulting training step improves AU validation loss (Li et al., 2021). This is not symmetric multi-task learning. The auxiliary FER objective is retained only insofar as it improves the primary AU task, which makes the method a primary-task-driven instance of MAL rather than a generic equal-status multi-task model.

A third pattern is explicit multi-objective bilevel optimization. "Multi-Objective Meta Learning" formulates the general problem as

$$\min_{\alpha\in \mathcal{A}, \omega \in \mathbb{R}^p} F(\omega,\alpha) \quad \mathrm{s.t.} \quad \omega\in \mathcal{S}(\alpha), \qquad \mathcal{S}(\alpha)=\arg\min_\omega f(\omega,\alpha),$$

and uses MGDA to compute a common descent direction across upper-level objectives without fixing scalarization coefficients in advance (Ye et al., 2021). Although the paper is not affect-specific, it directly supplies a generic MAL template when affective adaptation must be balanced against robustness, fairness, calibration, privacy, or cross-modal consistency.

A fourth pattern is Pareto search or vectorized alignment rather than fixed scalarization. MOPO uses NSGA-II and non-domination over domain-specific emotion objectives rather than a weighted sum, retaining a set of prompts corresponding to different trade-offs (Resendiz et al., 2024). By contrast, MORLAIF explicitly scalarizes principle-specific preference-model outputs for PPO training, while "Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards" keeps objective dimensions separate through multi-action heads and only mixes them at inference time through

$$\pi_{\text{MAH}}(y_t \mid x,y_{<t}) = \sum_{i=1}^{H} w_i \pi_{\theta_b,W_i}(y_t \mid x,y_{<t})$$

(Williams, 2024, Shen et al., 1 Oct 2025). Together these papers show that MAL includes both Pareto-preserving and scalarization-based regimes, with the design choice depending on whether downstream control should expose a menu of trade-offs or commit to a single scalar objective.

3. Affective prediction under fairness, auxiliary supervision, and structured sharing

The clearest affective fairness exemplar is bias-aware emotion classification from text on the Equity Evaluation Corpus. The task is single-label emotion prediction from sentences, the identity attributes are gender and race/ethnicity, and the sensitive output categories are selected rather than global. The paper explicitly notes that some emotion-identity pairs, such as joy versus gender, have nearly uniform posterior distributions and are therefore not treated as sensitive pairs for debiasing (Sen et al., 2020). This output-specificity is conceptually important: in MAL, fairness need not be imposed uniformly over every affective label.

The main empirical result is that balanced affective data do not prevent stereotypical outputs. On the balanced EEC split, Bias-Agnstc-L2 attains $A=0.8620$ while showing $\alpha_{\mathrm{female}}=0.7914$ for fear and $\alpha_{\mathrm{male}}=0.3806$ for anger; Bias-Agnstc gives $A=0.8237$ with $\alpha_{\mathrm{female}}=0.7971$ for fear and $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$0 for anger (Sen et al., 2020). The bias-aware variant Bias-Awr-Gndr moves the female-fear posterior to $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$1, near parity, while increasing accuracy to $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$2. The same paper evaluates fairness with

$$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$3

and combines accuracy and fairness as

$$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$4

This suggests a characteristic MAL lesson: affective and normative objectives may be competitive in principle but empirically cooperative when the model is forced to discard identity-correlated nuisance structure (Sen et al., 2020).

A different but closely related form of MAL appears in AU detection with FER as auxiliary supervision. Here the affective outputs are heterogeneous: AU detection is multi-label and FER is single-label. The central problem is negative transfer, since FER images can be classified by gaze, head pose, gesture, or facial appearance rather than AU-relevant muscle activations (Li et al., 2021). Meta Auxiliary Learning addresses this by reserving a small AU validation split and learning sample weights for both AU and FE examples so that the weighted training update improves AU validation performance. On BP4D, DISFA, and GFT, MAL improves average F1 to 62.2, 58.0, and 59.8, compared with STL baselines of 59.2, 51.3, and 53.0 respectively (Li et al., 2021). The paper’s own acronym “MAL” therefore names a specific meta-learning mechanism, but it also exemplifies the broader MAL pattern of optimizing a hard affective task with the controlled help of an easier related one.

Structured sharing also appears in real-world driving stress detection. "Detection of Real-world Driving-induced Affective State Using Physiological Signals and Multi-view Multi-task Machine Learning" treats high/low stress recognition as a multi-view, multi-task problem: EDA and HR are two views, and latent drive-profile clusters become tasks (Lopez-Martinez et al., 2019). The model learns task-specific kernel mixtures over views while coupling tasks through regularization on the modality-weight vectors. Best accuracies rise from 0.85 to 0.93 on MIT drivedb, from 0.64 to 0.71 on HciLab, and from 0.70 to 0.83 on AffectiveROAD. In MAL terms, the important point is that “multiple objectives” may be realized not only as multiple labels but also as joint optimization of affect classification, modality weighting, and profile-aware sharing (Lopez-Martinez et al., 2019).

4. Multimodal, cross-domain, and representation-centric MAL

A large portion of the MAL literature is representation-centric rather than explicitly vector-optimized. "Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Data" learns separate stacked convolutional autoencoders for ECG and EDA, concatenates their 80-dimensional latent vectors into a 160-dimensional fused representation, and trains a random forest for binary arousal classification on an aggregated dataset comprising AMIGOS, ASCERTAIN, CLEAS, and MAHNOB-HCI (Ross et al., 2020). The method is two-stage rather than end-to-end multi-objective, but it is strongly MAL-relevant because it combines multiple modalities, multiple corpora, reduced reliance on labels, and improved generalization. Its best feature-level fusion result reaches overall $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$5 accuracy and $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$6 F1, outperforming separated per-dataset training and both hand-crafted and supervised CNN baselines (Ross et al., 2020). This suggests that, in affective settings with heterogeneous annotation schemes, unsupervised modality-specific objectives can function as practical building blocks for broader MAL systems.

A related multimodal benchmark perspective appears in "Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing." The paper evaluates EEG, ECG or PPG, GSR, and facial video across DEAP, AMIGOS, MAHNOB-HCI, and DREAMER, on valence, arousal, liking, and 4-class emotion classification (Siddharth et al., 2019). The key design move is to convert heterogeneous biosignals into common image-like representations: EEG power spectral densities become RGB scalp topographies processed by VGG-16, while cardiac and GSR signals become spectrogram images. Although the targets are trained largely in parallel rather than with a single joint loss, the work is important for MAL because it shows that shared multimodal representations can be reused across several affective objectives and across inconsistent datasets. On DEAP, the temporal EEG+Face LSTM reaches $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$7 for valence, $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$8 for arousal, $$\mathcal{L}=P(y|\vec{x};\Theta_p,\Theta_s) - \sum_{i=1}^{n} P(y^B_i|\vec{x};\Theta^B_{i},\Theta_s).$$9 for liking, and $$\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$$0 for 4-class emotion, surpassing previous best reported results on the dataset (Siddharth et al., 2019).

These representation-centric papers also highlight a recurrent MAL difficulty: cross-corpus affect learning is often limited less by model capacity than by incompatible label spaces, sensor layouts, trial lengths, and missing modalities. The practical harmonization strategies in this literature are deliberately coarse—binary valence/arousal, quadrant emotion classes, pooled latent spaces, modality-specific encoders—and this suggests that many current MAL pipelines are still solving the preconditions for true joint optimization rather than the final optimization problem itself (Ross et al., 2020, Siddharth et al., 2019).

5. Generative, alignment-oriented, and preference-conditioned MAL

MAL increasingly extends beyond recognition into controllable affective generation. MOPO treats affective text generation as optimization over three domain-specific emotion objectives derived from RoBERTa classifiers trained on ISEAR, TEC, and AffectiveText, with the shared five-way emotion set anger, disgust, fear, joy, and sadness (Resendiz et al., 2024). Rather than returning one best prompt, MOPO uses NSGA-II to preserve a set of non-dominated prompts, each representing a different trade-off among domain-specific norms of emotional expression. For GPT-3.5, the best seed prompt averages .63 across the three objectives, while the best multi-objective prompt reaches .97; compared with the best single-objective prompts, the multi-objective solution improves the average by 15 percentage points while incurring at most 1–2 percentage points loss on any single diagonal objective (Resendiz et al., 2024). In MAL terms, this is a direct demonstration that affective generation quality is domain-relative and that the object of optimization may be a Pareto set rather than a unique optimum.

A second alignment-style formulation decomposes human-centered response quality into principle-specific evaluators. MORLAIF trains separate preference models for principles such as helpfulness, ethicality, factuality, toxicity, sycophancy, empathy, relevance, context, bias, understandability, repetitiveness, detail, and conciseness, then scalarizes their outputs for PPO fine-tuning (Williams, 2024). The paper is not framed as affective computing, but it is MAL-relevant because it replaces a monolithic preference signal with multiple evaluators, some of which are directly affective or interpersonal. One of its most striking empirical claims is that the precise scalarization family did not significantly impact results; the main gain came from decomposition itself. This suggests that, for some MAL problems, reliable objective-specific critics may matter more than elaborate aggregation rules (Williams, 2024).

The same decomposition logic is extended in "Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards," which keeps objective dimensions separate through process reward models and objective-specific action heads (Shen et al., 1 Oct 2025). The paper spans mathematical correctness, human values, and tutoring dialogue; its directly affect-relevant dimensions include engagement and subjective human-value objectives. On the Human Values benchmark, the base model scores 0.5800 for helpfulness, 0.3042 for honesty, and 0.1888 for truthfulness, whereas the MAH-DPO ensemble reaches 0.6389, 0.3687, and 0.2478. With PRM-guided decoding, these values increase further—for example, MAH-DPO + Honest PRM reaches 0.6968, 0.5196, and 0.4107 (Shen et al., 1 Oct 2025). Within a broad MAL interpretation, this paper shows how subjective and interactive qualities can be factorized into separate heads while preserving inference-time control.

MORAL offers a related, though normatively rather than affectively framed, route. It learns multiple AIRL reward components from different experts and then maintains a posterior over scalarization weights updated by pairwise trajectory preferences, training a single PPO policy on the posterior-mean scalarization (Peschl et al., 2021). The formal overlap with MAL is substantial: multiple human-derived objectives are preserved as separate reward channels, pairwise preferences identify the desired trade-off, and the resulting policy aligns to one preferred point on the convex coverage set rather than to a single precommitted scalar reward. The paper’s explicit objectives are social norms rather than affective states, but its architecture is directly reusable wherever affective and non-affective objectives must be combined interactively (Peschl et al., 2021).

6. Evaluation, nonstationarity, and open problems

A persistent issue in MAL is that objective specification is often easier than objective assessment. "Assessing Affective Objectives for Communicative Visualizations" addresses this directly by arguing that assessments for affective objectives should be timely, actionable, and valid (Lee-Robbins et al., 1 Apr 2026). The paper adopts a taxonomy in which affective learning objectives are defined by verb categories—Observe, Position, Strengthen, Connect, Behave—and noun categories—Appraisal, Attitude, Value, Value System. In its Somalia case study, the five objectives are operationalized as crisis observation, perceived importance, personal efficacy, moral responsibility, and actual donation behavior. The resulting measurements show why direct behavioral endpoints matter: the human narrative yields an average donation of \$\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$12 bonus, compared with \$\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$20.76 for the mixed narrative, even though several attitudinal measures differ little across conditions (Lee-Robbins et al., 1 Apr 2026). For MAL, the implication is that proxy choice is itself a core technical problem; optimizing the wrong assessment can distort the entire objective vector.

A second open problem is temporal locality under distribution shift. "Locally Adaptive Multi-Objective Learning" shows that controlling multi-objective performance only over the full horizon can hide serious local failures, and proposes replacing Hedge with Fixed Share to obtain local adaptivity in online multi-objective prediction (Kaur et al., 16 Feb 2026). Its main interval bound is of order

$\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$3

and its empirical evaluations show improved subgroup-unbiased predictions under distribution shift (Kaur et al., 16 Feb 2026). Although the paper is not affective, it suggests that MAL systems deployed in dialogue, wellbeing monitoring, or multimodal interaction should evaluate fairness, calibration, and robustness on local windows, sessions, or interaction phases rather than only on aggregate corpus metrics.

Current MAL methods also remain limited by proxy dependence and problem specification. Bias-aware emotion classification depends on known identity attributes, predefined sets of sensitive outputs, and manually specified $\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$4 versus $\mathcal{L}_{total} = \sum^{B}_{i=1} w^{AU}_{i} \mathcal{L}^{AU}_{i} + \sum^{B}_{i=1} w^{FE}_{i} \mathcal{L}^{FE}_{i},$5 partitions; MOPO depends on domain-specific emotion classifiers whose biases shape the search objective; MORLAIF and MAH-DPO depend on principle-specific preference models or PRMs whose calibration and cultural validity are not guaranteed (Sen et al., 2020, Resendiz et al., 2024, Williams, 2024, Shen et al., 1 Oct 2025). This suggests that many MAL systems still optimize surrogate objectives whose epistemic and normative adequacy is only partially established.

Methodological precursors outside affective computing push toward richer future MAL formulations. In multi-objective normal-form games with non-linear utilities, opponent modelling and opponent learning awareness alter equilibrium selection and can still yield meaningful solutions when Nash equilibria are absent (Rădulescu et al., 2020). In dynamic robotics MOMDPs, intrinsically motivated generic skills improve hypervolume and median reward under changing dynamics, supporting hierarchical transfer across shifts (Abdelfattah et al., 2023). In pluralistic alignment, policy selection over a learned multi-objective repertoire is proposed as a way to adapt to diverse and shifting user preferences without retraining the underlying policy set (Harland et al., 2024). These lines of work suggest that future MAL systems may need non-linear user utilities, latent preference inference, reusable affective skills, and policy-set adaptation rather than a single static scalarized objective.

Overall, MAL is best understood as an emerging synthesis rather than a closed methodology. The literature shows that affective learning can be coupled productively with fairness, auxiliary supervision, multimodal fusion, domain-specific generation criteria, pluralistic alignment, and local robustness. It also shows that the difficult part of MAL is rarely only optimization. Objective selection, proxy validity, temporal adaptivity, and user- or context-specific trade-offs are equally central, and current systems often succeed precisely when they keep those dimensions explicit instead of collapsing them too early into one score.