Descriptive History Representations (DHRs)

• Descriptive History Representations are learned, task-focused summaries of past interactions that support answering semantically rich, domain-specific queries.
• They integrate an encoder, decision agent, and QA modules to jointly optimize reward maximization with sufficient query-answering capabilities.
• DHRs enhance interpretability and efficiency in partially observable environments, improving predictions and decision-making in applications like recommendations and robotics.

Descriptive History Representations (DHRs) are a class of learned representations for partially observable decision-making settings, characterized by their ability to encode the information necessary to answer a domain-relevant set of questions about a past interaction history and prospects for future outcomes. DHRs formalize the principle that, for effective control, prediction, and interpretability, an agent should summarize its observation–action history in a way that is not merely compressive or predictive in a generic sense, but tailored to answering questions that matter for the task at hand—potentially including richly structured, semantic, or natural language queries. This framework provides a unifying and extensible approach to history summarization with rigorous sufficiency guarantees for optimizing downstream policies or queries.

1. Core Definition and Distinction from Prior Methods

A Descriptive History Representation is a function $E(h) = z$ mapping an interaction history $h$ into a representation $z$ that suffices to answer a (potentially broad and semantically complex) set of queries $\mathcal{Q}$ about past and future behavior:

$E: \mathcal{H} \rightarrow \mathcal{Z}$

$$\nu_A: \mathcal{Z} \times \mathcal{Q} \to \Delta_{\mathcal{Y}}$$

where for all histories $h$ and all queries $q$, $\nu_A(E(h), q) = \nu(h, q)$, with $\nu$ giving the ground-truth answer distribution in the “QA-space.”

This approach differs from:

• Belief states (which summarize the history as a distribution over environment states, optimal when a full Markov model is available and the state space is known),
• Predictive State Representations (PSRs) (which summarize history via predictions of future low-level action-observation sequences), and
• Latent state or “value-irrelevant” abstractions (which may only preserve information for value/policy computation).

Instead, DHRs are explicitly aligned with answering chosen, task-relevant questions, which can be arbitrarily expressive, including natural language queries, preference-based questions, or any domain-specific functional on the history.

2. Learning Framework and Components

The DHR construction is formalized as a multi-agent learning process, with the following cooperative modules:

1. Representation Encoder ($E$): Compresses the observed trajectory $h$ into the DHR $z$.
2. Decision Agent ($\pi_D$): Selects actions based only on $z$, trained to maximize the expected cumulative reward.
3. Question-Answer Modules:
   • QA Generator ($\nu_{QA}^*$): Generates relevant question–answer pairs drawn from the observed history and future. It formalizes what queries are important for the domain.
   • Answer Agent ($\nu_A$): Receives $z$ and a question $q$, and outputs a distribution over answers $y$.

The workflow, for each episode or user trajectory, is:

• Generate question–answer pairs about held-out or possible future outcomes (sampled via the QA generator).
• Encode the observed history with $E$ to get $z$.
• Train $\nu_A$ to answer questions about both the past and the (counterfactual) future using only $z$.
• Train the policy $\pi_D$ to act optimally, using only $z$, to maximize reward.

3. Joint Optimization Objective

The learning goal is to obtain representations that are both reward-optimal and QA-sufficient. This is achieved via a joint objective:

$$\max_{E, \nu_A, \pi_D} (1-\lambda) V(\pi) - \lambda D_f\big(d^{\nu^*_A} \| d^{\nu_A}\big)$$

where:

• $V(\pi)$ is the expected reward of the policy using DHRs.
• $D_f(\cdot\|\cdot)$ is an $f$-divergence (e.g., total variation, KL, $\chi^2$) between the distributions of (question, answer, history) tuples for the QA-generator (ground truth) and the answer agent induced by the current DHR.
• $\lambda$ governs the tradeoff between control and query-answering power.

A variational (minimax) form is used in practical training:

$$\max_{E, \nu_A, \pi_D} \min_{g} \ \mathbb{E}[(1-\lambda) r(h, a) + \lambda \mathbb{E}_{y \sim \nu_A}[f^*(g(h, q, y))] - \lambda \mathbb{E}_{y \sim \nu^*_A}[g(h, q, y)]]$$

where $g$ is a discriminator function associated with the dual form of the divergence.

4. Empirical Validation and Interpretability

The DHR approach was empirically validated in user modeling domains using MovieLens (movie recommendation) and Amazon Reviews (shopping). For each user:

• The encoder generates a concise textual user profile (the DHR) summarizing their interaction history.
• The QA generator asks about preferences or likely ratings for items not yet seen (“Would this user enjoy Item X?”).
• The answer agent predicts answers from the DHR only (without looking at full history).
• The recommendation policy recommends items using only the DHR profile, with rewards measured from real user behavior.

Key empirical findings include:

• DHRs as Sufficient Statistics: Profiles produced by the DHR encoder serve as sufficient statistics for predicting user preferences, session abandonment, and review content, outperforming both baseline LLM and embedding models.
• Interpretability: Generated DHRs, especially as textual profiles, provide interpretable, human-readable summaries that can answer semantically rich questions (e.g., “What styles does this user prefer?”).
• Reward/Prediction Balance: By controlling $\lambda$, one can balance reward maximization with QA sufficiency, including the possibility of learning minimally sufficient, highly interpretable DHRs.
• Superior Performance: DHR-based decision agents achieve higher recommendation effectiveness and prediction accuracy across tested datasets.

5. Theoretical Properties and Mathematical Guarantees

Mathematically, DHRs are characterized as follows:

• Let the QA-space be $(\mathcal{Q}, \mathcal{Y}, \mathcal{H}, \nu)$, where $\mathcal{Q}$ is the query set, $\mathcal{H}$ set of histories, and $$\nu : \mathcal{H} \times \mathcal{Q} \to \Delta_{\mathcal{Y}}$$ gives answer distributions.
• The DHR encoder $E$ is sufficient if, for some decoder $\nu_A$, for all $h, q$:

$\nu_A(E(h), q) = \nu(h, q)$

• A Sufficiency Proposition (see paper) states that a DHR sufficing for QA answers is also sufficient for any downstream functional $f^\pi : \mathcal{H} \to \mathbb{R}$, such that

$f^\pi(h) = g^\pi(E(h))$

for some function $g^\pi$.

• The joint objective guarantees that, at optimum, the DHRs provide all information needed for both optimal control and QA-answering.

6. Applications, Implications, and Directions

Applications:

• Personalized recommendation systems, where interpretable profiles improve user trust and transparency.
• Any partially observable control/problem domain requiring compact, task-aligned histories: dialogue, healthcare, robotics, education, and user modeling.
• Regulatory compliance scenarios (“right to explanation”) where interpretability is legally required.

Implications:

• Interpretability: DHRs enable direct explanations of agent behavior as the information needed to answer high-level, human-specified questions is made explicit.
• Task-Alignment: Representations are explicitly trained to focus on information relevant for practical or user-specified questions—promoting relevance and robustness.
• Efficiency and Generalization: By discarding unnecessary details, DHRs can increase learning efficiency and generalization in high-dimensional, sparse, or complex environments.

Future Research Directions:

• Learning/Adapting QA-Spaces: Instead of hand-specifying critical questions, adaptively discover them through adversarial training or with LLM assistance to optimize informativeness.
• Multi-domain, Multimodal DHRs: Extend to settings with text, vision, or sensor streams, where QA supervision and interpretability are still crucial.
• Automated Fairness/Privacy Guarantees: Incorporate controls to ensure that DHRs do not expose extraneous or privacy-sensitive information by controlling the QA-space definition and divergence regularization.
• Theoretical Extensions: Explore minimality, tradeoffs, and equivalence classes of DHRs for different tasks/domains.

7. Algorithmic Skeleton

A canonical DHR training loop follows:

for epoch in range(num_epochs): for trajectory in dataset: # Split trajectory into history (h) and possible futures (omega) h, omega = split_trajectory(trajectory) # Generate question–answer pairs using QA generator qa_pairs = qa_generator.sample_pairs(h, omega) # Encode history z = encoder(h) for (q, y_star) in qa_pairs: # Predict answer from DHR encoding y_pred = answer_agent(z, q) # Compute QA divergence loss (e.g., KL or chi-squared) loss_qa += divergence(y_star, y_pred) # Take action using policy action = decision_agent(z) # Compute reward loss loss_policy += reward_loss(action, actual_outcome) # Jointly update model parameters total_loss = (1 - lambda_) * loss_policy + lambda_ * loss_qa total_loss.backward() optimizer.step()

Here, every step aligns the encoder to maximize both the task performance and QA sufficiency.

---

In summary, Descriptive History Representations supply a rigorous, practically validated, and interpretable approach to history summarization and reasoning in decision-making systems. By directly optimizing the ability to answer meaningful questions about past and future outcomes, DHRs advance the state-of-the-art in representation learning for partially observed and complex real-world domains.