---
title: Descriptive History Representations (DHRs)
url: https://www.emergentmind.com/topics/descriptive-history-representations-dhrs
type: topic
---

# Descriptive History Representations (DHRs)

Descriptive History Representations (DHRs) are a class of learned representations for partially observable decision-making settings, characterized by their ability to encode the information necessary to answer a domain-relevant set of questions about a past interaction history and prospects for future outcomes. DHRs formalize the principle that, for effective control, prediction, and interpretability, an agent should summarize its observation–action history in a way that is not merely compressive or predictive in a generic sense, but tailored to answering questions that matter for the task at hand—potentially including richly structured, semantic, or natural language queries. This framework provides a unifying and extensible approach to history summarization with rigorous sufficiency guarantees for optimizing downstream policies or queries.

## 1. Core Definition and Distinction from Prior Methods

A Descriptive History Representation is a function \( E(h) = z \) mapping an interaction history \( h \) into a representation \( z \) that suffices to answer a (potentially broad and semantically complex) set of queries \( \mathcal{Q} \) about past and future behavior:

\[
E: \mathcal{H} \rightarrow \mathcal{Z}
\]
\[
\nu_A: \mathcal{Z} \times \mathcal{Q} \to \Delta_{\mathcal{Y}}
\]
where for all histories \( h \) and all queries \( q \), \( \nu_A(E(h), q) = \nu(h, q) \), with \( \nu \) giving the ground-truth answer distribution in the “QA-space.”

This approach differs from:
- **Belief states** (which summarize the history as a distribution over environment states, optimal when a full Markov model is available and the state space is known),
- **Predictive State Representations (PSRs)** (which summarize history via predictions of future low-level action-observation sequences), and
- **Latent state or “value-irrelevant” abstractions** (which may only preserve information for value/policy computation).

Instead, DHRs are explicitly aligned with answering chosen, task-relevant questions, which can be arbitrarily expressive, including natural language queries, preference-based questions, or any domain-specific functional on the history.

## 2. Learning Framework and Components

The DHR construction is formalized as a multi-agent learning process, with the following cooperative modules:

1. **Representation Encoder (\( E \))**: Compresses the observed trajectory \( h \) into the DHR \( z \).
2. **Decision Agent (\( \pi_D \))**: Selects actions based only on \( z \), trained to maximize the expected cumulative reward.
3. **Question-Answer Modules**:
   - **QA Generator (\( \nu_{QA}^* \))**: Generates relevant question–answer pairs drawn from the observed history and future. It formalizes what queries are important for the domain.
   - **Answer Agent (\( \nu_A \))**: Receives \( z \) and a question \( q \), and outputs a distribution over answers \( y \).

The workflow, for each episode or user trajectory, is:
- Generate question–answer pairs about held-out or possible future outcomes (sampled via the QA generator).
- Encode the observed history with \( E \) to get \( z \).
- Train \( \nu_A \) to answer questions about both the past and the (counterfactual) future using only \( z \).
- Train the policy \( \pi_D \) to act optimally, using only \( z \), to maximize reward.

## 3. Joint Optimization Objective

The learning goal is to obtain representations that are both reward-optimal and QA-sufficient. This is achieved via a joint objective:

\[
\max_{E, \nu_A, \pi_D} (1-\lambda) V(\pi) - \lambda D_f\big(d^{\nu^*_A} \| d^{\nu_A}\big)
\]

where:
- \( V(\pi) \) is the expected reward of the policy using DHRs.
- \( D_f(\cdot\|\cdot) \) is an \( f \)-divergence (e.g., total variation, KL, \(\chi^2\)) between the distributions of (question, answer, history) tuples for the QA-generator (ground truth) and the answer agent induced by the current DHR.
- \( \lambda \) governs the tradeoff between control and query-answering power.
  
A variational (minimax) form is used in practical training:

\[
\max_{E, \nu_A, \pi_D} \min_{g} \ \mathbb{E}[(1-\lambda) r(h, a) + \lambda \mathbb{E}_{y \sim \nu_A}[f^*(g(h, q, y))] - \lambda \mathbb{E}_{y \sim \nu^*_A}[g(h, q, y)]]
\]

where \( g \) is a discriminator function associated with the dual form of the divergence.

## 4. Empirical Validation and Interpretability

The DHR approach was empirically validated in user modeling domains using MovieLens (movie recommendation) and Amazon Reviews (shopping). For each user:

- The encoder generates a concise textual user profile (the DHR) summarizing their interaction history.
- The QA generator asks about preferences or likely ratings for items not yet seen (“Would this user enjoy Item X?”).
- The answer agent predicts answers from the DHR only (without looking at full history).
- The recommendation policy recommends items using only the DHR profile, with rewards measured from real user behavior.

Key empirical findings include:

- **DHRs as Sufficient Statistics**: Profiles produced by the DHR encoder serve as sufficient statistics for predicting user preferences, session abandonment, and review content, outperforming both baseline LLM and embedding models.
- **Interpretability**: Generated DHRs, especially as textual profiles, provide interpretable, human-readable summaries that can answer semantically rich questions (e.g., “What styles does this user prefer?”).
- **Reward/Prediction Balance**: By controlling \( \lambda \), one can balance reward maximization with QA sufficiency, including the possibility of learning minimally sufficient, highly interpretable DHRs.
- **Superior Performance**: DHR-based decision agents achieve higher recommendation effectiveness and prediction accuracy across tested datasets.

## 5. Theoretical Properties and Mathematical Guarantees

Mathematically, DHRs are characterized as follows:

- Let the QA-space be \((\mathcal{Q}, \mathcal{Y}, \mathcal{H}, \nu)\), where \(\mathcal{Q}\) is the query set, \(\mathcal{H}\) set of histories, and \(\nu : \mathcal{H} \times \mathcal{Q} \to \Delta_{\mathcal{Y}}\) gives answer distributions.
- The DHR encoder \(E\) is sufficient if, for some decoder \(\nu_A\), for all \(h, q\):
  \[
  \nu_A(E(h), q) = \nu(h, q)
  \]
- A **Sufficiency Proposition** (see paper) states that a DHR sufficing for QA answers is also sufficient for any downstream functional \(f^\pi : \mathcal{H} \to \mathbb{R}\), such that
  \[
  f^\pi(h) = g^\pi(E(h))
  \]
  for some function \(g^\pi\).
- The joint objective guarantees that, at optimum, the DHRs provide all information needed for both optimal control and QA-answering.

## 6. Applications, Implications, and Directions

**Applications**:
- Personalized recommendation systems, where interpretable profiles improve user trust and transparency.
- Any partially observable control/problem domain requiring compact, task-aligned histories: dialogue, healthcare, robotics, education, and user modeling.
- Regulatory compliance scenarios (“right to explanation”) where interpretability is legally required.

**Implications**:
- **Interpretability**: DHRs enable direct explanations of agent behavior as the information needed to answer high-level, human-specified questions is made explicit.
- **Task-Alignment**: Representations are explicitly trained to focus on information relevant for practical or user-specified questions—promoting relevance and robustness.
- **Efficiency and Generalization**: By discarding unnecessary details, DHRs can increase learning efficiency and generalization in high-dimensional, sparse, or complex environments.

**Future Research Directions**:
- **Learning/Adapting QA-Spaces**: Instead of hand-specifying critical questions, adaptively discover them through adversarial training or with LLM assistance to optimize informativeness.
- **Multi-domain, Multimodal DHRs**: Extend to settings with text, vision, or sensor streams, where QA supervision and interpretability are still crucial.
- **Automated Fairness/Privacy Guarantees**: Incorporate controls to ensure that DHRs do not expose extraneous or privacy-sensitive information by controlling the QA-space definition and divergence regularization.
- **Theoretical Extensions**: Explore minimality, tradeoffs, and equivalence classes of DHRs for different tasks/domains.

## 7. Algorithmic Skeleton

A canonical DHR training loop follows:

```python
# Pseudocode for DHR training framework

for epoch in range(num_epochs):
    for trajectory in dataset:
        # Split trajectory into history (h) and possible futures (omega)
        h, omega = split_trajectory(trajectory)
        
        # Generate question–answer pairs using QA generator
        qa_pairs = qa_generator.sample_pairs(h, omega)
        
        # Encode history
        z = encoder(h)
        
        for (q, y_star) in qa_pairs:
            # Predict answer from DHR encoding
            y_pred = answer_agent(z, q)
            # Compute QA divergence loss (e.g., KL or chi-squared)
            loss_qa += divergence(y_star, y_pred)
    
        # Take action using policy
        action = decision_agent(z)
        # Compute reward loss
        loss_policy += reward_loss(action, actual_outcome)
    
    # Jointly update model parameters
    total_loss = (1 - lambda_) * loss_policy + lambda_ * loss_qa
    total_loss.backward()
    optimizer.step()
```

Here, every step aligns the encoder to maximize both the task performance and QA sufficiency.

---

**In summary**, Descriptive History Representations supply a rigorous, practically validated, and interpretable approach to history summarization and reasoning in decision-making systems. By directly optimizing the ability to answer meaningful questions about past and future outcomes, DHRs advance the state-of-the-art in representation learning for partially observed and complex real-world domains.

Source: https://www.emergentmind.com/topics/descriptive-history-representations-dhrs