DistantTimex: Neural Temporal Reasoning

• DistantTimex is a neural temporal reasoning framework that learns timex representations from synthetically generated labeled pairs to support event ordering.
• It employs a character-level biLSTM to encode time expressions and integrates the resulting embeddings into event temporal relation models.
• Empirical results show notable gains in low-resource conditions where explicit event-timex interactions are frequent, despite modest improvements on standard datasets.

DistantTimex, in the context of neural temporal reasoning, denotes a distant-supervision setting in which explicit temporal expressions are used to support event temporal ordering by learning timex representations that can be integrated directly into deep models. The formulation introduced in "Embedding time expressions for deep temporal ordering models" addresses a central limitation of data-driven temporal relation systems: state-of-the-art models often overlook explicit temporal signals such as dates and time windows, while rule-based methods can identify links between time expressions but fail to capture timexes' interactions with events and are hard to integrate with the distributed representations of neural net models (Goyal et al., 2019). The resulting framework learns dedicated timex embeddings from synthetic timex pairs and applies them in event temporal ordering, yielding a small increase in performance on MATRES and more substantial gains on an automatically collected dataset with more frequent event-timex interactions.

1. Conceptual basis

The core problem is temporal ordering of events in text. In that setting, explicit temporal expressions—timexes such as dates, times, and relative expressions—encode strong constraints on event order, but standard neural architectures do not necessarily exploit them effectively. DistantTimex-style modeling addresses this gap by making timex information available as learned distributed representations rather than leaving it in a purely symbolic or rule-based form (Goyal et al., 2019).

The framework rests on a division between two kinds of temporal knowledge. First, timex–timex relations can often be computed deterministically once expressions are normalized to a standard time scale. Second, event–timex interactions require contextual modeling because events inherit temporal information from associated modifiers. The paper’s contribution is to bridge these layers by learning temporally informed timex embeddings and injecting them into an event temporal relation classifier.

A common misconception is that contextual word embeddings already solve explicit temporal modeling. The reported masking experiments do not support that interpretation. They indicate that standard embeddings do pick up some temporal information, but explicit timex embeddings are superior, especially in low-resource settings. Conversely, another misconception would be that explicit timex modeling always produces large gains; on MATRES, the improvement is modest because only a small fraction of event pairs have computable timex links.

2. Synthetic supervision for timex relations

The timex embedding model is trained on synthetic data generated from a grammar designed to produce a wide variety of time expressions (Goyal et al., 2019). The data generation process has two main classes:

• Explicit datetimes: examples include “Sept 12, 1993”, “1998”, and “9th January”.
• Natural language indicators: examples include “two months ago” and “next year”.

Generic templates such as [mm dd, yy] and [xx units ago/later] are instantiated by sampling plausible slot values including years, months, and quantities. For each synthetically generated pair of timexes, the temporal relation is deterministically computed by converting the expressions to a standard time scale. The label space for this intrinsic task is three-way: before, after, or simultaneous.

For intrinsic evaluation, the reported synthetic corpus contains 50,000 training pairs and 5,000 test pairs. This design is significant because it replaces expensive manual annotation with automatically generated supervision while preserving exact temporal labels for timex pairs. A plausible implication is that the supervision distills symbolic temporal normalization into a form usable by neural encoders.

3. Timex embedding model

The timex encoder operates at the character level. Given a pair of time expressions, $t_1$ and $t_2$, each string is encoded by a biLSTM over characters, and the output states are averaged:

$$h_1 = \frac{1}{L_1}\sum_{i=1}^{L_1}\mathrm{biLSTM}(c^{(1)}_i)\quad h_2 = \frac{1}{L_2}\sum_{i=1}^{L_2}\mathrm{biLSTM}(c^{(2)}_i)$$

The two averaged vectors are concatenated as $z=[h_1; h_2]$ and passed through multiple feed-forward layers with ReLU nonlinearity, followed by a softmax over the three temporal relation classes. Training uses cross-entropy loss (Goyal et al., 2019).

The same computation is summarized for a single timex as:

$$h_\mathrm{timex} = \frac{1}{L}\sum_{i=1}^{L} \mathrm{biLSTM}(c_i)$$

The reported intrinsic results show that the dedicated character-level model effectively learns temporal ordering properties of time expressions.

Model	Task	Accuracy
Char biLSTM (proposed)	Synthetic timex relation classification	97.3%
ELMo-based	Synthetic timex relation classification	97.6%
GloVe-based	Synthetic timex relation classification	~88%

The baseline linear pooling model is also reported as lower than the full biLSTM. The comparison indicates that character-level encoding is substantially stronger than word-level embeddings for this task, which is consistent with the compositional and orthographic variability of timex strings.

4. Injection into event temporal ordering

The downstream event ordering model is an improved version of the model from Cheng & Miyao (2017), designed to classify the temporal relation between two events $(e_1, e_2)$, possibly in separate sentences $(s_1, s_2)$ (Goyal et al., 2019). At the token level, each contextual representation is

$\tilde{x}_k = [v_w; v_p; v_t]$

where $v_w$ is a word embedding such as GloVe or ELMo, $v_p$ is a randomized POS tag embedding, and $t_2$0 is the timex embedding if the token is a time expression or a zero vector otherwise.

A distinctive design choice is timex embedding assignment. The timex vector detected by the timex biLSTM is attached not only to the timex token itself but also broadcast to events it modifies according to grammatical heuristics derived from the dependency parse. In this way, event representations inherit temporal information from associated timex modifiers.

Contextualization proceeds with a bottom-level biLSTM over tokens in context, using shared parameters for both sentences. For each event, the representations along the dependency path from the event to the root or lowest common ancestor are fed into an upper biLSTM:

$t_2$1

The two path summaries are then concatenated,

$t_2$2

and passed through feedforward and softmax layers to classify the event relation as before, after, simultaneous, or vague. The full network is trained with cross-entropy on available annotation.

5. Empirical results

On MATRES, the standard split uses TimeBank plus AQUAINT for training and Platinum for test, and results are reported as accuracy averaged over 3 runs (Goyal et al., 2019).

Embedding	Variant	Accuracy
GloVe	Baseline (Cheng+17)	59.53
GloVe	Ours w/o timex embed	62.83
GloVe	Ours w/ timex embed	63.22
ELMo	Baseline (Cheng+17)	65.50
ELMo	Ours w/o timex embed	68.45
ELMo	Ours w/ timex embed	68.61

These results show a modest improvement from timex embeddings. The paper attributes the limited effect to low coverage: only a small fraction of event pairs in MATRES have computable timex links.

The distant-supervision setting is more favorable to explicit timex modeling. Using high-precision, low-recall rule-based classifiers from Chambers et al. (2014) over Gigaword, the framework extracts large numbers of event pairs with strong timex anchoring. Models are trained with limited supervision of 2000, 3000, or 4000 samples and evaluated on a constant 1000-sample test set.

Setting	Variant	Accuracy
GloVe, 2000	Ours w/o timex	74.0
GloVe, 2000	Ours w/ masked timex	73.9
GloVe, 2000	Ours w/ timex embed	81.6
GloVe, 4000	Ours w/o timex	78.2
GloVe, 4000	Ours w/ masked timex	77.1
GloVe, 4000	Ours w/ timex embed	83.1
ELMo, 2000	Ours w/o timex	80.1
ELMo, 2000	Ours w/ masked timex	79.8
ELMo, 2000	Ours w/ timex embed	82.3
ELMo, 4000	Ours w/o timex	84.3
ELMo, 4000	Ours w/ masked timex	80.7
ELMo, 4000	Ours w/ timex embed	84.8

The reported gains are substantial, particularly in low-resource conditions: for example, approximately +7 points for GloVe and +2 for ELMo at small data sizes. Masking timexes hurts the models, especially ELMo, reinforcing the claim that explicit temporal expressions carry useful signal beyond generic contextualization.

6. Significance for distant supervision and explicit temporal modeling

The distant supervision scenario is described as mimicking classic DistantTimex systems, but with a different representational strategy (Goyal et al., 2019). Instead of relying only on manual rule-based linking or shallow features, the model learns to encode temporal information in the embeddings themselves, making the signal directly compatible with deep architectures.

This is significant for two reasons. First, the approach offers a differentiable mechanism for combining symbolic regularities in timex interpretation with neural event modeling. Second, the empirical pattern suggests that the method is especially beneficial when event-timex interactions are common and annotated data is limited, which is typical of distantly or lazily supervised scenarios.

The paper characterizes the work as the first to propose learning dedicated timex embeddings from temporal-order-labeled synthetic data and to demonstrate how they can be broadcast to events and directly integrated into modern neural architectures. A plausible implication is that DistantTimex-style systems need not choose between brittle rule systems and purely distributional models; the framework instead distills hand-engineered temporal reasoning into trainable representations.

7. Workflow and limitations

The full workflow is summarized in five steps (Goyal et al., 2019): generate labeled synthetic timex pairs using templates and grammar; train a character-level biLSTM to embed timexes and classify their pairwise relation; use these embeddings in an event ordering model alongside word and POS features by assigning them to both time tokens and the events they modify; evaluate on MATRES and on event-timex-rich, distantly labeled data; and observe small gains on MATRES but larger gains where event-timex interactions are frequent.

The principal limitation stated in the results is coverage. On MATRES, the benefit is constrained because only a small fraction of event pairs have computable timex links. This indicates that explicit timex embedding is not a universal replacement for broader contextual temporal reasoning. Rather, its strongest contribution emerges in corpora where events are strongly anchored by dates or relative time expressions.

Within that scope, DistantTimex provides a scalable mechanism for explicit temporal modeling in neural temporal relation systems. By synthesizing supervision for timex-timex and event-timex relations and training character-level models, it moves beyond the brittle coverage and scope of traditional rule systems while preserving the temporal structure those systems capture (Goyal et al., 2019).