Temporal Granularity Optimization

• Temporal Granularity Optimization (TGO) is a framework that treats temporal resolution as a controllable variable to optimize inference across multiple granularities.
• It employs methods like adaptive partitioning, multi-scale fusion, and learned balancing to reduce errors and manage computational trade-offs.
• Empirical findings show that the optimal choice of temporal granularity is task-dependent, influencing prediction accuracy, efficiency, and system robustness.

Temporal Granularity Optimization (TGO) is best understood as an umbrella designation for methods that treat temporal scale as a first-class modeling variable and optimize how coarse and fine temporal information is represented, partitioned, weighted, or reconciled for downstream inference. In current arXiv literature, TGO spans explicit optimization problems—such as adaptive repartitioning of continuous-time inference intervals and minimization of periodic calendar representations—as well as architecture-level strategies in which multiple fixed temporal granularities are preserved and fused end-to-end rather than collapsed to a single timescale (Saria et al., 2012, Bettini et al., 2011, Zhang et al., 2024, Akdag et al., 2024).

1. Conceptual scope and representative forms

Across recent work, TGO does not denote a single canonical algorithm. It instead names a recurring design problem: temporal information can be encoded at several resolutions, and the choice of resolution affects approximation error, computational cost, predictive accuracy, robustness, privacy leakage, and coherence across outputs. Some papers optimize granularity directly through a stated criterion; others optimize the contribution of several granularities implicitly through learned fusion or through an evaluation framework that compares candidate resolutions (Saria et al., 2012, Qian et al., 2021, Jiang et al., 2023, Radovanovic et al., 2 Jun 2026).

Domain	Temporal object	Representative mechanism
Continuous-time inference	Message intervals	KL-based adaptive splitting
Temporal knowledge graphs	Year, month, day	GRL and AGB
Surveillance video	Short, medium, long chunks	Cross-attention and self-attention
Event-based deblurring	Voxel bins and event points	AMM and AFDM
Smart-meter privacy	15-minute to 7-day profiles	Cross-granularity evaluation

A representative symbolic formulation appears in temporal knowledge graph completion, where a temporal knowledge graph is defined as

$$\mathcal{G}=\{(s,r,o,t)\mid s,o\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\},$$

and interpolation queries take the form

$$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$

The TGO question there is not whether time should be embedded at all, but whether a timestamp should be treated as a monolithic symbol or decomposed into year-, month-, and day-level semantics with query-dependent balancing (Zhang et al., 2024).

An equally representative continuous-time formulation appears in approximate inference for continuous-time Bayesian networks. There, temporal granularity refers to whether a message or cluster is represented over one long interval by a single homogeneous Markov process or over several shorter subintervals by a piecewise homogeneous approximation. The corresponding optimization problem is to decide where additional temporal partitioning is justified by the reduction in approximation error (Saria et al., 2012).

2. Optimization criteria and mathematical objectives

The most explicit TGO objective in the corpus is the adaptive cluster-partitioning criterion for continuous-time Bayesian networks. Given one-piece and two-piece KL approximation costs,

$$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$

the split point is chosen by

$t^* = \arg\min_t C_{KL}^{2}(t),$

and the split is made if

$C_{KL}^{1} - C_{KL}^{2}(t^*) > k^*.$

This formulation makes temporal refinement an online, local approximation-selection problem rather than a fixed preprocessing choice (Saria et al., 2012).

A different explicit criterion appears in calendar reasoning. There the optimization target is not predictive performance but the minimal period length of a periodic-set representation compiled from Calendar Algebra. A representation of a granularity $H$ in terms of $G$ is minimal when its period length $P$ is the smallest among all valid pairs $(P_H^G, N_H^G)$. The period length is treated as the key cost parameter because it dominates the efficiency of granule conversion, set operations, and temporal-constraint reasoning (Bettini et al., 2011).

In self-supervised video representation learning, TGO is parameterized by a convex combination of fine-grained and persistent temporal learning objectives:

$$\mathcal{L} = \alpha \mathcal{L}_f + (1-\alpha)\mathcal{L}_p.$$

Here $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$0 is an explicit temporal-granularity control knob: larger values emphasize temporally local discrimination, while smaller values emphasize temporally persistent invariance (Qian et al., 2021).

Event-based motion deblurring gives a complementary error-analysis view. After discretizing blur inversion with step size

$$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$1

the paper shows that the reconstruction error includes terms proportional to $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$2 and $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$3, so smaller $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$4 yields smaller approximation error. In that setting, finer temporal granularity is mathematically tied to better inversion of fast motion, but only if the resulting fine-grained representation remains usable for dense image restoration (Lin et al., 2024).

A broader implication is that TGO objectives are domain-specific. In some settings they minimize KL error or period length; in others they maximize downstream discrimination, reduce privacy leakage, or recover fine-grained predictions from coarse-grained storage. The unifying theme is not one loss function, but the treatment of temporal resolution itself as an optimization variable.

3. Multi-granularity representation learning and fusion

A dominant contemporary pattern is to preserve several temporal granularities simultaneously and learn how to combine them. In temporal knowledge graph completion, LGRe decomposes each timestamp into year-month-day, learns time-aware embeddings with a GRU, produces granularity-specific query representations through time-specific multi-layer CNNs, and then uses Adaptive Granularity Balancing to compute weights $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$5 over the year-, month-, and day-level representations. The model’s two named components are Granularity Representation Learning (GRL) and Adaptive Granularity Balancing (AGB) (Zhang et al., 2024).

Traffic forecasting uses a closely related coarse-to-fine hierarchy. STMGF defines recent-data sequences at fine, middle, and coarse granularities, denoted $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$6, $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$7, and $$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$8, and refines lower-scale predictions with higher-scale context through cross-attention:

$$\mathcal{Q}=\{(s,r,?,t)\mid s\in\mathcal{E},\ r\in\mathcal{R},\ t\in\mathcal{T}\}.$$9

The same framework adds day-level and week-level periodic history through similarity-based matching weights $$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$0, so temporal granularity is coupled to both recent-context compression and periodic retrieval (Zhao et al., 2024).

A more explicitly systems-oriented coarse-to-fine design appears in fine-grained traffic prediction from coarse-grained data. STRP assumes input granularity $$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$1 and target granularity $$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$2, and defines two tasks: Window-Based Fine-Grained Prediction and Duration-Based Fine-Grained Prediction. Its temporal decoder is an Inverse Dilated Convolution module,

$$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$3

which recursively expands coarse histories into fine-grained future trajectories. Here TGO is tied to storage-efficient coarse sensing combined with learned refinement at inference time (Li et al., 8 Jun 2026).

Event-based motion deblurring employs perhaps the clearest representation-level duality. MTGNet uses a temporally coarse but spatially dense voxel tensor

$$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$4

and a temporally fine but spatially sparse point-cloud event representation

$$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$5

Its Aggregation and Mapping Module (AMM) aligns point-based features to frame-based features, and its Adaptive Feature Diffusion Module (AFDM) spreads sparse point features over dense grids using a diffusion range

$$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$6

The model therefore treats TGO as a fusion problem between fine temporal fidelity and dense spatial support (Lin et al., 2024).

Generative human video compression applies the same idea to motion coding. MTTF transmits compact motion vectors $$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$7 and reconstructs fine-grained motion fields from a key-frame latent basis, so temporal granularity is split between a low-rate coded representation and a richer decoder-side motion field. The contribution is therefore closer to temporal granularity-aware representation learning than to adaptive scale selection (Yin et al., 2024).

4. Explicit adaptation, repartitioning, and hierarchical temporal scheduling

Some work goes beyond fixed multi-scale fusion and modifies temporal partitions directly during inference or system execution. In continuous-time Bayesian networks, the cluster-graph architecture allows clusters and sepsets to overlap in both variable scope and time interval. Different parts of the same model can therefore operate at different temporal resolutions, and the resolution can change online as information is transmitted. This is the most explicit realization of TGO as heterogeneous, adaptive temporal partitioning (Saria et al., 2012).

Calendar compilation provides a symbolic counterpart. The conversion from Calendar Algebra to periodic sets is organized as a hybrid procedure that interleaves operator-wise conversion with period minimization. The method computes period length and period label distance, aligns labels in one canonical period, and minimizes intermediate results before reusing them. In practical terms, TGO here is a compilation-and-normalization pipeline that preserves semantics while reducing the period length that downstream reasoning must manipulate (Bettini et al., 2011).

Hierarchical temporal scheduling also appears in long-form video retrieval. The moment-retrieval system built around frame-level indexing uses TransNetV2 for scene detection, samples four evenly spaced frames per detected scene, removes near-duplicates when cosine similarity exceeds 0.9, retrieves top-$$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$8 frames per model, reranks candidates using neighboring frames, and then localizes a moment with separate start and end queries by searching up to 20 relevant frames on each side of an anchor frame. The resulting temporal stack comprises scene-level partitioning, sparse keyframe indexing, frame-level retrieval, micro-neighborhood reranking, and segment-level output (Tran et al., 11 Apr 2025).

Other systems remain fixed rather than adaptive, but they still make temporal scale the central modeling variable. TeG for weakly supervised anomaly detection divides each clip into 32 temporal segments and uses three manually chosen chunk sizes—8, 32, and 64 frames—followed by multi-head cross-attention and multi-head self-attention. The paper explicitly distinguishes this from learned scale selection: it is a multi-scale fusion method, not a scale-selection method (Akdag et al., 2024).

5. Empirical regularities and trade-offs

A recurring empirical result is that finer granularity is not universally better. In self-supervised video learning, fine-grained TeG improves VidSitu from 28.3 to 31.1 and Kinetics-GEBD from 69.9 to 71.4, but persistent TeG outperforms fine-grained TeG on Kinetics (67.8 vs 65.0), Something-Something-v2 (61.4 vs 60.5), and Diving48 (83.6 vs 81.5). The paper’s central conclusion is that different downstream tasks require different temporal granularities (Qian et al., 2021).

Dynamic graph learning reaches a similar conclusion. The empirical study over second, minute, hour, and day resolutions reports that a sophisticated memory mechanism and proper time granularity are crucial for competitive and robust dynamic link prediction, and that there is no universal rule that the finest timestamps are best. It also finds that TGN is the most robust model across granularity choices, largely because its GRU-based memory is less brittle than vanilla RNN-style memory under coarsening and granularity mismatch (Jiang et al., 2023).

Smart-meter privacy analysis reveals a different but equally sharp structure: predictive performance forms two plateaus, one between 15 minutes and 1 hour and another between 1 day and 7 days. The study interprets this as evidence that temporal resolution can often be coarsened substantially without sacrificing utility for some attributes, while dynamic attributes such as swimming-pool usage depend much more strongly on fine-grained temporal signals (Radovanovic et al., 2 Jun 2026).

Multi-scale processing also carries direct compute costs. In surveillance anomaly detection, feature extraction costs 104.96 ms per temporal granularity for a segment of 2,133 ms, so using three granularities raises extraction time to 314.88 ms; adding attention-based fusion yields 317.38 ms total per segment, corresponding to 14.87% of the segment’s temporal duration and about 1.5 s delay for a 10 s video segment. The paper therefore quantifies a standard TGO trade-off: preserving more temporal scales increases robustness to heterogeneous event duration, but the dominant cost grows roughly linearly with the number of scales unless features are shared (Akdag et al., 2024).

A plausible synthesis is that TGO surfaces are often non-monotone and task-specific. High-frequency motifs matter strongly for some tasks, while others are dominated by persistent low-frequency structure or by coarse semantic context. This helps explain why both explicit adaptation and multi-scale retention recur across otherwise unrelated domains.

6. Conceptual boundaries and open problems

One common misconception is that TGO means choosing one globally optimal time step. The literature instead repeatedly favors query-dependent balancing, task-dependent weighting, or piecewise adaptation. In temporal knowledge graph completion, the optimization target is not a single best granularity but adaptive balancing of year-, month-, and day-level semantics for each query (Zhang et al., 2024). In self-supervised video learning, the best $$C_{KL}^{1} = D(P_C \,\|\, P_S), \qquad C_{KL}^{2}(t) = D(P_C \,\|\, P_S^L) + D(P_C \,\|\, P_S^R),$$9 depends on the downstream task rather than on a universal preference for either fine-grained or persistent features (Qian et al., 2021).

A second misconception is that any multi-scale architecture already solves TGO. Several papers explicitly do not learn temporal granularity itself: TeG anomaly detection fixes the three granularities $t^* = \arg\min_t C_{KL}^{2}(t),$0; STMGF uses predefined $t^* = \arg\min_t C_{KL}^{2}(t),$1, $t^* = \arg\min_t C_{KL}^{2}(t),$2, $t^* = \arg\min_t C_{KL}^{2}(t),$3, $t^* = \arg\min_t C_{KL}^{2}(t),$4, and $t^* = \arg\min_t C_{KL}^{2}(t),$5; the dynamic-graph study evaluates four preset resolutions rather than learning them (Akdag et al., 2024, Zhao et al., 2024, Jiang et al., 2023). In these cases, the optimization occurs through end-to-end training or empirical selection, not through adaptive search over a granularity space.

Open directions are correspondingly consistent. Video anomaly detection suggests learned scale weights, input-dependent routing, differentiable temporal pooling, or neural architecture search over chunk sizes (Akdag et al., 2024). Dynamic graph learning explicitly identifies learnable time granularity as a promising direction beyond deterministic preset intervals (Jiang et al., 2023). Continuous-time Bayesian network inference proposes extending temporal adaptation to dynamic changes in the spatial clustering structure itself (Saria et al., 2012). Calendar reasoning points to reverse compilation from optimized periodic sets back to readable algebraic expressions and to richer low-level targets that handle finite exceptions more naturally (Bettini et al., 2011).

In aggregate, the literature suggests that TGO is emerging as a cross-domain principle rather than a single method family. Its central claim is stable: temporal resolution should be treated neither as a fixed preprocessing artifact nor as a purely architectural afterthought, but as a controllable resource whose allocation shapes both the fidelity and the cost structure of inference.