General Information Metrics Evaluation (GIME)

• GIME is a domain-agnostic framework that quantifies, compares, and optimizes information properties using Objective Information Theory.
• It employs a multi-criteria, threshold-based approach to select optimal datasets by balancing metrics like volume, scope, and granularity.
• Experimental validations in CTR prediction, legal analysis, and weather forecasting confirm its efficiency in reducing training time and resource usage.

General Information Metrics Evaluation (GIME) encompasses a rigorously formalized, domain-agnostic approach to quantifying, comparing, and optimizing the ^^^^2^^^^ properties of datasets, models, and outputs. Drawing on Objective Information Theory (OIT), self-correlation analyses, and advanced metric evaluation frameworks, GIME serves as both a methodological foundation for efficient AI data pipeline design and a meta-framework for the standardization and empirical validation of evaluation metrics across complex, multi-modal, and information-rich domains.

1. Objective Information Theory and the OIT Metric Set

GIME’s analytics are grounded in Objective Information Theory, which models any information event as a sextuple: $I = \langle o, T_h, f, c, T_m, g \rangle$. Here, $o$ is the noumenon (objective entity), $T_h$ its occurrence interval, $f(o, T_h)$ the source state evolution, $c$ the carrier, $T_m$ the reflection interval, and $g(c, T_m)$ the reflected states. For "restorable" information flows (i.e., injective mappings), OIT defines eleven core metrics:

Metric	Formula / Interpretation	OIT Notation
Volume	"How much" reflected	$V(I) = |g(c, T_m)|$
Scope	Carrier size	$S(I) = |c|$
Duration	Source time span	$D_h(I) = |T_h|$
Delay	Lag before reflection	$\Delta t(I) = \min T_m - \max T_h$
Variety	Distinct source states	$Var(I) = |f(o, T_h)|$
Granularity	Time resolution	$G(I) = |T_m| / |g(c, T_m)|$
Sampling Rate	Rate of reflection	$SR(I) = |g(c, T_m)| / |T_h|$
Aggregation	Fraction unique	$Agg(I) = |\mathrm{Distinct}~g|/|g|$
Coverage	Source-state coverage	$Cov(I) = |f(o, T_h) \cap I^{-1}(g)|/|f|$
Distortion	Avg. error	$\mathrm{Dist}(I) = \frac{1}{|T_h|} \int d(...)$
Mismatch	Fraction missed/extra	$MM(I) = |f \Delta I^{-1}(g)| / |f|$

Under typical operating conditions, volume can be tuned through scope, variety, duration, sampling rate, and granularity, while the remaining ten metrics are mutually independent (Xu et al., 2 Jan 2025).

2. GIME Selection Objective and Algorithmic Implementation

GIME operationalizes dataset selection by partitioning the eleven metrics into three classes: high-sensitivity (must exactly match or exceed full pool), moderate-sensitivity (constrained to a designer-specified window), and low-sensitivity (ignored). For a data pool $P$, threshold-based selection seeks a subset $S \subseteq P$ such that:

• $\forall i \in H: \text{metr}_i(S) = T_i$ (high-sensitivity exact match)
• $$\forall i \in M: \alpha_i \leq \text{metr}_i(S) \leq \beta_i$$ (moderate-sensitivity window)

No single aggregate objective is used; the approach is strictly multi-criteria. Typically, $S$ is incrementally grown until all target metrics are satisfied. Since the evaluation of these metrics relies on basic set operations, the computational overhead is negligible relative to downstream model training costs (Xu et al., 2 Jan 2025).

3. Experimental Validation and Domain Applications

GIME’s efficacy has been demonstrated across heterogeneous application domains:

• CTR Prediction (deep-FM/DNN, Avazu dataset):
  • GIME yielded a $53.8\%$ reduction in training time with only $\Delta$AUC $= -0.0045$ (from $0.7522$ to $0.7488$); random subsets of equal size underperformed GIME (mean AUC drop $\sim0.005$).
• Civil Case Prediction (ERNIE, 100k legal documents):
  • Average accuracy drop less than $2\%$, and $39.7\%$ less training time.
• Weather Forecasting (CNN-LSTM family, European cities):
  • Error increases in MRMSE limited to $0.17$–$6.91\%$, time savings up to $59.6\%$.
• Judicial AI Program (six commercial-scale models):
  • GIME reduced human effort by $23,500$ person-hours and total training expenses by $\approx 39.6\%$, with only negligible accuracy degradation (Xu et al., 2 Jan 2025).

These results empirically support the monotonic correlation between optimized GIME metrics and predictive performance (AUC, accuracy, MRMSE).

4. Self-Correlation and Codebook-Free Information Assessment

GIME integrates codebook-free, self-correlation-based observables, particularly for binary or bitstream data, following Viznyuk (0803.1695). Key metrics include:

• CR(n): Shifted Hamming distance
• MF(n): $M - 2 \cdot CR(n)$ (first-order correlation)
• MF: Aggregate mean of MF(n) across all shifts
• DF: Second moment (dispersion) of MF(n)
• AdjMF: Adjusted aggregate (grows with information content)

For random data, MF $\rightarrow 0$, DF $\approx 1$; for structured/low-entropy data, MF $\gg 0$, DF $\gg 1$. These metrics do not require symbol-probability assumptions (unlike entropy), are $O(M^2)$, and scale to tens of megabits. When combined with traditional entropy/compression proxies, they afford robust multi-variate classification of information richness—a critical GIME submodule for binary or high-throughput environments (0803.1695).

5. Evaluating Metric Quality: Measurement Theory, Resolving Power, and Unified Benchmarks

GIME also provides a lens for evaluating the evaluation metrics themselves.

• Measurement Scales (Stevens’ Hierarchy):
  • Information retrieval (IR) metrics map categorical data (e.g., SERPs) to real-valued scales. Provided these mappings reflect externally grounded user utility (monetary value, willingness to pay), IR metrics are valid ratio-scale quantities. Uniform spacing of metric values is not required, and forced uniformization may sever the mapping from real-world utility (Moffat, 2022).
• Resolving Power Framework:
  • For threshold-free metrics (AUROC, AUPRC), distinguishing ability is quantified as the signal-to-noise ratio:

  $$\text{Resolving Power}(M) = \frac{\beta_1}{\sigma_M}$$

  where $\beta_1$ is the local derivative of $M$ with respect to a reference scale $Q$ (e.g., AUROC) and $\sigma_M$ is its sampling standard deviation. Empirically, AUROC usually dominates AUPRC in resolving power, except when searching among high-quality classifiers for low-prevalence outcomes (Beam, 2023).

• BEAMetrics and Human Alignment:

  • Robust evaluation requires assessing not only metric-to-metric relationships but also metric-to-human-judgment correlation across tasks, languages, and dimensions. Pearson/Spearman correlations between leading NLG metrics (BLEURT, ROUGE, BERTScore, etc.) and normalized human ratings show that no metric universally dominates; task-dependence and knowledge requirements are major factors in alignment (Scialom et al., 2021).

6. Recommendations and Limitations

GIME's design principles include:

• Explicit multi-criteria optimization—eschewing monolithic score aggregation in favor of domain- and developer-specified trade-off surfaces.
• Standardized notation and protocol documentation (e.g., data cards, annotator metadata) for reproducibility and extensibility.
• Orthogonality of metric dimensions to ensure comprehensive coverage of information richness/fidelity.
• Empirical validation of all metric–performance correlations, supported by ablation against random/active learning baselines.
• Modular integration of codebook-free, entropy/compression-based, and self-correlation-based metrics for broad applicability.

Limitations noted in specific domains include reduced class-separation for short bitstrings (self-correlation methods), potential adversarial circumvention of shift-based metrics, and the necessity to balance interpretability against maximizing resolving power in metric choice (Xu et al., 2 Jan 2025, 0803.1695, Beam, 2023).

7. Conceptual and Practical Impact

GIME reifies "information" into a rigorously quantifiable, multidimensional construct, shifting data selection and model evaluation from ad hoc, domain-specific heuristics to a principled, performance-driven paradigm. This yields verifiable efficiency gains (e.g., substantial GPU, human, and energy savings in judicial AI), and standardizes the benchmarking and adoption of evaluation metrics themselves, including in human-grounded, context-dependent settings. The framework is currently validated across CTR prediction, legal decision-making, meteorological forecasting, and natural language generation but is, by construction, extensible to new domains and future evaluation challenges (Xu et al., 2 Jan 2025, Scialom et al., 2021).