TANGOS Framework: Neural Reg. & Cosmo DB

• TANGOS is a dual-purpose framework that addresses overfitting in tabular neural networks and streamlines cosmological simulation data management.
• It employs specialized attribution regularizers to enforce sparsity and orthogonality in hidden representations, enhancing model diversity and accuracy.
• In cosmological applications, Tangos transforms simulation workflows with a modular, schema-free database platform enabling rapid, reproducible analyses.

The TANGOS framework refers to two distinct research efforts: (1) a regularization strategy for tabular neural networks based on gradient orthogonalization and specialization, and (2) a database-driven analysis platform for cosmological simulations. Both frameworks address data management and interpretability challenges in their respective domains, employing advanced computational methods and extensible architectures.

1. Gradient Orthogonalization and Specialization in Tabular Neural Networks

TANGOS—Tabular Neural Gradient Orthogonalization and Specialization—addresses the persistent issue of overfitting in deep learning models for structured tabular data. Unlike unstructured domains (e.g., images, language), tabular data lack intrinsic spatial or temporal inductive biases, leaving fully-connected networks susceptible to redundant co-adaptation across hidden units. Standard regularizers such as weight decay, dropout, and input noise fail to directly control the degree to which individual neurons attend to specific features.

The key innovation in TANGOS is to regularize the latent unit attributions—the gradients of hidden activations with respect to input features—by encouraging each neuron to both specialize (focus on a sparse subset of features) and be orthogonal (uncorrelated) to other units' feature attributions. This is realized by embedding attribution-based regularizers directly into the loss function, thus enforcing sparsity and diversity in the hidden representations (Jeffares et al., 2023).

2. Mathematical Formulation

Let $x\in\mathbb{R}^{d_X}$ be the input, $y$ the target, and $f(x;\theta)$ the network with hidden layer activations $h(x) = (h_1(x),\dots,h_{d_H}(x))$. Define the attribution vector of hidden unit $i$ as $$g_i(x) = (\partial h_i(x)/\partial x_1, \dots, \partial h_i(x)/\partial x_{d_X})$$.

Two distinct regularization penalties are introduced:

• Specialization Regularizer: Enforces sparsity in each hidden unit's gradient attribution vector, using a mini-batch $\ell_1$ penalty:

$$\mathcal{R}_{\text{spec}} = \frac{1}{B} \sum_{b=1}^{B} \frac{1}{d_H} \sum_{i=1}^{d_H} \|g_i(x_b)\|_1$$

• Orthogonalization Regularizer: Penalizes pairwise cosine similarity (or squared dot product) between the attribution vectors of different units, encouraging them not to attend to overlapping feature sets. In cosine form:

$$\mathcal{R}_{\text{orth}} = \frac{1}{B} \sum_{b=1}^B \frac{1}{C} \sum_{1 \leq i < j \leq d_H} \cos(g_i(x_b), g_j(x_b))$$

with $C = d_H(d_H-1)/2$.

The overall training objective is:

$y$0

where $y$1 is the standard predictive loss (Jeffares et al., 2023).

3. Practical Implementation and Computational Considerations

Computation of the orthogonalization penalty is quadratic in hidden unit count, but in practice TANGOS samples a fixed subset $y$2 of unit pairs (e.g., $y$3), reducing per-batch cost to linear in $y$4. Attribution vectors are computed via automatic differentiation within deep learning libraries. Training integrates seamlessly with auto-grad engines, and the approach remains compatible with existing regularizers such as L1/L2 weight decay, dropout, batch normalization, input noise, and MixUp. Hyperparameter grids typically use $y$5 and $y$6 (Jeffares et al., 2023).

4. Empirical Performance and Theoretical Justification

Benchmarked on 20 UCI datasets (10 regression, 10 classification, 5-fold CV, held-out test):

• TANGOS outperforms or matches state-of-the-art regularizers, attaining best test error on 10/20 datasets and second best on 6 more.
• Statistically, Wilcoxon tests versus L2-regularization yield $y$7 (regression) and $y$8 (classification).
• When combined with standard regularizers, TANGOS consistently improves generalization.
• Ablation studies indicate both specialization and orthogonalization are critical; removing either degrades performance.
• For large $y$9, subsampling as few as $f(x;\theta)$0 pairwise terms maintains $f(x;\theta)$198% of the regularization benefit at minimal compute overhead.
• Applied to FT-Transformer architectures on Jannis and Higgs datasets, TANGOS narrows the accuracy gap to tree-based models (XGBoost, CatBoost) by 0.003–0.007 (Jeffares et al., 2023).

The theoretical intuition leverages the ensemble perspective: the penultimate layer acts as an ensemble of weak learners $f(x;\theta)$2, where TANGOS lifts prediction diversity by decorrelating unit attributions, reducing ensemble variance and error according to Krogh & Vedelsby decomposition.

5. Practitioner Guidance and Extensions

For practical deployment:

• Initial settings: $f(x;\theta)$3, $f(x;\theta)$4, $f(x;\theta)$5, batch size $f(x;\theta)$6.
• TANGOS is applied on top of conventional regularization, not as a replacement.
• The approach is extensible to multiple hidden layers and to non-ReLU networks via alternative attribution metrics (e.g., Integrated Gradients, SmoothGrad).
• Domain-specific extensions include constraining the specialization penalty to enforce coverage over particular feature groups.
• Limitations involve increased compute/memory for Jacobian calculations and the need for hyperparameter tuning per domain. Compute overhead is mitigated by pairwise subsampling (Jeffares et al., 2023).

6. TANGOS in Cosmological Simulation Data Management

A separate framework, Tangos (the Agile Numerical Galaxy Organization System), addresses the organization and analysis of large-scale cosmological simulation outputs (Pontzen et al., 2018). Tangos decouples the reduction of raw simulation data from the subsequent organization and query of derived physical quantities, providing:

• A Python and web-based interface for database-driven analysis.
• Modular architecture with six sub-packages: core (relational schema), live_calculation (mini-language for queries), relation_finding (merger-tree traversal), properties (custom property class hierarchy), parallel_tasks (MPI/multiprocessing), input_handlers (snapshot format abstraction), and web (browser UI).
• Schemaless key–value property storage, weighted merger-tree links, and efficient multi-hop SQL strategies.
• Support for custom observable definitions via subclassing, parallelized data extraction, and succinct single-line queries.
• Integration with Pynbody, yt, and custom data formats.
• Built-in reproducibility and sharing mechanisms via single-file databases, versionable pipelines, and portable environments.

Performance benchmarks demonstrate linear scaling and more than $f(x;\theta)$7 speedups over naive scripting approaches. The framework transforms the traditional simulation analysis workflow, enhancing transparency, reproducibility, and collaborative science (Pontzen et al., 2018).

7. Distinction and Nomenclature

Despite sharing an acronym, the TANGOS frameworks described above are unrelated conceptually and technically. In machine learning, TANGOS refers to a regularization strategy for latent space diversity in tabular neural networks; in computational astrophysics, it is an extensible database and analysis toolkit for organizing cosmological simulation outputs. Each system is widely cited within its respective research community, but they should not be conflated.