Are UFOs Real? Introducing Unified Factored Organizers
Abstract: Implicit in much research in deep learning is the assumption that if a network achieves low loss, then it must have learned a good representation of its training data. Recent work by Kumar et al. (2025, arXiv:2505.11581) questions this assumption, arguing that networks trained with conventional gradient descent achieve low loss despite learning fractured entangled representations (FER), leaving related concepts disconnected and unrelated concepts entangled inappropriately. Kumar et al. further show that dramatically better representations are possible: a network evolved through an open-ended, human-in-the-loop process learned unified factored representations (UFR) that capture the regularities of the domain far better than a counterpart with nearly identical outputs trained with conventional gradient descent. Inspired by UFR, this paper introduces unified factored organizers (UFOs), networks trained to model regularities in the representations of other networks. A network can then be trained both to achieve low loss on its task objective and to align its representations with the model learned by the UFO. Networks trained with UFOs are shown to learn qualitatively superior representations in an image-regression domain where representations are easily interpretable. To provide quantitative evidence for the impact of UFOs on representation learning, a transformer is trained with and without UFO regularization to grok a modular arithmetic problem set. When combined with weight decay, UFOs can reduce the number of transformer training steps needed to grok by more than 10x relative to weight decay alone. Without weight decay, UFOs can reduce the number of transformer training steps needed to grok by more than 100x relative to an unregularized baseline. If UFOs are real, they suggest a new paradigm where loss functions themselves are trained to learn an effective model of good representation.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
This paper asks a big question: Can we train neural networks to learn better internal patterns, not just produce correct answers?
Neural networks are computer systems inspired by the brain. They learn by adjusting many numbers called weights. Usually, researchers judge a network by checking how many answers it gets right. This is called its loss: a lower loss usually means better performance.
However, the paper argues that a network can get good answers while still having messy and confusing internal patterns. The authors introduce a new method called a Unified Factored Organizer, or UFO, to encourage neural networks to develop cleaner, more organized internal representations.
Despite the name, UFOs here do not mean flying saucers. They are additional neural networks that help organize another neural network’s thoughts.
2. What questions are the researchers asking?
The paper focuses on several main questions:
- Can a neural network be encouraged to learn simple, regular, and meaningful internal patterns?
- Can UFOs make a network’s hidden representations more organized?
- Can this method help a network learn faster?
- Can UFOs help a transformer discover a general rule instead of simply memorizing examples?
The authors are especially interested in the difference between two kinds of internal representations:
- Fractured entangled representations, or FER: related ideas are kept apart, while unrelated ideas become mixed together.
- Unified factored representations, or UFR: related ideas are grouped together, and different features are represented separately and clearly.
An everyday analogy is a messy bedroom compared with a well-organized room. In a messy room, socks may be mixed with books and related objects may be far apart. In an organized room, similar things are stored together, making it easier to find and use them.
3. How did the researchers study this?
The researchers used two main experiments.
Experiment 1: Creating pictures
They trained neural networks to produce images, such as a butterfly. They used a type of network related to Picbreeder, an experiment where people evolved pictures by repeatedly choosing interesting images.
These networks take information about a pixel’s position, such as its horizontal and vertical coordinates, and calculate the pixel’s color. By checking every pixel, the network creates a complete picture.
The researchers could also examine what each hidden part of the network did. They made a picture showing how strongly each hidden node responded at different pixel positions. These pictures acted like maps of the network’s internal activity.
A well-organized network might have hidden activity maps showing smooth curves, symmetry, or repeated patterns. A poorly organized network might have noisy and random-looking maps.
The researchers compared:
- A normal network trained using ordinary gradient descent.
- A network trained with UFO regularization.
Gradient descent is a common learning method. Imagine trying to walk downhill in fog: at each step, you check which direction goes downward and move that way. In machine learning, the “downhill” direction is the direction that reduces mistakes.
A UFO is a smaller autoencoder network that tries to recreate the main network’s internal activity. An autoencoder is like a student who compresses a long message into a short summary and then tries to rebuild the original message from that summary.
If the UFO can reconstruct the hidden activity easily, that suggests the activity has a simple, regular structure. The main network is therefore encouraged to create internal patterns that are easier for the UFO to understand.
Experiment 2: Learning modular arithmetic
The researchers also trained small transformers, the type of neural network used in systems such as LLMs.
The transformers had to solve modular arithmetic problems, such as division using numbers that wrap around after reaching a certain limit. Modular arithmetic is similar to a clock: after 12 comes 1 again.
The researchers studied a phenomenon called grokking. Grokking happens when a network first memorizes its training examples but later suddenly learns the general rule and performs well on new examples.
For example, a student might memorize the answers to a few practice problems. Later, the student understands the multiplication rule and can solve many new problems. That deeper understanding is similar to grokking.
The researchers compared transformers trained:
- Without special regularization.
- With weight decay, a common technique that discourages overly complicated weights.
- With UFO regularization.
- With both UFO regularization and weight decay.
They measured how many training steps each transformer needed before it could correctly solve almost all new modular arithmetic problems.
4. What did the researchers find?
UFOs produced more organized internal patterns
In the image experiments, networks trained with UFOs created hidden activity maps that were generally more regular and structured than those created by ordinary training.
This happened even when both networks produced nearly identical final images. In other words, two networks could draw the same butterfly, but one could have a much cleaner internal understanding of the butterfly’s structure.
The UFO-trained network was more likely to represent features such as symmetry and repeated shapes. This supports the idea that a network’s internal organization matters, even when its final answers look correct.
UFOs helped transformers learn general rules faster
The transformer experiments showed an even more practical result: UFOs could greatly speed up grokking.
According to the paper:
- When combined with weight decay, UFOs reduced the number of training steps needed for grokking by more than 10 times compared with weight decay alone.
- Without weight decay, UFOs reduced the number of training steps by more than 100 times compared with an unregularized network.
For example, one comparison showed that a UFO-trained transformer needed about 12 times fewer training steps than a transformer using weight decay alone.
This suggests that UFOs may help a network move away from memorizing individual examples and toward discovering the deeper pattern behind them.
UFOs work by encouraging compressible representations
The UFO does not simply force the main network to have fewer neurons. Instead, it encourages the network’s activity to have patterns that can be summarized and reconstructed.
The researchers also gradually increase the UFO’s influence during training. This is called a warm-up schedule. It is similar to beginning exercise gently before increasing the difficulty. If the organizing pressure were too strong at the beginning, the network might struggle to learn the task at all.
The UFO is also updated selectively. It mostly learns from the easier-to-reconstruct patterns instead of immediately adapting to every messy pattern. This helps preserve pressure on the main network to improve its irregular representations.
5. Why are these results important?
The findings challenge the idea that a low error score automatically means a neural network has learned well.
A network may perform well on familiar examples while using confusing internal shortcuts. Such a network might:
- Fail when it sees new kinds of data.
- Forget older knowledge when learning something new.
- Be very confident while giving incorrect answers.
- Have difficulty discovering general rules.
If UFOs help networks form more organized representations, they might eventually improve these abilities.
The method could possibly help with:
- Generalization: solving new problems rather than memorizing old ones.
- Continual learning: learning new information without forgetting earlier knowledge.
- Creativity: combining learned features in useful new ways.
- LLMs: developing clearer internal structures for language and ideas.
However, the paper does not prove that UFOs solve all these problems. The strongest evidence comes from relatively small image networks and transformer arithmetic tasks. More research is needed to test whether the method works on much larger and more complicated systems.
Conclusion
This paper introduces UFOs, extra neural networks that encourage another network to develop simpler and more organized internal representations.
The main lesson is that getting the right answer is not the whole story. A neural network may also need to represent information in a clear and useful way. The experiments suggest that UFOs can improve these internal patterns and help networks learn general rules much faster.
If future studies confirm these results, UFOs could lead to a new approach in artificial intelligence: instead of only designing a fixed loss function, researchers could train a system that learns what good internal representations should look like.
Practical Applications
Immediate Applications
The paper’s results are primarily demonstrated on Picbreeder-style image-regression networks and small transformers solving modular arithmetic. Therefore, the most credible immediate applications are research, development, and diagnostic workflows rather than production deployment in high-stakes systems.
- Neural-network training acceleration for algorithmic and synthetic tasks — Software/AI research.
- extracts hidden activations or attention weights;
- feeds normalized representations to an auxiliary autoencoder;
- adds the reconstruction error to the primary loss;
- applies warm-up and selective-update schedules.
Dependencies: The speedups were measured on small, synthetic grokking problems. Equivalent gains on language, vision, or commercial-scale models are not established. The method also introduces extra computation and hyperparameters, including UFO capacity, regularization strength, and selectivity.
- A representation-quality benchmark for model development — Machine learning engineering and academia.
- task loss and validation performance;
- UFO reconstruction loss;
- representation dimensionality;
- layer-wise and head-wise regularity over time.
This would complement standard metrics such as accuracy, calibration, robustness, and loss rather than replace them.
Dependencies: Low reconstruction error is only a proxy for desirable representation quality. A UFO may reward compressibility that is unrelated to robustness, fairness, interpretability, or downstream utility.
- Visualization and interpretability of hidden representations — AI interpretability and education.
- ordinary gradient-descent training;
- UFO-regularized training;
- evolutionary or human-guided training;
- different regularization schedules.
Such a tool could help researchers and students see how symmetry, smoothness, and other domain regularities emerge inside a model.
Dependencies: The visual interpretability demonstrated in the paper depends heavily on CPPNs, where inputs correspond naturally to pixel coordinates. The same visualization is not directly available for LLMs.
- Faster experimentation with grokking and representation learning — Academic research.
- generalization transitions;
- weight decay;
- intrinsic dimensionality;
- attention-pattern formation;
- memorization versus algorithm learning.
Dependencies: Results may depend strongly on the modular arithmetic task, transformer size, optimizer, batch construction, and the definition of “grokking,” here operationalized as high validation accuracy.
- Regularization of attention patterns in small transformers — NLP and software tooling. The method can be applied not only to hidden activations but also to attention edges or groups of attention weights. A prototype training library could selectively regularize attention patterns that are irregular across examples or across heads.
Potential benefit: This may encourage more systematic attention behavior and provide a quantitative signal for comparing heads during training.
Dependencies: Regular attention is not necessarily meaningful attention. The paper does not show that UFO-regularized attention improves language-model quality, factuality, interpretability, or safety.
- Representation auditing for research models — AI safety and model evaluation.
- distribution shift;
- memorization;
- anomalous inputs;
- unstable internal computation;
- poorly learned subdomains.
Dependencies: High UFO error does not uniquely identify a safety problem. It may instead reflect legitimate complexity, rare examples, or insufficient UFO capacity.
- Improved teaching materials for neural-network regularization — Education.
- the distinction between loss and representation quality;
- auxiliary losses;
- autoencoder bottlenecks;
- selective optimization;
- grokking and delayed generalization.
Dependencies: The educational example should be presented as an empirical hypothesis and visualization aid, not as proof that all compressible representations are superior.
- Policy and governance guidance for model evaluation — AI policy. The paper supports a practical policy principle: model assessment should not rely exclusively on task loss or benchmark accuracy. For models used in consequential settings, evaluation programs could additionally examine representation stability, distribution-shift behavior, calibration, forgetting, and internal regularity.
Dependencies: No direct evidence in the paper shows that UFO regularization improves these governance-relevant properties. Additional validation would be required before incorporating UFO metrics into certification standards.
Long-Term Applications
The following possibilities are suggested by the paper’s hypothesis that unified, factored, and compressible representations may support generalization, continual learning, creativity, and interpretability. They require substantial research beyond the reported experiments.
- More robust out-of-distribution generalization — Healthcare, finance, robotics, and autonomous systems.
- medical-imaging models that organize features around anatomical structure;
- financial models that separate stable economic factors from transient noise;
- robotic perception systems that represent object geometry and symmetry;
- vision systems designed to generalize across lighting, pose, or environment changes.
Dependencies: The paper does not test distribution shift. Compressibility may improve or harm robustness depending on which regularities the UFO captures. Domain-specific representation targets and external robustness benchmarks would be necessary.
- Continual-learning systems with reduced catastrophic forgetting — Robotics, personalized software, and adaptive agents.
- UFO regularization during sequential task learning;
- replay or rehearsal buffers;
- parameter-efficient adapters;
- layer-wise monitoring of representation drift.
Dependencies: The paper provides no continual-learning experiments. Preventing forgetting may require explicit modularity, task identifiers, memory mechanisms, or replay in addition to UFO regularization.
- More interpretable LLMs — NLP and foundation-model engineering.
- representation dashboards for tracking regularity by layer and token type;
- candidate circuits for mechanistic interpretability;
- auxiliary models that predict or summarize activation structure;
- training objectives that combine language-model loss with representation constraints.
Dependencies: LLM activation spaces are much larger and more heterogeneous than the paper’s experimental domains. Intracomponent representations are difficult to scale because they depend on covering many inputs; intercomponent UFOs may be more practical but remain unvalidated for LLMs. Computational overhead, sequence-length dependence, and possible suppression of useful diversity are major concerns.
- Automated discovery of reusable neural representations — Software and foundation-model platforms.
A successful UFO system could encourage representations that transfer across tasks, enabling reusable latent modules or adapters. A future platform might:
- train a base model with UFO regularization;
- identify compact latent factors;
- attach task-specific heads or adapters;
- measure whether new tasks reuse existing factors rather than relearning them.
Dependencies: The paper does not measure transfer learning, factor disentanglement, or modular reuse. A small reconstruction error does not guarantee semantic factorization.
- Improved model calibration and uncertainty estimation — Healthcare, finance, and decision support.
- UFO reconstruction error as an internal anomaly signal;
- joint prediction and representation-quality monitoring;
- abstention systems that defer when both predictive confidence and representation regularity are low.
Dependencies: The relationship between representation compressibility and calibrated uncertainty is speculative. Any deployment would require calibration tests, subgroup analysis, adversarial evaluation, and clear failure-mode handling.
- Creative generative systems with structured variation — Design, media, and scientific visualization.
- interactive image-generation systems;
- evolutionary design tools;
- procedural graphics and animation;
- scientific visualization of structured objects.
Dependencies: The paper does not evaluate creativity or generative diversity. Excessive regularization could reduce novelty, mode coverage, or desirable irregularity. Human evaluation and diversity-sensitive metrics would be required.
- Neural controllers for robotics and embodied agents — Robotics.
- world models;
- reinforcement learning;
- equivariant neural architectures;
- memory and continual-learning mechanisms.
Dependencies: Robotic observations are high-dimensional, nonstationary, and temporally correlated. The current batch-based formulation may need temporal, multimodal, and online adaptations. Real-world safety testing would be essential.
- Scientific and engineering surrogate models — Energy, climate, materials, and physics.
- energy-demand forecasting;
- fluid and climate emulation;
- materials-property prediction;
- reduced-order engineering simulation.
Dependencies: Explicit physics-informed losses, equivariant architectures, or conservation constraints may be more appropriate than generic compressibility. The UFO would need to be tested against known physical laws and extrapolation regimes.
- Compute-efficient training of large models — Cloud infrastructure and AI hardware.
- low-frequency UFO updates;
- shared UFOs across layers;
- intercomponent regularization for scalability;
- compressed activation sampling;
- asynchronous auxiliary training.
Dependencies: The auxiliary autoencoder itself adds memory, computation, and communication costs. Net energy and cost savings must be measured rather than inferred from fewer optimization steps.
- Policy standards for representation-aware AI evaluation — Regulation and institutional governance.
- representation stability under perturbations;
- behavior under distribution shift;
- forgetting during updates;
- calibration and abstention;
- internal anomaly indicators.
UFO-based measures could become one component of such evaluations if they are shown to predict operationally important properties.
Dependencies: A policy standard should not mandate UFOs specifically until independent studies establish causal links to safety, robustness, fairness, or reliability. Metrics must also be resistant to gaming and comparable across architectures.
- Learned loss functions for representation quality — Long-term machine-learning research.
- UFOs trained across multiple tasks and architectures;
- organizers based on predictive coding, contrastive learning, or diffusion models;
- human- or domain-supervised notions of “good” representation;
- meta-learning of regularization schedules;
- multi-objective organizers balancing compressibility, diversity, robustness, and interpretability.
Dependencies: The central unresolved issue is whether compressibility is a sufficiently reliable definition of representation quality. A learned organizer could encode undesirable biases, collapse useful complexity, or favor artifacts of the training distribution. Independent benchmarks and causal evaluations are necessary before treating UFOs as general-purpose learned loss functions.
Glossary
- Attention edge: A query–key connection between two fixed sequence positions within an attention head. “an attention edge describes the query-key connection between two fixed sequence positions in a given attention head”
- Autoencoder: A neural network trained to reconstruct its input, typically through a compressed internal representation. “This auxiliary loss function is an autoencoder”
- Auxiliary loss: An additional optimization objective used alongside the primary task loss to shape model behavior. “ then serves as an auxiliary loss for the primary network .”
- Bottleneck: A deliberately low-dimensional intermediate layer that limits the information passing through a network. “the UFO autoencoder can feature a small, aggressive bottleneck”
- Catastrophic forgetting: The loss of previously learned knowledge when a model is trained on new data. “catastrophically forget what it previously learned when tasked with integrating new data”
- Compositional pattern producing network (CPPN): A neural network that generates structured patterns, such as images, by mapping coordinates to values. “a separate compositional pattern producing network (CPPN), a kind of neural network inspired by evolutionary developmental biology”
- Composite loss: A single objective formed by combining a task loss with one or more additional loss terms. “The primary network is trained to minimize the composite loss”
- Continual learning: A learning setting in which a model integrates new information over time while retaining earlier knowledge. “networks evolved in Picbreeder are adaptable, evolvable, and amenable to continual learning”
- Cross entropy: A loss function that measures the discrepancy between predicted probability distributions and target labels. “cross entropy for next token prediction”
- Effective dimensionality: The number of dimensions that meaningfully characterize a representation, which may be smaller than its nominal layer width. “The intrinsic or effective dimensionality of hidden layer activations”
- Entangled representation: A representation in which unrelated concepts are mixed together rather than encoded independently. “fractured entangled representations (FER)”
- Evolutionary developmental biology: A field studying how evolutionary processes and developmental mechanisms jointly shape biological form. “inspired by evolutionary developmental biology”
- Fractured entangled representations (FER): Representations in which related concepts are disconnected and unrelated concepts are inappropriately entangled. “FER means that the underlying representations of trained networks leave related concepts disconnected, while unrelated concepts are entangled inappropriately.”
- Gradient descent: An optimization method that updates parameters in the direction that reduces a loss function. “Neural networks trained with conventional gradient descent”
- Grokking: Delayed generalization in which a model fits its training data before eventually generalizing to held-out data. “Grokking, the phenomenon of delayed generalization after initially fitting the training data”
- Hidden node activation: The numerical response produced by an internal network unit for a particular input. “the activations of any hidden node can be visualized as an image”
- Hyperparameter: A training configuration selected externally rather than learned directly by the model. “Two transformers with matching hyperparameters are trained on division modulo $97$”
- Intrinsic dimensionality: The dimensionality of the lower-dimensional structure on or near which data representations lie. “the intrinsic dimension that can be orders of magnitude smaller than the layer width”
- Layerization: The process of reorganizing a network into explicit computational layers while preserving its function. “This MLP version is constructed through a layerization process.”
- Low-dimensional manifold: A lower-dimensional geometric surface or structure embedded within a higher-dimensional representation space. “UFOs may encourage representations to lie near a low-dimensional activation manifold”
- Mechanistic interpretability: The study of the internal mechanisms, features, and circuits that produce a neural network’s behavior. “Although mechanistic interpretability has made some progress in describing specific features or circuits”
- Mean-squared error: A loss measure calculated as the average squared difference between predicted and target values. “where be the mean-squared reconstruction error”
- Modular arithmetic: Arithmetic involving remainders under a fixed modulus. “When UFOs are combined with weight decay, they can reduce the number of transformer training steps needed to grok modular division problems”
- Modular division: Division performed within modular arithmetic, typically over a finite modular domain. “a transformer is trained with and without UFO regularization to grok a modular arithmetic problem set”
- NeuroEvolution of Augmenting Topologies (NEAT): An evolutionary algorithm that evolves neural networks by modifying their connection weights and topologies. “based on a version of the NeuroEvolution of Augmenting Topologies (NEAT) algorithm”
- One-hot vector: A vector in which one position is active and all other positions are zero, commonly used to encode categorical information. “ can be a one-hot vector describing the layer from which was extracted.”
- Open-ended search: An evolutionary or optimization process without a fixed target endpoint, allowing novelty and complexity to continue developing. “an explicitly open-ended search process like in Picbreeder”
- Primary network: The network trained to perform the main task while also satisfying the auxiliary regularization objective. “The primary network is trained with gradient descent both to achieve low loss on some task objective”
- Regularization: A technique that adds constraints or penalties to training to encourage desirable model properties and reduce overfitting. “The UFO regularization approach described first aims to directly encourage individual hidden node activation maps”
- Representational optimism: The assumption that achieving low loss implies that a network has learned a good representation. “This presumption that the field need only develop a descriptive theory of representation reflects widespread representational optimism”
- Selective update: An optimization procedure that updates a model using only a chosen subset of examples or errors. “In an attempt to avoid training away the UFO's regularization pressure on irregular activations, the UFO parameters are updated selectively.”
- Stop-gradient: An operation that prevents gradients from propagating through a specified tensor during backpropagation. “a stop-gradient is placed on the representations so that the primary network is not updated a second time.”
- Transformer: A neural network architecture that uses attention mechanisms to process sequences and model relationships among their elements. “a transformer is trained with and without UFO regularization”
- Unified factored organizer (UFO): An auxiliary autoencoder trained to model and encourage regular, compressible representations in another neural network. “we refer to the autoencoder in this context as a unified factored organizer (UFO).”
- Unified factored representation (UFR): A representation that captures the underlying regularities of a domain in a structured and separable form. “neural networks should learn unified factored representations (UFR) that respect the underlying regularities of the training data.”
- Validation accuracy: The proportion of held-out examples that a model classifies or predicts correctly. “the transformer trained with UFO regularization groks (i.e.\ achieves validation accuracy)”
- Weight decay: A regularization method that penalizes large parameter values during optimization. “When UFOs are combined with weight decay”
- Z-score normalization: Rescaling data by subtracting its mean and dividing by its standard deviation. “each hidden node's activations are -score normalized independently over to have zero mean and unit variance.”



