Perceptual Taxonomy

• Perceptual taxonomy is a formal system that organizes entities based on perceptual features like shape, color, and motion.
• It integrates cues from vision, audition, and material recognition using spatialization and pre-attentive encoding for objective classification.
• The framework advances machine learning and cognitive research by providing measurable, operational models for concept differentiation.

A perceptual taxonomy is a formal system for organizing concepts, entities, or phenomena according to the principles by which they are distinguished, grouped, or classified by perceptual or cognitive mechanisms. Unlike taxonomies based solely on semantic, ontological, or engineering criteria, perceptual taxonomies ground their divisions in the distinctive features, operations, or representations relevant to human or machine perception. Major applications span vision, audition, material recognition, olfaction, multimodal concept learning, and cognitive architecture, encompassing both descriptive analyses and operational frameworks suitable for machine learning and computational neuroscience.

1. Conceptual Basis and Motivations

The foundational rationale for a perceptual taxonomy is to move beyond arbitrary or legacy-driven classification schemes, instead anchoring all groupings and distinctions in the operations or axes available to perceptual systems. In human vision and visualization research, this approach emphasizes the pre-attentive channels (position, shape, color, motion) as the basic building blocks through which information is mapped, detected, and interpreted (Jr. et al., 2015, Jr et al., 2015). In cognitive science and machine learning, perceptual taxonomies seek to uncover the geometric, probabilistic, or logical constraints that organize object or concept spaces so as to reflect the discriminative capacities and invariances of an observer or model (Chung et al., 2017, Sanders et al., 22 Oct 2025, Victor et al., 2023).

This approach contrasts with purely inventory-based taxonomies (which enumerate categories) or purely semantic/hierarchical taxonomies (which may be motivated by language or external logic), by instead explicitly relating taxonomic groupings to measurable properties of perception, similarity, and feature extraction.

2. Core Structures Across Modalities

Visual and Data Visualization Domains

Perceptual taxonomies for visualization and vision consistently structure techniques as compositions of spatialization modes and pre-attentive encodings (Jr. et al., 2015, Jr et al., 2015):

• Spatialization: The mapping from data to position, instantiated as structure exposition (exposes relationships through layouts), patterned positioning (regular grids/sequences), projection (mapping via explicit axes or functions), and reproduction (using inherent coordinates, as in geographic maps).
• Pre-attentive Stimuli: Further distinctions are driven by the assignment of shape (for differentiation, correspondence, meaning, relationship) and color (for class-labeling and value correspondence). Motion or animation acts as a dynamic channel.

The design grammar resulting from these criteria formalizes every visualization as a sequence of spatialization operations and their associated perceptual encodings.

Dimensionality in Perceptual Geometry

In computational cognitive science, a perceptual taxonomy is also realized as a set of interpretable axes in a psychological or neural feature space (Sanders et al., 22 Oct 2025):

• Human and model-derived similarity judgments elicit a latent space with axes such as lightness, grain, surface texture, chromaticity, and organization.
• Multidimensional scaling (MDS) combined with Procrustes alignment enables rigorous matching between human and model “perceptual spaces,” allowing the recovered dimensions to serve as the basis for objective classification and generative models.

Sensory-Specific and Multimodal Domains

In audition, sound scene analysis employs an extensible cluster-graph model in which atomic perceptual labels (e.g., “speech,” “street,” “office”) form nodes connected in overlapping clusters (Environment, Event, Context). This structure is designed for open-set extensibility and synonym resolution (Bear et al., 2018). For olfaction, expert-derived and data-driven hierarchies organize molecular odor descriptors into nested families based on perceptual similarity or empirical co-occurrence, validated through both expert consensus and machine learning performance (Sajan et al., 11 Aug 2025).

3. Formal Models and Learning Frameworks

Probabilistic and Geometric Models

Taxonomy induction from multi-modal data leverages probabilistic models that encode tree structures as latent variables, learning edge potentials that combine visual similarity (deep image embeddings) and textual/semantic similarity (word embeddings) (Zhang et al., 2016). Taxonomic structure is enforced via global priors, regularized likelihoods, and maximum spanning tree extraction.

For continuous representations, perceptual manifolds generated by neural responses to object variations are formalized as sets in high-dimensional spaces. Taxonomy is grounded in quantitative descriptors such as manifold radius $R_M$ and dimension $D_M$; classification capacity is predicted via statistical-mechanical theory, relating taxonomic separability to manifold geometry and label sparsity (Chung et al., 2017).

Hierarchical and Interactive Induction

Frameworks for incremental or interactive taxonomy construction recapitulate human concept formation by recursively decomposing object recognition into genus (most general class) and differentia (features distinguishing siblings). Algorithms employing self-supervised embeddings and extreme value theory enable recursive, interpretable mappings in open-world settings (Erculiani et al., 2023).

Ordinal and Tree Compatibility Indices

To robustly characterize whether empirical similarity judgments support a tree-structured perceptual taxonomy, ordinal characterizations analyze rank-order constraints—testing local compatibility with symmetry, ultrametricity, and additive-tree metrics. This yields statistical indices (Bayes-factors) that quantitatively indicate the degree to which a dataset admits a hierarchy or more general tree (Victor et al., 2023).

4. Applications and Benchmarks

Perceptual taxonomies are integral to the construction and evaluation of structured benchmarks for scene understanding in vision-LLMs (Lee et al., 24 Nov 2025). These benchmarks go beyond surface recognition to require the inference of physically grounded object properties—material, physical attributes, affordance, and function—using challenging multi-step reasoning tasks. Empirical results consistently reveal gaps between associative/pattern-matching behaviors of large models and the structured, hierarchical reasoning explicit in perceptual-taxonomy-guided designs. This motivates the use of taxonomy-guided prompting and the formulation of new evaluation protocols focused on multi-attribute inference.

In image forensics, perceptual taxonomies categorize miscompressions by their impact on amplitude, geometry, or shape, and flag semantically critical errors (e.g., symbol-altering miscompressions) to guide risk mitigation, codec design, and forensic policy (Hofer et al., 9 Sep 2024).

Material recognition systems implement taxonomically structured representations, enabling robust few-shot generalization and interpretable attribute discovery aligned with human judgments (Schwartz et al., 2016).

5. Evaluation, Extension, and Analysis

Perceptual taxonomy systems are evaluated using metrics appropriate to their modality and representation:

• Visualization/Recognition: User studies measuring pre-attentive detection, analytic metrics for stimulus discriminability (Jr. et al., 2015).
• Taxonomic Learning: Ancestor-F1 scoring, tree-recall/precision, or classification accuracy over hierarchically defined properties (Zhang et al., 2016, Lee et al., 24 Nov 2025).
• Geometry and Similarity: Variance explained in MDS solutions; Procrustes correlations to human axis structure; tree compatibility indices $I_{\rm sym}$, $I_{\rm umi}$, $I_{\rm addtree}$ (Sanders et al., 22 Oct 2025, Victor et al., 2023).
• Machine Learning Benchmarks: AUROC, F1, precision-recall for class labels defined according to perceptual clusters or families (Sajan et al., 11 Aug 2025, Schwartz et al., 2016).

Extensibility is central: systems such as cluster-graph sound taxonomies and interactive visual concept hierarchies are explicitly constructed to absorb new labels, merge synonyms, and accommodate cross-modal signals (Bear et al., 2018, Erculiani et al., 2023). Error analysis and feature attribution further reveal where and when perceptual taxonomic boundaries correspond to salient chemical, semantic, or functional distinctions in downstream learning tasks (Sajan et al., 11 Aug 2025).

6. Limitations and Future Directions

Key challenges include:

• Modality Coverage: Underrepresentation of non-visual modalities (e.g., haptic and auditory) in both empirical foundations and operational taxonomies (Fagerholm et al., 2022).
• Semantic Drift and Discrepancy: Data-driven groupings may merge perceptually distinct descriptors due to co-labeling, while expert-crafted hierarchies risk embedding legacy biases (Sajan et al., 11 Aug 2025).
• Scalability and Supervision: Many incremental or interactive models require human-in-the-loop correction; automated step-wise induction of genus/differentia, or broad population studies for geometric/ordinal similarity data, remains resource-intensive (Erculiani et al., 2023, Victor et al., 2023).
• Theoretical Generality: Local pointwise compatibility with trees or ultrametrics does not guarantee a global fit; additional structure may be required to globally induce a perceptual taxonomy (Victor et al., 2023).
• Integration with Large Foundation Models: Taxonomy-guided prompting shows promise for enhancing human-like reasoning in vision-language systems, but further advances are needed for robust sim-to-real transfer, multi-attribute inference, and counterfactual reasoning (Lee et al., 24 Nov 2025).

Ongoing research seeks to unify perceptual taxonomy frameworks across modalities, develop automated discovery tools via MDS and deep embeddings, and formalize structure-aware training and evaluation protocols for artificial agents and cognitive models.