Avey Neural Architecture for Sequence Modeling

• Avey is a neural architecture family for sequence modeling that excludes traditional attention and recurrence, using a cosine similarity-based ranker to select the most relevant context splits.
• It comprises autoregressive and encoder-only variants that demonstrate high throughput and strong empirical performance, including >90% recall on long-context retrieval benchmarks.
• The design incorporates technical innovations such as selective retrieval, neural compression, and alternating static/dynamic layers, enabling efficient scaling to arbitrarily long sequences.

Avey is a neural architecture family for sequence modeling that systematically excludes both attention and recurrence, centering instead on explicit retrieval of salient context via a ranker and a compact but powerful neural processor. Its autoregressive formulation achieves strong performance on both short- and long-range language tasks, with recent encoder-only variants (Avey-B) matching or exceeding the empirical results and throughput of widely-used bidirectional Transformer encoders. Further, Avey serves as the foundational mechanism behind the Avey AI Benchmark in medical dialogue and structured entity annotation. This article presents the technical principles, mathematical mechanisms, and experimental findings that define Avey and its extended family.

1. Architectural Principles of Avey

Avey introduces a paradigm distinct from Transformer self-attention or RNN-style recurrence. The sequence is partitioned into non-overlapping splits, and a global ranker retrieves only the top-k most relevant splits for each position under processing. This retrieval is governed by an explicit cosine similarity-based MaxSim operator. Unlike self-attention, Avey decouples sequence length $N$ from its context width $C$, allowing efficient scaling to arbitrarily long contexts while focusing compute on relevant information.

Within each forward pass, the architecture comprises two core modules:

• Ranker: For each current split $S_c$, it computes $$r(S_c, S_p) = \sum_{e\in S_c}\max_{e'\in S_p}\cos(e, e')$$ for all previous splits $S_p$. The top-k splits by $r$ are selected as context.
• Neural Processor: A multitier network stacking Enricher, Contextualizer, and Fuser submodules. The Enricher projects and expands the concatenated context; the Contextualizer performs input-driven, cross-token mixing on a tail partition; the Fuser combines bypassed head and contextualized tail features with a projection.

Avey’s forward pass processes only those splits in the retrieved context, bounding per-position computation and memory. During training, ranking is $O(N^2d)$, but inference cost is amortized to $O(Nd)$, as the ranker is invoked only once per pass, not per layer (Hammoud et al., 12 Jun 2025).

2. Mathematical Foundations and Variants

The original Avey mechanism is strictly autoregressive. Training minimizes next-token cross-entropy: $$\mathcal L = -\sum_{t=1}^N \log \frac{\exp(z_{t,y_t})}{\sum_v \exp(z_{t,v})}$$ where logits $z_t$ are a function of the neural processor’s output.

In the Avey-B extension (Acharya et al., 17 Feb 2026), the design is reformulated for encoder-only, bidirectional pretraining:

• Ranker uses MaxSim scores similarly, but context retrieval is bidirectional.
• The processor alternates static (input-independent) and dynamic (cosine similarity-mixing only) layers:
  • Static layer: $C$0, learned parameters only.
  • Dynamic layer: $C$1 with $C$2.
  • Empirical findings favor an alternating pattern: Static→Dynamic→Static.

A learned neural compressor reduces the effective token count per split, maintaining constant throughput and memory as split width or k grows. Empirical results show a $C$3 throughput gain with $C$4–$C$5\% accuracy degradation on QA/IR (Acharya et al., 17 Feb 2026).

3. Empirical Benchmarks and Scaling Properties

In zero-shot reasoning on standard NLP benchmarks (ARC, HellaSwag, OBQA, etc.):

• With 153M to 1.52B parameters, Avey matches the performance of Transformer++ and lags behind Mamba or RWKV-7 by $C$6 on short-range zero-shot tasks (Hammoud et al., 12 Jun 2025).
• On “Single-NEEDLE” long-range retrieval (up to 64k tokens), Avey attains recall $C$7 where vanilla Transformers, Mamba, and RWKV-7 collapse at 2k context, demonstrating decoupled context width and practical extrapolation (Hammoud et al., 12 Jun 2025).

Avey-B demonstrates comparable or superior results on token classification (CoNLL: 92.88, OntoNotes: 93.80, UNER: 94.10) and information retrieval (NQ: 63.83, MLDR: 68.88, MS MARCO: 62.45), outperforming BERT, RoBERTa, ModernBERT, and NeoBERT on a majority of benchmarks (Acharya et al., 17 Feb 2026).

The scaling profile favors Avey-B for long input sequences, with measured latency $C$8 and throughput $C$9, outperforming ModernBERT ($S_c$0 for latency) and NeoBERT ($S_c$1) at $S_c$2k and beyond (Acharya et al., 17 Feb 2026).

4. Technical Innovations Underlying Avey

Avey’s architecture diverges fundamentally from prior approaches through key innovations:

• Selective Retrieval: The MaxSim-based ranker processes only the top-k relevant splits, rather than attending to the entire history, mitigating quadratic scaling.
• Autoregressive Neural Processor: Modularized into Enricher (positional expansion), Contextualizer (cross-token parametric mixing based on learned kernel and input norms), and Fuser (projection and residual), designed for both abstraction and fast inference.
• Partial-embedding Bypass: A subset of the enriched features is directly routed to mitigate information loss and improve downstream accuracy ($S_c$3 improvement).
• Alternating Static/Dynamic Layers (Avey-B): Avoids pathological gradient dynamics, preserves monotonicity, and allows signed/inhibitory relevance effects without instability.
• Stability-oriented Normalization: Row-sum normalization in place of softmax ensures bounded singular values and stability, outperforming alternatives in ablation studies (Acharya et al., 17 Feb 2026).
• Neural Compression: Efficient residual projection enables constant per-split compute and memory footprint.

5. Applications Beyond Sequence Modeling

Avey’s design, as a foundational sequence model, underpins both general and specialized applications:

• Medical Conversational AI: The Avey AI Benchmark suite provides structured patient vignettes for policy optimization by reinforcement learning. In the IGFT framework, Avey-annotated data is used to fine-tune Llama-3.1 and DeepSeek-R1-Distill-Qwen-7B models. The dataset includes 350 HPIs for training, 48 for testing; each is annotated for 10–15 entities, supporting fine-grained entity-level reinforcement feedback. Models trained on this data demonstrate $S_c$4 (Llama) and $S_c$5 (DeepSeek) F1 improvement over base and surpass OpenAI GPT-4o-mini and medical domain-specific baselines for HPI generation (Verma et al., 25 Jan 2026).
• Benchmarks in Token Classification and Retrieval: Avey-B’s performance on CoNLL, OntoNotes, UNER, MLDR, MS MARCO, NQ, and other large-scale benchmarks illustrates the approach’s suitability for high-throughput, bidirectional encoding tasks under constrained compute (Acharya et al., 17 Feb 2026).

A plausible implication is that Avey’s decoupling of context width and sequence length allows for broader, more memory-efficient deployment in domains characterized by very long-range dependencies.

6. Limitations, Extensions, and Research Directions

The original Avey release is text-only and autoregressive, with no published bidirectional or multimodal implementation. While PyTorch implementations lag behind hand-tuned Transformer/RNN kernels in raw GPU efficiency, the Avey-B variant narrows this gap. Notably, Avey’s retrieval machinery could in principle be extended to multimodal or crossmodal contexts, and future work envisions more sophisticated kernel optimizations and integration with quantization, pruning, or RL-driven context selection (Hammoud et al., 12 Jun 2025, Acharya et al., 17 Feb 2026).

Current limitations include:

• Absence of explicit attention or recurrence may impact nuanced language manipulation where both local and global cues interact sharply.
• The primary complexity at training remains $S_c$6 due to ranking, although amortized at inference.
• Dataset and code release remain partially restricted; performance may be subject to further ablation with open-source variants.

Ongoing research is evaluating multimodal retrieval, advanced controller regimes for adaptive context selection, and low-level acceleration. Empirical gains in medical question generation and dialogue structure suggest potential for domain-specific variants.

7. Related Architectures and Clarification

Avey is distinct from AVERY ("Adaptive VLM Split Computing…") (Bhattacharjya et al., 22 Nov 2025); the latter addresses resource-adaptive VLM deployment in edge/cloud disaster response, utilizing dual-stream split and self-aware controllers. The similarity in name is coincidental and does not reflect shared architectural lineage.

Key ablation findings within Avey demonstrate that dynamic parameterization, split weighting, partial-embedding bypass, and expansion each provide measurable empirical gains. The consistently strong performance on standard NLP tasks and long-context retrieval, coupled with superior scaling properties, support Avey and Avey-B as viable high-throughput, attention-free alternatives to dominant Transformer-based architectures.