Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction (2410.18160v1)

Abstract: Causal decoder-only transformer models used for generative LLMling, such as Generative Pre-trained Transformers (GPT), are trained to predict the next token in a sequence based only on its previous tokens. Despite this simple training objective, they have proved to be powerful AI tools. However, only predicting the next token results in top layer embedding vectors that are highly token-focused. There may be benefits in generating embedding vectors at each token position that better capture the overall meaning of longer sequences of future text. Recent studies matching brain scans with deep LLMs suggest that humans also predict upcoming words when listening or reading but consider multiple future tokens rather than just one. This research investigates a new pretraining method called Future Token Prediction (FTP). In FTP, a large transformer encoder generates top layer embedding vectors for each token position, which, instead of being passed to a language head, are linearly and expansively projected to a pseudo-sequence, which is cross attended to by a small transformer decoder to predict the next N tokens forward from that position in the sequence. The top layer embedding vectors from FTP models exhibit distinct properties compared to those from standard GPT models, varying smoothly along a text sequence as measured by cosine similarity between adjacent tokens. Text generated by FTP models show improved topic coherence compared to standard GPT-like models trained with the same prediction perplexity for the next single token. The vectors are shown to better represent the topic of text based on the results of text classification examples. On a toy, but complex, coding problem, FTP networks produce significantly better results than GPT networks.

• The paper introduces a Future Token Prediction (FTP) method that uses per-token semantic state vectors to predict multiple future tokens.
• It employs transformer encoders and decoders with cross-attention to generate richer embeddings and smoother token transitions.
• Empirical results show improved topic coherence and reduced drift compared to traditional GPT-style and masked language models.

Future Token Prediction: Advancements in Causal LLMing

The paper, "Future Token Prediction - Causal LLMling with Per-Token Semantic State Vector for Multi-Token Prediction," authored by Nicholas Walker, addresses a critical enhancement in causal LLMing by introducing a novel pretraining method dubbed Future Token Prediction (FTP). This research aims to improve the semantic coherence of LLMs over longer sequences, diverging from traditional methodologies that focus solely on immediate next-token prediction.

Key Insights and Methodology

This paper critiques the conventional generative LLMs like GPT, which—despite their efficacy—exhibit limitations such as topic drift over extended output sequences. Walker questions the one-dimensional view of future predictions—limited to the immediate next token—and explores FTP as a method for generating richer embeddings that can encapsulate a broader context.

FTP models employ a large transformer encoder to output embedding vectors at each token position. These vectors are linearly projected into a pseudo-sequence, which a transformer decoder then cross-attends to predict multiple (up to N) future tokens. This design aligns with findings in cognitive science suggesting humans consider numerous upcoming words rather than a singular next word.

Contrast with Existing Models

The paper surveys recent advancements in multi-token prediction techniques, such as:

• Modified Transformer Decoders: Compared to ProphetNet, which leverages future n-gram prediction, FTP aims to maintain coherence across longer sequences without leaning too heavily on local token correlations.
• Seq2Seq Models: These models, prominent in XLNet and T5, offer alternative ways of multi-token prediction without distillation of the context into single vectors.
• Masked LLMs: Traditional approaches like BERT have limitations in left-to-right generation, which Walker's method attempts to surpass through refined vector representations.

Numerical Outcomes and Comparative Performance

FTP models manifest distinctive vector properties, exhibiting smoother transitions between token embeddings compared to standard models. The empirical results underline FTP's strength, reporting superior topic coherence and outperforming GPT-style models even under identical perplexity settings for next-token prediction. On practical benchmarks, notably a toy coding problem, FTP networks demonstrate measurable improvements.

Implications and Future Trajectories

Walker’s work has notable implications in enhancing the semantic integrity of generated text. The refined model architecture not only suggests improvements in standard LLM applications but also implies potential progress in AI’s handling of complex, context-rich generation tasks such as coding, creative writing, and extended dialogues.

As AI models scale and become more integral in creativity-driven applications, FTP offers a promising step forward. It suggests a methodology where embedding generation aligns more closely with human-like semantic predictions, fostering advancements in how machines understand and generate language.

Continued exploration and scaling of FTP models, especially in domains requiring advanced coherence and contextual understanding, could significantly amplify the capabilities of future AI systems. The adaptation of FTP in large-scale models, especially within Iprova’s creativity-oriented datasets, could carve new pathways in the journey toward truly intelligent generative models.