Researchers present a new approach to improve transformers' memory capabilities by using landmark tokens to represent blocks of input.
This method allows access to the entire context while maintaining random-access flexibility, achieving comparable performance with Transformer-XL and extending context length capacity of LLaMA 7B up to 32k tokens.