ChatAnything: Facetime Chat with LLM-Enhanced Personas (2311.06772v1)

Abstract: In this technical report, we target generating anthropomorphized personas for LLM-based characters in an online manner, including visual appearance, personality and tones, with only text descriptions. To achieve this, we first leverage the in-context learning capability of LLMs for personality generation by carefully designing a set of system prompts. We then propose two novel concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for diverse voice and appearance generation. For MoV, we utilize the text-to-speech (TTS) algorithms with a variety of pre-defined tones and select the most matching one based on the user-provided text description automatically. For MoD, we combine the recent popular text-to-image generation techniques and talking head algorithms to streamline the process of generating talking objects. We termed the whole framework as ChatAnything. With it, users could be able to animate anything with any personas that are anthropomorphic using just a few text inputs. However, we have observed that the anthropomorphic objects produced by current generative models are often undetectable by pre-trained face landmark detectors, leading to failure of the face motion generation, even if these faces possess human-like appearances because those images are nearly seen during the training (e.g., OOD samples). To address this issue, we incorporate pixel-level guidance to infuse human face landmarks during the image generation phase. To benchmark these metrics, we have built an evaluation dataset. Based on it, we verify that the detection rate of the face landmark is significantly increased from 57.0% to 92.5% thus allowing automatic face animation based on generated speech content. The code and more results can be found at https://chatanything.github.io/.

• The paper introduces a framework that uses LLMs to generate anthropomorphic personas from text, improving face landmark detection from 57.0% to 92.5%.
• It employs a Mixture of Voices and Diffusers to transform textual descriptions into personalized vocal and visual outputs.
• The study demonstrates enhanced human-computer interaction through a zero-shot approach that eliminates the need for extensive model retraining.

An Overview of "ChatAnything: Facetime Chat with LLM-Enhanced Personas"

The paper "ChatAnything: Facetime Chat with LLM-Enhanced Personas" presents a framework designed to generate anthropomorphized personas using LLMs. This system facilitates the creation of characters with customized visual appearance, personality, and vocal tone using purely text-based input. The framework consists of several novel components focusing on enhancing user interaction by transforming textual descriptions into interactive, animated entities.

Framework Components and Methodology

The ChatAnything framework hinges on a well-structured system with several key components:

1. LLM-Controlled Personality Generation: The framework utilizes in-context learning capabilities of LLMs to define and simulate unique personalities for each persona. This is achieved by generating system prompts that guide the LLM to produce character-specific personality traits based on user-input text.
2. Mixture of Voices (MoV): This system focuses on diverse voice generation using a pool of text-to-speech (TTS) algorithms. Each voice is associated with a predefined tone that matches the user's text description, allowing the framework to select the optimal voice configuration automatically.
3. Mixture of Diffusers (MoD): Visual appearance and animation are handled using text-to-image generation techniques and talking head algorithms. The MoD combines these approaches to generate talking objects, utilizing recent advancements in diffusion-based generative models.

To address challenges related to the generation of anthropomorphic objects—particularly issues with current models' ability to detect these faces—ChatAnything incorporates pixel-level guidance to enhance the detection of face landmarks. This refinement significantly boosts face landmark detection rates from 57.0% to 92.5%, enabling effective face motion animation based on generated speech content.

Evaluation and Results

The framework introduces a comprehensive evaluation dataset to benchmark its performance in various categories such as realism, animal figures, and cartoon styles. The evaluation shows a marked improvement in landmark detection, validating the effectiveness of the MoV and MoD implementations. By employing a zero-shot approach, the system bridges the distribution gap between pre-trained generative models and talking head models, eliminating the need for computationally intensive retraining procedures.

Implications and Future Directions

Practically, ChatAnything has the potential to enhance human-computer interactivity by enabling more natural and personalized digital dialogues. Theoretically, the framework exemplifies the integration of LLMs with complementary AI models, shedding light on future explorations in modeling authentic and diverse human-like interactions.

Future developments could include optimizing and exploring alternative lightweight methods for generative model integration, leveraging newer advancements in diffusion models and LLMs. Additionally, expanding the stylistic and linguistic capabilities of the personas generated could further align the system with a wider array of user preferences and applications.

The paper effectively showcases the convergence of multiple AI technologies to produce a sophisticated framework that enriches user interaction and interaction customization through LLM-enhanced digital personas.