High-Dimensional Tail Index Regression: with An Application to Text Analyses of Viral Posts in Social Media (2403.01318v2)

Abstract: Motivated by the empirical observation of power-law distributions in the credits (e.g., "likes") of viral social media posts, we introduce a high-dimensional tail index regression model and propose methods for estimation and inference of its parameters. First, we present a regularized estimator, establish its consistency, and derive its convergence rate. Second, we introduce a debiasing technique for the regularized estimator to facilitate inference and prove its asymptotic normality. Third, we extend our approach to handle large-scale online streaming data using stochastic gradient descent. Simulation studies corroborate our theoretical findings. We apply these methods to the text analysis of viral posts on X (formerly Twitter) related to LGBTQ+ topics.

• The paper presents a novel HDTIR framework using an L1-regularized estimator to accurately model the tail distribution of post likes.
• It extends traditional tail index regression with debiasing techniques like sample splitting, ensuring reliable inference in high-dimensional text data.
• The empirical study on LGBTQ+++ posts shows that specific keywords, such as 'lgbt', significantly influence viral content performance.

High-Dimensional Tail Index Regression Applied to Social Media Analysis

Introduction

In the ever-evolving landscape of social media analytics, understanding the dynamics behind the virality of posts presents a unique set of challenges and opportunities. A paper has ventured into this terrain by theoretically establishing and empirically applying high-dimensional tail index regression (HDTIR) to analyze viral posts, specifically those related to LGBTQ+++ topics on a platform referred to as X (formerly recognized as Twitter). This novel approach aims to quantify the effect of specific keywords on the extreme distribution of likes, potentially offering invaluable insights into crafting viral content.

Methodology

The paper introduces HDTIR, extending the traditional tail index regression (TIR) to accommodate high-dimensional data scenarios that are prevalent in text analysis of social media posts. The method pivots on the Pareto distribution assumption for the tail of likes received by posts, a premise substantiated by empirical observations.

High-Dimensional Tail Index Regression (HDTIR)

• The paper proposes an L1-regularized maximum likelihood estimator for the tail index, accounting for the challenge posed by the high dimensionality of text data.
• To mitigate biases associated with regularization and enable inference, the authors introduce debiased estimators, employing techniques like sample splitting and cross-fitting.
• The theoretical contributions are solidified through the establishment of the consistency and convergence rates of the proposed estimators, providing a comprehensive theoretical framework for HDTIR.

Empirical Application

The application of HDTIR to LGBTQ+++ posts on the social media platform X illustrates the practical relevance and potency of the methodology. By focusing on the top and bottom keywords that significantly influence the number of likes (tail distribution), the paper reveals insights into the dynamics of virality within this context.

• Data: The analysis is based on a dataset consisting of 32,456 posts, with the dimensionality of the covariate vector set to 500 to represent the most frequently used words.
• Findings: The method identifies specific words that significantly affect the extremity of likes. For instance, the plain word 'lgbt' shows a significant impact, contrary to the usage of hashtags like '#lgbt', which appeared to have an adverse effect on virality.

Results

The simulation studies validate the efficacy of the proposed HDTIR method under various design scenarios, demonstrating its robust performance in estimating the tail index and thus the impact of keywords on post virality. The empirical application further underscores the potential utility of HDTIR in strategic content creation aimed at maximizing social media engagement.

Implications and Future Directions

This exploration into the high-dimensional tail index regression model unveils novel pathways for analyzing and understanding the vast and intricate data generated on social media platforms. The practical utility of discerning the impact of specific keywords or phrases on the virality of content offers strategic insights for content creators aiming to enhance their social media presence.

• Theoretical Implication: Beyond its immediate application, the paper contributes to the literature on high-dimensional statistical methodologies, especially in the context of tail behavior analysis in large datasets.
• Practical Implication: For practitioners in social media analytics and digital marketing, the insights offered by HDTIR can inform content strategy to leverage viral potential effectively.

The framework laid out by this paper opens numerous avenues for future research, particularly in refining and expanding the methodology to encompass other forms of social media engagement and analyzing the interaction effects among keywords. As social media continues to evolve as a dynamic and influential domain, methodologies like HDTIR will be instrumental in navigating its complexities and unlocking the secrets to viral content.