FLAIR: Federated Learning Annotated Image Repository

Abstract: Cross-device federated learning is an emerging ML paradigm where a large population of devices collectively train an ML model while the data remains on the devices. This research field has a unique set of practical challenges, and to systematically make advances, new datasets curated to be compatible with this paradigm are needed. Existing federated learning benchmarks in the image domain do not accurately capture the scale and heterogeneity of many real-world use cases. We introduce FLAIR, a challenging large-scale annotated image dataset for multi-label classification suitable for federated learning. FLAIR has 429,078 images from 51,414 Flickr users and captures many of the intricacies typically encountered in federated learning, such as heterogeneous user data and a long-tailed label distribution. We implement multiple baselines in different learning setups for different tasks on this dataset. We believe FLAIR can serve as a challenging benchmark for advancing the state-of-the art in federated learning. Dataset access and the code for the benchmark are available at \url{https://github.com/apple/ml-flair}.

• The paper introduces a novel large-scale image dataset tailored for federated learning, capturing real-world non-IID and imbalanced conditions.
• It benchmarks centralized, federated, and differentially private settings using a ResNet-18 architecture to highlight performance trade-offs.
• The paper emphasizes challenges like class imbalance and non-IID data handling, motivating further research in privacy-preserving federated algorithms.

An Analytical Overview of "FLAIR: Federated Learning Annotated Image Repository"

The paper introduces FLAIR, a novel dataset designed to address the specific challenges and requirements of cross-device federated learning in the domain of image classification. FLAIR is larger and more heterogeneous compared to existing federated learning datasets, representing a critical advancement for benchmarking federated learning algorithms aimed at privacy preservation and scalability.

The Motivation Behind FLAIR

Federated learning presents a unique paradigm where model training occurs across decentralized data sources without requiring data transfer to a central server, thus prioritizing user privacy. Despite its promising application, the development of federated learning has been impeded by a lack of representative datasets that simulate real-world data scenarios, including non-IID distributions, diverse user-generated content, and privacy concerns. This paper seeks to fill this gap by introducing FLAIR, which offers a large-scale multi-label image dataset with authentic user partitions, thus better reflecting the conditions encountered in practical federated learning scenarios.

Dataset Characteristics

FLAIR comprises 429,078 images sourced from 51,414 Flickr users. The dataset is annotated with two distinct sets of labels originating from a two-level hierarchy: 17 coarse-grained and 1,628 fine-grained classes. Its construction purposefully incorporates characteristics often overlooked in traditional datasets, such as:

• Imbalanced user data distributions, with a skew towards smaller datasets per user.
• Shifts in both feature and label distributions reflective of real-world conditions.
• Diverse device and contextual variations affecting data generation on devices.

This dataset configuration enables researchers to explore federated learning scenarios that include personalized model tuning, robustness against distributional shifts, and efficient handling of imbalanced data.

Experimental Framework

The paper benchmarks FLAIR under multiple setups: centralized learning, federated learning, and federated learning with differential privacy (DP) features. Utilizing a ResNet-18 architecture, the paper assesses both random initialization and pre-trained models, quantifying performance via standard multi-label classification metrics—precision, recall, F1 score, and averaged precision.

Notably, results indicate that federated and differentially private settings lag behind centralized setups, especially in scenarios requiring fine-grained label recognition. Training models with pre-existing knowledge, through pre-trained initialization, helps bridge the performance gap.

Challenges and Implications

FLAIR highlights several pressing challenges that remain in federated learning:

1. Class Imbalance: Federated learning models exhibit increased vulnerability to class imbalance, magnifying the disparities between high-frequency and low-frequency classes.
2. Non-IID Data Handling: The misalignment between centralized benchmarks and inherent federated learning conditions underscores the need for novel learning strategies robust to non-IID data.
3. Privacy-Preserving Techniques: While differentially private learning is vital for user data security, it significantly degrades model accuracy, necessitating research into more efficient noise-handling techniques.

FLAIR’s non-IID and skewed distribution reinforces its potential as a benchmark to test robust solutions against these challenges.

Prospective Research Directions

FLAIR fosters a myriad of research opportunities:

• Algorithm Development: Innovative federated learning algorithms tailored for heterogeneous data environments can be conceptualized and validated against FLAIR's diverse scenarios.
• Privacy and Utility Optimization: The dataset encourages exploration into balancing privacy and utility, leveraging adaptive clipping, cohort size management, and personalization techniques to enhance DP methods.
• Scalable Model Architectures: Investigating optimal model designs that are not only privacy-preserving but also computationally feasible for large-scale federated applications will be imperative.

In conclusion, FLAIR constitutes a pivotal addition to the toolkit for federated learning research, advocating for realistic, privacy-sensitive benchmarks that depict the complexities of real-world data environments. The dataset and outlined experiments both serve as baselines for exploring effective methodologies addressing the intricate challenges of federated learning.