Load Balancing for MapReduce-based Entity Resolution (1108.1631v1)

Abstract: The effectiveness and scalability of MapReduce-based implementations of complex data-intensive tasks depend on an even redistribution of data between map and reduce tasks. In the presence of skewed data, sophisticated redistribution approaches thus become necessary to achieve load balancing among all reduce tasks to be executed in parallel. For the complex problem of entity resolution, we propose and evaluate two approaches for such skew handling and load balancing. The approaches support blocking techniques to reduce the search space of entity resolution, utilize a preprocessing MapReduce job to analyze the data distribution, and distribute the entities of large blocks among multiple reduce tasks. The evaluation on a real cloud infrastructure shows the value and effectiveness of the proposed load balancing approaches.

• The paper introduces two load balancing strategies, BlockSplit and PairRange, to efficiently handle data skew in MapReduce-based entity resolution.
• BlockSplit handles large blocks by splitting them, while PairRange maps entity pairs to ranges, both distributing workloads across reduce nodes.
• Empirical results show both load balancing strategies significantly improve MapReduce runtime and scalability for entity resolution under data skew.

Load Balancing for MapReduce-based Entity Resolution

This paper focuses on optimizing the efficiency and scalability of MapReduce-based frameworks when applied to entity resolution (ER) tasks, which inherently involve complex data-intensive computations. Entity resolution is critical for maintaining data quality and integration through the identification of duplicates or matching records across datasets. Traditional approaches to ER that evaluate the full Cartesian product of data sets are computation-heavy and inefficient, particularly for large-scale datasets. To mitigate this, the authors explore two robust load balancing strategies designed to manage and balance the computational demands posed by skewed data distributions—BlockSplit and PairRange.

The paper begins by acknowledging the limitations of naive MapReduce implementations when handling skewed data distribution between map and reduce tasks. Such imbalances can lead to inefficiencies and increased costs in public cloud settings, as ineffective load distribution might prevent optimal utilization of computing resources. The core contribution of this work is the introduction of two load balancing approaches that effectively distribute tasks across reduce nodes, even under conditions of data skew.

1. Methodology and Load Balancing Approaches

The authors propose a general workflow utilizing MapReduce that begins with a preprocessing phase to generate a block distribution matrix (BDM). This BDM encapsulates the number of entities per block, segregated by input partitions, and is used to guide the two proposed load balancing schemes:

• BlockSplit: This strategy aims to handle large blocks by splitting them based on input partitions. It constructs sub-blocks and pairwise processing tasks to ensure that each reduce task has a balanced workload. The splitting is governed by heuristic measures that focus on reducing memory constraints and ensuring optimized load distribution among nodes.
• PairRange: This method involves a detailed enumeration scheme for mapping entity pairs to reduce tasks. By virtually dividing the comparison tasks into ranges, PairRange achieves uniform workload distribution among reduce nodes, minimizing computational skew that may arise from varied block sizes.

Both approaches were evaluated through implementation on a real cloud infrastructure using real-world datasets, demonstrating significant improvements in runtime efficiency over basic MapReduce implementations, irrespective of data skew levels.

2. Results and Evaluation

The empirical results highlighted the robustness of the proposed load balancing strategies. The tests revealed:

• Robustness to Data Skew: Both strategies exhibited stable performance across varying levels of data skew, with a noteworthy reduction in execution times as compared to basic MapReduce implementations.
• Scalability: BlockSplit and PairRange harnessed the additional computational resources provided by increased numbers of reduce tasks and nodes effectively, sustaining enhanced performance across different configuration setups.
• Efficiency: PairRange showed superior scalability and load distribution in larger datasets by uniformly balancing workloads despite increased computational overhead.

3. Implications and Future Directions

The implications of this research extend beyond entity resolution, with potential applications in any task requiring pairwise similarity computations within a MapReduce framework. This positions BlockSplit and PairRange as versatile strategies in contexts like document similarity assessments, set-similarity joins, and various scientific computing operations.

Future research as indicated by the authors will pivot towards extending the approaches for multi-pass blocking scenarios and optimizing their applicability in other data-intensive operations, such as join processes and data mining tasks.

Overall, this paper contributes valuable insights into the operationalization of load-balanced MapReduce workflows, enhancing computational efficacy and scalability in distributed data environments.