An Efficient Distributed Hierarchical-Clustering Algorithm for Large Scale Data

Abstract

The data-classification process can possibly involve a huge amount of data in today’s cloud computing environment. It could take a long time for processing, and could consume many resources for computation and storage. This study focuses on the problem of using the traditional hierarchical agglomerative clustering algorithm on a distributed environment since hierarchical agglomerative clustering has high applicability and efficiency. A parallel hierarchical agglomerative clustering algorithm is proposed in this study. The proposed algorithm divides the whole computation into several small tasks, distribute the tasks to message-passing processes, and merge the results to form a hierarchical cluster. A threshold is used to reduce the storage requirement during the computation. To evaluate the performance and limitation of our algorithm, this study has conducted several experiments using real astronomical data, the main asteroid belt catalog. The experimental results confirm that the proposed parallel algorithm is efficient.