Big Data Approach for Organization and Forecast of Student Consequence using MapReduce

Abstract

This paper takes a novel perspective on the problem of clustering high-dimensional data. Instead of attempting to avoid the curse of dimensionality by observing a lower dimensional feature subspace, this project embraces dimensionality by taking advantage of inherently high-dimensional phenomena. More specifically, it is showed that hubness, i.e., the tendency of highdimensional data to contain points (hubs) that frequently occur in k-nearest-neighbor lists of other points, can be successfully exploited in clustering. The proposed system demonstrates that hubness is a good measure of point centrality within a high-dimensional data cluster, and by proposing several hubness-based clustering algorithms, showing that major hubs can be used effectively as cluster prototypes or as guides during the search for centroidbased cluster configurations. The proposed methods are tailored mostly for detecting approximately hyper spherical clusters and need to be extended to properly handle clusters of arbitrary shapes.