File(s) not publicly available

Is high performance computing (HPC) ready to handle big data?

conference contribution
posted on 15.05.2018, 00:00 by Biplob Ray, M Chowdhury, U Atif
In recent years big data has emerged as a universal term and its management has become a crucial research topic. The phrase ‘big data’ refers to data sets so large and complex that the processing of them requires collaborative High Performance Computing (HPC). How to effectively allocate resources is one of the prime challenges in HPC. This leads us to the question: are the existing HPC resource allocation techniques effective enough to support future big data challenges? In this context, we have investigated the effectiveness of HPC resource allocation using the Google cluster dataset and a number of data mining tools to determine the correlational coefficient between resource allocation, resource usages and priority. Our analysis initially focused on correlation between resource allocation and resource uses. The finding shows that a high volume of resources that are allocated by the system for a job are not being used by that same job. To investigate further, we analyzed the correlation between resource allocation, resource usages and priority. Our clustering, classification and prediction techniques identified that the allocation and uses of resources are very loosely correlated with priority of the jobs. This research shows that our current HPC scheduling needs improvement in order to accommodate the big data challenge efficiently.

History

Editor

Doss R; Piramuthu S; Zhou W

Parent Title

Future Network Systems and Security: Third International Conference, FNSS 2017, proceedings

Volume

759

Start Page

97

End Page

112

Number of Pages

16

Start Date

31/08/2017

Finish Date

02/09/2017

ISSN

1865-0929

ISBN-13

9783319655475

Location

Gainesville, Florida, USA

Publisher

Springer

Place of Publication

Cham, Switzerland

Peer Reviewed

Yes

Open Access

No

External Author Affiliations

Deakin University

Author Research Institute

Centre for Intelligent Systems

Era Eligible

Yes

Name of Conference

International Conference on Future Network Systems and Security 2017

Exports

CQUniversity

Exports