File(s) not publicly available

The scalability of volunteer computing for MapReduce big data applications

conference contribution
posted on 13.06.2018, 00:00 by Wei LiWei Li, Wanwu GuoWanwu Guo
Volunteer Computing (VC) has been successfully applied to many compute-intensive scientific projects to solve embarrassingly parallel computing prob-lems. There exist some efforts in the current literature to apply VC to data-intensive (i.e. big data) applications, but none of them has confirmed the scalability of VC for the applications in the opportunistic volunteer envi-ronments. This paper chooses MapReduce as a typical computing paradigm in coping with big data processing in distributed environments and models it on DHT (Distributed Hash Table) P2P overlay to bring this computing para-digm into VC environments. The modelling results in a distributed prototype implementation and a simulator. The experimental evaluation of this paper has confirmed that the scalability of VC for the MapReduce big data (up to 10TB) applications in the cases, where the number of volunteers is fairly large (up to 10K), they commit high churn rates (up to 90%), and they have heterogeneous compute capacities (the fastest is 6 times of the slowest) and bandwidths (the fastest is up to 75 times of the slowest).

History

Editor

Zou B; LI M; Wang H; Song X; Xi W; Lu Z

Parent Title

Data science: third International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2017, Changsha, China, September 22-24, 2017, Proceedings. Part I

Volume

CCIS 727

Start Page

153

End Page

165

Number of Pages

13

Start Date

22/09/2017

Finish Date

24/09/2017

eISSN

1865-0937

ISSN

1865-0929

ISBN-13

9789811063855

Location

Changsha, China

Publisher

Springer

Place of Publication

Singapore

Peer Reviewed

Yes

Open Access

No

Author Research Institute

Centre for Intelligent Systems

Era Eligible

Yes

Name of Conference

Third International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2017