ReStore: Reusing Results of MapReduce Jobs
Analyzing large scale data has emerged as an important activity for many organizations
in the past few years. This large scale data analysis is facilitated
by the MapReduce programming and execution model and its implementations,
most notably Hadoop. Users of MapReduce often have analysis tasks that are
too complex to express as individual MapReduce jobs. Instead, they use highlevel
query languages such as Pig, Hive, or Jaql to express their complex tasks.
The compilers of these languages translate queries into workflows of MapReduce
jobs. Each job in these workflows reads its input from the distributed file
system used by the MapReduce system and produces output that is stored in
this distributed file system and read as input by the next job in the workflow.
The current practice is to delete these intermediate results from the distributed
file system at the end of executing the workflow. One way to improve the performance
of workflows of MapReduce jobs is to keep these intermediate results
and reuse them for future workflows submitted to the system. In this project
we build ReStore, a system that manages the storage and reuse of such intermediate
results by employing execution coat and monetary cost optimization
techniques. ReStore optimizes these workflows of MapReduce jobs by exploiting
the following: (1) store the outputs of jobs and sub-jobs in the workflow
to reuse them to rewrite future queries, (2) use a machine learning model to
predict the cost of alternate query rewritings and choose the the plan with best
expected performance, and (3) use estimated output sizes of jobs and sub-jobs
in the workflow, their predicted execution time, and monetary cost of executing
jobs in the cluster and storing their output to decide on which plans to materialize.
We have implemented ReStore as an extension to the Pig dataflow system
on top of Hadoop, and have conducted experiments on a private cloud and on
EC2 and we uses data sets and queries from benchmarks such as PigMix and
a Pig Latin representation of TPC-H.
Iman Elghandour, Assistant Professor, Alexandria University, Egypt.
Ahmed E. Khalifa, Research Assistant, Alexandria University, Egypt.
Ashraf Aboulnaga, Associate Professor, University of Waterloo, Canada (Supervisor at UW).
Ahmed Khalifa, Iman Elghandour, and Nagwa El-Makky. IncReStore: Incremental Computation of MapReduce Workows . In Proc. IEEE Int. Conf. on Data Engineering Workshops (ICDEW) , Helsinki, Finland, 2016.
Iman Elghandour and Ashraf
Aboulnaga. “ReStore: Reusing results of MapReduce jobs,” In
Proceedings of the VLDB Endowment (PVLDB), 5(6):586–597, 2012.
Presented at the International Conference on Very Large Data Bases
(VLDB '12), Istanbul, Turkey, August 2012.
Iman Elghandour and Ashraf
Aboulnaga. “ReStore: Reusing results of MapReduce jobs in Pig,”
Demonstration at the ACM SIGMOD International Conference on
Management of Data (SIGMOD ‘12), Scottsdale, USA, May 2012.