Building and Deploying MR2
From the post:
A number of architectural changes have been added to Hadoop MapReduce. The new MapReduce system is called MR2 (AKA MR.next). The first release version to include these changes will be Hadoop 0.23.
A key change in the new architecture is the disappearance of the centralized JobTracker service. Previously, the JobTracker was responsible for provisioning the resources across the whole cluster, in addition to managing the life cycle of all submitted MapReduce applications; this typically included starting, monitoring and retrying the applications individual tasks. Throughout the years and from a practical perspective, the Hadoop community has acknowledged the problems that inherently exist in this functionally aggregated design (See MAPREDUCE-279).
In MR2, the JobTracker aggregated functionality is separated across two new components:
- Central Resource Manager (RM): Management of resources in the cluster.
- Application Master (AM): Management of the life cycle of an application and its tasks. Think of the AM as a per-application JobTracker.
The new design enables scaling Hadoop to run on much larger clusters, in addition to the ability to run non-mapreduce applications on the same Hadoop cluster. For more architecture details, the interested reader may refer to the design document at: https://issues.apache.org/jira/secure/attachment/12486023/MapReduce_NextGen_Architecture.pdf.
The objective of this blog is to outline the steps for building, configuring, deploying and running a single-node NextGen MR cluster.
…(see the post for the rest)
If you want to get a jump on experience with the next generation of Hadoop, here is a place to start!