NickName:OckhamsRazor Ask DateTime:2013-03-03T02:31:55

Hadoop on EC2 vs Elastic Map Reduce

I'm trying to evaluate the differences between these two options. Here are some pros and cons I can think of :

Elastic Map Reduce => Better support from Amazon, No need to administer cluster, More Expensive (?) EC2 + Hadoop => More control of your hadoop configuration, Cheaper (?)

I'm wondering if anyone might have benchmarked the performance of EC2 + Hadoop vis a vis EMR? Is there any significant difference in cost for large cluster deployments? What other differences exist?

Copyright Notice：Content Author:「OckhamsRazor」，Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article：https://stackoverflow.com/questions/15177908/hadoop-on-ec2-vs-elastic-map-reduce

Answers

WestCoastProjects 2013-03-02T23:28:42

We use both approaches (EMR and EC2) at my job. \n\nThe advantages of EMR that Amar mentioned are more or less true: so if you want simplicity it may be the way to go.\n\nBut there are other considerations:\n\n\nthe version of EMR is far behind apache head. it is approximately 0.20.205 whereas head is at 2.X, which is essentially 3 versions up (1.0, 1.1, 2.0..) \n\n\nhadoop@domU-12-31-39-07-B9-97:~$ ll hadoop*.jar\nlrwxrwxrwx 1 hadoop hadoop 73 Feb 5 12:00 hadoop-examples-0.20.205.jar -> /home/hadoop/.versions/0.20.205/share/hadoop/hadoop-examples-0.20.205.jar\nlrwxrwxrwx 1 hadoop hadoop 69 Feb 5 12:00 hadoop-test-0.20.205.jar -> /home/hadoop/.versions/0.20.205/share/hadoop/hadoop-test-0.20.205.jar\nlrwxrwxrwx 1 hadoop hadoop 69 Feb 5 12:00 hadoop-core-0.20.205.jar -> /home/hadoop/.versions/0.20.205/share/hadoop/hadoop-core-0.20.205.jar\nlrwxrwxrwx 1 hadoop hadoop 70 Feb 5 12:00 hadoop-tools-0.20.205.jar -> /home/hadoop/.versions/0.20.205/share/hadoop/hadoop-tools-0.20.205.jar\nlrwxrwxrwx 1 hadoop hadoop 68 Feb 5 12:00 hadoop-ant-0.20.205.jar -> /home/hadoop/.versions/0.20.205/share/hadoop/hadoop-ant-0.20.205.jar\n\n\nAs a direct consequence I had to re-code /restructure my Map/reduce program due to missing contrib modules in the older version running on EMR\nYou do not have as much of an opportunity to use non-Map/Reduce algorithms as if you were using updated version of M/R.\nFlexibility to mix and match versions of hadoop ecosystem. \n",

Amar 2013-03-02T20:10:38

Well, administering/monitoring/maintaining a cluster isn't a small task in itself.\nUsing EMR really you could get machines configured and up and running with your custom bootstrap code in no time.\nApart from doing all these things EMR provides a A lot of other tools/options/facilities too.\n\nHere you don't have to worry about terminating a cluster after the jobs are done, you can surely implement a way for yourself in the EC2+Hadoop setup, but EMR does this for you in a neat way.\n\nAlso you have facility to resize the cluster size even while your jobs are running!\n\nThe Pig and Hive that are available with EMR also contain patches which make it easier to work with files in S3.\n\nEven here in this answer you may find that EMR has been given an upper hand.",