Recommendations for Hadoop on EC2?
NickName:Mike Baranczak Ask DateTime:2011-02-11T09:55:31

Recommendations for Hadoop on EC2?

When running Hadoop in EC2, I seem to have two options:

  • A: Manage the cluster myself, using the EC2-specific shell scripts that come with Hadoop.
  • B: Use Elastic MapReduce, and pay a little extra for the convenience.

I'm leaning towards B, but I'd appreciate some advice from people with more experience. Here are my questions:

  1. Are there any tasks that can be done with one of these methods but not the other?
  2. Are there other options besides these two that I'm overlooking?
  3. If I choose B, how easy would it be to go back to A? That is, what's the danger of vendor lock-in?

Copyright Notice:Content Author:「Mike Baranczak」,Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/4964885/recommendations-for-hadoop-on-ec2

Answers
Helmut Zechmann 2011-02-13T14:30:08

Third option: \nYou can use apache whirr to set up an hadoop cluster on ec2 (rackspace is also supported)",


Steve Wilhelm 2011-02-11T06:50:31

I have been told by people close to the Amazon Elastic MapReduce (EMR) development team that there are at least two other advantages to using EMR: a) Amazon is actively applying bug fixes and performance enhancements to the Hadoop code base used on EMR, and b) Amazon employs a high performance network between EMR servers and S3 servers that may not be available between EC2 servers and S3 servers. \n\nUPDATE: See @mat's comments that refute the rumored advantages of using EMR. ",


More about “Recommendations for Hadoop on EC2?” related questions

Recommendations for Hadoop on EC2?

When running Hadoop in EC2, I seem to have two options: A: Manage the cluster myself, using the EC2-specific shell scripts that come with Hadoop. B: Use Elastic MapReduce, and pay a little extra f...

Show Detail

getting started with the latest version of hadoop and ec2

I am following Hadoop in Action to get started with hadoop with ec2. I'm running on ubuntu and have downloaded and installed the latest version of Hadoop. I am hitting a road block at this command:

Show Detail

EC2 Reservation recommendations - Normalized units

I have been working on reducing server costs today and had a look at the reservations recommendations within amazon EC2 recommendations. I have noticed the recommendations are for instance types th...

Show Detail

Script to set up Hadoop on EC2

The script to setup Hadoop on EC2 as described in https://wiki.apache.org/hadoop/AmazonEC2 has been removed from recent hadoop release. Google points me to an alternative http://whirr.apache.org/...

Show Detail

Hadoop on EC2 vs Elastic Map Reduce

I'm trying to evaluate the differences between these two options. Here are some pros and cons I can think of : Elastic Map Reduce => Better support from Amazon, No need to administer cluster, More

Show Detail

Hadoop initial setup on AWS EC2?

I am just a newbie of Hadoop. does anyone know how to setup hadoop on EC2(micro) instance? or It would be helpful if anyone knows good reference on-line links for Hadoop on EC2. Thank you

Show Detail

Configuring hadoop manually on ec2

Can someone please point me in the direction of any resources that will help me manually setup/configure Hadoop (1.0.4) on EC2. I agree that there are lots of resources for accomplishing this using...

Show Detail

Hadoop on EC2 vs. ElasticMapReduce/S3

I have used ElasticMapReduce for some time. It is quite convenient but I can't run HBase since Hadoop cluster is only temporarily available (I have asked somewhat related question at HBase and Hado...

Show Detail

Where can I find AMI for Hadoop on EC2?

I am trying to set up Hadoop permanently on Amazon EC2. Currently what I am doing is every morning launch EC2 instances and set up Hadoop. Is there any way i can avoid this tedious step? I am looki...

Show Detail

Installing Hbase / Hadoop on EC2 cluster

I know that I can spin off a EC2 cluster with Hadoop installed (unless I am wrong about that). How about Hbase? Can I have the Hadoop and Hbase premade, ready to go? Or do I need to get my hands di...

Show Detail