├── .gitignore ├── README.md └── Vagrantfile /.gitignore: -------------------------------------------------------------------------------- 1 | ### Vagrant ### 2 | .vagrant/ 3 | vagrant_ansible_inventory* -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # A working virtual Hadoop cluster 2 | 3 | With these files you can setup and provision a locally running, virtual Hadoop cluster in real distributed fashion for trying out Hadoop and related technologies. It runs the latest Cloudera Hadoop distribution: **CDH5**. It also allows you to practise the use of [Cloudera Manager](http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/cloudera-manager.html) for installing the Hadoop stack. If you're looking for a fully automated install, without user intervention, look elsewhere. I specifically made this with the goal of creating an environment ideally suited for Cloudera Manager to do its job. This gives you the freedom to actually install the services you want, and change the configuration how you see fit. 4 | 5 | This README describes how to get the cluster with Cloudera Manager up and running. For more detailed instructions on how to install the whole Hadoop stack on that, you can use [this guide](http://dandydev.net/blog/installing-virtual-hadoop-cluster). 6 | 7 | ## Specs 8 | 9 | The cluster conists of 4 nodes: 10 | 11 | * Master node with 4GB of RAM (Running the NameNode, Hue, ResourceManager etc. after installing the Hadoop services) 12 | * 3 slaves with 2GB of RAM each (Running DataNodes) 13 | 14 | As you can see, you'll need at least 10GB of free RAM to run this. If you have less, you can try to remove one machine from the Vagrantfile. This will lead to worse performance though! 15 | 16 | ## Usage 17 | 18 | Depending on the hardware of your computer, installation will probably take between 15 and 25 minutes. 19 | 20 | First install [VirtualBox](https://www.virtualbox.org/) and [Vagrant](http://www.vagrantup.com/). 21 | 22 | Install the Vagrant [Hostmanager plugin](https://github.com/smdahlen/vagrant-hostmanager) 23 | 24 | ```bash 25 | $ vagrant plugin install vagrant-hostmanager 26 | ``` 27 | 28 | Clone this repository. 29 | 30 | ```bash 31 | $ git clone https://github.com/DandyDev/virtual-hadoop-cluster.git 32 | ``` 33 | 34 | Provision the bare cluster. It will ask you to enter your password, so it can modify your `/etc/hosts` file for easy access in your browser. It uses the Vagrant Hostmanager plugin to do this. 35 | 36 | ```bash 37 | $ cd virtual-hadoop-cluster 38 | $ vagrant up 39 | ``` 40 | 41 | Go to the [Cloudera Manager web console](http://vm-cluster-node1:7180) and follow the installation instructions. For more detailed instructions on how to do that, you can use [this guide](http://dandydev.net/blog/installing-virtual-hadoop-cluster). 42 | 43 | **Done!** Have fun with your Hadoop cluster. 44 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | $master_script = <