69 |
70 |
86 |
87 |
88 |
--------------------------------------------------------------------------------
/pages/explanation.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: Test
4 | ignore: true
5 | ---
6 |
7 | # How tutorials.github.com Works
8 |
9 | This site works by using [Github Pages](http://pages.github.com/) to automatically generate and serve content. Github Pages works by using by using [Jekyll](https://github.com/mojombo/jekyll), a "blog-aware, static site generator in Ruby." Since it's a completely static site, all interactions are done through javascript on a per-page basis.
10 |
11 | ## Location and Markup Language
12 |
13 | All tutorials are stored in the `/pages` subdirectory of the repo. Although jekyll supports several different markup languages such as markdown, textile, etc., tutorials.github.com has only been tested using markdown (for now).
14 |
15 | ## Syntax
16 |
17 | Since Github Pages are (unfortunately) not parsed using [Github Flavored Markdown](http://github.github.com/github-flavored-markdown/), we use the [kramdown](https://github.com/gettalong/kramdown)'s extended markdown syntax to enable things like syntax highlighting and allowing markdown parsing _within_ an html tag.
18 |
19 | ## Structure
20 |
21 | Tutorials have, at a minimum, the following structure:
22 |
23 | {% highlight html %}
24 |
25 | some _markdown_ content
26 |
27 | {% endhighlight %}
28 |
29 | Let's explore each in more detail.
30 |
31 | ### data-facets
32 |
33 | These data attributes that are used to list key-value pairs that javascript uses to "facet" the document with. We store JSON because jQuery automatically converts HTML5 data attributes into `camelCase`, so we'd lose formatting.
34 |
35 | ### markdown="1"
36 |
37 | The `markdown="1"` attribute to tell kramdown to parse the content of the divs as markdown instead. Because of this internal parsing, is it recommended to outdent the content inside the div (that is, type your content without leading whitespace), like in the example above.
38 |
39 | ### class="tutorial"
40 |
41 | Tutorial divs should have `class="tutorial"` to separate them from internally-nested divs. I haven't tested nesting divs, but they should work.
42 |
43 | ## Faceting
44 |
45 | We parse all the tutorials on a page and generate a sidebar so that users can drill-down to the most relevant tutorial. For example, a document like this:
46 |
47 | {% highlight html %}
48 |
49 | ...
50 |
51 |
52 |
53 | ...
54 |
55 |
56 |
57 | ...
58 |
59 | {% endhighlight %}
60 |
61 | should show the following sidebar:
62 |
63 | Operating System (2)
64 | OS X (2)
65 | Ubuntu (1)
66 |
67 | Package Management (3)
68 | Homebrew (1)
69 | Macports (1)
70 | Source (1)
71 |
72 | If you clicked on the "Source (1)" `li` then you should see this:
73 |
74 | Operating System (1)
75 | Ubuntu (1)
76 |
77 | Package Management (3)
78 | Source
79 |
80 | ## Installing tutorials.github.com Locally
81 |
82 | First you need to [fork the repo](https://github.com/tutorials/tutorials.github.com/fork_select). Then you need to clone your fork and checkout a topic-branch:
83 |
84 | {% highlight sh %}
85 | git clone git@github.com:some_user/tutorials.github.com.git
86 | cd tutorials.github.com
87 | git checkout -b my-awesome-tutorial
88 | {% endhighlight %}
89 |
90 | Now we need to install some gems using bundler:
91 |
92 | {% highlight sh %}
93 | bundle
94 | {% endhighlight %}
95 |
96 | Then start the jekyll server:
97 |
98 | {% highlight sh %}
99 | jekyll --server --auto
100 | {% endhighlight %}
101 |
102 | Now you can browse to [http://localhost:4000](http://localhost:4000) and see the site running locally.
103 |
104 | ## Creating a New Tutorial
105 |
106 | Creating a new tutorial is easy. All you need to do is create a new file in the `pages` directory, but it's a lot easier if you just start from the `template.md` file:
107 |
108 | {% highlight sh %}
109 | cp pages/template.md pages/my-new-tutorial.md
110 | {% endhighlight %}
111 |
112 | Then just follow the guide and `pages/hello-world.md` as an example.
113 |
114 | Once you're done (and it looks good when running locally), send me a pull request and I'll merge in your new page, regenerate the table of contents and other cleanup, and then push it to github.
115 |
--------------------------------------------------------------------------------
/pages/controlling-ec2-from-the-console.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: Controlling EC2 from the Console
4 | ---
5 |
6 | # Controlling EC2 from the Console
7 |
8 | [Amazon Web Services'](http://aws.amazon.com/) [Elastic Compute Cloud](http://aws.amazon.com/ec2/) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.
9 |
10 | You can control your EC2 instances through the [AWS Management Console](https://console.aws.amazon.com/ec2/), but clicking around can be a chore. This tutorial will help you setup your machine so you can spawn new instances, query existing instances, and do general maintainence without leaving the comfort of a terminal.
11 |
12 |
18 |
19 | ## Assumptions
20 | We assume you've [signed up for EC2](https://aws-portal.amazon.com/gp/aws/developer/registration). We also assume your machine has a public key setup (commonly `~/.ssh/id_rsa` and `~/id_rsa.pub`).
21 |
22 | ## Download Credential Files
23 | First, you need to download some credential files to your machine. Go to the [AWS Security Credentials](https://portal.aws.amazon.com/gp/aws/securityCredentials) page, click "X.509 Certificates", create a new certificate if you need to, download the files when prompted, and then put them into your `~/.ec2` directory:
24 |
25 | {% highlight sh %}
26 | mkdir ~/.ec2
27 | cp ~/Downloads/{cert,pk}-*.pem ~/.ec2
28 | {% endhighlight %}
29 |
30 | ## Install the EC2 Command Line Tools
31 | Now let's install the two packages we need:
32 |
33 | {% highlight sh %}
34 | brew install ec2-{api,ami}-tools
35 | {% endhighlight %}
36 |
37 | Copy the following to your `~/.bash_profile`:
38 |
39 | {% highlight sh %}
40 | export JAVA_HOME="$(/usr/libexec/java_home)"
41 | export EC2_PRIVATE_KEY="$(/bin/ls "$HOME"/.ec2/pk-*.pem | /usr/bin/head -1)"
42 | export EC2_CERT="$(/bin/ls "$HOME"/.ec2/cert-*.pem | /usr/bin/head -1)"
43 | export EC2_HOME="/usr/local/Library/LinkedKegs/ec2-api-tools/jars"
44 | export AWS_ACCESS_KEY_ID="..."
45 | export AWS_SECRET_ACCESS_KEY"=..."
46 | {% endhighlight %}
47 |
48 | Where `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are available from the "Access Keys" tab on the [https://portal.aws.amazon.com/gp/aws/securityCredentials](Amazon Security Credentials page).
49 |
50 | Don't forget to reload your `.bash_profile`:
51 |
52 | {% highlight sh %}
53 | source ~/.bash_profile
54 | {% endhighlight %}
55 |
56 | ## Uploading your Access Keys
57 | Some people use the public/private keys that Amazon gives them, but I find it easier to use the same public key on my laptop that I already use for things like Github and servers at work. To upload, you can use [this helpful script from Eric Hammond](http://alestic.com/2010/10/ec2-ssh-keys) (slightly modified):
58 |
59 | {% highlight sh %}
60 | keypair=macbook # Choose something that makes sense, like your computer name
61 | publickeyfile=$HOME/.ssh/id_rsa.pub # Point this to the public key you want to use
62 | regions=$(ec2-describe-regions | cut -f2)
63 |
64 | for region in $regions; do
65 | echo $region
66 | ec2-import-keypair --region $region --public-key-file $publickeyfile $keypair
67 | done
68 | {% endhighlight %}
69 |
70 | If you get an error like this, you have to wait a little while until Amazon processes your account:
71 |
72 | Client.OptInRequired: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.
73 |
74 | ## Test it All Works
75 | Once we've done all that, we can test it by doing:
76 |
77 | {% highlight sh %}
78 | ec2-describe-instances
79 | {% endhighlight %}
80 |
81 | If you don't see an error, you're all set!
82 |
83 | ## Further Reading
84 | Since there are almost 300 commands available to you now, your best bet is to take a look at the [EC2 Command Line Reference](http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/Welcome.html). One command that you'll want to memorize is [ec2-run-instances](http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html), which will spawn a new ec2 instance when you give it an amazon machine image (AMI) id.
85 |
86 | ## References
87 | These links were helpful in getting my machine setup and writing this tutorial:
88 |
89 | * [Starting Amazon EC2 with Mac OS X](http://www.robertsosinski.com/2008/01/26/starting-amazon-ec2-with-mac-os-x/)
90 | * [Uploading Personal ssh Keys to Amazon EC2](http://alestic.com/2010/10/ec2-ssh-keys)
91 |
92 |
--------------------------------------------------------------------------------
/pages/creating-a-production-storm-cluster.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: Creating a Production Storm Cluster
4 | ---
5 |
6 | # Creating a Production Storm Cluster
7 |
8 | [Storm](http://storm-project.net/) is a [free and open source](http://storm-project.net/about/free-and-open-source.html) distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is [simple](http://storm-project.net/about/simple-api.html), can be used with [any programming language](http://storm-project.net/about/multi-language.html), and is a lot of fun to use!
9 |
10 | This tutorial will help you set up a production storm cluster from scratch.
11 |
12 |
18 |
19 | ## Assumptions
20 | We assume you have two machines that you can ssh into as the `deploy` user, and that user has `sudo` privleges. We'll call these machines `storm` and `zookeeper`. It's ok if you only have one machine, this tutorial will handle that case as well.
21 |
22 | ## Java
23 |
24 | We need to install the JDK (which includes the JRE). Oracle requires you accept the license agreement, so I prefer to download this locally and then `scp` the file to my host. To download the JDK, go to [the jdk download page](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1637583.html), accept the license agreement, and download the file called `jdk-7u5-linux-x64.rpm`. Then copy it to the `deploy` user's home directory using `scp`:
25 |
26 | {% highlight sh %}
27 | scp -C jdk-7u5-linux-x64.rpm deploy@zookeeper:/home/deploy
28 | scp -C jdk-7u5-linux-x64.rpm deploy@storm:/home/deploy
29 | {% endhighlight %}
30 |
31 | Then we install it on each machine (and setup our `JAVA_HOME` and `PATH` to find all the java binaries):
32 |
33 | {% highlight sh %}
34 | ssh zookeeper
35 | sudo rpm -Uvh jdk-7u5-linux-x64.rpm
36 | echo "export JAVA_HOME=/usr/java/default \
37 | export PATH=$PATH:$JAVA_HOME/bin:$HOME/bin" > ~/.bash_profile
38 | logout
39 |
40 | ssh storm
41 | sudo rpm -Uvh jdk-7u5-linux-x64.rpm
42 | echo "export JAVA_HOME=/usr/java/default \
43 | export PATH=$PATH:$JAVA_HOME/bin:$HOME/bin" > ~/.bash_profile
44 | logout
45 | {% endhighlight %}
46 |
47 | ## Zookeeper
48 |
49 | ### Installing Dependencies
50 |
51 | First we need to install some dependencies and setup a place to keep our source files:
52 |
53 | {% highlight sh %}
54 | ssh deploy@zookeeper
55 | mkdir -p ~/src
56 | sudo yum install -y libtool libuuid-devel gcc-c++ make
57 | {% endhighlight %}
58 |
59 | ### Installing Zookeeper
60 |
61 | Now we're ready to install zookeeper:
62 |
63 | {% highlight sh %}
64 | cd ~/src
65 | wget http://mirrors.axint.net/apache/zookeeper/zookeeper-3.4.3/zookeeper-3.4.3.tar.gz
66 | tar xzf zookeeper-3.4.3.tar.gz
67 | {% endhighlight %}
68 |
69 | ### Running Zookeeper
70 |
71 | Zookeeper is configured by a file located in `conf/zookeper.conf`. We'll use this as our zookeeper config when we start the zookeeper server:
72 |
73 | {% highlight sh %}
74 | ~/src/zookeeper-3.4.3/bin/zkServer.sh start ~/src/zookeeper-3.4.3/conf/zoo_sample.cfg
75 | {% endhighlight %}
76 |
77 | ## Storm
78 |
79 | Now that zookeeper is running, we can setup our storm servers. First we need to [install native dependencies](https://github.com/nathanmarz/storm/wiki/Installing-native-dependencies):
80 |
81 | ### Installing Dependencies
82 |
83 | {% highlight sh %}
84 | ssh deploy@zookeeper
85 | sudo yum install -y git libtool libuuid-devel gcc-c++ make
86 | mkdir -p ~/src /tmp/storm
87 | {% endhighlight %}
88 |
89 | ### Installing ZeroMQ
90 |
91 | {% highlight sh %}
92 | cd ~/src
93 | wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
94 | tar xzf zeromq-2.1.7.tar.gz
95 | cd zeromq-2.1.7
96 | ./configure
97 | make
98 | sudo make install
99 | {% endhighlight %}
100 |
101 | ### Installing JZMQ
102 |
103 | {% highlight sh %}
104 | cd ~/src
105 | git clone https://github.com/nathanmarz/jzmq.git
106 | cd jzmq
107 | ./autogen.sh
108 | ./configure
109 | make
110 | sudo make install
111 | {% endhighlight %}
112 |
113 | ### Installing Storm
114 |
115 | {% highlight sh %}
116 | cd ~/src
117 | wget https://github.com/downloads/nathanmarz/storm/storm-0.7.0.zip
118 | unzip storm-0.7.0.zip
119 | {% endhighlight %}
120 |
121 | ## Configuring Storm
122 |
123 | We need to point storm to the correct zookeeper servers, as well as [setup other config options](https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster). For now, we'll just modify the `conf/storm.yaml` file to look like this:
124 |
125 | {% highlight yaml %}
126 | storm.local.dir: "/tmp/storm"
127 | storm.zookeeper.servers:
128 | - "zookeeper"
129 | nimbus.host: "localhost"
130 | {% endhighlight %}
131 |
132 | You'll want to put some real values in there depending on the size of your cluster, and you'll definitely want to change `/tmp/storm` to something more persistent, but this sufices for a demo.
133 |
134 | ## Running Storm
135 |
136 | In order to run storm, you need both nimbus and supervisor processes running. We can run them with `nohup`:
137 |
138 | {% highlight sh %}
139 | nohup ~/src/storm-0.7.0/bin/storm nimbus &
140 | nohup ~/src/storm-0.7.0/bin/storm supervisor &
141 | {% endhighlight %}
142 |
143 | Now it should be ready to process topologies!
144 |
145 | ## Running the Storm UI
146 |
147 | Storm comes with a nifty dashboard to view cluster stats. To run it, just do:
148 |
149 | {% highlight sh %}
150 | nohup ~/src/storm-0.7.0/bin/storm ui &
151 | {% endhighlight %}
152 |
");
35 | }
36 | });
37 |
38 | dict = _.chain(tutorials)
39 | .reduce(function(lookup, tutorial) {
40 | _($(tutorial).data().facets).each(function(val, key) {
41 | if(!_(lookup).has(key)) { lookup[key] = {}; }
42 | // We want to be able to have multiple values for vertain
43 | // facets, so we convert strings to arrays and iterate over each
44 | if(_(val).isString()) { val = [val]; }
45 | _(val).each(function(v) {
46 | if(!_(lookup[key]).has(v)) { lookup[key][v] = []; }
47 | lookup[key][v].push(tutorial);
48 | });
49 | });
50 | return lookup;
51 | }, {})
52 | .value();
53 |
54 | // Draw the sidebar
55 | redraw();
56 |
57 | //Fix sidebar
58 | function sidebar_fix() {
59 | var header_height = $('header').height(),
60 | footer_height = 150, //fixed
61 | window_height = $(window).height(),
62 | $nav_sidebar = $('nav#sidebar'),
63 | sidebar_height= $nav_sidebar.height(),
64 | difference = window_height - (header_height + footer_height);
65 |
66 | if ($(window).width()>970){
67 | var existence = sidebar_height+difference;
68 | $('header h1').css('paddingTop', 0);
69 | if (existence>250) {
70 | $nav_sidebar.height(existence);
71 | if ($('header').width()==250) {
72 | $('header').css('position', 'fixed');
73 | $('footer').css('position', 'fixed');
74 | }
75 | }
76 | } else {
77 | $nav_sidebar.css('height', 'auto');
78 | $('header').css('position', 'static');
79 | $('footer').css('position', 'static');
80 | $('header h1').css('paddingTop', $('section#main h1').height());
81 | }
82 |
83 | }
84 | sidebar_fix();
85 | $(window).resize(function() { sidebar_fix() });
86 |
87 | // title fix
88 | var $section = $('section#main'),
89 | $title = $section.find('h1'),
90 | title_height = $title.height(),
91 | set_padding = title_height + 20;
92 | $section.css('paddingTop',set_padding);
93 |
94 | // Hide tutorials if the user clicks on a facet
95 | $("nav li").live("click", function(e) {
96 | target = $(this).data();
97 | existing_filter = _(filters).find(function(filter) {
98 | return filter.name === target.name && filter.value === target.value;
99 | });
100 | if(existing_filter) {
101 | _gaq.push(['_trackEvent', 'filters - remove', existing_filter.name, existing_filter.value, filters.size]);
102 | filters = _(filters).without(existing_filter);
103 | } else {
104 | _gaq.push(['_trackEvent', 'filters - apply', target.name, target.value, filters.size]);
105 | filters = _(filters).reject(function(filter) { return filter.name === target.name && filter.value === target.value; });
106 | filters.push(target);
107 | }
108 | redraw();
109 | });
110 | });
111 |
112 | function redraw() {
113 | update_visibility();
114 | update_sidebar();
115 | var template = $("#sidebar_template").text();
116 | var list = _.template(template);
117 | $("#sidebar").html(list);
118 | };
119 |
120 | function update_visibility() {
121 | // Grab all the tutorials for each facet into an array
122 | all = _.chain(filters).map(function(filter) {
123 | return dict[filter.name][filter.value];
124 | }).flatten().value();
125 |
126 | // Only keep the tutorials that are visible
127 | keep = _(tutorials).select(function(tutorial) {
128 | return all.length - _(all).without(tutorial).length === filters.length
129 | });
130 |
131 | // Make DOM elements visible or not
132 | $(keep).show();
133 | $(_(tutorials).difference(keep)).hide();
134 | };
135 |
136 | function update_sidebar() {
137 | visible = _.chain($("div.tutorial:visible"))
138 | // Collect all the facets
139 | .map(function(tutorial) { return $(tutorial).data().facets; })
140 | // and combine all facets into a nested hash
141 | .reduce(function(counts, facets) {
142 | // by iterating over each pair and aggregating the counts of each
143 | _(facets).each(function(val, key) {
144 | // Initialize the names
145 | if(!_(counts).has(key)) { counts[key] = {}; }
146 | // and the values/counts
147 | if(_(val).isString()) { val = [val]; }
148 | _(val).each(function(v) {
149 | if(!_(counts[key]).has(v)) { counts[key][v] = 0; }
150 | // then increment the counts
151 | counts[key][v] += 1;
152 | });
153 | });
154 | return counts;
155 | // This is where we're storing the new data
156 | }, {})
157 | // Get the final counts
158 | .value();
159 |
160 | // Generate counts for the sidebar
161 | var counts = {};
162 | _(visible).each(function(val, key) {
163 | counts[key] = _(val).keys().length;
164 | });
165 | visible_counts = counts;
166 | };
167 |
--------------------------------------------------------------------------------
/pages/retrieving-storm-data-from-nimbus.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: Retrieving Storm Cluster Statistics from Nimbus
4 | ---
5 |
6 | # Retrieving Storm Cluster Statistics from Nimbus
7 |
8 | This tutorial will show how to obtain basic storm topology statistics from nimbus (similar to the information show in the storm ui dashboard) so you can store that data in a separate location -- such as graphite -- for monitoring. Since storm topologies are thrift-based, you can use almost any language to connect to nimbus to download your data.
9 |
10 |
16 |
17 | ## Assumptions
18 |
19 | We assume you'll store your data in `~/code`. We also assume that your homebrew is newer enough that the `/usr/local` directory is homebrew's git repo.
20 |
21 | ## Setup
22 |
23 | Let's make a location to store our project:
24 |
25 | {% highlight sh %}
26 | mkdir -p ~/code/nimbus-thrift-demo
27 | {% endhighlight %}
28 |
29 | ## Download the Storm Source Code
30 |
31 | In order to connect to nimbus, we need to get the `storm.thrift` file from the storm source, and use that to generate the thrift bindings in ruby:
32 |
33 | {% highlight sh %}
34 | cd ~/code
35 | wget https://github.com/downloads/nathanmarz/storm/storm-0.7.0.zip
36 | unzip storm-0.7.0.zip
37 | {% endhighlight %}
38 |
39 | ## Install Thrift 0.7.0 and the Thrift Gem
40 |
41 | Now let's install the same version of thrift that storm 0.7.0 uses:
42 |
43 | {% highlight sh %}
44 | cd /usr/local
45 | brew update
46 | brew versions thrift
47 | git checkout 141ddb6 /usr/local/Library/Formula/thrift.rb
48 | brew install thrift
49 | {% endhighlight %}
50 |
51 | And the thrift gem:
52 |
53 | {% highlight sh %}
54 | gem install thrift
55 | {% endhighlight %}
56 |
57 | ## Generate the Ruby Bindings
58 |
59 | {% highlight sh %}
60 | cd ~/code/nimbus-thrift-demo
61 | cp ~/code/storm-0.7.0/src/storm.thrift .
62 | thrift --gen rb storm.thrift
63 | {% endhighlight %}
64 |
65 | ## Connect to Nimbus
66 |
67 | Now that we have the ruby bindings in place, we can create a sample app to connect to our nimbus cluster. Copy the following to `test.rb`, replacing `localhost` with the hostname of your nimbus instance:
68 |
69 | {% highlight ruby %}
70 | $:.push('gen-rb')
71 |
72 | require 'rubygems'
73 | require 'thrift'
74 | require 'nimbus'
75 |
76 | begin
77 | socket = Thrift::Socket.new('localhost', 6627)
78 | transport = Thrift::FramedTransport.new(socket)
79 | protocol = Thrift::BinaryProtocol.new(transport)
80 | client = Nimbus::Client.new(protocol)
81 |
82 | transport.open
83 |
84 | summary = client.getClusterInfo
85 | puts summary.inspect
86 |
87 | transport.close
88 |
89 | rescue Thrift::Exception => tx
90 | print 'Thrift::Exception: ', tx.message, "\n"
91 | end
92 | {% endhighlight %}
93 |
94 | Now we can run our test app:
95 |
96 | {% highlight sh %}
97 | ruby test.rb
98 | {% endhighlight %}
99 |
100 | Now you can take that data and write it to where you'd like. Enjoy!
101 |
102 |
103 |
104 |
110 |
111 | ## Assumptions
112 |
113 | We assume you'll store your data in `~/code`.
114 |
115 | ## Setup
116 |
117 | Let's make a location to store our project:
118 |
119 | {% highlight sh %}
120 | mkdir -p ~/code/nimbus-thrift-demo
121 | {% endhighlight %}
122 |
123 | ## Download the Storm Source Code
124 |
125 | In order to connect to nimbus, we need to get the `storm.thrift` file from the storm source, and use that to generate the thrift bindings in python:
126 |
127 | {% highlight sh %}
128 | cd ~/code
129 | wget https://github.com/downloads/nathanmarz/storm/storm-0.7.0.zip
130 | unzip storm-0.7.0.zip
131 | {% endhighlight %}
132 |
133 | ## Install Thrift 0.7.0 and the Thrift Python Package
134 |
135 | Now let's install the same version of thrift that storm 0.7.0 uses:
136 |
137 | {% highlight sh %}
138 | cd /usr/local
139 | brew update
140 | brew versions thrift
141 | git checkout 141ddb6 /usr/local/Library/Formula/thrift.rb
142 | brew install thrift
143 | {% endhighlight %}
144 |
145 | And the thrift python package:
146 |
147 | {% highlight sh %}
148 | pip install thrift==0.7.0
149 | {% endhighlight %}
150 |
151 | ## Generate the Python Bindings
152 |
153 | {% highlight sh %}
154 | cd ~/code/nimbus-thrift-demo
155 | cp ~/code/storm-0.7.0/src/storm.thrift .
156 | thrift --gen py storm.thrift
157 | {% endhighlight %}
158 |
159 | ## Connect to Nimbus
160 |
161 | Now that we have the python bindings in place, we can create a sample app to connect to our nimbus cluster. Copy the following to `test.py`, replacing `localhost` with the hostname of your nimbus instance:
162 |
163 | {% highlight python %}
164 | import sys
165 |
166 | sys.path.append('gen-py')
167 |
168 | from thrift import Thrift
169 | from thrift.transport import TSocket
170 | from thrift.transport import TTransport
171 | from thrift.protocol import TBinaryProtocol
172 |
173 | from storm import Nimbus
174 | from storm.ttypes import *
175 | from storm.constants import *
176 |
177 | try:
178 | socket = TSocket.TSocket('localhost', 6627)
179 | transport = TTransport.TFramedTransport(socket)
180 | protocol = TBinaryProtocol.TBinaryProtocol(transport)
181 | client = Nimbus.Client(protocol)
182 |
183 | transport.open()
184 |
185 | summary = client.getClusterInfo()
186 | print summary
187 |
188 | transport.close()
189 |
190 | except Thrift.TException, tx:
191 | print "%s" % (tx.message)
192 | {% endhighlight %}
193 |
194 | Now we can run our test app:
195 |
196 | {% highlight sh %}
197 | python test.py
198 | {% endhighlight %}
199 |
200 | Now you can take that data and write it to where you'd like. Enjoy!
201 |
202 |
16 | To install MySQL, run the following command from a terminal prompt:
17 |
18 | {% highlight sh %}
19 | sudo apt-get install mysql-server
20 | {% endhighlight %}
21 |
22 | During the installation process you will be prompted to enter a password for the __MySQL__ root user.
23 |
24 | Once the installation is complete, the MySQL server should be started automatically. You can run the following command from a terminal prompt to check whether the MySQL server is running:
25 |
26 | {% highlight sh %}
27 | sudo netstat -tap | grep mysql
28 | {% endhighlight %}
29 |
30 | When you run this command, you should see the following line or something similar:
31 |
32 | {% highlight sh %}
33 | tcp 0 0 localhost.localdomain:mysql *:* LISTEN -
34 | {% endhighlight %}
35 |
36 | If the server is not running correctly, you can type the following command to start it:
37 |
38 | {% highlight sh %}
39 | sudo /etc/init.d/mysql restart
40 | {% endhighlight %}
41 |
42 | Configuration
43 | -------------
44 | You can edit the `/etc/mysql/my.cnf` file to configure the basic settings -- log file, port number, etc. For example, to configure __MySQL__ to listen for connections from network hosts, change the `bind_address` directive to the server's IP address:
45 |
46 | {% highlight sh %}
47 | bind-address = 192.168.0.5
48 | {% endhighlight %}
49 |
50 | Note Replace 192.168.0.5 with the appropriate address.
51 |
52 | After making a change to `/etc/mysql/my.cnf` the __mysql__ daemon will need to be restarted:
53 |
54 | {% highlight sh %}
55 | sudo /etc/init.d/mysql restart
56 | {% endhighlight %}
57 |
58 | ---------------------------------------------------------
59 |
60 |
61 |
62 |
68 | These are instructions for compiling and installing a 64-bit version of [MySQL](http://www.mysql.com/), the world's most popular open source database, on Mac OS X 10.6 (Snow Leopard).
69 |
70 | The benefits of manually building MySQL yourself in `/usr/local` are detailed [here](http://hivelogic.com/articles/using_usr_local/). I also wrote a in-depth explanation of why you might want to build MySQL yourself in my [Compiling MySQL on Leopard](http://hivelogic.com/articles/installing-mysql-on-mac-os-x/) article.
71 |
72 | NO SUPPORT
73 | ----------
74 | These instructions will probably work just fine and you won't run into any trouble, but if you do, **please don't email me about it**. I wish I could help everybody who might have an issue, but I just don't have the time to help you troubleshoot hundreds of potential variables with your specific configuration, and I probably won't even reply to the email, and I'm sorry in advance for that.
75 |
76 | PREREQUISITES
77 | -------------
78 | Before following these instructions, you will need:
79 |
80 | - Mac OS X 10.6 Snow Leopard
81 | - The latest Xcode Tools (from the Snow Leopard DVD or downloaded from [Apple](http://developer.apple.com/) - the 10.5 version _won't work_)
82 | - Confidence running UNIX commands using the Terminal
83 |
84 | If you want to learn more about UNIX and the command line, check out [my PeepCode screencast](http://peepcode.com/products/meet-the-command-line) on this topic.
85 |
86 | STEP 1: SET THE PATH
87 | --------------------
88 | Launch Terminal.app from the `/Applications/Utilities` folder.
89 |
90 | We need to set your shell's `PATH` variable first. The `PATH` variable determines where your system searches for command-line programs. Using the editor of your choice, create and edit a file in your home directory named `.profile` (note the "." preceding the filename).
91 |
92 | If you're using [TextMate](http://macromates.com/) like you should be and have [installed the UNIX `mate` command](http://manual.macromates.com/en/using_textmate_from_terminal.html), then you can create and start editing the file like this:
93 |
94 | {% highlight sh %}
95 | mate ~/.profile
96 | {% endhighlight %}
97 |
98 | To the end of this file, add the following line (or verify that it's already there):
99 |
100 | {% highlight sh %}
101 | export PATH="/usr/local/bin:/usr/local/sbin:/usr/local/mysql/bin:$PATH"
102 | {% endhighlight %}
103 |
104 | Close and save the file and run this command to load the new setting into your current shell:
105 |
106 | source ~/.profile
107 | To verify that you've updated your path, enter the following command:
108 |
109 | {% highlight sh %}
110 | echo $PATH
111 | {% endhighlight %}
112 |
113 | You should see `/usr/local/bin` at the beginning of the line returned by the system.
114 |
115 | STEP 2: DOWNLOAD
116 | ----------------
117 |
118 | We're going to create a folder to contain the files we're about to download and compile. If you want, you can delete this folder when you're done, but keeping it around makes it easier to re-install (or uninstall) these apps later.
119 |
120 | Make the new folder:
121 |
122 | {% highlight sh %}
123 | mkdir ~/src
124 | cd ~/src
125 | {% endhighlight %}
126 |
127 | I used to provide a link to download the latest version of MySQL, but they update frequently and sometimes the links die, so instead, please go to the MySQL site, and download the latest version from [this page](http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.39.tar.gz/from/pick#mirrors), picking a mirror near you. Then move or copy it to your `src` folder.
128 |
129 | For those of you who expect the world, you can try this command, but please don't email me if it's broken:
130 |
131 | {% highlight sh %}
132 | curl -O http://mysql.mirrors.pair.com/Downloads/MySQL-5.1/mysql-5.1.39.tar.gz
133 | {% endhighlight %}
134 |
135 | STEP 3: COMPILE AND INSTALL
136 | ---------------------------
137 |
138 | Build and install MySQL like this:
139 |
140 | {% highlight sh %}
141 | tar xzvf mysql-5.1.37.tar.gz
142 | cd mysql-5.1.37
143 | ./configure --prefix=/usr/local/mysql --with-extra-charsets=complex --enable-thread-safe-client --enable-local-infile --enable-shared --with-plugins=innobase
144 | make
145 | sudo make install
146 | cd /usr/local/mysql
147 | sudo ./bin/mysql_install_db --user=mysql
148 | sudo chown -R mysql ./var
149 | cd ..
150 | {% endhighlight %}
151 |
152 | STARTING (AND AUTO-STARTING) MYSQL
153 | ----------------------------------
154 |
155 | In most cases, you'll want MySQL to auto-start every time you boot (or reboot) your Mac. The easiest way to do this is using launchd, Mac OS X's infrastructure for managing system processes.
156 |
157 | I've prepared a launchd plist file that will allow you to manage MySQL, starting it at boot and stopping it cleanly at shutdown. Save the plist file to your `~/src` directory and then move it to the proper place with the following commands:
158 |
159 | {% highlight sh %}
160 | cd ~/src
161 | curl -O http://hivelogic.com/downloads/com.mysql.mysqld.plist
162 | sudo mv ~/src/com.mysql.mysqld.plist /Library/LaunchDaemons
163 | sudo chown root /Library/LaunchDaemons/com.mysql.mysqld.plist
164 | {% endhighlight %}
165 |
166 | Finally, tell `launchd` to load and startup MySQL:
167 |
168 | {% highlight sh %}
169 | sudo launchctl load -w /Library/LaunchDaemons/com.mysql.mysqld.plist
170 | {% endhighlight %}
171 |
172 | If you see no response, it probably means that everything worked correctly, and that MySQL is running. You can verify this by launching MySQL's command-line monitor:
173 |
174 | {% highlight sh %}
175 | mysql -uroot
176 | {% endhighlight %}
177 |
178 | You should see something like this:
179 |
180 | {% highlight sh %}
181 | Welcome to the MySQL monitor. Commands end with ; or g.
182 | Your MySQL connection id is 1
183 | Server version: 5.1.37 Source distribution
184 | Type 'help;' or 'h' for help. Type 'c' to clear the buffer.
185 | mysql>
186 | {% endhighlight %}
187 |
188 | If this prompt appears, MySQL has been installed correctly. Type `exit` and press return to quit the MySQL monitor.
189 |
190 | If you didn't see that message, it's possible that something "bad" happened, or that you may have missed a step above. You can always try running through the instructions again.
191 |
192 | You now have a custom-built 64-bit version of MySQL.
193 |
194 | STARTING AND STOPPING MYSQL MANUALLY
195 | ------------------------------------
196 |
197 | If you ever want to stop MySQL manually, use this command:
198 |
199 | {% highlight sh %}
200 | sudo launchctl unload -w /Library/LaunchDaemons/com.mysql.mysqld.plist
201 | {% endhighlight %}
202 |
203 | To (re)start MySQL manually, use this command:
204 |
205 | {% highlight sh %}
206 | sudo launchctl load -w /Library/LaunchDaemons/com.mysql.mysqld.plist
207 | {% endhighlight %}
208 |
209 | EXTRA CREDIT: THE RAILS GEM
210 | ---------------------------
211 |
212 | You can install the Rails MySQL Gem by pointing it at your new MySQL installation direcory:
213 |
214 | {% highlight sh %}
215 | sudo gem install mysql -- --with-mysql-dir=/usr/local/mysql
216 | {% endhighlight %}
217 |
218 | That should do it.
219 |
220 | ---------------------------------------------------------
221 |
222 |
223 |
224 |
230 | If you already have a `/usr/local` folder and it's not owned by your user:
231 |
232 | {% highlight sh %}
233 | sudo chown -R `whoami` /usr/local
234 | {% endhighlight %}
235 |
236 | Install Homebrew:
237 |
238 | {% highlight sh %}
239 | cd /usr/local
240 | git init
241 | git remote add origin git://github.com/mxcl/homebrew.git
242 | git pull origin master
243 | {% endhighlight %}
244 |
245 | This is kind of odd -- you install Homebrew right into the base of your `/usr/local` folder. It nicely ignores other folders that already exists there. Just do it.
246 |
247 | Install MySQL:
248 |
249 | {% highlight sh %}
250 | brew install mysql
251 | {% endhighlight %}
252 |
253 | Yeah, it's really that easy. This will take a while.
254 |
255 | Now warm it up:
256 |
257 | {% highlight sh %}
258 | mysql_install_db
259 | {% endhighlight %}
260 |
261 | And make sure it automatically starts again on login:
262 |
263 | {% highlight sh %}
264 | launchctl load -w /usr/local/Cellar/mysql/5.1.43/com.mysql.mysqld.plist
265 | {% endhighlight %}
266 |
267 | ---------------------------------------------------------
268 |
269 |
270 |
--------------------------------------------------------------------------------
/pages/hello-world.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: Hello, World!
4 | ---
5 |
6 | Hello, World!
7 | =============
8 |
9 | Here are some examples of "Hello, World!" in various programming languages.
10 |
11 |
37 |