├── .gitignore ├── .rvmrc ├── Capfile ├── Gemfile ├── Procfile ├── README.md ├── bin ├── async_client.rb ├── client.rb ├── dispatcher.rb └── worker.rb ├── config └── deploy.rb ├── controllers ├── dispatcher_controller.rb ├── skeeter_controller.rb └── worker_controller.rb ├── images ├── moose-ascii.jpg └── moose.png ├── service ├── config │ ├── .service.rb.swp │ └── skeeter.rb └── skeeter.rb └── test_server ├── public └── images │ ├── InBed.jpg │ └── programming-motherfuckers.jpg └── test_server.rb /.gitignore: -------------------------------------------------------------------------------- 1 | *.DS_Store 2 | *.swp 3 | -------------------------------------------------------------------------------- /.rvmrc: -------------------------------------------------------------------------------- 1 | rvm ruby-1.9.2-p180 2 | -------------------------------------------------------------------------------- /Capfile: -------------------------------------------------------------------------------- 1 | load 'deploy' if respond_to?(:namespace) # cap2 differentiator 2 | Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) } 3 | 4 | load 'config/deploy' # remove this line to skip loading any of the default tasks -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source :gemcutter 2 | 3 | gem 'ffi', '1.0.7' 4 | gem 'ffi-rzmq', '0.8.0' 5 | gem 'em-http-request', '1.0.0.beta.3' 6 | gem 'em-synchrony', '0.3.0.beta.1' 7 | gem 'em-zeromq', '0.2.1' 8 | gem 'goliath', :git => 'https://github.com/postrank-labs/goliath.git', :ref => 'bb8d049d15eaf78bc062' 9 | gem 'sinatra', '1.2.3' 10 | gem 'daemons' 11 | gem 'capistrano' 12 | gem 'foreman' 13 | -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | worker: ruby bin/worker.rb 2 | dispatcher: ruby bin/dispatcher.rb 3 | skeeter: ruby service/skeeter.rb 4 | test_server: ruby test_server/test_server.rb 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Skeeter 2 | 3 | ## What is it? 4 | 5 | Skeeter is a small asynchronous web service that takes in image urls and returns ascii art. 6 | Convert this: 7 | 8 | ![Original 9 | image](https://github.com/blakesmith/skeeter/raw/master/images/moose.png) 10 | 11 | Into this: 12 | 13 | ![Converted 14 | image](https://github.com/blakesmith/skeeter/raw/master/images/moose-ascii.jpg) 15 | 16 | You make a request to it like so: 17 | 18 | http://skeeter.blakesmith.me/?image_url=http://www.softicons.com/download/animal-icons/animal-icons-by-martin-berube/png/128/moose.png&width=100 19 | 20 | And it spits out the ascii art! Magic! 21 | 22 | ## Why do this? 23 | 24 | I wanted a way to have images pasted in campfire via my 25 | [flamethrower](http://github.com/blakesmith/flamethrower) IRC gateway converted 26 | to ascii for inline display. Rather than add extra dependencies and additional 27 | overhead to flamethrower itself, it makes a simple non-blocking service call 28 | using EM-HttpRequest. 29 | 30 | This was also an exercise in learning more about Ruby 1.9 Fibers, the 31 | [Goliath](http://www.igvita.com/2011/03/08/goliath-non-blocking-ruby-19-web-server/) 32 | webserver, and last but certainly not least [ZeroMQ](http://www.zeromq.org/) 33 | (Only the sweetest most awesomely mind expanding piece of software ever). 34 | Put all these components together and you can build something pretty freaking 35 | sweet. 36 | 37 | ## How does it work? 38 | 39 | ### All the moving pieces 40 | 41 | Each one of the following represents an independent ruby process that 42 | communicates via message passing with ZeroMQ. 43 | 44 | - skeeter.rb - The Goliath webserver definition. Similar in style to a Rack app. 45 | Feeds requests into dispatcher.rb 46 | - dispatcher.rb - Lightweight process that takes requests from the webserver via 47 | a ZeroMQ socket and evenly distributes them to a pool of workers connected to 48 | a backend ZeroMQ socket. 49 | - worker.rb - Worker process. Listens for requests on a ZeroMQ socket and shells 50 | out to jp2a to do the actual image conversion. Responds on the socket with the 51 | converted ascii. 52 | - jp2a - C program that does the actual ascii conversion. 53 | 54 | ### Request lifecycle 55 | 56 | A user makes a request via HTTP to the Goliath webserver (skeeter.rb). Each web 57 | request is wrapped in a Ruby 1.9 Fiber. In this simplified case, think of a 58 | Fiber as a lightweight Thread that can be scheduled manually. The Fiber is 59 | created, parses the request, puts a JSON message onto the ZeroMQ socket and then 60 | goes to sleep. The message is routed via the dispatcher to a worker on the 61 | backend using fair queueing (think of it like a load balancer). The worker takes 62 | the request off the socket, does the work (converting the image) and then sends 63 | a response back across the socket. The dispatcher routes the response back to 64 | the original requester. At this point, EventMachine wakes up the sleeping 65 | request Fiber with the response, which is sent to the client. Voilà! ASCII 66 | Magic! 67 | 68 | ## Why it's awesome 69 | 70 | Here's why I get so excited about this stuff: Using ZeroMQ and an awesome async 71 | webserver, we are able to glue together a relatively complicated architecture 72 | with very little code. Not only that, but the webserver itself can be completely 73 | non-blocking while the backend can make blocking calls without hurting request 74 | throughput. Scaling this system is really simple: we just add more worker 75 | processes (not even any config changes needed). A new worker simply joins the 76 | ZeroMQ socket and begins processing requests. This is all transparently handled 77 | by ZeroMQ. Workers can join and leave the worker pool at will and ZeroMQ will 78 | handle it all for us. 79 | 80 | When our service really starts to take off, we can even spin up more worker 81 | processes on other nodes in the networke and ZeroMQ can handle communicating 82 | with them transparently over TCP. 83 | 84 | If you haven't played with ZeroMQ, I suggest you make up a good reason to try it 85 | out. 86 | 87 | ## Install 88 | 89 | ### Native dependencies 90 | 91 | - jp2a (get it here: http://csl.sublevel3.org/jp2a/) 92 | - zeromq (OS X users can 'brew install zeromq') 93 | 94 | ### Ruby dependencies 95 | 96 | Make sure you have the bundler gem installed and do 'bundle install'. This will 97 | automatically install all the necessary ruby dependencies. 98 | 99 | ### Running 100 | 101 | Each ruby process has its own controller script (in the controllers/ direction) 102 | that will launch it as a daemon. You must use Ruby 1.9 (for fiber support). 103 | 104 | - skeeter_controller.rb - Start the Goliath web server on port 9000. (ruby 105 | controllers/skeeter_controller.rb start) 106 | - dispatcher_controller.rb - Start the dispatcher. (ruby 107 | controllers/dispatcher_controller.rb) 108 | - worker_controller.rb - Start the forked worker processes, 2 by default (ruby 109 | controllers/worker_controller.rb) 110 | 111 | There are corresponding capistrano tasks to start and stop each one of these 112 | daemon processes. 113 | 114 | ## Author 115 | 116 | Skeeter is written by Blake Smith . 117 | 118 | -------------------------------------------------------------------------------- /bin/async_client.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'em-zeromq' 3 | require 'json' 4 | require 'fiber' 5 | 6 | class Handler 7 | attr_reader :received 8 | 9 | def on_readable(socket, messages) 10 | messages.each do |m| 11 | puts m.copy_out_string 12 | end 13 | end 14 | end 15 | 16 | images = [ 17 | "http://localhost:4567/images/InBed.jpg", 18 | "http://localhost:4567/images/programming-motherfuckers.jpg" 19 | ] 20 | 21 | EM.run do 22 | context = EM::ZeroMQ::Context.new(1) 23 | req_socket = context.connect(ZMQ::REQ, "ipc:///tmp/dispatch-front.ipc", Handler.new) 24 | 25 | 26 | EM::PeriodicTimer.new(3.3) do 27 | json = {:message => 'convert', :url => images[rand(images.size)], :width => 60}.to_json 28 | puts "Sending #{json}" 29 | 30 | req_socket.send_msg(json) 31 | req_socket.register_readable 32 | end 33 | end 34 | 35 | -------------------------------------------------------------------------------- /bin/client.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'ffi-rzmq' 3 | require 'json' 4 | 5 | context = ZMQ::Context.new 6 | 7 | client = context.socket(ZMQ::REQ) 8 | client.connect("ipc:///tmp/dispatch-front.ipc") 9 | 10 | images = [ 11 | "http://localhost:4567/images/InBed.jpg", 12 | "http://localhost:4567/images/programming-motherfuckers.jpg" 13 | ] 14 | 15 | puts "Press enter when the workers are ready..." 16 | gets 17 | 18 | while true 19 | json = {:message => 'convert', :url => images[rand(images.size)], :width => 60}.to_json 20 | puts "Sending #{json}" 21 | 22 | client.send_string(json) 23 | response = client.recv_string 24 | puts response 25 | sleep(rand(0.7)) 26 | end 27 | 28 | client.close 29 | context.terminate 30 | -------------------------------------------------------------------------------- /bin/dispatcher.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'ffi-rzmq' 3 | 4 | context = ZMQ::Context.new 5 | 6 | front_addr = "ipc:///tmp/dispatch-front.ipc" 7 | back_addr = "ipc:///tmp/dispatch-back.ipc" 8 | 9 | frontend = context.socket(ZMQ::XREP) 10 | frontend.bind(front_addr) 11 | 12 | backend = context.socket(ZMQ::XREQ) 13 | backend.bind(back_addr) 14 | 15 | queue = ZMQ::Device.new(ZMQ::QUEUE, frontend, backend) 16 | 17 | frontend.close 18 | backend.close 19 | context.terminate 20 | -------------------------------------------------------------------------------- /bin/worker.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'ffi-rzmq' 3 | require 'open-uri' 4 | require 'json' 5 | 6 | WORKER_COUNT = ARGV.size > 0 ? ARGV[0].to_i : 2 7 | 8 | pids = [] 9 | 10 | def fetch_image(url, width) 11 | response = `curl -s "#{url}" | convert - jpg:- | jp2a - --width=#{width}` 12 | end 13 | 14 | def die(pids) 15 | puts "Killing all workers..." 16 | pids.map {|p| Process.kill("INT", p) } 17 | end 18 | 19 | WORKER_COUNT.times do |i| 20 | pids << fork do 21 | trap("INT") { exit } 22 | puts "Starting worker #{i}..." 23 | context = ZMQ::Context.new(1) 24 | 25 | receiver = context.socket(ZMQ::REP) 26 | receiver.connect("ipc:///tmp/dispatch-back.ipc") 27 | 28 | loop do 29 | str = receiver.recv_string 30 | 31 | message = JSON.parse(str) 32 | puts "Worker #{i}: Got message #{message.inspect}" 33 | 34 | # Do some work 35 | if message['message'] == "convert" 36 | width = message['width'] || 80 37 | image = fetch_image(message['url'], width) 38 | receiver.send_string(image) 39 | end 40 | end 41 | end 42 | end 43 | 44 | trap("INT") { die(pids) } 45 | trap("TERM") { die(pids) } 46 | 47 | Process.wait 48 | -------------------------------------------------------------------------------- /config/deploy.rb: -------------------------------------------------------------------------------- 1 | $:.unshift(File.expand_path('./lib', ENV['rvm_path'])) # Add RVM's lib directory to the load path. 2 | 3 | require "rvm/capistrano" # Load RVM's capistrano plugin. 4 | set :rvm_ruby_string, "1.9.2-p180" 5 | set :rvm_type, :user # Copy the exact line. I really mean :user here 6 | 7 | set :user, "rails" 8 | set :primary_server, "stiletto.blakesmith.me" 9 | 10 | set :application, "skeeter" 11 | set :repository, "ssh://#{user}@#{primary_server}/home/blake/pubgit/#{application}.git" 12 | 13 | set :scm, "git" 14 | set :branch, "master" 15 | set :deploy_to, "/home/rails/#{application}" 16 | set :deploy_via, :remote_cache 17 | 18 | set :runner, nil 19 | 20 | role :app, "#{primary_server}" # This may be the same as your `Web` server 21 | 22 | before('deploy:cleanup') { set :use_sudo, false } 23 | before('deploy:setup') { set :use_sudo, false } 24 | after 'deploy:update_code', 'bundler:bundle_new_release' 25 | 26 | namespace :bundler do 27 | task :create_symlink, :roles => :app do 28 | shared_dir = File.join(shared_path, 'bundle') 29 | release_dir = File.join(current_release, '.bundle') 30 | run("mkdir -p #{shared_dir} && ln -s #{shared_dir} #{release_dir}") 31 | end 32 | 33 | task :bundle_new_release, :roles => :app do 34 | # bundler.create_symlink 35 | run "cd #{release_path} && bundle install --without test" 36 | end 37 | end 38 | 39 | namespace :deploy do 40 | task :start_dispatcher do 41 | run "cd #{deploy_to}/current/controllers && ruby dispatcher_controller.rb start" 42 | end 43 | 44 | task :start_workers do 45 | run "cd #{deploy_to}/current/controllers && ruby worker_controller.rb start" 46 | end 47 | 48 | task :start_server do 49 | run "cd #{deploy_to}/current/controllers && ruby skeeter_controller.rb start" 50 | end 51 | 52 | task :stop_dispatcher do 53 | run "cd #{deploy_to}/current/controllers && ruby dispatcher_controller.rb stop" 54 | end 55 | 56 | task :stop_workers do 57 | run "cd #{deploy_to}/current/controllers && ruby worker_controller.rb stop" 58 | end 59 | 60 | task :stop_server do 61 | run "cd #{deploy_to}/current/controllers && ruby skeeter_controller.rb stop" 62 | end 63 | 64 | task :start do 65 | start_dispatcher 66 | start_workers 67 | start_server 68 | end 69 | 70 | task :stop do 71 | stop_dispatcher 72 | stop_workers 73 | stop_server 74 | end 75 | 76 | task :restart do 77 | stop 78 | start 79 | end 80 | end 81 | -------------------------------------------------------------------------------- /controllers/dispatcher_controller.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'daemons' 4 | 5 | dispatcher = File.join(File.dirname(__FILE__), '..', 'bin', 'dispatcher.rb') 6 | 7 | Daemons.run(dispatcher) 8 | -------------------------------------------------------------------------------- /controllers/skeeter_controller.rb: -------------------------------------------------------------------------------- 1 | def start 2 | puts "Starting Skeeter..." 3 | `ruby service/skeeter.rb -d -e prod -v -P ../tmp/pids/skeeter.pid` 4 | end 5 | 6 | def stop 7 | puts "Killing Skeeter..." 8 | pid_path = File.join(File.dirname(__FILE__), '..', 'tmp', 'pids', 'skeeter.pid') 9 | pid = File.read(pid_path).to_i 10 | status = Process.kill("TERM", pid) 11 | puts "Killed pid #{pid}" if status == 1 12 | end 13 | 14 | if ARGV[0] == "start" 15 | start 16 | elsif ARGV[0] == "stop" 17 | stop 18 | elsif ARGV[0] == "restart" 19 | stop 20 | start 21 | else 22 | puts "Unknown command" 23 | end 24 | -------------------------------------------------------------------------------- /controllers/worker_controller.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'daemons' 4 | 5 | worker = File.join(File.dirname(__FILE__), '..', 'bin', 'worker.rb') 6 | 7 | Daemons.run(worker) 8 | -------------------------------------------------------------------------------- /images/moose-ascii.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blakesmith/skeeter/aa32043ccdfba4d3c06b1d85c9f5b70bbb4bfbef/images/moose-ascii.jpg -------------------------------------------------------------------------------- /images/moose.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blakesmith/skeeter/aa32043ccdfba4d3c06b1d85c9f5b70bbb4bfbef/images/moose.png -------------------------------------------------------------------------------- /service/config/.service.rb.swp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blakesmith/skeeter/aa32043ccdfba4d3c06b1d85c9f5b70bbb4bfbef/service/config/.service.rb.swp -------------------------------------------------------------------------------- /service/config/skeeter.rb: -------------------------------------------------------------------------------- 1 | require 'em-zeromq' 2 | require 'em-synchrony' 3 | 4 | context = EM::ZeroMQ::Context.new(1) 5 | config['context'] = context 6 | 7 | config['connection_pool'] = EM::Synchrony::ConnectionPool.new(:size => 20) do 8 | context.connect(ZMQ::REQ, "ipc:///tmp/dispatch-front.ipc") 9 | end 10 | -------------------------------------------------------------------------------- /service/skeeter.rb: -------------------------------------------------------------------------------- 1 | require 'bundler' 2 | Bundler.setup 3 | 4 | require 'goliath' 5 | require 'json' 6 | 7 | class EM::Protocols::ZMQConnectionHandler 8 | attr_reader :received 9 | 10 | def initialize(connection) 11 | @connection = connection 12 | @client_fiber = Fiber.current 13 | @connection.setsockopt(ZMQ::IDENTITY, "req-#{@client_fiber.object_id}") 14 | @connection.handler = self 15 | end 16 | 17 | def send_msg(*parts) 18 | queued = @connection.send_msg(*parts) 19 | @connection.register_readable 20 | messages = Fiber.yield 21 | messages.map(&:copy_out_string) 22 | end 23 | 24 | def on_readable(socket, messages) 25 | @client_fiber.resume(messages) 26 | end 27 | end 28 | 29 | class Skeeter < Goliath::API 30 | use Goliath::Rack::Params 31 | use Goliath::Rack::Validation::RequiredParam, {:key => 'image_url', :message => 'query param is required'} 32 | 33 | def response(env) 34 | json = {:message => 'convert', :width => params['width'], :url => params['image_url']}.to_json 35 | puts "Sending #{json}" 36 | 37 | connection_pool.execute(false) do |conn| 38 | handler = EM::Protocols::ZMQConnectionHandler.new(conn) 39 | resp = handler.send_msg(json).first 40 | [200, {}, resp] 41 | end 42 | end 43 | end 44 | 45 | -------------------------------------------------------------------------------- /test_server/public/images/InBed.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blakesmith/skeeter/aa32043ccdfba4d3c06b1d85c9f5b70bbb4bfbef/test_server/public/images/InBed.jpg -------------------------------------------------------------------------------- /test_server/public/images/programming-motherfuckers.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blakesmith/skeeter/aa32043ccdfba4d3c06b1d85c9f5b70bbb4bfbef/test_server/public/images/programming-motherfuckers.jpg -------------------------------------------------------------------------------- /test_server/test_server.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'sinatra' 3 | 4 | get "/" do 5 | "Test service" 6 | end 7 | 8 | --------------------------------------------------------------------------------