├── README.md ├── input.sh ├── dtr.rb └── thread_pool.rb /README.md: -------------------------------------------------------------------------------- 1 | dtr 2 | === 3 | 4 | From a long list of commandlines, run just some of them in parallel. -------------------------------------------------------------------------------- /input.sh: -------------------------------------------------------------------------------- 1 | sleep 20 2 | sleep 20 3 | sleep 20 4 | sleep 20 5 | sleep 20 6 | sleep 20 7 | sleep 30 8 | sleep 2 9 | sleep 2 10 | sleep 2 11 | sleep 2 -------------------------------------------------------------------------------- /dtr.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/ruby 2 | 3 | # depends on thread pool from http://burgestrand.se/code/ruby-thread-pool/ 4 | require_relative 'thread_pool' 5 | 6 | p = Pool.new(8) 7 | ctr = 0 8 | ARGF.each do |l| 9 | i = ARGF.lineno 10 | p.schedule do 11 | puts "Starting job #{i} to run command \"#{l.chomp}\" by thread #{Thread.current[:id]}" 12 | system(l) 13 | puts "Job #{i} finished by thread #{Thread.current[:id]}" 14 | end 15 | end 16 | 17 | at_exit { p.shutdown } 18 | -------------------------------------------------------------------------------- /thread_pool.rb: -------------------------------------------------------------------------------- 1 | # Ruby Thread Pool 2 | # ================ 3 | # A thread pool is useful when you wish to do some work in a thread, but do 4 | # not know how much work you will be doing in advance. Spawning one thread 5 | # for each task is potentially expensive, as threads are not free. 6 | # 7 | # In this case, it might be more beneficial to start a predefined set of 8 | # threads and then hand off work to them as it becomes available. This is 9 | # the pure essence of what a thread pool is: an array of threads, all just 10 | # waiting to do some work for you! 11 | # 12 | # Prerequisites 13 | # ------------- 14 | 15 | # We need the [Queue](http://rdoc.info/stdlib/thread/1.9.2/Queue), as our 16 | # thread pool is largely dependent on it. Thanks to this, the implementation 17 | # becomes very simple! 18 | require 'thread' 19 | 20 | # Public Interface 21 | # ---------------- 22 | 23 | # `Pool` is our thread pool class. It will allow us to do three operations: 24 | # 25 | # - `.new(size)` creates a thread pool of a given size 26 | # - `#schedule(*args, &job)` schedules a new job to be executed 27 | # - `#shutdown` shuts down all threads (after letting them finish working, of course) 28 | class Pool 29 | 30 | # ### initialization, or `Pool.new(size)` 31 | # Creating a new `Pool` involves a certain amount of work. First, however, 32 | # we need to define its’ `size`. It defines how many threads we will have 33 | # working internally. 34 | # 35 | # Which size is best for you is hard to answer. You do not want it to be 36 | # too low, as then you won’t be able to do as many things concurrently. 37 | # However, if you make it too high Ruby will spend too much time switching 38 | # between threads, and that will also degrade performance! 39 | def initialize(size) 40 | # Before we do anything else, we need to store some information about 41 | # our pool. `@size` is useful later, when we want to shut our pool down, 42 | # and `@jobs` is the heart of our pool that allows us to schedule work. 43 | @size = size 44 | @jobs = Queue.new 45 | 46 | # #### Creating our pool of threads 47 | # Once preparation is done, it’s time to create our pool of threads. 48 | # Each thread store its’ index in a thread-local variable, in case we 49 | # need to know which thread a job is executing in later on. 50 | @pool = Array.new(@size) do |i| 51 | Thread.new do 52 | Thread.current[:id] = i 53 | 54 | # We start off by defining a `catch` around our worker loop. This 55 | # way we’ve provided a method for graceful shutdown of our threads. 56 | # Shutting down is merely a `#schedule { throw :exit }` away! 57 | catch(:exit) do 58 | # The worker thread life-cycle is very simple. We continuously wait 59 | # for tasks to be put into our job `Queue`. If the `Queue` is empty, 60 | # we will wait until it’s not. 61 | loop do 62 | # Once we have a piece of work to be done, we will pull out the 63 | # information we need and get to work. 64 | job, args = @jobs.pop 65 | job.call(*args) 66 | end 67 | end 68 | end 69 | end 70 | end 71 | 72 | # ### Work scheduling 73 | 74 | # To schedule a piece of work to be done is to say to the `Pool` that you 75 | # want something done. 76 | def schedule(*args, &block) 77 | # Your given task will not be run immediately; rather, it will be put 78 | # into the work `Queue` and executed once a thread is ready to work. 79 | @jobs << [block, args] 80 | end 81 | 82 | # ### Graceful shutdown 83 | 84 | # If you ever wish to close down your application, I took the liberty of 85 | # making it easy for you to wait for any currently executing jobs to finish 86 | # before you exit. 87 | def shutdown 88 | # A graceful shutdown involves threads exiting cleanly themselves, and 89 | # since we’ve defined a `catch`-handler around the threads’ worker loop 90 | # it is simply a matter of throwing `:exit`. Thus, if we throw one `:exit` 91 | # for each thread in our pool, they will all exit eventually! 92 | @size.times do 93 | schedule { throw :exit } 94 | end 95 | 96 | # And now one final thing: wait for our `throw :exit` jobs to be run on 97 | # all our worker threads. This call will not return until all worker threads 98 | # have exited. 99 | @pool.map(&:join) 100 | end 101 | end 102 | 103 | # Demonstration 104 | # ------------- 105 | # Running this file will display how the thread pool works. 106 | if $0 == __FILE__ 107 | # - First, we create a new thread pool with a size of 10. This number is 108 | # lower than our planned amount of work, to show that threads do not 109 | # exit once they have finished a task. 110 | p = Pool.new(10) 111 | 112 | # - Next we simulate some workload by scheduling a large amount of work 113 | # to be done. The actual time taken for each job is randomized. This 114 | # is to demonstrate that even if two tasks are scheduled approximately 115 | # at the same time, the one that takes less time to execute is likely 116 | # to finish before the other one. 117 | 20.times do |i| 118 | p.schedule do 119 | sleep rand(4) + 2 120 | puts "Job #{i} finished by thread #{Thread.current[:id]}" 121 | end 122 | end 123 | 124 | # - Finally, register an `at_exit`-hook that will wait for our thread pool 125 | # to properly shut down before allowing our script to completely exit. 126 | at_exit { p.shutdown } 127 | end 128 | 129 | # License (X11 License) 130 | # ===================== 131 | # 132 | # Copyright (c) 2012, Kim Burgestrand 133 | # 134 | # Permission is hereby granted, free of charge, to any person obtaining a copy 135 | # of this software and associated documentation files (the "Software"), to deal 136 | # in the Software without restriction, including without limitation the rights 137 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 138 | # copies of the Software, and to permit persons to whom the Software is 139 | # furnished to do so, subject to the following conditions: 140 | # 141 | # The above copyright notice and this permission notice shall be included in 142 | # all copies or substantial portions of the Software. 143 | # 144 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 145 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 146 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 147 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 148 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 149 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 150 | # SOFTWARE. 151 | --------------------------------------------------------------------------------