├── .gitignore ├── .rspec ├── .ruby-gemset ├── .ruby-version ├── .travis.yml ├── .yardopts ├── Gemfile ├── LICENSE ├── README.md ├── Rakefile ├── lib ├── sidekiq_status.rb └── sidekiq_status │ ├── client_middleware.rb │ ├── container.rb │ ├── version.rb │ ├── web.rb │ └── worker.rb ├── log └── .gitkeep ├── sidekiq_status.gemspec ├── spec ├── client_middleware_spec.rb ├── container_spec.rb ├── dummy │ ├── app │ │ └── workers │ │ │ ├── test_worker1.rb │ │ │ └── test_worker2.rb │ └── boot.rb ├── integration │ └── sidekiq_spec.rb ├── spec_helper.rb └── worker_spec.rb └── web └── views ├── status.erb └── statuses.erb /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | *.rbc 3 | .bundle 4 | .config 5 | .yardoc 6 | Gemfile.lock 7 | InstalledFiles 8 | _yardoc 9 | coverage 10 | doc/ 11 | lib/bundler/man 12 | log/*.log 13 | pkg 14 | rdoc 15 | spec/reports 16 | tmp 17 | -------------------------------------------------------------------------------- /.rspec: -------------------------------------------------------------------------------- 1 | --color 2 | --format documentation 3 | -------------------------------------------------------------------------------- /.ruby-gemset: -------------------------------------------------------------------------------- 1 | sidekiq_status 2 | -------------------------------------------------------------------------------- /.ruby-version: -------------------------------------------------------------------------------- 1 | ruby-2.2.4 2 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: ruby 2 | rvm: 3 | #- rbx 4 | - 2.2.5 5 | - 2.3.1 6 | env: 7 | global: 8 | - SHOW_SIDEKIQ=true 9 | matrix: 10 | - SIDEKIQ_VERSION="~>5.1.1" 11 | - SIDEKIQ_VERSION="~>5.0.3" 12 | - SIDEKIQ_VERSION="~>4.2.0" 13 | - SIDEKIQ_VERSION="~>4.1.0" 14 | - SIDEKIQ_VERSION="~>4.0.2" 15 | - SIDEKIQ_VERSION="~>3.5.4" 16 | - SIDEKIQ_VERSION="~>3.4.2" 17 | 18 | before_install: 19 | - sudo apt-get -qq update 20 | - sudo apt-get install libgmp3-dev 21 | - gem update --system 22 | - gem update bundler 23 | - gem --version 24 | - bundle --version 25 | 26 | script: bundle exec rake 27 | services: 28 | - redis-server 29 | -------------------------------------------------------------------------------- /.yardopts: -------------------------------------------------------------------------------- 1 | -m markdown 2 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in sidekiq_status.gemspec 4 | gemspec 5 | 6 | gem 'sidekiq', ENV['SIDEKIQ_VERSION'] if ENV['SIDEKIQ_VERSION'] 7 | gem 'activesupport', '< 4.0.0' if RUBY_VERSION < '1.9.3' 8 | gem 'coveralls', require: false 9 | 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 Artem Ignatyev 2 | 3 | MIT License 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining 6 | a copy of this software and associated documentation files (the 7 | "Software"), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit persons to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be 14 | included in all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 19 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 20 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 22 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # SidekiqStatus 2 | 3 | [![Build Status](https://travis-ci.org/cryo28/sidekiq_status.png?branch=master)](https://travis-ci.org/cryo28/sidekiq_status) 4 | [![Dependency Status](https://gemnasium.com/cryo28/sidekiq_status.png)](https://gemnasium.com/cryo28/sidekiq_status) 5 | [![Test coverage](https://coveralls.io/repos/cryo28/sidekiq_status/badge.png?branch=master)](https://coveralls.io/r/cryo28/sidekiq_status) 6 | 7 | Sidekiq extension to track job execution statuses and returning job results back to the client in a convenient manner 8 | 9 | ## Installation 10 | 11 | Add this line to your application's Gemfile: 12 | 13 | ```ruby 14 | gem 'sidekiq_status' 15 | ``` 16 | 17 | And then execute: 18 | 19 | $ bundle 20 | 21 | ## Usage 22 | 23 | ### Basic 24 | 25 | Create a status-friendly worker by include SidekiqStatus::Worker module having #perform method with Sidekiq worker-compatible signature: 26 | 27 | ```ruby 28 | class MyWorker 29 | include SidekiqStatus::Worker 30 | 31 | def perform(arg1, arg2) 32 | # do something 33 | end 34 | end 35 | ``` 36 | 37 | Now you can enqueue some jobs for this worker 38 | 39 | ```ruby 40 | jid = MyWorker.perform_async('val_for_arg1', 'val_for_arg2') 41 | ``` 42 | 43 | If a job is rejected by some Client middleware, #perform_async returns false (as it does with ordinary Sidekiq worker). 44 | 45 | Now, you can easily track the status of the job execution: 46 | 47 | ```ruby 48 | status_container = SidekiqStatus::Container.load(jid) 49 | status_container.status # => 'waiting' 50 | ``` 51 | 52 | When a jobs is scheduled its status is *waiting*. As soon sidekiq worker begins job execution its status is changed to *working*. 53 | If the job successfully finishes (i.e. doesn't raise an unhandled exception) its status is *complete*. Otherwise its status is *failed*. 54 | 55 | ### Communication from Worker to Client 56 | 57 | *SidekiqStatus::Container* has some attributes and *SidekiqStatus::Worker* module extends your Worker class with a few methods which allow Worker to leave 58 | some info for the subsequent fetch by a Client. For example you can notify client of the worker progress via *at* and *total=* methods 59 | 60 | ```ruby 61 | class MyWorker 62 | include SidekiqStatus::Worker 63 | 64 | def perform(arg1, arg2) 65 | objects = Array.new(200) { 'some_object_to_process' } 66 | self.total = objects.count 67 | objects.each_with_index do |object, index| 68 | at(index, "Processing object #{object}") 69 | object.process! 70 | end 71 | end 72 | end 73 | ``` 74 | 75 | Lets presume a client refreshes container at the middle of job execution (when it's processing the object number 50): 76 | 77 | ```ruby 78 | container = SidekiqStatus::Container.load(jid) # or container.reload 79 | 80 | container.status # => 'working' 81 | container.at # => 50 82 | container.total # => 200 83 | container.pct_complete # => 25 84 | container.message # => 'Processing object #{50}' 85 | ``` 86 | 87 | Also, a job can leave for the client any custom payload. The only requirement is json-serializeability 88 | 89 | ```ruby 90 | class MyWorker 91 | include SidekiqStatus::Worker 92 | 93 | def perform(arg1, arg2) 94 | objects = Array.new(5) { |i| i } 95 | self.total = objects.count 96 | result = objects.inject([]) do |accum, object| 97 | accum << "result #{object}" 98 | accum 99 | end 100 | 101 | self.payload= result 102 | end 103 | end 104 | ``` 105 | 106 | 107 | Then a client can fetch the result payload 108 | 109 | ```ruby 110 | container = SidekiqStatus::Container.load(jid) 111 | container.status # => 'complete' 112 | container.payload # => ["result 0", "result 1", "result 2", "result 3", "result 4"] 113 | ``` 114 | 115 | SidekiqStatus stores all container attributes in a separate redis key until it's explicitly deleted via container.delete method 116 | or until redis key expires (see SidekiqStatus::Container.ttl class_attribute). 117 | 118 | ### Job kill 119 | 120 | Any job which is waiting or working can be killed. A working job is killed at the moment of container access. 121 | 122 | ```ruby 123 | container = SidekiqStatus::Container.load(jid) 124 | container.status # => 'working' 125 | container.killable? # => true 126 | container.should_kill # => false 127 | 128 | container.request_kill 129 | 130 | container.status # => 'working' 131 | container.killable? # => false 132 | container.should_kill # => true 133 | 134 | sleep(1) 135 | 136 | container.reload 137 | container.status # => 'killed' 138 | ``` 139 | 140 | ### Sidekiq web integration 141 | 142 | SidekiqStatus also provides an extension to Sidekiq web interface with /statuses page where you can track and kill jobs 143 | and clean status containers. 144 | 145 | 1. Setup Sidekiq web interface according to Sidekiq documentation 146 | 2. Add "require 'sidekiq_status/web'" beneath "require 'sidekiq/web'" 147 | 148 | ## Changelog 149 | 150 | ### 1.2.0 151 | 152 | * Support for sidekiq 4.2, integration with the new non-sinatra-based Sidekiq::Web (ncuesta) 153 | 154 | ### 1.1.0 155 | 156 | * Support for sidekiq 4.1, 4.0, 3.5, 3.4 157 | * No more replacement of original job arguments with generated unique jid. 158 | This is not needed anymore as Sidekiq started to do it. 159 | This change should make integration with other middlewares easier. 160 | * Dropped support for sidekiq versions older than 3.3 161 | * Dropped support for ruby 1.9.x, 2.0.x 162 | * Experimental support for Rubinius 163 | 164 | 165 | ### 1.0.7 166 | 167 | * Sidekiq 2.16 support 168 | 169 | ### 1.0.6 170 | 171 | * Sidekiq 2.15 support 172 | 173 | ### 1.0.5 174 | 175 | * Sidekiq 2.14 support 176 | * Do not create (and display in sidekiq_status/web) status containers 177 | for the jobs scheduled to run in the future 178 | by the means of perform_at/perform_in (mhfs) 179 | * Sidekiq web templates converted from .slim to .erb 180 | * Allow specifying worker name as a String (gumayunov) 181 | * Added ruby 2.0 to travis build matrix 182 | * Don't be too smart in extending Sinatra template search path (springbok) 183 | * Show worker names and adjust sidekiq-web template tags to conform 184 | to Sidekiq conventions (mhfs) 185 | 186 | ### 1.0.4 187 | 188 | * Sidekiq 2.10 and 2.11 support 189 | 190 | ### 1.0.3 191 | 192 | * Include SidekiqStatus::Web app into Sidekiq::Web app unobtrusively (pdf) 193 | * sidekiq 2.8.0 and 2.9.0 support 194 | 195 | ### 1.0.2 196 | 197 | * sidekiq 2.7.0 support 198 | * sidekiq integration tests 199 | * Display progress bar and last message in sidekiq-web tab (leandrocg) 200 | 201 | ### 1.0.1 202 | 203 | * sidekiq 2.6.x support 204 | 205 | ### 1.0.0 206 | 207 | * First release 208 | 209 | ## Roadmap 210 | 211 | * Add some sidekiq-web specs 212 | * Support running inline with sidekiq/testing 213 | 214 | ## Contributing 215 | 216 | 1. Fork it 217 | 2. Create your feature branch (`git checkout -b my-new-feature`) 218 | 3. Don't forget to write specs. Make sure rake spec passes 219 | 4. Commit your changes (`git commit -am 'Added some feature'`) 220 | 5. Push to the branch (`git push origin my-new-feature`) 221 | 6. Create new Pull Request 222 | 223 | ## Copyright 224 | 225 | SidekiqStatus © 2012-2013 by Artem Ignatyev. SidekiqStatus is licensed under the MIT license 226 | 227 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env rake 2 | # -*- encoding : utf-8 -*- 3 | 4 | require "bundler/gem_tasks" 5 | 6 | require 'rspec/core/rake_task' 7 | spec = RSpec::Core::RakeTask.new 8 | 9 | task :default => :spec -------------------------------------------------------------------------------- /lib/sidekiq_status.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | require 'sidekiq' 3 | 4 | require 'securerandom' 5 | require "sidekiq_status/version" 6 | require "sidekiq_status/client_middleware" 7 | require "sidekiq_status/container" 8 | require "sidekiq_status/worker" 9 | Sidekiq.client_middleware do |chain| 10 | chain.add SidekiqStatus::ClientMiddleware 11 | end 12 | 13 | require 'sidekiq_status/web' if defined?(Sidekiq::Web) -------------------------------------------------------------------------------- /lib/sidekiq_status/client_middleware.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | 3 | module SidekiqStatus 4 | class ClientMiddleware 5 | def call(worker, item, queue, redis_pool = nil) 6 | worker = worker.constantize if worker.is_a?(String) 7 | return yield unless worker < SidekiqStatus::Worker 8 | 9 | # Don't start reporting status if the job is scheduled for the future 10 | # When perform_at/perform_in is called this middleware is invoked within the client process 11 | # and job arguments have 'at' parameter. If all middlewares pass the job 12 | # Sidekiq::Client#raw_push puts the job into 'schedule' sorted set. 13 | # 14 | # Later, Sidekiq server ruby process periodically polls this sorted sets and repushes all 15 | # scheduled jobs which are due to run. This repush invokes all client middlewares, but within 16 | # sidekiq server ruby process. 17 | # 18 | # Luckily for us, when job is repushed, it doesn't have 'at' argument. 19 | # So we can distinguish the condition of the middleware invokation: we don't create SidekiqStatus::Container 20 | # when job is scheduled to run in the future, but we create status container when previously scheduled 21 | # job is due to run. 22 | return yield if item['at'] 23 | 24 | jid = item['jid'] 25 | args = item['args'] 26 | 27 | SidekiqStatus::Container.create( 28 | 'jid' => jid, 29 | 'worker' => worker.name, 30 | 'queue' => queue, 31 | 'args' => args 32 | ) 33 | 34 | yield 35 | rescue Exception => exc 36 | SidekiqStatus::Container.load(jid).delete rescue nil 37 | raise exc 38 | end 39 | end 40 | end 41 | -------------------------------------------------------------------------------- /lib/sidekiq_status/container.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | require 'active_support/core_ext/class' 3 | 4 | # Sidekiq extension to track job execution statuses and returning job results back to the client in a convenient manner 5 | module SidekiqStatus 6 | # SidekiqStatus job container. Contains all job attributes, redis storage/retrieval logic, 7 | # some syntactical sugar, such as status predicates and some attribute writers 8 | # Doesn't hook into Sidekiq worker 9 | class Container 10 | # Exception raised if SidekiqStatus job being loaded is not found in Redis 11 | class StatusNotFound < RuntimeError; 12 | end 13 | 14 | # Possible SidekiqStatus job statuses 15 | STATUS_NAMES = %w(waiting working complete failed killed).freeze 16 | 17 | # A list of statuses jobs in which are not considered pending 18 | FINISHED_STATUS_NAMES = %w(complete failed killed).freeze 19 | 20 | # Redis SortedSet key containing requests to kill {SidekiqStatus} jobs 21 | KILL_KEY = 'sidekiq_status_kill'.freeze 22 | 23 | # Redis SortedSet key to track existing {SidekiqStatus} jobs 24 | STATUSES_KEY = 'sidekiq_statuses'.freeze 25 | 26 | class_attribute :ttl 27 | self.ttl = 60*60*24*30 # 30 days 28 | 29 | # Default attribute values (assigned to a newly created container if not explicitly defined) 30 | DEFAULTS = { 31 | 'args' => [], 32 | 'worker' => 'SidekiqStatus::Worker', 33 | 'queue' => '', 34 | 'status' => 'waiting', 35 | 'at' => 0, 36 | 'total' => 100, 37 | 'message' => nil, 38 | 'payload' => {} 39 | }.freeze 40 | 41 | attr_reader :jid, :args, :worker, :queue 42 | attr_reader :status, :at, :total, :message, :last_updated_at 43 | attr_accessor :payload 44 | 45 | # @param [#to_s] jid SidekiqStatus job id 46 | # @return [String] redis key to store/fetch {SidekiqStatus::Container} for the given job 47 | def self.status_key(jid) 48 | "sidekiq_status:#{jid}" 49 | end 50 | 51 | # @return [String] Redis SortedSet key to track existing {SidekiqStatus} jobs 52 | def self.statuses_key 53 | STATUSES_KEY 54 | end 55 | 56 | # @return [String] Redis SortedSet key containing requests to kill {SidekiqStatus} jobs 57 | def self.kill_key 58 | KILL_KEY 59 | end 60 | 61 | # Delete all {SidekiqStatus} jobs which are in given status 62 | # 63 | # @param [String,Array,nil] status_names List of status names. If nil - delete jobs in any status 64 | def self.delete(status_names = nil) 65 | status_names ||= STATUS_NAMES 66 | status_names = [status_names] unless status_names.is_a?(Array) 67 | 68 | self.statuses.select { |container| status_names.include?(container.status) }.map(&:delete) 69 | end 70 | 71 | 72 | # Retrieve {SidekiqStatus} job identifiers 73 | # It's possible to perform some pagination by specifying range boundaries 74 | # 75 | # @param [Integer] start 76 | # @param [Integer] stop 77 | # @return [Array<[String,jid]>] Array of hash-like arrays of job id => last_updated_at (unixtime) pairs 78 | # @see *Redis#zrange* for details on return values format 79 | def self.status_jids(start = 0, stop = -1) 80 | Sidekiq.redis do |conn| 81 | conn.zrange(self.statuses_key, start, stop, :with_scores => true) 82 | end 83 | end 84 | 85 | # Retrieve {SidekiqStatus} jobs 86 | # It's possible to perform some pagination by specifying range boundaries 87 | # 88 | # @param [Integer] start 89 | # @param [Integer] stop 90 | # @return [Array] 91 | def self.statuses(start = 0, stop = -1) 92 | jids = status_jids(start, stop) 93 | jids = Hash[jids].keys 94 | load_multi(jids) 95 | end 96 | 97 | # @return [Integer] Known {SidekiqStatus} jobs amount 98 | def self.size 99 | Sidekiq.redis do |conn| 100 | conn.zcard(self.statuses_key) 101 | end 102 | end 103 | 104 | # Create (initialize, generate unique jid and save) a new {SidekiqStatus} job with given arguments. 105 | # 106 | # @overload 107 | # @param [String] jid job identifier 108 | # @overload 109 | # @param [Hash] data 110 | # @option data [String] jid (SecureRandom.hex(12)) optional job id to create status container for 111 | # @option data [Array] args job arguments 112 | # @option data [String] worker job worker class name 113 | # @option data [String] queue job queue 114 | # 115 | # @return [SidekiqStatus::Container] 116 | def self.create(data = {}) 117 | jid = data.delete('jid') if data.is_a?(Hash) 118 | jid ||= SecureRandom.hex(12) 119 | 120 | new(jid, data).tap(&:save) 121 | end 122 | 123 | # Load {SidekiqStatus::Container} by job identifier 124 | # 125 | # @param [String] jid job identifier 126 | # @raise [StatusNotFound] if there's no info about {SidekiqStatus} job with given *jid* 127 | # @return [SidekiqStatus::Container] 128 | def self.load(jid) 129 | data = load_data(jid) 130 | new(jid, data) 131 | end 132 | 133 | # Load a list of {SidekiqStatus::Container SidekiqStatus jobs} from Redis 134 | # 135 | # @param [Array] jids A list of job identifiers to load 136 | # @return [Array>] 137 | def self.load_multi(jids) 138 | data = load_data_multi(jids) 139 | data.map do |jid, data| 140 | new(jid, data) 141 | end 142 | end 143 | 144 | # Load {SidekiqStatus::Container SidekiqStatus job} {SidekiqStatus::Container#dump serialized data} from Redis 145 | # 146 | # @param [String] jid job identifier 147 | # @raise [StatusNotFound] if there's no info about {SidekiqStatus} job with given *jid* 148 | # @return [Hash] Job container data (as parsed json, but container is not yet initialized) 149 | def self.load_data(jid) 150 | load_data_multi([jid])[jid] or raise StatusNotFound.new(jid.to_s) 151 | end 152 | 153 | # Load multiple {SidekiqStatus::Container SidekiqStatus job} {SidekiqStatus::Container#dump serialized data} from Redis 154 | # 155 | # As this method is the most frequently used one, it also contains expire job clean up logic 156 | # 157 | # @param [Array<#to_s>] jids a list of job identifiers to load data for 158 | # @return [Hash{String => Hash}] A hash of job-id to deserialized data pairs 159 | def self.load_data_multi(jids) 160 | keys = jids.map(&method(:status_key)) 161 | 162 | return {} if keys.empty? 163 | 164 | threshold = Time.now - self.ttl 165 | 166 | data = Sidekiq.redis do |conn| 167 | conn.multi do 168 | conn.mget(*keys) 169 | 170 | conn.zremrangebyscore(kill_key, 0, threshold.to_i) # Clean up expired unprocessed kill requests 171 | conn.zremrangebyscore(statuses_key, 0, threshold.to_i) # Clean up expired statuses from statuses sorted set 172 | end 173 | end 174 | 175 | data = data.first.map do |json| 176 | json ? Sidekiq.load_json(json) : nil 177 | end 178 | 179 | Hash[jids.zip(data)] 180 | end 181 | 182 | # Initialize a new {SidekiqStatus::Container} with given unique job identifier and attribute data 183 | # 184 | # @param [String] jid 185 | # @param [Hash] data 186 | def initialize(jid, data = {}) 187 | @jid = jid 188 | load(data) 189 | end 190 | 191 | # Reload current container data from JSON (in case they've changed) 192 | def reload 193 | data = self.class.load_data(jid) 194 | load(data) 195 | self 196 | end 197 | 198 | # @return [String] redis key to store current {SidekiqStatus::Container container} 199 | # {SidekiqStatus::Container#dump data} 200 | def status_key 201 | self.class.status_key(jid) 202 | end 203 | 204 | # Save current container attribute values to redis 205 | def save 206 | data = dump 207 | data = Sidekiq.dump_json(data) 208 | 209 | Sidekiq.redis do |conn| 210 | conn.multi do 211 | conn.setex(status_key, self.ttl, data) 212 | conn.zadd(self.class.statuses_key, Time.now.to_f.to_s, self.jid) 213 | end 214 | end 215 | end 216 | 217 | # Delete current container data from redis 218 | def delete 219 | Sidekiq.redis do |conn| 220 | conn.multi do 221 | conn.del(status_key) 222 | 223 | conn.zrem(self.class.kill_key, self.jid) 224 | conn.zrem(self.class.statuses_key, self.jid) 225 | end 226 | end 227 | end 228 | 229 | # Request kill for the {SidekiqStatus::Worker SidekiqStatus job} 230 | # which parameters are tracked by the current {SidekiqStatus::Container} 231 | def request_kill 232 | Sidekiq.redis do |conn| 233 | conn.zadd(self.class.kill_key, Time.now.to_f.to_s, self.jid) 234 | end 235 | end 236 | 237 | # @return [Boolean] if job kill is requested 238 | def kill_requested? 239 | Sidekiq.redis do |conn| 240 | conn.zrank(self.class.kill_key, self.jid) 241 | end 242 | end 243 | 244 | # Reflect the fact that a job has been killed in redis 245 | def kill 246 | self.status = 'killed' 247 | 248 | Sidekiq.redis do |conn| 249 | conn.multi do 250 | save 251 | conn.zrem(self.class.kill_key, self.jid) 252 | end 253 | end 254 | end 255 | 256 | # @return [Boolean] can the current job be killed 257 | def killable? 258 | !kill_requested? && %w(waiting working).include?(self.status) 259 | end 260 | 261 | # @return [Integer] Job progress in percents (reported solely by {SidekiqStatus::Worker job}) 262 | def pct_complete 263 | (at.to_f / total * 100).round 264 | rescue FloatDomainError, ZeroDivisionError 265 | 0 266 | end 267 | 268 | # @param [Fixnum] at Report the progress of a job which is tracked by the current {SidekiqStatus::Container} 269 | def at=(at) 270 | raise ArgumentError, "at=#{at.inspect} is not a scalar number" unless at.is_a?(Numeric) 271 | @at = at 272 | @total = @at if @total < @at 273 | end 274 | 275 | # Report the estimated upper limit of {SidekiqStatus::Container#at= job items} 276 | # 277 | # @param [Fixnum] total 278 | def total=(total) 279 | raise ArgumentError, "total=#{total.inspect} is not a scalar number" unless total.is_a?(Numeric) 280 | @total = total 281 | end 282 | 283 | # Report current job execution status 284 | # 285 | # @param [String] status Current job {SidekiqStatus::STATUS_NAMES status} 286 | def status=(status) 287 | raise ArgumentError, "invalid status #{status.inspect}" unless STATUS_NAMES.include?(status) 288 | @status = status 289 | end 290 | 291 | # Report side message for the client code 292 | # 293 | # @param [String] message 294 | def message=(message) 295 | @message = message && message.to_s 296 | end 297 | 298 | # Assign multiple values to {SidekiqStatus::Container} attributes at once 299 | # 300 | # @param [Hash{#to_s => #to_json}] attrs Attribute=>value pairs 301 | def attributes=(attrs = {}) 302 | attrs.each do |attr_name, value| 303 | setter = "#{attr_name}=" 304 | send(setter, value) 305 | end 306 | end 307 | 308 | # Assign multiple values to {SidekiqStatus::Container} attributes at once and save to redis 309 | # @param [Hash{#to_s => #to_json}] attrs Attribute=>value pairs 310 | def update_attributes(attrs = {}) 311 | self.attributes = attrs 312 | save 313 | end 314 | 315 | STATUS_NAMES.each do |status_name| 316 | define_method("#{status_name}?") do 317 | status == status_name 318 | end 319 | end 320 | 321 | protected 322 | 323 | # Merge-in given data to the current container 324 | # 325 | # @private 326 | # @param [Hash] data 327 | def load(data) 328 | data = DEFAULTS.merge(data) 329 | 330 | @args, @worker, @queue = data.values_at('args', 'worker', 'queue') 331 | @status, @at, @total, @message = data.values_at('status', 'at', 'total', 'message') 332 | @payload = data['payload'] 333 | @last_updated_at = data['last_updated_at'] && Time.at(data['last_updated_at'].to_i) 334 | end 335 | 336 | # Dump current container attribute values to json-serializable hash 337 | # 338 | # @private 339 | # @return [Hash] Data for subsequent json-serialization 340 | def dump 341 | { 342 | 'args' => self.args, 343 | 'worker' => self.worker, 344 | 'queue' => self.queue, 345 | 346 | 'status' => self.status, 347 | 'at' => self.at, 348 | 'total' => self.total, 349 | 'message' => self.message, 350 | 351 | 'payload' => self.payload, 352 | 'last_updated_at' => Time.now.to_i 353 | } 354 | end 355 | end 356 | end 357 | 358 | -------------------------------------------------------------------------------- /lib/sidekiq_status/version.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | module SidekiqStatus 3 | # SidekiqStatus version 4 | VERSION = "1.2.0" 5 | end 6 | -------------------------------------------------------------------------------- /lib/sidekiq_status/web.rb: -------------------------------------------------------------------------------- 1 | module SidekiqStatus 2 | # Hook into *Sidekiq::Web* Sinatra app which adds a new "/statuses" page 3 | module Web 4 | # Location of SidekiqStatus::Web view templates 5 | VIEW_PATH = File.expand_path('../../../web/views', __FILE__) 6 | 7 | # @param [Sidekiq::Web] app 8 | def self.registered(app) 9 | app.helpers do 10 | def sidekiq_status_template(name) 11 | path = File.join(VIEW_PATH, name.to_s) + ".erb" 12 | File.open(path).read 13 | end 14 | 15 | def redirect_to(subpath) 16 | if respond_to?(:to) 17 | # Sinatra-based web UI 18 | redirect to(subpath) 19 | else 20 | # Non-Sinatra based web UI (Sidekiq 4.2+) 21 | redirect "#{root_path}#{subpath}" 22 | end 23 | end 24 | end 25 | 26 | app.get '/statuses' do 27 | @count = (params[:count] || 25).to_i 28 | 29 | @current_page = (params[:page] || 1).to_i 30 | @current_page = 1 unless @current_page > 0 31 | 32 | @total_size = SidekiqStatus::Container.size 33 | 34 | pageidx = @current_page - 1 35 | @statuses = SidekiqStatus::Container.statuses(pageidx * @count, (pageidx + 1) * @count) 36 | 37 | erb(sidekiq_status_template(:statuses)) 38 | end 39 | 40 | app.get '/statuses/:jid' do 41 | @status = SidekiqStatus::Container.load(params[:jid]) 42 | erb(sidekiq_status_template(:status)) 43 | end 44 | 45 | app.get '/statuses/:jid/kill' do 46 | SidekiqStatus::Container.load(params[:jid]).request_kill 47 | redirect_to :statuses 48 | end 49 | 50 | app.get '/statuses/delete/all' do 51 | SidekiqStatus::Container.delete 52 | redirect_to :statuses 53 | end 54 | 55 | app.get '/statuses/delete/complete' do 56 | SidekiqStatus::Container.delete('complete') 57 | redirect_to :statuses 58 | end 59 | 60 | app.get '/statuses/delete/finished' do 61 | SidekiqStatus::Container.delete(SidekiqStatus::Container::FINISHED_STATUS_NAMES) 62 | redirect_to :statuses 63 | end 64 | end 65 | end 66 | end 67 | 68 | require 'sidekiq/web' unless defined?(Sidekiq::Web) 69 | Sidekiq::Web.register(SidekiqStatus::Web) 70 | if Sidekiq::Web.tabs.is_a?(Array) 71 | # For sidekiq < 2.5 72 | Sidekiq::Web.tabs << "statuses" 73 | else 74 | Sidekiq::Web.tabs["Statuses"] = "statuses" 75 | end 76 | -------------------------------------------------------------------------------- /lib/sidekiq_status/worker.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | module SidekiqStatus 3 | module Worker 4 | def self.included(base) 5 | base.class_eval do 6 | include Sidekiq::Worker 7 | 8 | include(InstanceMethods) 9 | 10 | base.define_singleton_method(:new) do |*args, &block| 11 | super(*args, &block).extend(Prepending) 12 | end 13 | end 14 | end 15 | 16 | module Prepending 17 | def perform(*args) 18 | @status_container = SidekiqStatus::Container.load(jid) 19 | 20 | begin 21 | catch(:killed) do 22 | set_status('working') 23 | super(*args) 24 | set_status('complete') 25 | end 26 | rescue Exception => exc 27 | set_status('failed', exc.class.name + ': ' + exc.message + " \n\n " + exc.backtrace.join("\n ")) 28 | raise exc 29 | end 30 | end 31 | end 32 | 33 | module InstanceMethods 34 | def status_container 35 | kill if @status_container.kill_requested? 36 | @status_container 37 | end 38 | alias_method :sc, :status_container 39 | 40 | def kill 41 | # NOTE: status_container below should be accessed by instance var instead of an accessor method 42 | # because the second option will lead to infinite recursing 43 | @status_container.kill 44 | throw(:killed) 45 | end 46 | 47 | def set_status(status, message = nil) 48 | self.sc.update_attributes('status' => status, 'message' => message) 49 | end 50 | 51 | def at(at, message = nil) 52 | self.sc.update_attributes('at' => at, 'message' => message) 53 | end 54 | 55 | def total=(total) 56 | self.sc.update_attributes('total' => total) 57 | end 58 | 59 | def payload=(payload) 60 | self.sc.update_attributes('payload' => payload) 61 | end 62 | end 63 | end 64 | end 65 | -------------------------------------------------------------------------------- /log/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cryo28/sidekiq_status/afa8bcb85c3c68ad9ffdf445bb1d4da34df83f27/log/.gitkeep -------------------------------------------------------------------------------- /sidekiq_status.gemspec: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | require File.expand_path('../lib/sidekiq_status/version', __FILE__) 3 | 4 | Gem::Specification.new do |gem| 5 | gem.authors = ["Artem Ignatyev"] 6 | gem.email = ["cryo28@gmail.com"] 7 | gem.description = "Job status tracking extension for Sidekiq" 8 | gem.summary = "A Sidekiq extension to track job execution statuses and return job results back to the client in a convenient manner" 9 | gem.homepage = "https://github.com/cryo28/sidekiq_status" 10 | gem.licenses = ["MIT"] 11 | 12 | gem.files = `git ls-files`.split($\) 13 | gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) } 14 | gem.test_files = gem.files.grep(%r{^(test|spec|features)/}) 15 | gem.name = "sidekiq_status" 16 | gem.require_paths = ["lib"] 17 | gem.version = SidekiqStatus::VERSION 18 | 19 | gem.add_runtime_dependency("activesupport") 20 | gem.add_runtime_dependency("sidekiq", ">= 3.3", "< 6") 21 | 22 | gem.add_development_dependency("rspec", '>= 3.4.0') 23 | gem.add_development_dependency("rspec-its") 24 | gem.add_development_dependency("simplecov") 25 | gem.add_development_dependency("rake") 26 | gem.add_development_dependency("timecop") 27 | 28 | gem.add_development_dependency("yard") 29 | gem.add_development_dependency("maruku") 30 | end 31 | -------------------------------------------------------------------------------- /spec/client_middleware_spec.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | require 'spec_helper' 3 | 4 | describe SidekiqStatus::ClientMiddleware do 5 | describe "cryo28/sidekiq_status#11 regression" do 6 | describe "#call" do 7 | before do 8 | expect(SidekiqStatus::Container).to receive(:create).with(hash_including('worker' => 'TestWorker1')) 9 | end 10 | 11 | it "accepts a worker class" do 12 | subject.call(TestWorker1, {}, nil) do 13 | end 14 | end 15 | 16 | it "accepts a worker name string" do 17 | subject.call("TestWorker1", {}, nil) do 18 | end 19 | end 20 | end 21 | end 22 | 23 | it "does not create container for scheduled job" do 24 | expect(SidekiqStatus::Container).to_not receive(:create) 25 | 26 | subject.call("TestWorker1", { "at" => Time.now }, nil) do 27 | end 28 | end 29 | end 30 | -------------------------------------------------------------------------------- /spec/container_spec.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | require 'spec_helper' 3 | 4 | def test_container(container, hash, jid = nil) 5 | hash.reject { |k, v| k == :last_updated_at }.find do |k, v| 6 | container.send(k).should == v 7 | end 8 | 9 | container.last_updated_at.should == Time.at(hash['last_updated_at']) if hash['last_updated_at'] 10 | container.jid.should == jid if jid 11 | end 12 | 13 | 14 | describe SidekiqStatus::Container do 15 | let(:jid) { "c2db8b1b460608fb32d76b7a" } 16 | let(:status_key) { described_class.status_key(jid) } 17 | let(:sample_json_hash) do 18 | { 19 | 'args' => ['arg1', 'arg2'], 20 | 'worker' => 'SidekiqStatus::Worker', 21 | 'queue' => '', 22 | 23 | 'status' => "completed", 24 | 'at' => 50, 25 | 'total' => 200, 26 | 'message' => "Some message", 27 | 28 | 'payload' => {}, 29 | 'last_updated_at' => 1344855831 30 | } 31 | end 32 | 33 | specify ".status_key" do 34 | jid = SecureRandom.base64 35 | described_class.status_key(jid).should == "sidekiq_status:#{jid}" 36 | end 37 | 38 | specify ".kill_key" do 39 | described_class.kill_key.should == described_class::KILL_KEY 40 | end 41 | 42 | 43 | context "finders" do 44 | let!(:containers) do 45 | described_class::STATUS_NAMES.inject({}) do |accum, status_name| 46 | container = described_class.create() 47 | container.update_attributes(:status => status_name) 48 | 49 | accum[status_name] = container 50 | accum 51 | end 52 | end 53 | 54 | specify ".size" do 55 | described_class.size.should == containers.size 56 | end 57 | 58 | specify ".status_jids" do 59 | expected = containers.values.map(&:jid).map{ |jid| [jid, anything()] } 60 | described_class.status_jids.should =~ expected 61 | described_class.status_jids(0, 0).size.should == 1 62 | end 63 | 64 | specify ".statuses" do 65 | described_class.statuses.should be_all{|st| st.is_a?(described_class) } 66 | described_class.statuses.size.should == containers.size 67 | described_class.statuses(0, 0).size.should == 1 68 | end 69 | 70 | describe ".delete" do 71 | before do 72 | described_class.status_jids.map(&:first).should =~ containers.values.map(&:jid) 73 | end 74 | 75 | specify "deletes jobs in specific status" do 76 | statuses_to_delete = ['waiting', 'complete'] 77 | described_class.delete(statuses_to_delete) 78 | 79 | described_class.status_jids.map(&:first).should =~ containers. 80 | reject{ |status_name, container| statuses_to_delete.include?(status_name) }. 81 | values. 82 | map(&:jid) 83 | end 84 | 85 | specify "deletes jobs in all statuses" do 86 | described_class.delete() 87 | 88 | described_class.status_jids.should be_empty 89 | end 90 | end 91 | end 92 | 93 | specify ".create" do 94 | expect(SecureRandom).to receive(:hex).with(12).and_return(jid) 95 | args = ['arg1', 'arg2', {arg3: 'val3'}] 96 | 97 | container = described_class.create('args' => args) 98 | container.should be_a(described_class) 99 | container.args.should == args 100 | 101 | # Check default values are set 102 | test_container(container, described_class::DEFAULTS.reject{|k, v| k == 'args' }, jid) 103 | 104 | Sidekiq.redis do |conn| 105 | conn.exists(status_key).should be true 106 | end 107 | end 108 | 109 | describe ".load" do 110 | it "raises StatusNotFound exception if status is missing in Redis" do 111 | expect { described_class.load(jid) }.to raise_exception(described_class::StatusNotFound, jid) 112 | end 113 | 114 | it "loads a container from the redis key" do 115 | json = Sidekiq.dump_json(sample_json_hash) 116 | Sidekiq.redis { |conn| conn.set(status_key, json) } 117 | 118 | container = described_class.load(jid) 119 | test_container(container, sample_json_hash, jid) 120 | end 121 | 122 | it "cleans up unprocessed expired kill requests as well" do 123 | Sidekiq.redis do |conn| 124 | conn.zadd(described_class.kill_key, [ 125 | [(Time.now - described_class.ttl - 1).to_i, 'a'], 126 | [(Time.now - described_class.ttl + 1).to_i, 'b'], 127 | ] 128 | ) 129 | end 130 | 131 | json = Sidekiq.dump_json(sample_json_hash) 132 | Sidekiq.redis { |conn| conn.set(status_key, json) } 133 | described_class.load(jid) 134 | 135 | Sidekiq.redis do |conn| 136 | conn.zscore(described_class.kill_key, 'a').should be_nil 137 | conn.zscore(described_class.kill_key, 'b').should_not be_nil 138 | end 139 | end 140 | end 141 | 142 | specify "#dump" do 143 | hash = sample_json_hash.reject{ |k, v| k == 'last_updated_at' } 144 | container = described_class.new(jid, hash) 145 | dump = container.send(:dump) 146 | dump.should == hash.merge('last_updated_at' => Time.now.to_i) 147 | end 148 | 149 | specify "#save saves container to Redis" do 150 | hash = sample_json_hash.reject{ |k, v| k == 'last_updated_at' } 151 | described_class.new(jid, hash).save 152 | 153 | result = Sidekiq.redis{ |conn| conn.get(status_key) } 154 | result = Sidekiq.load_json(result) 155 | 156 | result.should == hash.merge('last_updated_at' => Time.now.to_i) 157 | 158 | Sidekiq.redis{ |conn| conn.ttl(status_key).should >= 0 } 159 | end 160 | 161 | specify "#delete" do 162 | Sidekiq.redis do |conn| 163 | conn.set(status_key, "something") 164 | conn.zadd(described_class.kill_key, 0, jid) 165 | end 166 | 167 | container = described_class.new(jid) 168 | container.delete 169 | 170 | Sidekiq.redis do |conn| 171 | conn.exists(status_key).should be false 172 | conn.zscore(described_class.kill_key, jid).should be_nil 173 | end 174 | end 175 | 176 | specify "#request_kill, #should_kill?, #killable?" do 177 | container = described_class.new(jid) 178 | container.kill_requested?.should be_falsey 179 | container.should be_killable 180 | 181 | Sidekiq.redis do |conn| 182 | conn.zscore(described_class.kill_key, jid).should be_nil 183 | end 184 | 185 | 186 | container.request_kill 187 | 188 | Sidekiq.redis do |conn| 189 | conn.zscore(described_class.kill_key, jid).should == Time.now.to_i 190 | end 191 | container.should be_kill_requested 192 | container.should_not be_killable 193 | end 194 | 195 | specify "#kill" do 196 | container = described_class.new(jid) 197 | container.request_kill 198 | Sidekiq.redis do |conn| 199 | conn.zscore(described_class.kill_key, jid).should == Time.now.to_i 200 | end 201 | container.status.should_not == 'killed' 202 | 203 | 204 | container.kill 205 | 206 | Sidekiq.redis do |conn| 207 | conn.zscore(described_class.kill_key, jid).should be_nil 208 | end 209 | 210 | container.status.should == 'killed' 211 | described_class.load(jid).status.should == 'killed' 212 | end 213 | 214 | specify "#pct_complete" do 215 | container = described_class.new(jid) 216 | container.at = 1 217 | container.total = 100 218 | container.pct_complete.should == 1 219 | 220 | container.at = 5 221 | container.total = 200 222 | container.pct_complete.should == 3 # 2.5.round(0) => 3 223 | 224 | container.at = Float::INFINITY 225 | container.pct_complete.should == 0 # FloatDomainError 226 | 227 | container.at = 5 228 | container.total = 0 229 | container.pct_complete.should == 0 # ZeroDivisionError 230 | end 231 | 232 | 233 | 234 | context "setters" do 235 | let(:container) { described_class.new(jid) } 236 | 237 | describe "#at=" do 238 | it "sets numeric value" do 239 | container.total = 100 240 | container.at = 3 241 | container.at.should == 3 242 | container.total.should == 100 243 | end 244 | 245 | it "raises ArgumentError otherwise" do 246 | expect{ container.at = "Wrong" }.to raise_exception(ArgumentError) 247 | end 248 | 249 | it "adjusts total if its less than new at" do 250 | container.total = 200 251 | container.at = 250 252 | container.total.should == 250 253 | end 254 | end 255 | 256 | describe "#total=" do 257 | it "sets numeric value" do 258 | container.total = 50 259 | container.total.should == 50 260 | end 261 | 262 | it "raises ArgumentError otherwise" do 263 | expect{ container.total = "Wrong" }.to raise_exception(ArgumentError) 264 | end 265 | end 266 | 267 | describe "#status=" do 268 | described_class::STATUS_NAMES.each do |status| 269 | it "sets status #{status.inspect}" do 270 | container.status = status 271 | container.status.should == status 272 | end 273 | end 274 | 275 | it "raises ArgumentError otherwise" do 276 | expect{ container.status = 'Wrong' }.to raise_exception(ArgumentError) 277 | end 278 | end 279 | 280 | specify "#message=" do 281 | container.message = 'abcd' 282 | container.message.should == 'abcd' 283 | 284 | container.message = nil 285 | container.message.should be_nil 286 | 287 | message = double('Message', :to_s => 'to_s') 288 | container.message = message 289 | container.message.should == 'to_s' 290 | end 291 | 292 | specify "#payload=" do 293 | container.should respond_to(:payload=) 294 | end 295 | 296 | specify "update_attributes" do 297 | container.update_attributes(:at => 1, 'total' => 3, :message => 'msg', 'status' => 'working') 298 | reloaded_container = described_class.load(container.jid) 299 | 300 | reloaded_container.at.should == 1 301 | reloaded_container.total.should == 3 302 | reloaded_container.message.should == 'msg' 303 | reloaded_container.status.should == 'working' 304 | 305 | expect{ container.update_attributes(:at => 'Invalid') }.to raise_exception(ArgumentError) 306 | end 307 | end 308 | 309 | context "predicates" do 310 | described_class::STATUS_NAMES.each do |status_name1| 311 | context "status is #{status_name1}" do 312 | subject{ described_class.create().tap{|c| c.status = status_name1} } 313 | 314 | its("#{status_name1}?") { should be true } 315 | 316 | (described_class::STATUS_NAMES - [status_name1]).each do |status_name2| 317 | its("#{status_name2}?") { should be false } 318 | end 319 | end 320 | end 321 | end 322 | end 323 | -------------------------------------------------------------------------------- /spec/dummy/app/workers/test_worker1.rb: -------------------------------------------------------------------------------- 1 | class TestWorker1 2 | include SidekiqStatus::Worker 3 | 4 | def perform(arg1) 5 | self.payload = arg1 6 | self.total = 200 7 | end 8 | end -------------------------------------------------------------------------------- /spec/dummy/app/workers/test_worker2.rb: -------------------------------------------------------------------------------- 1 | class TestWorker2 2 | include SidekiqStatus::Worker 3 | 4 | def perform(redis_key) 5 | signal = nil 6 | while signal != 'stop' 7 | signal = Sidekiq.redis{ |conn| conn.get(redis_key) } 8 | i = signal.to_i 9 | self.at(i, "Some message at #{i}") 10 | sleep(0.1) 11 | end 12 | end 13 | end -------------------------------------------------------------------------------- /spec/dummy/boot.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'bundler' 3 | Bundler.require 4 | 5 | require 'active_support' 6 | require 'active_support/dependencies' 7 | 8 | DUMMY_APP_ROOT = Pathname.new(File.expand_path('../', __FILE__)) 9 | Sidekiq.redis = {:url => "redis://localhost/15", :size => 5} 10 | 11 | ActiveSupport::Dependencies.autoload_paths += Dir.glob(DUMMY_APP_ROOT.join('app/*')) 12 | -------------------------------------------------------------------------------- /spec/integration/sidekiq_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe SidekiqStatus::Worker do 4 | def run_sidekiq(show_sidekiq_output = ENV['SHOW_SIDEKIQ']) 5 | log_to = show_sidekiq_output ? STDOUT : GEM_ROOT.join('log/spawned_sidekiq.log').to_s 6 | command = 'bundle exec sidekiq -r ./boot.rb --concurrency 1' 7 | 8 | Process.spawn( 9 | command, 10 | :chdir => DUMMY_APP_ROOT, 11 | :err => :out, 12 | :out => log_to, 13 | :pgroup => true 14 | ) 15 | end 16 | 17 | def with_sidekiq_running 18 | pid = run_sidekiq 19 | 20 | begin 21 | yield(pid) 22 | ensure 23 | Process.kill('TERM', -Process.getpgid(pid)) 24 | Process.wait(pid) 25 | end 26 | end 27 | 28 | context "integrates seamlessly with sidekiq and" do 29 | it "allows to query for complete job status and request payload" do 30 | some_value = 'some_value' 31 | jid = TestWorker1.perform_async(some_value) 32 | container = SidekiqStatus::Container.load(jid) 33 | container.should be_waiting 34 | 35 | with_sidekiq_running do 36 | wait{ container.reload.complete? } 37 | 38 | container.total.should == 200 39 | container.payload.should == some_value 40 | end 41 | end 42 | 43 | it "allows to query for working job status and request payload" do 44 | redis_key = 'SomeRedisKey' 45 | 46 | jid = TestWorker2.perform_async(redis_key) 47 | container = SidekiqStatus::Container.load(jid) 48 | container.should be_waiting 49 | 50 | with_sidekiq_running do 51 | wait{ container.reload.working? } 52 | 53 | Sidekiq.redis{ |conn| conn.set(redis_key, 10) } 54 | wait{ container.reload.at == 10 } 55 | container.message.should == 'Some message at 10' 56 | 57 | Sidekiq.redis{ |conn| conn.set(redis_key, 50) } 58 | wait{ container.reload.at == 50 } 59 | container.message.should == 'Some message at 50' 60 | 61 | Sidekiq.redis{ |conn| conn.set(redis_key, 'stop') } 62 | wait{ container.reload.complete? } 63 | container.should be_complete 64 | container.message.should be_nil 65 | end 66 | end 67 | end 68 | end 69 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # -*- encoding : utf-8 -*- 2 | require 'bundler' 3 | Bundler.setup 4 | ENV['RACK_ENV'] = ENV['RAILS_ENV'] = 'test' 5 | GEM_ROOT = Pathname.new(File.expand_path('../..', __FILE__)) 6 | 7 | 8 | require 'simplecov' 9 | SimpleCov.start do 10 | root GEM_ROOT 11 | end 12 | 13 | require 'coveralls' 14 | Coveralls.wear! 15 | 16 | require 'rspec/its' 17 | require 'sidekiq_status' 18 | require 'sidekiq/util' 19 | 20 | require 'timecop' 21 | 22 | Sidekiq.logger.level = Logger::ERROR 23 | 24 | 25 | require GEM_ROOT.join('spec/dummy/boot.rb') 26 | 27 | RSpec.configure do |c| 28 | c.expect_with :rspec do |expectations| 29 | expectations.syntax = [:should, :expect] 30 | end 31 | 32 | c.before do 33 | Sidekiq.redis{ |conn| conn.flushdb } 34 | end 35 | 36 | c.around do |example| 37 | Timecop.freeze(Time.utc(2012)) do 38 | example.call 39 | end 40 | end 41 | 42 | def wait(&block) 43 | Timeout.timeout(15) do 44 | sleep(0.5) while !block.call 45 | end 46 | end 47 | end 48 | -------------------------------------------------------------------------------- /spec/worker_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe Sidekiq::Worker do 4 | class SomeWorker 5 | include SidekiqStatus::Worker 6 | 7 | def perform(*args) 8 | some_method(*args) 9 | end 10 | 11 | def some_method(*args); end 12 | end 13 | 14 | let(:args) { ['arg1', 'arg2', {'arg3' => 'val3'}]} 15 | 16 | describe ".perform_async" do 17 | it "invokes middleware which creates sidekiq_status container with the same jid" do 18 | jid = SomeWorker.perform_async(*args) 19 | jid.should be_a(String) 20 | 21 | container = SidekiqStatus::Container.load(jid) 22 | container.args.should == args 23 | end 24 | end 25 | 26 | describe "#perform (Worker context)" do 27 | let(:worker) { SomeWorker.new } 28 | 29 | it "loads container using @jid and runs original perform" do 30 | expect(worker).to receive(:some_method).with(*args) 31 | jid = SomeWorker.perform_async(*args) 32 | worker.jid = jid.freeze 33 | worker.perform(*args) 34 | end 35 | 36 | it "changes status to working" do 37 | has_been_run = false 38 | worker.extend(Module.new do 39 | define_method(:some_method) do |*args| 40 | status_container.status.should == 'working' 41 | has_been_run = true 42 | end 43 | end) 44 | 45 | jid = SomeWorker.perform_async(*args) 46 | worker.jid = jid.freeze 47 | worker.perform(*args) 48 | 49 | has_been_run.should be true 50 | worker.status_container.reload.status.should == 'complete' 51 | end 52 | 53 | it "intercepts failures and set status to 'failed' then re-raises the exception" do 54 | exc = RuntimeError.new('Some error') 55 | allow(worker).to receive(:some_method).and_raise(exc) 56 | 57 | jid = SomeWorker.perform_async(*args) 58 | worker.jid = jid.freeze 59 | expect{ worker.perform(*args) }.to raise_exception{ |error| error.object_id.should == exc.object_id } 60 | 61 | container = SidekiqStatus::Container.load(jid) 62 | container.status.should == 'failed' 63 | end 64 | 65 | it "sets status to 'complete' if finishes without errors" do 66 | jid = SomeWorker.perform_async(*args) 67 | worker.jid = jid.freeze 68 | worker.perform(*args) 69 | 70 | container = SidekiqStatus::Container.load(jid) 71 | container.status.should == 'complete' 72 | end 73 | 74 | it "handles kill requests if kill requested before job execution" do 75 | jid = SomeWorker.perform_async(*args) 76 | worker.jid = jid.freeze 77 | 78 | container = SidekiqStatus::Container.load(jid) 79 | container.request_kill 80 | 81 | worker.perform(*args) 82 | 83 | container.reload 84 | container.status.should == 'killed' 85 | end 86 | 87 | it "handles kill requests if kill requested amid job execution" do 88 | jid = SomeWorker.perform_async(*args) 89 | worker.jid = jid.freeze 90 | 91 | container = SidekiqStatus::Container.load(jid) 92 | container.status.should == 'waiting' 93 | 94 | i = 0 95 | i_mut = Mutex.new 96 | 97 | worker.extend(Module.new do 98 | define_method(:some_method) do |*args| 99 | loop do 100 | i_mut.synchronize do 101 | i += 1 102 | end 103 | 104 | status_container.at = i 105 | end 106 | end 107 | end) 108 | 109 | worker_thread = Thread.new{ worker.perform(jid) } 110 | 111 | 112 | killer_thread = Thread.new do 113 | sleep(0.01) while i < 100 114 | container.reload.status.should == 'working' 115 | container.request_kill 116 | end 117 | 118 | worker_thread.join(2) 119 | killer_thread.join(1) 120 | 121 | container.reload 122 | container.status.should == 'killed' 123 | container.at.should >= 100 124 | end 125 | 126 | it "allows to set at, total and customer payload from the worker" do 127 | jid = SomeWorker.perform_async(*args) 128 | worker.jid = jid.freeze 129 | 130 | container = SidekiqStatus::Container.load(jid) 131 | 132 | lets_stop = false 133 | 134 | worker.extend(Module.new do 135 | define_method(:some_method) do |*args| 136 | self.total=(200) 137 | self.at(50, "25% done") 138 | self.payload = 'some payload' 139 | wait{ lets_stop } 140 | end 141 | end) 142 | 143 | worker_thread = Thread.new{ worker.perform(jid) } 144 | checker_thread = Thread.new do 145 | wait{ container.reload.working? && container.at == 50 } 146 | 147 | container.at.should == 50 148 | container.total.should == 200 149 | container.message.should == '25% done' 150 | container.payload == 'some payload' 151 | 152 | lets_stop = true 153 | end 154 | 155 | worker_thread.join(15) 156 | checker_thread.join(15) 157 | 158 | wait{ container.reload.complete? } 159 | 160 | container.payload.should == 'some payload' 161 | container.message.should be_nil 162 | end 163 | end 164 | end 165 | -------------------------------------------------------------------------------- /web/views/status.erb: -------------------------------------------------------------------------------- 1 |

2 | Job <%= @status.jid %> is <%= @status.status %> (<%= @status.pct_complete %>% done) 3 |

4 |

5 | 6 |
7 |

Details

8 |
9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 52 | 53 | 54 | 55 | 60 | 61 | 62 |
AttributeValue
jid<%= @status.jid %>
worker<%= @status.worker %>
status<%= @status.status %>
last updated at<%= @status.last_updated_at %>
at<%= @status.at %>
total<%= @status.total %>
message<%= @status.message %>
payload 48 | 49 | <%= @status.payload.to_json %> 50 | 51 |
job args 56 | 57 | <%= @status.args.to_json %> 58 | 59 |
63 | Back 64 |
65 |
66 | -------------------------------------------------------------------------------- /web/views/statuses.erb: -------------------------------------------------------------------------------- 1 |

Recent job statuses

2 | 3 |
4 | Delete jobs in  5 | 6 | complete 7 | 8 | ,  9 | 10 | finished 11 | 12 | ,  13 | 14 | all 15 | 16 |  statuses 17 |
18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | <% @statuses.each do |container| %> 29 | 30 | 37 | 38 | 39 | 46 | 53 | 54 | <% end %> 55 | <% if @statuses.empty? %> 56 | 57 | 58 | 59 | <% end %> 60 |
Worker/jidStatusLast Updated ↆProgressMessageActions
31 | 32 | <%= container.worker %> 33 |
34 | <%= container.jid %> 35 | 36 | 		<%= container.status %><%= container.last_updated_at %> 40 |
41 |
42 | <%= container.pct_complete %>% 43 |
44 |
45 | 		<%= container.message %> 47 | <% if container.killable? %> 48 | Kill 49 | <% elsif container.kill_requested? %> 50 | Kill requested 51 | <% end %> 52 |
61 | 62 | 63 | <%= erb :_paging, :locals => { :url => "#{root_path}statuses" } %> 64 | --------------------------------------------------------------------------------