├── .gitignore ├── .gitmodules ├── .python-version ├── .travis.yml ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.md ├── Vagrantfile ├── bin └── qless-py-worker ├── forgetful-bench.py ├── provision.sh ├── qless ├── __init__.py ├── config.py ├── exceptions.py ├── job.py ├── listener.py ├── profile.py ├── queue.py ├── util.py └── workers │ ├── __init__.py │ ├── forking.py │ ├── greenlet.py │ └── serial.py ├── requirements.txt ├── setup.cfg ├── setup.py ├── test ├── common.py ├── test_client.py ├── test_config.py ├── test_events.py ├── test_forking.py ├── test_greenlet.py ├── test_job.py ├── test_queue.py ├── test_serial.py └── test_worker.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *~ 3 | .coverage* 4 | build/* 5 | dist/* 6 | htmlcov/* 7 | qless_py.egg-info/* 8 | .vagrant 9 | MANIFEST 10 | .tox/ 11 | venv/ 12 | *.sublime-project 13 | *.sublime-workspace 14 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "qless/qless-core"] 2 | path = qless/qless-core 3 | url = git://github.com/seomoz/qless-core.git 4 | -------------------------------------------------------------------------------- /.python-version: -------------------------------------------------------------------------------- 1 | 2.7.15 2 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: false 2 | services: 3 | - redis-server 4 | before_install: 5 | - make qless-core 6 | language: python 7 | python: 8 | - "2.7" 9 | - "3.3" 10 | - "3.4" 11 | - "3.5" 12 | install: pip install tox tox-travis 13 | script: tox 14 | addons: 15 | apt: 16 | packages: 17 | - libevent-dev 18 | cache: 19 | directories: 20 | - .tox 21 | - $HOME/.cache/pip 22 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 SEOmoz 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include qless/qless-core/*.lua 2 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: test 2 | 3 | .PHONY: clean 4 | clean: 5 | # Remove the build 6 | rm -rf build dist 7 | # And all of our pyc files 8 | find . -name '*.pyc' -delete 9 | # And lastly, .coverage files 10 | find . -name .coverage -delete 11 | 12 | .PHONY: qless-core 13 | qless-core: 14 | # Ensure qless is built 15 | make -C qless/qless-core/ 16 | 17 | .PHONY: nose 18 | nose: qless-core 19 | nosetests --with-coverage 20 | 21 | requirements: 22 | pip freeze | grep -v -e qless-py > requirements.txt 23 | 24 | .PHONY: test 25 | test: nose 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | qless [![Build Status](https://travis-ci.org/seomoz/qless-py.svg?branch=master)](https://travis-ci.org/seomoz/qless-py) 2 | ===== 3 | Qless is a powerful `Redis`-based job queueing system inspired by 4 | [resque](https://github.com/defunkt/resque#readme), 5 | but built on a collection of Lua scripts, maintained in the 6 | [qless-core](https://github.com/seomoz/qless-core) repo. Be sure to check the 7 | changelog below. 8 | 9 | Philosophy and Nomenclature 10 | =========================== 11 | A `job` is a unit of work identified by a job id or `jid`. A `queue` can 12 | contain several jobs that are scheduled to be run at a certain time, several 13 | jobs that are waiting to run, and jobs that are currently running. A `worker` 14 | is a process on a host, identified uniquely, that asks for jobs from the 15 | queue, performs some process associated with that job, and then marks it as 16 | complete. When it's completed, it can be put into another queue. 17 | 18 | Jobs can only be in one queue at a time. That queue is whatever queue they 19 | were last put in. So if a worker is working on a job, and you move it, the 20 | worker's request to complete the job will be ignored. 21 | 22 | A job can be `canceled`, which means it disappears into the ether, and we'll 23 | never pay it any mind every again. A job can be `dropped`, which is when a 24 | worker fails to heartbeat or complete the job in a timely fashion, or a job 25 | can be `failed`, which is when a host recognizes some systematically 26 | problematic state about the job. A worker should only fail a job if the error 27 | is likely not a transient one; otherwise, that worker should just drop it and 28 | let the system reclaim it. 29 | 30 | Features 31 | ======== 32 | 33 | 1. __Jobs don't get dropped on the floor__ -- Sometimes workers drop jobs. 34 | Qless automatically picks them back up and gives them to another worker 35 | 1. __Tagging / Tracking__ -- Some jobs are more interesting than others. Track 36 | those jobs to get updates on their progress. Tag jobs with meaningful 37 | identifiers to find them quickly in the UI. 38 | 1. __Job Dependencies__ -- One job might need to wait for another job to 39 | complete 40 | 1. __Stats__ -- `qless` automatically keeps statistics about how long jobs wait 41 | to be processed and how long they take to be processed. Currently, we keep 42 | track of the count, mean, standard deviation, and a histogram of these 43 | times. 44 | 1. __Job data is stored temporarily__ -- Job info sticks around for a 45 | configurable amount of time so you can still look back on a job's history, 46 | data, etc. 47 | 1. __Priority__ -- Jobs with the same priority get popped in the order they 48 | were inserted; a higher priority means that it gets popped faster 49 | 1. __Retry logic__ -- Every job has a number of retries associated with it, 50 | which are renewed when it is put into a new queue or completed. If a job 51 | is repeatedly dropped, then it is presumed to be problematic, and is 52 | automatically failed. 53 | 1. __Web App__ -- With the advent of a Ruby client, there is a Sinatra-based 54 | web app that gives you control over certain operational issues 55 | 1. __Scheduled Work__ -- Until a job waits for a specified delay (defaults to 56 | 0), jobs cannot be popped by workers 57 | 1. __Recurring Jobs__ -- Scheduling's all well and good, but we also support 58 | jobs that need to recur periodically. 59 | 1. __Notifications__ -- Tracked jobs emit events on pubsub channels as they get 60 | completed, failed, put, popped, etc. Use these events to get notified of 61 | progress on jobs you're interested in. 62 | 63 | Interest piqued? Then read on! 64 | 65 | Installation 66 | ============ 67 | Install from pip: 68 | 69 | pip install qless-py 70 | 71 | Alternatively, install qless-py from source by checking it out from github, 72 | and checking out the qless-core submodule: 73 | 74 | ```bash 75 | git clone git://github.com/seomoz/qless-py.git 76 | cd qless-py 77 | # qless-core is a submodule 78 | git submodule init 79 | git submodule update 80 | sudo python setup.py install 81 | ``` 82 | 83 | Business Time! 84 | ============== 85 | You've read this far -- you probably want to write some code now and turn them 86 | into jobs. Jobs are described essentially by two pieces of information -- a 87 | class` and `data`. The class should have static methods that know how to 88 | process this type of job depending on the queue it's in. For those thrown for 89 | a loop by this example, it's in reference to a 90 | [South Park](http://en.wikipedia.org/wiki/Gnomes_%28South_Park%29#Plot) 91 | episode where a group of enterprising gnomes set on world domination through 92 | three steps: 1) collect underpants, 2) ? 3) profit! 93 | 94 | ```python 95 | # In gnomes.py 96 | class GnomesJob(object): 97 | # This would be invoked when a GnomesJob is popped off the 'underpants' queue 98 | @staticmethod 99 | def underpants(job): 100 | # 1) Collect Underpants 101 | ... 102 | # Complete and advance to the next step, 'unknown' 103 | job.complete('unknown') 104 | 105 | @staticmethod 106 | def unknown(job): 107 | # 2) ? 108 | ... 109 | # Complete and advance to the next step, 'profit' 110 | job.complete('profit') 111 | 112 | @staticmethod 113 | def profit(job): 114 | # 3) Profit 115 | ... 116 | # Complete the job 117 | job.complete() 118 | ``` 119 | 120 | This makes it easy to describe how a `GnomesJob` might move through a pipeline, 121 | first in the 'underpants' step, then 'unknown', and lastly 'profit.' 122 | Alternatively, you can define a single method `process` that knows how to 123 | complete the job, no matter what queue it was popped from. The above is just 124 | meant as a convenience for pipelines: 125 | 126 | ```python 127 | # Alternative gnomes.py 128 | class GnomesJob(object): 129 | # This method would be invoked at every stage 130 | @staticmethod 131 | def process(job): 132 | if job['queue'] == 'underpants': 133 | ... 134 | job.complete('underpants') 135 | elif job['queue'] == 'unknown': 136 | ... 137 | job.complete('profit') 138 | elif job['queue'] == 'profit': 139 | ... 140 | job.complete() 141 | else: 142 | job.fail('unknown-stage', 'What what?') 143 | ``` 144 | 145 | Jobs have user data associated with them that can be modified as it goes 146 | through a pipeline. In general, you should make this data a dictionary, in 147 | which case it's accessible through `__getitem__` and `__setitem__`. Otherwise, 148 | it's accessible through `job.data`. For example, you might update the data... 149 | 150 | ```python 151 | @staticmethod 152 | def underpants(job): 153 | # Record how many underpants we collected 154 | job['collected'] = ... 155 | 156 | @staticmethod 157 | def unknown(job): 158 | # Make some decision based on how many we've collected. 159 | if job['collected'] ...: 160 | ... 161 | ``` 162 | 163 | Great! With all this in place, let's put them in the queue so that they can 164 | get run 165 | 166 | ```python 167 | import qless 168 | # Connecting to localhost on 6379 169 | client = qless.Client() 170 | # Connecting to a remote machine 171 | client = qless.Client('redis://foo.bar.com:1234') 172 | ``` 173 | 174 | Now, reference a queue, and start putting your gnomes to work: 175 | 176 | ```python 177 | queue = client.queues['underpants'] 178 | 179 | import gnomes 180 | for i in range(1000): 181 | queue.put(gnomes.GnomesJob, {}) 182 | ``` 183 | 184 | Alternatively, if the job class is not importable from where you're adding 185 | jobs, you can use the full path of the job class as a string: 186 | 187 | ```python 188 | ... 189 | for i in range(1000): 190 | queue.put('gnomes.GnomesJob', {}) 191 | ``` 192 | 193 | __By way of a quick note__, it's important that your job class can be imported 194 | -- you can't create a job class in an interactive prompt, for example. You can 195 | _add_ jobs in an interactive prompt, but just can't define new job types. 196 | 197 | Running 198 | ======= 199 | All that remains is to have workers actually run these jobs. This distribution 200 | comes with a script to help with this: 201 | 202 | ```bash 203 | qless-py-worker -q underpants -q unknown -q profit 204 | ``` 205 | 206 | This script actually forks off several subprocesses that perform the work, and 207 | the original process keeps tabs on them to ensure that they are all up and 208 | running. In the future, the parent process might also perform other sanity 209 | checks, but for the time being, it's just that the process is still alive. You 210 | can specify the `host` and `port` you want to use for the qless server as well: 211 | 212 | ```bash 213 | qless-py-worker --host foo.bar --port 1234 ... 214 | ``` 215 | 216 | In the absence of the `--workers` argument, qless will spawn as many workers 217 | as there are cores on the machine. The interval specifies how often to poll 218 | in seconds) for work items. Future versions may have a mechanism to support 219 | blocking pop. 220 | 221 | ```bash 222 | qless-py-worker --workers 4 --interval 10 223 | ``` 224 | 225 | Because this works on a forked process model, it can be convenient to import 226 | large modules _before_ subprocesses are forked. Specify these with `--import`: 227 | 228 | ```bash 229 | qless-py-worker --import my.really.bigModule 230 | ``` 231 | 232 | Filesystem 233 | ---------- 234 | Previous versions of `qless-py` included a feature to have each worker process 235 | run in its own sandbox directory. We've removed this feature because since 236 | greenlets can't run in their own directory, the 'regular' and greenlet workers 237 | behave differently. 238 | 239 | In lieu of this behavior, each child process runs in its own sandboxed directory 240 | and each job is given a `sandbox` attribute which is the name of a directory for 241 | the sole use of that job. It's guaranteed to be clean by the time the job is 242 | performed, and it cleaned up afterwards. 243 | 244 | For example, if you invoke: 245 | 246 | ```bash 247 | qless-py-worker --workers 4 --greenlets 5 --workdir foo 248 | ``` 249 | 250 | Then four child processes will be spawned using the directories: 251 | 252 | ``` 253 | foo/qless-py-workers/sandbox-{0,1,2,3} 254 | ``` 255 | 256 | The jobs run by the greenlets in the first process are given their own sandboxes 257 | of the form: 258 | 259 | ``` 260 | foo/qless-py-workers/sandbox-0/greenlet-{0,1,2,3,4} 261 | ``` 262 | 263 | Gevent 264 | ------ 265 | Some jobs are I/O-bound, and might want to, say, make use of a greenlet pool. 266 | If you have a class where you've, say, monkey-patched `socket`, you can ask 267 | qless to create a pool of greenlets to run you job inside each process. To run 268 | 5 processes with 50 greenlets each: 269 | 270 | ```bash 271 | qless-py-worker --workers 5 --greenlets 50 272 | ``` 273 | 274 | Signals 275 | ------- 276 | With a worker running, you can send signals to child processes to: 277 | 278 | - `USR1` - Get the current stack trace in that worker 279 | - `USR2` - Enter a debugger in that worker 280 | 281 | So, for example, if one of the worker child processes is `PID 1234`, then you 282 | can invoke `kill -USR1 1234` to get the backtrace in the logs (and console 283 | output). 284 | 285 | Resuming Jobs 286 | ------------- 287 | This is an __experimental__ feature, but you can start workers `--resume` flag 288 | to have the worker begin its processing with the jobs it left off with. For 289 | instance, during deployments, it's common to restart the worker processes, and 290 | the `--resume` flag has the worker first perform a check with `qless` to see 291 | which jobs it had last been running (and still has locks for). 292 | 293 | This flag should be used with some caution. In particular, if two workers are 294 | running with the same worker name, then this should not be used. The reason is 295 | that through the `qless` interface, it's impossible to differentiate the two, 296 | and currently-running jobs may be confused with jobs that were simply dropped 297 | when the worker was stopped. 298 | 299 | Debugging / Developing 300 | ====================== 301 | Whenever a job is processed, it checks to see if the file in which your job is 302 | defined has been updated since its last import. If it has, it automatically 303 | reimports it. We think of this as a feature. 304 | 305 | With this in mind, when I start a new project and want to make use of qless, I 306 | first start up the web app locally (see 307 | [`qless`](http://github.com/seomoz/qless) for more), take a first pass, and 308 | enqueue a single job while the worker is running: 309 | 310 | # Supposing that I have /my/awesome/project/awesomeproject.py 311 | # In one terminal... 312 | qless-py-worker --path /my/awesome/project --queue foo --workers 1 --interval 10 --verbose 313 | 314 | # In another terminal... 315 | >>> import qless 316 | >>> import awesomeproject 317 | >>> qless.Client().queues['foo'].put(awesomeproject.Job, {'key': 'value')) 318 | 319 | From there, I watch the output on the worker, adjust my job class, save it, 320 | watch again, etc., but __without restarting the worker__ -- in general it 321 | shouldn't be necessary to restart the worker. 322 | 323 | Internals and Additional Features 324 | ================================= 325 | While in many cases the above is sufficient, there are also many cases where 326 | you may need something more. Hopefully after this section many of your 327 | questions will be answered. 328 | 329 | Priority 330 | -------- 331 | Jobs can optionally have priority associated with them. Jobs of equal priority 332 | are popped in the order in which they were put in a queue. The higher the 333 | priority, the sooner it will be processed. If, for example, you get a new job 334 | to collect some really valuable underpants: 335 | 336 | ```python 337 | queue.put(qless.gnomes.GnomesJob, {'address': '123 Brief St.'}, priority = 10) 338 | ``` 339 | 340 | You can also adjust a job's priority while it's waiting: 341 | 342 | ```python 343 | job = client.jobs['83da4d32a0a811e1933012313b062cf1'] 344 | job.priority = 25 345 | ``` 346 | 347 | Scheduled Jobs 348 | -------------- 349 | Jobs can also be scheduled for the future with a delay (in seconds). If for 350 | example, you just learned of an underpants heist opportunity, but you have to 351 | wait until later: 352 | 353 | ```python 354 | queue.put(qless.gnomes.GnomesJob, {}, delay=3600) 355 | ``` 356 | 357 | It's worth noting that it's not guaranteed that this job will run at that time. 358 | It merely means that this job will only be considered valid after the delay 359 | has passed, at which point it will be subject to the normal constraints. If 360 | you want it to be processed very soon after the delay expires, you could also 361 | boost its priority: 362 | 363 | ``` 364 | queue.put(qless.gnomes.GnomesJob, {}, delay=3600, priority=100) 365 | ``` 366 | 367 | Recurring Jobs 368 | -------------- 369 | Whether it's nightly maintainence, or weekly customer updates, you can have a 370 | job of a certain configuration set to recur. Recurring jobs still support 371 | priority, and tagging, and are attached to a queue. Let's say, for example, I 372 | need some global maintenance to run, and I don't care what machine runs it, so 373 | long as someone does: 374 | 375 | ```python 376 | client.queues['maintenance'].recur(myJob, {'tasks': ['sweep', 'mop', 'scrub']}, interval=60 * 60 * 24) 377 | ``` 378 | 379 | That will spawn a job right now, but it's possible you'd like to have it recur, 380 | but maybe the first job should wait a little bit: 381 | 382 | ```python 383 | client.queues['maintenance'].recur(..., interval=86400, offset=3600) 384 | ``` 385 | 386 | You can always update the tags, priority and even the interval of a recurring job: 387 | 388 | ```python 389 | job = client.jobs['83da4d32a0a811e1933012313b062cf1'] 390 | job.priority = 20 391 | job.tag('foo', 'bar') 392 | job.untag('hello') 393 | job.interval = 7200 394 | ``` 395 | 396 | These attributes aren't attached to the recurring jobs, per se, but it's used 397 | as the template for the job that it creates. In the case where more than one 398 | interval passes before a worker tries to pop the job, __more than one job is 399 | created__. The thinking is that while it's 400 | completely client-managed, the state should not be dependent on how often 401 | workers are trying to pop jobs. 402 | 403 | ```python 404 | # Recur every minute 405 | queue.recur(..., {'lots': 'of jobs'}, 60) 406 | # Wait 5 minutes 407 | len(queue.pop(10)) 408 | # => 5 jobs got popped 409 | ``` 410 | 411 | Configuration Options 412 | ===================== 413 | You can get and set global (read: in the context of the same Redis instance) 414 | configuration to change the behavior for heartbeating, and so forth. There 415 | aren't a tremendous number of configuration options, but an important one is 416 | how long job data is kept around. Job data is expired after it has been 417 | completed for `jobs-history` seconds, but is limited to the last 418 | `jobs-history-count` completed jobs. These default to 50k jobs, and 30 days, 419 | but depending on volume, your needs may change. To only keep the last 500 jobs 420 | for up to 7 days: 421 | 422 | ```python 423 | client.config['jobs-history'] = 7 * 86400 424 | client.config['jobs-history-count'] = 500 425 | ``` 426 | 427 | Tagging / Tracking 428 | ------------------ 429 | In qless, 'tracking' means flagging a job as important. Tracked jobs have a 430 | tab reserved for them in the web interface, and they also emit subscribable 431 | events as they make progress (more on that below). You can flag a job from the 432 | web interface, or the corresponding code: 433 | 434 | ```python 435 | client.jobs['b1882e009a3d11e192d0b174d751779d'].track() 436 | ``` 437 | 438 | Jobs can be tagged with strings which are indexed for quick searches. For 439 | example, jobs might be associated with customer accounts, or some other key 440 | that makes sense for your project. 441 | 442 | ```python 443 | queue.put(qless.gnomes.GnomesJob, {'tags': 'aplenty'}, tags=['12345', 'foo', 'bar']) 444 | ``` 445 | 446 | This makes them searchable in the web interface, or from code: 447 | 448 | ```python 449 | jids = client.jobs.tagged('foo') 450 | ``` 451 | 452 | You can add or remove tags at will, too: 453 | 454 | ```python 455 | job = client.jobs['b1882e009a3d11e192d0b174d751779d'] 456 | job.tag('howdy', 'hello') 457 | job.untag('foo', 'bar') 458 | ``` 459 | 460 | Job Dependencies 461 | ---------------- 462 | Jobs can be made dependent on the completion of another job. For example, if 463 | you need to buy eggs, and buy a pan before making an omelete, you could say: 464 | 465 | ```python 466 | eggs_jid = client.queues['buy_eggs'].put(myJob, {'count': 12}) 467 | pan_jid = client.queues['buy_pan' ].put(myJob, {'coating': 'non-stick'}) 468 | client.queues['omelete'].put(myJob, {'toppings': ['onions', 'ham']}, depends=[eggs_jid, pan_jid]) 469 | ``` 470 | 471 | That way, the job to make the omelete can't be performed until the pan and eggs 472 | purchases have been completed. 473 | 474 | Notifications 475 | ------------- 476 | Tracked jobs emit events on specific pubsub channels as things happen to them. 477 | Whether it's getting popped off of a queue, completed by a worker, etc. The 478 | jist of it goes like this, though: 479 | 480 | ```python 481 | def callback(evt, jid): 482 | print '%s => %s' % (jid, evt) 483 | 484 | from functools import partial 485 | for evt in ['canceled', 'completed', 'failed', 'popped', 'put', 'stalled', 'track', 'untrack']: 486 | client.events.on(evt, partial(callback, evt)) 487 | client.events.listen() 488 | ``` 489 | 490 | If you're interested in, say, getting growl or campfire notifications, you 491 | should check out the `qless-growl` and `qless-campfire` ruby gems. 492 | 493 | Retries 494 | ------- 495 | Workers sometimes die. That's an unfortunate reality of life. We try to 496 | mitigate the effects of this by insisting that workers heartbeat their jobs to 497 | ensure that they do not get dropped. That said, qless will automatically 498 | requeue jobs that do get 'stalled' up to the provided number of retries 499 | (default is 5). Since underpants profit can sometimes go awry, maybe you want 500 | to retry a particular heist several times: 501 | 502 | ```python 503 | queue.put(qless.gnomes.GnomesJob, {}, retries=10) 504 | ``` 505 | 506 | Pop 507 | --- 508 | A client pops one or more jobs from a queue: 509 | 510 | ```python 511 | # Get a single job 512 | job = queue.pop() 513 | # Get 20 jobs 514 | jobs = queue.pop(20) 515 | ``` 516 | 517 | Heartbeating 518 | ------------ 519 | Each job object has a notion of when you must either check in with a heartbeat 520 | or turn it in as completed. You can get the absolute time until it expires, or 521 | how long you have left: 522 | 523 | ```python 524 | # When I have to heartbeat / complete it by (seconds since epoch) 525 | job.expires_at 526 | # How long until it expires 527 | job.ttl 528 | ``` 529 | 530 | If your lease on the job will expire before you have a chance to complete it, 531 | then you should heartbeat it to make sure that no other worker gets access to 532 | it. Or, if you are done, you should complete it so that the job can move on: 533 | 534 | ```python 535 | # I call stay-offsies! 536 | job.heartbeat() 537 | # I'm done! 538 | job.complete() 539 | # I'm done with this step, but need to go into another queue 540 | job.complete('anotherQueue') 541 | ``` 542 | 543 | Stats 544 | ----- 545 | One of the selling points of qless is that it keeps stats for you about your 546 | underpants hijinks. It tracks the average wait time, number of jobs that have 547 | waited in a queue, failures, retries, and average running time. It also keeps 548 | histograms for the number of jobs that have waited _x_ time, and the number 549 | that took _x_ time to run. 550 | 551 | Frankly, these are best viewed using the web app. 552 | 553 | Lua 554 | --- 555 | Qless is a set of client language bindings, but the majority of the work is 556 | done in a collection of Lua scripts that comprise the 557 | [core](https://github.com/seomoz/qless-core) functionality. These scripts run 558 | on the Redis 2.6+ server atomically and allow for portability with the same 559 | functionality guarantees. Consult the documentation for `qless-core` to learn 560 | more about its internals. 561 | 562 | Web App 563 | ------- 564 | `Qless` also comes with a web app for administrative tasks, like keeping tabs 565 | on the progress of jobs, tracking specific jobs, retrying failed jobs, etc. 566 | It's available in the [`qless`](https://github.com/seomoz/qless) library as a 567 | mountable [`Sinatra`](http://www.sinatrarb.com/) app. The web app is language 568 | agnostic and was one of the major desires out of this project, so you should 569 | consider using it even if you're not planning on using the Ruby client. 570 | 571 | Changelog 572 | ========= 573 | Things that have changed over time. 574 | 575 | v0.10.0 576 | ------- 577 | The major change was the switch to `unified` qless. This change is 578 | semi-incompatibile. In particular, it changes the job history format but the new 579 | version knows how to convert the old format forward. Upgrades to your workers 580 | should be made from the end of pipelines towards the start. It will also be 581 | necessary to upgrade your `qless-web` install if you're using it. 582 | 583 | - Preempts workers running jobs for which they've lost their lock 584 | - Improved coverage (98%, up from 71%), all of which was worker code 585 | - Debugging signals 586 | - Resumable workers 587 | - Redis URL interface. When specifying a qless client, the default is still to 588 | point to `localhost:6379`, but rather than specify `host` and `port`, you 589 | should provide a single `host` argument of a Redis URL format. For example, 590 | `redis://user:auth@host:port/db`. Many of these paremeters are optional, but 591 | it seems to be the convention recently. 592 | 593 | Upgrading to qless-py 0.10.0 594 | ============================ 595 | Some notes, instructions and potential road blocks to the upgrade. This version 596 | has much better coverage, and a few added features, including stalled job 597 | preemption, pauseable queues, unified sandboxing and the ability to use the 598 | cleaner web interface. 599 | 600 | Road Blocks 601 | =========== 602 | Before we talk about how to install the updated client, here are a couple 603 | potential road blocks that will need to be addressed before you can make the 604 | switch. 605 | 606 | Sandboxes 607 | --------- 608 | If you were using sandboxes (if using the non-greenlet client) and relying on 609 | using the current working directory as the sandbox, that interface has been 610 | done away with. The replacement is that each job comes with a `sandbox` 611 | attribute which is guaranteed to be a directory that exists and empty at the 612 | start of the job, and which is cleaned up after the job. It's a great place for 613 | temporary files. __This only applies if you are running a qless-worker, and not 614 | if you are using the qless client directly to work on jobs.__ 615 | 616 | The directories are made up of subdirectories under the directory provided as 617 | `--workdir`, defaulting to the current directory. 618 | 619 | Client Rename 620 | ------------- 621 | If you are using the `qless` client directly, all instances of `qless.client` 622 | will have to change to `qless.Client`. It was an unfortunate mistake that it was 623 | ever named `client` to begin with, but hopefully this change won't be painful. 624 | 625 | Redis Server Spec 626 | ----------------- 627 | There was a feature request to be able to provide redis auth credentials, and 628 | rather than support any new attributes to the redis client that might come 629 | along, we'll now use a [redis url](http://redis-py.readthedocs.org/en/latest/#redis.StrictRedis.from_url). 630 | 631 | For example: 632 | 633 | ```python 634 | # Instead of this 635 | client = qless.Client(host='foo', port=6380) 636 | # Now it's this 637 | client = qless.Client(url='redis://foo:6380') 638 | ``` 639 | 640 | This allows users to provide auth, select a database, etc. Remember to change 641 | this in __worker invocations__ and __config files__. 642 | 643 | 644 | Installation 645 | ============ 646 | With an existing copy of `qless-py` checked out 647 | 648 | ```bash 649 | # Get the most recent version 650 | git fetch 651 | git checkout v0.10.0 652 | 653 | # Checkout, update and build the submodule 654 | git submodule init 655 | git submodule update 656 | make -C qless/qless-core 657 | 658 | # Install dependencies and then qless 659 | sudo pip install -r requirements.txt 660 | sudo python setup.py install 661 | ``` 662 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # Encoding: utf-8 2 | # -*- mode: ruby -*- 3 | # vi: set ft=ruby : 4 | 5 | ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox' 6 | 7 | # http://docs.vagrantup.com/v2/ 8 | Vagrant.configure('2') do |config| 9 | config.vm.box = 'ubuntu/trusty64' 10 | config.vm.hostname = 'qless-py' 11 | config.ssh.forward_agent = true 12 | 13 | config.vm.provider :virtualbox do |vb| 14 | vb.customize ["modifyvm", :id, "--memory", "1024"] 15 | vb.customize ["modifyvm", :id, "--cpus", "2"] 16 | end 17 | 18 | config.vm.provision :shell, path: 'provision.sh', privileged: false 19 | end 20 | -------------------------------------------------------------------------------- /bin/qless-py-worker: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | import multiprocessing as mp 4 | import argparse 5 | 6 | # First off, read the arguments 7 | parser = argparse.ArgumentParser(description='Run qless workers.', 8 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 9 | 10 | # Options specific to the client we're making' 11 | parser.add_argument('--host', dest='host', default='redis://localhost:6379', 12 | help='The redis:// url to connect to') 13 | parser.add_argument('-n', '--name', default=None, type=str, 14 | help='The hostname to identify your worker as') 15 | 16 | # Options that we consume in this binary 17 | parser.add_argument('-p', '--path', action='append', default=[], 18 | help='Path(s) to include when loading jobs') 19 | parser.add_argument('-v', '--verbose', default=False, action='store_true', 20 | help='Be extra talkative') 21 | parser.add_argument('-l', '--logger', default=False, type=str, 22 | help='Log out to this file') 23 | parser.add_argument('-m', '--import', action='append', default=[], 24 | help='The modules to preemptively import') 25 | parser.add_argument('-d', '--workdir', default='.', 26 | help='The base work directory path') 27 | 28 | # Options specific to the worker we're instantiating 29 | parser.add_argument('-w', '--workers', default=mp.cpu_count(), type=int, 30 | help='How many processes to run. Set to 0 to use all available cores') 31 | parser.add_argument('-q', '--queue', action='append', default=[], 32 | help='The queues to pull work from') 33 | 34 | # Options specific to forking greenlet workers 35 | parser.add_argument('-g', '--greenlets', default=0, type=int, 36 | help='How many greenlets to run in each process (if used, uses gevent)') 37 | parser.add_argument('-i', '--interval', default=60, type=int, 38 | help='The polling interval') 39 | parser.add_argument('-r', '--resume', default=False, action='store_true', 40 | help='Try to resume jobs that this worker had previously been working on') 41 | args = parser.parse_args() 42 | 43 | # Build up the kwargs that we'll pass to the worker 44 | kwargs = { 45 | 'workers': args.workers, 46 | 'interval': args.interval, 47 | 'resume': args.resume 48 | } 49 | 50 | # If we're supposed to use greenlets... 51 | if args.greenlets: 52 | from gevent import monkey 53 | monkey.patch_all() 54 | kwargs.update({ 55 | 'klass': 'qless.workers.greenlet.GeventWorker', 56 | 'greenlets': args.greenlets 57 | }) 58 | 59 | import os 60 | import sys 61 | import qless 62 | from qless import logger 63 | from qless.workers.forking import ForkingWorker 64 | 65 | # Add each of the paths to the python search path 66 | sys.path = [os.path.abspath(p) for p in args.path] + sys.path 67 | 68 | # Be verbose if need be. 'Nuff said 69 | if args.verbose: 70 | import logging 71 | logger.setLevel(logging.DEBUG) 72 | 73 | # Log to the provided file, if need be 74 | if args.logger: 75 | import logging 76 | from qless import formatter 77 | handler = logging.FileHandler(args.logger) 78 | handler.setFormatter(formatter) 79 | handler.setLevel(logging.DEBUG) 80 | logger.addHandler(handler) 81 | 82 | # Import all the modules and packages we've been asked to import 83 | for module in getattr(args, 'import'): 84 | try: 85 | logger.info('Loaded %s' % repr(__import__(module))) 86 | except Exception: 87 | logger.exception('Failed to import %s' % module) 88 | 89 | # Change path to our working directory 90 | os.chdir(args.workdir) 91 | 92 | # And now run the worker 93 | ForkingWorker( 94 | args.queue, qless.Client(args.host, hostname=args.name), **kwargs).run() 95 | -------------------------------------------------------------------------------- /forgetful-bench.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | from __future__ import print_function 4 | 5 | import argparse 6 | 7 | # First off, read the arguments 8 | parser = argparse.ArgumentParser(description='Run forgetful workers on contrived jobs.') 9 | 10 | parser.add_argument('--forgetfulness', dest='forgetfulness', default=0.1, type=float, 11 | help='What portion of jobs should be randomly dropped by workers') 12 | parser.add_argument('--host', dest='host', default='localhost', 13 | help='The host to connect to as the Redis server') 14 | parser.add_argument('--port', dest='port', default=6379, type=int, 15 | help='The port to connect on as the Redis server') 16 | parser.add_argument('--stages', dest='stages', default=1, type=int, 17 | help='How many times to requeue jobs') 18 | parser.add_argument('--jobs', dest='numJobs', default=1000, type=int, 19 | help='How many jobs to schedule for the test') 20 | parser.add_argument('--workers', dest='numWorkers', default=10, type=int, 21 | help='How many workers should do the work') 22 | parser.add_argument('--retries', dest='retries', default=5, type=int, 23 | help='How many retries to give each job') 24 | parser.add_argument('--quiet', dest='verbose', default=True, action='store_false', 25 | help='Reduce all the output') 26 | parser.add_argument('--no-flush', dest='flush', default=True, action='store_false', 27 | help='Don\'t flush Redis after running') 28 | 29 | args = parser.parse_args() 30 | 31 | import time 32 | import qless 33 | import random 34 | import logging 35 | import threading 36 | 37 | logger = logging.getLogger('qless-bench') 38 | formatter = logging.Formatter('[%(asctime)s] %(threadName)s => %(message)s') 39 | handler = logging.StreamHandler() 40 | handler.setLevel(logging.DEBUG) 41 | handler.setFormatter(formatter) 42 | logger.addHandler(handler) 43 | if args.verbose: 44 | logger.setLevel(logging.DEBUG) 45 | else: 46 | logger.setLevel(logging.WARN) 47 | 48 | # Our qless client 49 | client = qless.client(host=args.host, port=args.port) 50 | 51 | class ForgetfulWorker(threading.Thread): 52 | def __init__(self, *a, **kw): 53 | threading.Thread.__init__(self, *a, **kw) 54 | # This is to fake out thread-level workers 55 | tmp = qless.client(host=args.host, port=args.port) 56 | tmp.worker += '-' + self.getName() 57 | self.q = tmp.queue('testing') 58 | 59 | def run(self): 60 | while len(self.q): 61 | job = self.q.pop() 62 | if not job: 63 | # Sleep a little bit 64 | time.sleep(0.1) 65 | logger.debug('No jobs available. Sleeping.') 66 | continue 67 | # Randomly drop a job? 68 | if random.random() < args.forgetfulness: 69 | logger.debug('Randomly dropping job!') 70 | continue 71 | else: 72 | logger.debug('Completing job!') 73 | job['stages'] -= 1 74 | if job['stages'] > 0: 75 | job.complete('testing') 76 | else: 77 | job.complete() 78 | 79 | # Make sure that the redis instance is empty first 80 | if len(client.redis.keys('*')): 81 | print('Must begin on an empty Redis instance') 82 | exit(1) 83 | 84 | client.config.set('heartbeat', 1) 85 | # This is how much CPU Redis had used /before/ 86 | cpuBefore = client.redis.info()['used_cpu_user'] + client.redis.info()['used_cpu_sys'] 87 | # This is how long it took to add the jobs 88 | putTime = -time.time() 89 | # Alright, let's make a bunch of jobs 90 | testing = client.queue('testing') 91 | jids = [testing.put(qless.Job, {'test': 'benchmark', 'count': c, 'stages':args.stages}, retries=args.retries) for c in range(args.numJobs)] 92 | putTime += time.time() 93 | 94 | # This is how long it took to run the workers 95 | workTime = -time.time() 96 | # And now let's make some workers to deal with 'em! 97 | workers = [ForgetfulWorker() for i in range(args.numWorkers)] 98 | for worker in workers: 99 | worker.start() 100 | 101 | for worker in workers: 102 | worker.join() 103 | 104 | workTime += time.time() 105 | 106 | def histo(l): 107 | count = sum(l) 108 | l = list(o for o in l if o) 109 | for i in range(len(l)): 110 | print('\t\t%2i, %10.9f, %i' % (i, float(l[i]) / count, l[i])) 111 | 112 | # Now we'll print out some interesting stats 113 | stats = client.queue('testing').stats() 114 | print('Wait:') 115 | print('\tCount: %i' % stats['wait']['count']) 116 | print('\tMean : %fs' % stats['wait']['mean']) 117 | print('\tSDev : %f' % stats['wait']['std']) 118 | print('\tWait Time Histogram:') 119 | histo(stats['wait']['histogram']) 120 | 121 | print('Run:') 122 | print('\tCount: %i' % stats['run']['count']) 123 | print('\tMean : %fs' % stats['run']['mean']) 124 | print('\tSDev : %f' % stats['run']['std']) 125 | print('\tCompletion Time Histogram:') 126 | histo(stats['run']['histogram']) 127 | 128 | print('=' * 50) 129 | print('Put jobs : %fs' % putTime) 130 | print('Do jobs : %fs' % workTime) 131 | info = client.redis.info() 132 | print('Redis Mem: %s' % info['used_memory_human']) 133 | print('Redis Lua: %s' % info['used_memory_lua']) 134 | print('Redis CPU: %fs' % (info['used_cpu_user'] + info['used_cpu_sys'] - cpuBefore)) 135 | 136 | # Flush the database when we're done 137 | if args.flush: 138 | print('Flushing') 139 | client.redis.flushdb() 140 | -------------------------------------------------------------------------------- /provision.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | # Some dependencies 6 | sudo apt-get update 7 | sudo apt-get install -y git libhiredis-dev libevent-dev python-pip python-dev 8 | 9 | # Install redis in support of qless 10 | sudo apt-get install -y redis-server 11 | 12 | # Libraries required to build a complete python with pyenv: 13 | # https://github.com/yyuu/pyenv/wiki 14 | sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ 15 | libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev 16 | 17 | # Install pyenv 18 | git clone https://github.com/yyuu/pyenv.git ~/.pyenv 19 | echo ' 20 | # Pyenv 21 | export PYENV_ROOT="$HOME/.pyenv" 22 | export PATH="$PYENV_ROOT/bin:$PATH" 23 | eval "$(pyenv init -)" 24 | ' >> ~/.bash_profile 25 | source ~/.bash_profile 26 | hash 27 | 28 | # Install dependencies and the thing itself 29 | pushd /vagrant 30 | 31 | # Install our python version 32 | pyenv install 33 | pyenv rehash 34 | 35 | # Install a virtualenv 36 | pip install virtualenv 37 | if [ ! -d venv ]; then 38 | virtualenv venv 39 | fi 40 | source venv/bin/activate 41 | 42 | # Lastly, our dependencies 43 | pip install -r requirements.txt 44 | 45 | popd 46 | -------------------------------------------------------------------------------- /qless/__init__.py: -------------------------------------------------------------------------------- 1 | '''Main qless business''' 2 | 3 | import time 4 | import redis 5 | import pkgutil 6 | import logging 7 | import decorator 8 | import simplejson as json 9 | import sys 10 | 11 | from six import PY3 12 | 13 | # Internal imports 14 | from .exceptions import QlessException 15 | 16 | 17 | # Our logger 18 | logger = logging.getLogger('qless') 19 | formatter = logging.Formatter( 20 | '%(asctime)s | PID %(process)d | [%(levelname)s] %(message)s') 21 | handler = logging.StreamHandler() 22 | handler.setLevel(logging.DEBUG) 23 | handler.setFormatter(formatter) 24 | logger.addHandler(handler) 25 | logger.setLevel(logging.FATAL) 26 | 27 | 28 | def retry(*excepts): 29 | '''A decorator to specify a bunch of exceptions that should be caught 30 | and the job retried. It turns out this comes up with relative frequency''' 31 | @decorator.decorator 32 | def new_func(func, job): 33 | '''No docstring''' 34 | try: 35 | func(job) 36 | except tuple(excepts): 37 | job.retry() 38 | return new_func 39 | 40 | 41 | class Jobs(object): 42 | '''Class for accessing jobs and job information lazily''' 43 | def __init__(self, client): 44 | self.client = client 45 | 46 | def complete(self, offset=0, count=25): 47 | '''Return the paginated jids of complete jobs''' 48 | return self.client('jobs', 'complete', offset, count) 49 | 50 | def tracked(self): 51 | '''Return an array of job objects that are being tracked''' 52 | results = json.loads(self.client('track')) 53 | results['jobs'] = [Job(self, **job) for job in results['jobs']] 54 | return results 55 | 56 | def tagged(self, tag, offset=0, count=25): 57 | '''Return the paginated jids of jobs tagged with a tag''' 58 | return json.loads(self.client('tag', 'get', tag, offset, count)) 59 | 60 | def failed(self, group=None, start=0, limit=25): 61 | '''If no group is provided, this returns a JSON blob of the counts of 62 | the various types of failures known. If a type is provided, returns 63 | paginated job objects affected by that kind of failure.''' 64 | if not group: 65 | return json.loads(self.client('failed')) 66 | else: 67 | results = json.loads( 68 | self.client('failed', group, start, limit)) 69 | results['jobs'] = self.get(*results['jobs']) 70 | return results 71 | 72 | def get(self, *jids): 73 | '''Return jobs objects for all the jids''' 74 | if jids: 75 | return [ 76 | Job(self.client, **j) for j in 77 | json.loads(self.client('multiget', *jids))] 78 | return [] 79 | 80 | def __getitem__(self, jid): 81 | '''Get a job object corresponding to that jid, or ``None`` if it 82 | doesn't exist''' 83 | results = self.client('get', jid) 84 | if not results: 85 | results = self.client('recur.get', jid) 86 | if not results: 87 | return None 88 | return RecurringJob(self.client, **json.loads(results)) 89 | return Job(self.client, **json.loads(results)) 90 | 91 | 92 | class Workers(object): 93 | '''Class for accessing worker information lazily''' 94 | def __init__(self, clnt): 95 | self.client = clnt 96 | 97 | def __getattr__(self, attr): 98 | '''What workers are workers, and how many jobs are they running''' 99 | if attr == 'counts': 100 | return json.loads(self.client('workers')) 101 | raise AttributeError('qless.Workers has no attribute %s' % attr) 102 | 103 | def __getitem__(self, worker_name): 104 | '''Which jobs does a particular worker have running''' 105 | result = json.loads( 106 | self.client('workers', worker_name)) 107 | result['jobs'] = result['jobs'] or [] 108 | result['stalled'] = result['stalled'] or [] 109 | return result 110 | 111 | 112 | class Queues(object): 113 | '''Class for accessing queues lazily''' 114 | def __init__(self, clnt): 115 | self.client = clnt 116 | 117 | def __getattr__(self, attr): 118 | '''What queues are there, and how many jobs do they have running, 119 | waiting, scheduled, etc.''' 120 | if attr == 'counts': 121 | return json.loads(self.client('queues')) 122 | raise AttributeError('qless.Queues has no attribute %s' % attr) 123 | 124 | def __getitem__(self, queue_name): 125 | '''Get a queue object associated with the provided queue name''' 126 | return Queue(queue_name, self.client, self.client.worker_name) 127 | 128 | 129 | class Client(object): 130 | '''Basic qless client object.''' 131 | def __init__(self, url='redis://localhost:6379', hostname=None, **kwargs): 132 | import socket 133 | # This is our unique idenitifier as a worker 134 | self.worker_name = hostname or socket.gethostname() 135 | if PY3: 136 | kwargs['decode_responses'] = True 137 | # This is just the redis instance we're connected to conceivably 138 | # someone might want to work with multiple instances simultaneously. 139 | self.redis = redis.Redis.from_url(url, **kwargs) 140 | self.jobs = Jobs(self) 141 | self.queues = Queues(self) 142 | self.config = Config(self) 143 | self.workers = Workers(self) 144 | 145 | # We now have a single unified core script. 146 | data = pkgutil.get_data('qless', 'qless-core/qless.lua') 147 | self._lua = self.redis.register_script(data) 148 | 149 | def __getattr__(self, key): 150 | if key == 'events': 151 | self.events = Events(self.redis) 152 | return self.events 153 | raise AttributeError('%s has no attribute %s' % ( 154 | self.__class__.__module__ + '.' + self.__class__.__name__, key)) 155 | 156 | def __call__(self, command, *args): 157 | lua_args = [command, repr(time.time())] 158 | lua_args.extend(args) 159 | try: 160 | return self._lua(keys=[], args=lua_args) 161 | except redis.ResponseError as exc: 162 | raise QlessException(str(exc)) 163 | 164 | def track(self, jid): 165 | '''Begin tracking this job''' 166 | return self('track', 'track', jid) 167 | 168 | def untrack(self, jid): 169 | '''Stop tracking this job''' 170 | return self('track', 'untrack', jid) 171 | 172 | def tags(self, offset=0, count=100): 173 | '''The most common tags among jobs''' 174 | return json.loads(self('tag', 'top', offset, count)) 175 | 176 | def unfail(self, group, queue, count=500): 177 | '''Move jobs from the failed group to the provided queue''' 178 | return self('unfail', queue, group, count) 179 | 180 | from .job import Job, RecurringJob 181 | from .queue import Queue 182 | from .config import Config 183 | from .listener import Events 184 | -------------------------------------------------------------------------------- /qless/config.py: -------------------------------------------------------------------------------- 1 | '''All our configuration operations''' 2 | 3 | import simplejson as json 4 | 5 | 6 | class Config(object): 7 | '''A class that allows us to change and manipulate qless config''' 8 | def __init__(self, client): 9 | self._client = client 10 | 11 | def __getattr__(self, attr): 12 | if attr == 'all': 13 | return json.loads(self._client('config.get')) 14 | raise AttributeError('qless.Config has no attribute %s' % attr) 15 | 16 | def __len__(self): 17 | return len(self.all) 18 | 19 | def __getitem__(self, option): 20 | result = self._client('config.get', option) 21 | if not result: 22 | return None 23 | try: 24 | return json.loads(result) 25 | except TypeError: 26 | return result 27 | 28 | def __setitem__(self, option, value): 29 | return self._client('config.set', option, value) 30 | 31 | def __delitem__(self, option): 32 | return self._client('config.unset', option) 33 | 34 | def __contains__(self, option): 35 | return dict.__contains__(self.all, option) 36 | 37 | def __iter__(self): 38 | return iter(self.all) 39 | 40 | def clear(self): 41 | '''Remove all keys''' 42 | for key in self.all.keys(): 43 | self._client('config.unset', key) 44 | 45 | def get(self, option, default=None): 46 | '''Get a particular option, or the default if it's missing''' 47 | val = self[option] 48 | return (val is None and default) or val 49 | 50 | def items(self): 51 | '''Just like `dict.items`''' 52 | return self.all.items() 53 | 54 | def keys(self): 55 | '''Just like `dict.keys`''' 56 | return self.all.keys() 57 | 58 | def pop(self, option, default=None): 59 | '''Just like `dict.pop`''' 60 | val = self[option] 61 | del self[option] 62 | return (val is None and default) or val 63 | 64 | def update(self, other=(), **kwargs): 65 | '''Just like `dict.update`''' 66 | _kwargs = dict(kwargs) 67 | _kwargs.update(other) 68 | for key, value in _kwargs.items(): 69 | self[key] = value 70 | 71 | def values(self): 72 | '''Just like `dict.values`''' 73 | return self.all.values() 74 | -------------------------------------------------------------------------------- /qless/exceptions.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | '''Some exception classes''' 4 | 5 | 6 | class QlessException(Exception): 7 | '''Any and all qless exceptions''' 8 | pass 9 | 10 | 11 | class LostLockException(QlessException): 12 | '''Lost lock on a job''' 13 | pass 14 | -------------------------------------------------------------------------------- /qless/job.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | '''Both the regular Job and RecurringJob classes''' 4 | 5 | import os 6 | import time 7 | import types 8 | import traceback 9 | import simplejson as json 10 | from six.moves import reload_module 11 | 12 | # Internal imports 13 | from qless import logger 14 | from qless.exceptions import LostLockException, QlessException 15 | 16 | 17 | class BaseJob(object): 18 | '''This is a dictionary of all the classes that we've seen, and 19 | the last load time for each of them. We'll use this either for 20 | the debug mode or the general mechanism''' 21 | _loaded = {} 22 | 23 | def __init__(self, client, **kwargs): 24 | self.client = client 25 | for att in ['jid', 'priority']: 26 | object.__setattr__(self, att, kwargs[att]) 27 | object.__setattr__(self, 'klass_name', kwargs['klass']) 28 | object.__setattr__(self, 'queue_name', kwargs['queue']) 29 | # Because of how Lua parses JSON, empty tags comes through as {} 30 | object.__setattr__(self, 'tags', kwargs['tags'] or []) 31 | object.__setattr__(self, 'data', json.loads(kwargs['data'])) 32 | 33 | def __setattr__(self, key, value): 34 | if key == 'priority': 35 | return self.client('priority', self.jid, 36 | value) and object.__setattr__(self, key, value) 37 | else: 38 | return object.__setattr__(self, key, value) 39 | 40 | def __getattr__(self, key): 41 | if key == 'queue': 42 | # An actual queue instance 43 | object.__setattr__(self, 'queue', 44 | self.client.queues[self.queue_name]) 45 | return self.queue 46 | elif key == 'klass': 47 | # Get a reference to the provided klass 48 | object.__setattr__(self, 'klass', self._import(self.klass_name)) 49 | return self.klass 50 | raise AttributeError('%s has no attribute %s' % ( 51 | self.__class__.__module__ + '.' + self.__class__.__name__, key)) 52 | 53 | @staticmethod 54 | def reload(klass): 55 | '''Force a reload of this klass on next import''' 56 | BaseJob._loaded[klass] = 0 57 | 58 | @staticmethod 59 | def _import(klass): 60 | '''1) Get a reference to the module 61 | 2) Check the file that module's imported from 62 | 3) If that file's been updated, force a reload of that module 63 | return it''' 64 | mod = __import__(klass.rpartition('.')[0]) 65 | for segment in klass.split('.')[1:-1]: 66 | mod = getattr(mod, segment) 67 | 68 | # Alright, now check the file associated with it. Note that clases 69 | # defined in __main__ don't have a __file__ attribute 70 | if klass not in BaseJob._loaded: 71 | BaseJob._loaded[klass] = time.time() 72 | if hasattr(mod, '__file__'): 73 | try: 74 | mtime = os.stat(mod.__file__).st_mtime 75 | if BaseJob._loaded[klass] < mtime: 76 | mod = reload_module(mod) 77 | except OSError: 78 | logger.warn('Could not check modification time of %s', 79 | mod.__file__) 80 | 81 | return getattr(mod, klass.rpartition('.')[2]) 82 | 83 | def cancel(self): 84 | '''Cancel a job. It will be deleted from the system, the thinking 85 | being that if you don't want to do any work on it, it shouldn't be in 86 | the queuing system.''' 87 | return self.client('cancel', self.jid) 88 | 89 | def tag(self, *tags): 90 | '''Tag a job with additional tags''' 91 | return self.client('tag', 'add', self.jid, *tags) 92 | 93 | def untag(self, *tags): 94 | '''Remove tags from a job''' 95 | return self.client('tag', 'remove', self.jid, *tags) 96 | 97 | 98 | class Job(BaseJob): 99 | '''The Job class''' 100 | def __init__(self, client, **kwargs): 101 | BaseJob.__init__(self, client, **kwargs) 102 | self.client = client 103 | for att in ['state', 'tracked', 'failure', 104 | 'history', 'dependents', 'dependencies']: 105 | object.__setattr__(self, att, kwargs[att]) 106 | 107 | # The reason we're using object.__setattr__ directly is because 108 | # we have __setattr__ defined for this class, and we're actually 109 | # just interested in setting these members directly 110 | object.__setattr__(self, 'expires_at', kwargs['expires']) 111 | object.__setattr__(self, 'original_retries', kwargs['retries']) 112 | object.__setattr__(self, 'retries_left', kwargs['remaining']) 113 | object.__setattr__(self, 'worker_name', kwargs['worker']) 114 | # Because of how Lua parses JSON, empty lists come through as {} 115 | object.__setattr__(self, 'dependents', kwargs['dependents'] or []) 116 | object.__setattr__(self, 'dependencies', kwargs['dependencies'] or []) 117 | 118 | def __getattr__(self, key): 119 | if key == 'ttl': 120 | # How long until this expires, in seconds 121 | return self.expires_at - time.time() 122 | return BaseJob.__getattr__(self, key) 123 | 124 | def __getitem__(self, key): 125 | return self.data.get(key) 126 | 127 | def __setitem__(self, key, value): 128 | self.data[key] = value 129 | 130 | def __repr__(self): 131 | return '<%s %s>' % (self.klass_name, self.jid) 132 | 133 | def process(self): 134 | '''Load the module containing your class, and run the appropriate 135 | method. For example, if this job was popped from the queue 136 | ``testing``, then this would invoke the ``testing`` staticmethod of 137 | your class.''' 138 | try: 139 | method = getattr(self.klass, self.queue_name, 140 | getattr(self.klass, 'process', None)) 141 | except Exception as exc: 142 | # We failed to import the module containing this class 143 | logger.exception('Failed to import %s', self.klass_name) 144 | return self.fail(self.queue_name + '-' + exc.__class__.__name__, 145 | 'Failed to import %s' % self.klass_name) 146 | 147 | if method: 148 | if isinstance(method, types.FunctionType): 149 | try: 150 | logger.info('Processing %s in %s', 151 | self.jid, self.queue_name) 152 | method(self) 153 | logger.info('Completed %s in %s', 154 | self.jid, self.queue_name) 155 | except Exception as exc: 156 | # Make error type based on exception type 157 | logger.exception('Failed %s in %s: %s', 158 | self.jid, self.queue_name, repr(method)) 159 | self.fail(self.queue_name + '-' + exc.__class__.__name__, 160 | traceback.format_exc()) 161 | else: 162 | # Or fail with a message to that effect 163 | logger.error('Failed %s in %s : %s is not static', 164 | self.jid, self.queue_name, repr(method)) 165 | self.fail(self.queue_name + '-method-type', 166 | repr(method) + ' is not static') 167 | else: 168 | # Or fail with a message to that effect 169 | logger.error( 170 | 'Failed %s : %s is missing a method "%s" or "process"', 171 | self.jid, self.klass_name, self.queue_name) 172 | self.fail(self.queue_name + '-method-missing', self.klass_name + 173 | ' is missing a method "' + self.queue_name + '" or "process"') 174 | 175 | def move(self, queue, delay=0, depends=None): 176 | '''Move this job out of its existing state and into another queue. If 177 | a worker has been given this job, then that worker's attempts to 178 | heartbeat that job will fail. Like ``Queue.put``, this accepts a 179 | delay, and dependencies''' 180 | logger.info('Moving %s to %s from %s', 181 | self.jid, queue, self.queue_name) 182 | return self.client('put', self.worker_name, 183 | queue, self.jid, self.klass_name, 184 | json.dumps(self.data), delay, 'depends', json.dumps(depends or []) 185 | ) 186 | 187 | def complete(self, nextq=None, delay=None, depends=None): 188 | '''Turn this job in as complete, optionally advancing it to another 189 | queue. Like ``Queue.put`` and ``move``, it accepts a delay, and 190 | dependencies''' 191 | if nextq: 192 | logger.info('Advancing %s to %s from %s', 193 | self.jid, nextq, self.queue_name) 194 | return self.client('complete', self.jid, self.client.worker_name, 195 | self.queue_name, json.dumps(self.data), 'next', nextq, 196 | 'delay', delay or 0, 'depends', 197 | json.dumps(depends or [])) or False 198 | else: 199 | logger.info('Completing %s', self.jid) 200 | return self.client('complete', self.jid, self.client.worker_name, 201 | self.queue_name, json.dumps(self.data)) or False 202 | 203 | def heartbeat(self): 204 | '''Renew the heartbeat, if possible, and optionally update the job's 205 | user data.''' 206 | logger.debug('Heartbeating %s (ttl = %s)', self.jid, self.ttl) 207 | try: 208 | self.expires_at = float(self.client('heartbeat', self.jid, 209 | self.client.worker_name, json.dumps(self.data)) or 0) 210 | except QlessException: 211 | raise LostLockException(self.jid) 212 | logger.debug('Heartbeated %s (ttl = %s)', self.jid, self.ttl) 213 | return self.expires_at 214 | 215 | def fail(self, group, message): 216 | '''Mark the particular job as failed, with the provided type, and a 217 | more specific message. By `type`, we mean some phrase that might be 218 | one of several categorical modes of failure. The `message` is 219 | something more job-specific, like perhaps a traceback. 220 | 221 | This method should __not__ be used to note that a job has been dropped 222 | or has failed in a transient way. This method __should__ be used to 223 | note that a job has something really wrong with it that must be 224 | remedied. 225 | 226 | The motivation behind the `type` is so that similar errors can be 227 | grouped together. Optionally, updated data can be provided for the job. 228 | A job in any state can be marked as failed. If it has been given to a 229 | worker as a job, then its subsequent requests to heartbeat or complete 230 | that job will fail. Failed jobs are kept until they are canceled or 231 | completed. __Returns__ the id of the failed job if successful, or 232 | `False` on failure.''' 233 | logger.warn('Failing %s (%s): %s', self.jid, group, message) 234 | return self.client('fail', self.jid, self.client.worker_name, group, 235 | message, json.dumps(self.data)) or False 236 | 237 | def track(self): 238 | '''Begin tracking this job''' 239 | return self.client('track', 'track', self.jid) 240 | 241 | def untrack(self): 242 | '''Stop tracking this job''' 243 | return self.client('track', 'untrack', self.jid) 244 | 245 | def retry(self, delay=0, group=None, message=None): 246 | '''Retry this job in a little bit, in the same queue. This is meant 247 | for the times when you detect a transient failure yourself''' 248 | args = ['retry', self.jid, self.queue_name, self.worker_name, delay] 249 | if group is not None and message is not None: 250 | args.append(group) 251 | args.append(message) 252 | return self.client(*args) 253 | 254 | def depend(self, *args): 255 | '''If and only if a job already has other dependencies, this will add 256 | more jids to the list of this job's dependencies.''' 257 | return self.client('depends', self.jid, 'on', *args) or False 258 | 259 | def undepend(self, *args, **kwargs): 260 | '''Remove specific (or all) job dependencies from this job: 261 | 262 | job.remove(jid1, jid2) 263 | job.remove(all=True)''' 264 | if kwargs.get('all', False): 265 | return self.client('depends', self.jid, 'off', 'all') or False 266 | else: 267 | return self.client('depends', self.jid, 'off', *args) or False 268 | 269 | def timeout(self): 270 | '''Time out this job''' 271 | self.client('timeout', self.jid) 272 | 273 | 274 | class RecurringJob(BaseJob): 275 | '''Recurring Job object''' 276 | def __init__(self, client, **kwargs): 277 | BaseJob.__init__(self, client, **kwargs) 278 | for att in ['jid', 'priority', 'tags', 279 | 'retries', 'interval', 'count']: 280 | object.__setattr__(self, att, kwargs[att]) 281 | object.__setattr__(self, 'client', client) 282 | object.__setattr__(self, 'klass_name', kwargs['klass']) 283 | object.__setattr__(self, 'queue_name', kwargs['queue']) 284 | object.__setattr__(self, 'tags', self.tags or []) 285 | object.__setattr__(self, 'data', json.loads(kwargs['data'])) 286 | 287 | def __setattr__(self, key, value): 288 | if key in ('priority', 'retries', 'interval'): 289 | return self.client('recur.update', self.jid, key, 290 | value) and object.__setattr__(self, key, value) 291 | if key == 'data': 292 | return self.client('recur.update', self.jid, key, 293 | json.dumps(value)) and object.__setattr__(self, 'data', value) 294 | if key == 'klass': 295 | name = value.__module__ + '.' + value.__name__ 296 | return self.client('recur.update', self.jid, 'klass', 297 | name) and object.__setattr__(self, 'klass_name', 298 | name) and object.__setattr__(self, 'klass', value) 299 | return object.__setattr__(self, key, value) 300 | 301 | def __getattr__(self, key): 302 | if key == 'next': 303 | # The time (seconds since epoch) until the next time it's run 304 | return self.client.redis.zscore( 305 | 'ql:q:' + self.queue_name + '-recur', self.jid) 306 | return BaseJob.__getattr__(self, key) 307 | 308 | def move(self, queue): 309 | '''Make this recurring job attached to another queue''' 310 | return self.client('recur.update', self.jid, 'queue', queue) 311 | 312 | def cancel(self): 313 | '''Cancel all future recurring jobs''' 314 | self.client('unrecur', self.jid) 315 | 316 | def tag(self, *tags): 317 | '''Add tags to this recurring job''' 318 | return self.client('recur.tag', self.jid, *tags) 319 | 320 | def untag(self, *tags): 321 | '''Remove tags from this job''' 322 | return self.client('recur.untag', self.jid, *tags) 323 | -------------------------------------------------------------------------------- /qless/listener.py: -------------------------------------------------------------------------------- 1 | '''A class that listens to pubsub channels and can unlisten''' 2 | 3 | import logging 4 | import threading 5 | import contextlib 6 | 7 | # Our logger 8 | logger = logging.getLogger('qless') 9 | 10 | 11 | class Listener(object): 12 | '''A class that listens to pubsub channels and can unlisten''' 13 | def __init__(self, redis, channels): 14 | self._pubsub = redis.pubsub() 15 | self._channels = channels 16 | 17 | def listen(self): 18 | '''Listen for events as they come in''' 19 | try: 20 | self._pubsub.subscribe(self._channels) 21 | for message in self._pubsub.listen(): 22 | if message['type'] == 'message': 23 | yield message 24 | finally: 25 | self._channels = [] 26 | 27 | def unlisten(self): 28 | '''Stop listening for events''' 29 | self._pubsub.unsubscribe(self._channels) 30 | 31 | @contextlib.contextmanager 32 | def thread(self): 33 | '''Run in a thread''' 34 | thread = threading.Thread(target=self.listen) 35 | thread.start() 36 | try: 37 | yield self 38 | finally: 39 | self.unlisten() 40 | thread.join() 41 | 42 | 43 | class Events(Listener): 44 | '''A class for handling qless events''' 45 | namespace = 'ql:' 46 | events = ( 47 | 'canceled', 'completed', 'failed', 'popped', 48 | 'stalled', 'put', 'track', 'untrack' 49 | ) 50 | 51 | def __init__(self, redis): 52 | Listener.__init__( 53 | self, redis, [self.namespace + event for event in self.events]) 54 | self._callbacks = dict((k, None) for k in (self.events)) 55 | 56 | def listen(self): 57 | '''Listen for events''' 58 | for message in Listener.listen(self): 59 | logger.debug('Message: %s', message) 60 | # Strip off the 'namespace' from the channel 61 | channel = message['channel'][len(self.namespace):] 62 | func = self._callbacks.get(channel) 63 | if func: 64 | func(message['data']) 65 | 66 | def on(self, evt, func): 67 | '''Set a callback handler for a pubsub event''' 68 | if evt not in self._callbacks: 69 | raise NotImplementedError('callback "%s"' % evt) 70 | else: 71 | self._callbacks[evt] = func 72 | 73 | def off(self, evt): 74 | '''Deactivate the callback for a pubsub event''' 75 | return self._callbacks.pop(evt, None) 76 | -------------------------------------------------------------------------------- /qless/profile.py: -------------------------------------------------------------------------------- 1 | '''Some utilities for profiling''' 2 | 3 | from __future__ import print_function 4 | 5 | import redis 6 | from collections import defaultdict 7 | 8 | 9 | class Profiler(object): 10 | '''Profiling a series of requests. Initialized with a Qless client''' 11 | @staticmethod 12 | def clone(client): 13 | '''Clone the redis client to be slowlog-compatible''' 14 | kwargs = client.redis.connection_pool.connection_kwargs 15 | kwargs['parser_class'] = redis.connection.PythonParser 16 | pool = redis.connection.ConnectionPool(**kwargs) 17 | return redis.Redis(connection_pool=pool) 18 | 19 | @staticmethod 20 | def pretty(timings, label): 21 | '''Print timing stats''' 22 | results = [(sum(values), len(values), key) 23 | for key, values in timings.items()] 24 | print(label) 25 | print('=' * 65) 26 | print('%20s => %13s | %8s | %13s' % ( 27 | 'Command', 'Average', '# Calls', 'Total time')) 28 | print('-' * 65) 29 | for total, length, key in sorted(results, reverse=True): 30 | print('%20s => %10.5f us | %8i | %10i us' % ( 31 | key, float(total) / length, length, total)) 32 | 33 | def __init__(self, client): 34 | self._client = self.clone(client) 35 | self._configs = None 36 | self._timings = defaultdict(list) 37 | self._commands = {} 38 | 39 | def start(self): 40 | '''Get ready for a profiling run''' 41 | self._configs = self._client.config_get('slow-*') 42 | self._client.config_set('slowlog-max-len', 100000) 43 | self._client.config_set('slowlog-log-slower-than', 0) 44 | self._client.execute_command('slowlog', 'reset') 45 | 46 | def stop(self): 47 | '''Set everything back to normal and collect our data''' 48 | for key, value in self._configs.items(): 49 | self._client.config_set(key, value) 50 | logs = self._client.execute_command('slowlog', 'get', 100000) 51 | current = { 52 | 'name': None, 'accumulated': defaultdict(list) 53 | } 54 | for _, _, duration, request in logs: 55 | command = request[0] 56 | if command == 'slowlog': 57 | continue 58 | if 'eval' in command.lower(): 59 | subcommand = request[3] 60 | self._timings['qless-%s' % subcommand].append(duration) 61 | if current['name']: 62 | if current['name'] not in self._commands: 63 | self._commands[current['name']] = defaultdict(list) 64 | for key, values in current['accumulated'].items(): 65 | self._commands[current['name']][key].extend(values) 66 | current = { 67 | 'name': subcommand, 'accumulated': defaultdict(list) 68 | } 69 | else: 70 | self._timings[command].append(duration) 71 | if current['name']: 72 | current['accumulated'][command].append(duration) 73 | # Include the last 74 | if current['name']: 75 | if current['name'] not in self._commands: 76 | self._commands[current['name']] = defaultdict(list) 77 | for key, values in current['accumulated'].items(): 78 | self._commands[current['name']][key].extend(values) 79 | 80 | def display(self): 81 | '''Print the results of this profiling''' 82 | self.pretty(self._timings, 'Raw Redis Commands') 83 | print() 84 | for key, value in self._commands.items(): 85 | self.pretty(value, 'Qless "%s" Command' % key) 86 | print() 87 | 88 | def __enter__(self): 89 | self.start() 90 | return self 91 | 92 | def __exit__(self, typ, value, trace): 93 | self.stop() 94 | self.display() 95 | -------------------------------------------------------------------------------- /qless/queue.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | '''Our Queue and supporting classes''' 4 | 5 | import time 6 | import uuid 7 | from six import string_types 8 | 9 | from qless.job import Job 10 | import simplejson as json 11 | 12 | 13 | class Jobs(object): 14 | '''A proxy object for queue-specific job information''' 15 | def __init__(self, name, client): 16 | self.name = name 17 | self.client = client 18 | 19 | def running(self, offset=0, count=25): 20 | '''Return all the currently-running jobs''' 21 | return self.client('jobs', 'running', self.name, offset, count) 22 | 23 | def stalled(self, offset=0, count=25): 24 | '''Return all the currently-stalled jobs''' 25 | return self.client('jobs', 'stalled', self.name, offset, count) 26 | 27 | def scheduled(self, offset=0, count=25): 28 | '''Return all the currently-scheduled jobs''' 29 | return self.client('jobs', 'scheduled', self.name, offset, count) 30 | 31 | def depends(self, offset=0, count=25): 32 | '''Return all the currently dependent jobs''' 33 | return self.client('jobs', 'depends', self.name, offset, count) 34 | 35 | def recurring(self, offset=0, count=25): 36 | '''Return all the recurring jobs''' 37 | return self.client('jobs', 'recurring', self.name, offset, count) 38 | 39 | 40 | class Queue(object): 41 | '''The Queue class''' 42 | def __init__(self, name, client, worker_name): 43 | self.name = name 44 | self.client = client 45 | self.worker_name = worker_name 46 | self._hb = 60 47 | 48 | def __getattr__(self, key): 49 | if key == 'jobs': 50 | self.jobs = Jobs(self.name, self.client) 51 | return self.jobs 52 | if key == 'counts': 53 | return json.loads(self.client('queues', self.name)) 54 | if key == 'heartbeat': 55 | conf = self.client.config.all 56 | return int(conf.get( 57 | self.name + '-heartbeat', conf.get('heartbeat', 60))) 58 | raise AttributeError('qless.Queue has no attribute %s' % key) 59 | 60 | def __setattr__(self, key, value): 61 | if key == 'heartbeat': 62 | self.client.config[self.name + '-heartbeat'] = value 63 | else: 64 | object.__setattr__(self, key, value) 65 | 66 | def class_string(self, klass): 67 | '''Return a string representative of the class''' 68 | if isinstance(klass, string_types): 69 | return klass 70 | return klass.__module__ + '.' + klass.__name__ 71 | 72 | def pause(self): 73 | return self.client('pause', self.name) 74 | 75 | def unpause(self): 76 | return self.client('unpause', self.name) 77 | 78 | def put(self, klass, data, priority=None, tags=None, delay=None, 79 | retries=None, jid=None, depends=None): 80 | '''Either create a new job in the provided queue with the provided 81 | attributes, or move that job into that queue. If the job is being 82 | serviced by a worker, subsequent attempts by that worker to either 83 | `heartbeat` or `complete` the job should fail and return `false`. 84 | 85 | The `priority` argument should be negative to be run sooner rather 86 | than later, and positive if it's less important. The `tags` argument 87 | should be a JSON array of the tags associated with the instance and 88 | the `valid after` argument should be in how many seconds the instance 89 | should be considered actionable.''' 90 | return self.client('put', self.worker_name, 91 | self.name, 92 | jid or uuid.uuid4().hex, 93 | self.class_string(klass), 94 | json.dumps(data), 95 | delay or 0, 96 | 'priority', priority or 0, 97 | 'tags', json.dumps(tags or []), 98 | 'retries', retries or 5, 99 | 'depends', json.dumps(depends or []) 100 | ) 101 | 102 | '''Same function as above but check if the job already exists in the DB beforehand. 103 | You can re-queue for instance failed ones.''' 104 | def requeue(self, klass, data, priority=None, tags=None, delay=None, 105 | retries=None, jid=None, depends=None): 106 | return self.client('requeue', self.worker_name, 107 | self.name, 108 | jid or uuid.uuid4().hex, 109 | self.class_string(klass), 110 | json.dumps(data), 111 | delay or 0, 112 | 'priority', priority or 0, 113 | 'tags', json.dumps(tags or []), 114 | 'retries', retries or 5, 115 | 'depends', json.dumps(depends or []) 116 | ) 117 | 118 | def recur(self, klass, data, interval, offset=0, priority=None, tags=None, 119 | retries=None, jid=None): 120 | '''Place a recurring job in this queue''' 121 | return self.client('recur', self.name, 122 | jid or uuid.uuid4().hex, 123 | self.class_string(klass), 124 | json.dumps(data), 125 | 'interval', interval, offset, 126 | 'priority', priority or 0, 127 | 'tags', json.dumps(tags or []), 128 | 'retries', retries or 5 129 | ) 130 | 131 | def pop(self, count=None): 132 | '''Passing in the queue from which to pull items, the current time, 133 | when the locks for these returned items should expire, and the number 134 | of items to be popped off.''' 135 | results = [Job(self.client, **job) for job in json.loads( 136 | self.client('pop', self.name, self.worker_name, count or 1))] 137 | if count is None: 138 | return (len(results) and results[0]) or None 139 | return results 140 | 141 | def peek(self, count=None): 142 | '''Similar to the pop command, except that it merely peeks at the next 143 | items''' 144 | results = [Job(self.client, **rec) for rec in json.loads( 145 | self.client('peek', self.name, count or 1))] 146 | if count is None: 147 | return (len(results) and results[0]) or None 148 | return results 149 | 150 | def stats(self, date=None): 151 | '''Return the current statistics for a given queue on a given date. 152 | The results are returned are a JSON blob:: 153 | 154 | { 155 | 'total' : ..., 156 | 'mean' : ..., 157 | 'variance' : ..., 158 | 'histogram': [ 159 | ... 160 | ] 161 | } 162 | 163 | The histogram's data points are at the second resolution for the first 164 | minute, the minute resolution for the first hour, the 15-minute 165 | resolution for the first day, the hour resolution for the first 3 166 | days, and then at the day resolution from there on out. The 167 | `histogram` key is a list of those values.''' 168 | return json.loads( 169 | self.client('stats', self.name, date or repr(time.time()))) 170 | 171 | def __len__(self): 172 | return self.client('length', self.name) 173 | -------------------------------------------------------------------------------- /qless/util.py: -------------------------------------------------------------------------------- 1 | '''Some utility functions''' 2 | 3 | 4 | def import_class(klass): 5 | '''Import the named class and return that class''' 6 | mod = __import__(klass.rpartition('.')[0]) 7 | for segment in klass.split('.')[1:-1]: 8 | mod = getattr(mod, segment) 9 | return getattr(mod, klass.rpartition('.')[2]) 10 | -------------------------------------------------------------------------------- /qless/workers/__init__.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | '''Our base worker''' 4 | 5 | from __future__ import print_function 6 | 7 | import os 8 | import code 9 | import signal 10 | import shutil 11 | import sys 12 | import traceback 13 | import threading 14 | from contextlib import contextmanager 15 | 16 | from six import string_types 17 | from six.moves import zip_longest 18 | 19 | # Internal imports 20 | from qless.listener import Listener 21 | from qless import logger, exceptions 22 | 23 | # Try to use the fast json parser 24 | try: 25 | import simplejson as json 26 | except ImportError: # pragma: no cover 27 | import json 28 | 29 | # Setting the process title 30 | try: 31 | from setproctitle import setproctitle, getproctitle 32 | except ImportError: # pragma: no cover 33 | def setproctitle(title): 34 | pass 35 | 36 | def getproctitle(): 37 | return '' 38 | 39 | 40 | class Worker(object): 41 | '''Worker. For doing work''' 42 | @classmethod 43 | def title(cls, message=None): 44 | '''Set the title of the process''' 45 | if message == None: 46 | return getproctitle() 47 | else: 48 | setproctitle('qless-py-worker %s' % message) 49 | logger.info(message) 50 | 51 | @classmethod 52 | def divide(cls, jobs, count): 53 | '''Divide up the provided jobs into count evenly-sized groups''' 54 | jobs = list(zip(*zip_longest(*[iter(jobs)] * count))) 55 | # If we had no jobs to resume, then we get an empty list 56 | jobs = jobs or [()] * count 57 | for index in range(count): 58 | # Filter out the items in jobs that are Nones 59 | jobs[index] = [j for j in jobs[index] if j != None] 60 | return jobs 61 | 62 | @classmethod 63 | def clean(cls, path): 64 | '''Clean up all the files in a provided path''' 65 | for pth in os.listdir(path): 66 | pth = os.path.abspath(os.path.join(path, pth)) 67 | if os.path.isdir(pth): 68 | logger.debug('Removing directory %s' % pth) 69 | shutil.rmtree(pth) 70 | else: 71 | logger.debug('Removing file %s' % pth) 72 | os.remove(pth) 73 | 74 | @classmethod 75 | @contextmanager 76 | def sandbox(cls, path): 77 | '''Ensures path exists before yielding, cleans up after''' 78 | # Ensure the path exists and is clean 79 | try: 80 | os.makedirs(path) 81 | logger.debug('Making %s' % path) 82 | except OSError: 83 | if not os.path.isdir(path): 84 | raise 85 | finally: 86 | cls.clean(path) 87 | # Then yield, but make sure to clean up the directory afterwards 88 | try: 89 | yield 90 | finally: 91 | cls.clean(path) 92 | 93 | def __init__(self, queues, client, **kwargs): 94 | self.client = client 95 | # This should accept either queue objects, or string queue names 96 | self.queues = [] 97 | for queue in queues: 98 | if isinstance(queue, string_types): 99 | self.queues.append(self.client.queues[queue]) 100 | else: 101 | self.queues.append(queue) 102 | 103 | # Save our kwargs, since a common pattern to instantiate subworkers 104 | self.kwargs = kwargs 105 | # Check for any jobs that we should resume. If 'resume' is the actual 106 | # value 'True', we should find all the resumable jobs we can. Otherwise, 107 | # we should interpret it as a list of jobs already 108 | self.resume = kwargs.get('resume') or [] 109 | if self.resume == True: 110 | self.resume = self.resumable() 111 | # How frequently we should poll for work 112 | self.interval = kwargs.get('interval', 60) 113 | # To mark whether or not we should shutdown after work is done 114 | self.shutdown = False 115 | 116 | def resumable(self): 117 | '''Find all the jobs that we'd previously been working on''' 118 | # First, find the jids of all the jobs registered to this client. 119 | # Then, get the corresponding job objects 120 | jids = self.client.workers[self.client.worker_name]['jobs'] 121 | jobs = self.client.jobs.get(*jids) 122 | 123 | # We'll filter out all the jobs that aren't in any of the queues 124 | # we're working on. 125 | queue_names = set([queue.name for queue in self.queues]) 126 | return [job for job in jobs if job.queue_name in queue_names] 127 | 128 | def jobs(self): 129 | '''Generator for all the jobs''' 130 | # If we should resume work, then we should hand those out first, 131 | # assuming we can still heartbeat them 132 | for job in self.resume: 133 | try: 134 | if job.heartbeat(): 135 | yield job 136 | except exceptions.LostLockException: 137 | logger.exception('Cannot resume %s' % job.jid) 138 | while True: 139 | seen = False 140 | for queue in self.queues: 141 | job = queue.pop() 142 | if job: 143 | seen = True 144 | yield job 145 | if not seen: 146 | yield None 147 | 148 | @contextmanager 149 | def listener(self): 150 | '''Listen for pubsub messages relevant to this worker in a thread''' 151 | channels = ['ql:w:' + self.client.worker_name] 152 | listener = Listener(self.client.redis, channels) 153 | thread = threading.Thread(target=self.listen, args=(listener,)) 154 | thread.start() 155 | try: 156 | yield 157 | finally: 158 | listener.unlisten() 159 | thread.join() 160 | 161 | def listen(self, listener): 162 | '''Listen for events that affect our ownership of a job''' 163 | for message in listener.listen(): 164 | try: 165 | data = json.loads(message['data']) 166 | if data['event'] in ('canceled', 'lock_lost', 'put'): 167 | self.kill(data['jid']) 168 | except: 169 | logger.exception('Pubsub error') 170 | 171 | def kill(self, jid): 172 | '''Stop processing the provided jid''' 173 | raise NotImplementedError('Derived classes must override "kill"') 174 | 175 | def signals(self, signals=('QUIT', 'USR1', 'USR2')): 176 | '''Register our signal handler''' 177 | for sig in signals: 178 | signal.signal(getattr(signal, 'SIG' + sig), self.handler) 179 | 180 | def stop(self): 181 | '''Mark this for shutdown''' 182 | self.shutdown = True 183 | 184 | # Unfortunately, for most of this, it's not really practical to unit test 185 | def handler(self, signum, frame): # pragma: no cover 186 | '''Signal handler for this process''' 187 | if signum == signal.SIGQUIT: 188 | # QUIT - Finish processing, but don't do any more work after that 189 | self.stop() 190 | elif signum == signal.SIGUSR1: 191 | # USR1 - Print the backtrace 192 | message = ''.join(traceback.format_stack(frame)) 193 | message = 'Signaled traceback for %s:\n%s' % (os.getpid(), message) 194 | print(message, file=sys.stderr) 195 | logger.warn(message) 196 | elif signum == signal.SIGUSR2: 197 | # USR2 - Enter a debugger 198 | # Much thanks to http://stackoverflow.com/questions/132058 199 | data = {'_frame': frame} # Allow access to frame object. 200 | data.update(frame.f_globals) # Unless shadowed by global 201 | data.update(frame.f_locals) 202 | # Build up a message with a traceback 203 | message = ''.join(traceback.format_stack(frame)) 204 | message = 'Traceback:\n%s' % message 205 | code.InteractiveConsole(data).interact(message) 206 | -------------------------------------------------------------------------------- /qless/workers/forking.py: -------------------------------------------------------------------------------- 1 | '''A worker that forks child processes''' 2 | 3 | import os 4 | import multiprocessing 5 | import signal 6 | 7 | from six import string_types 8 | 9 | # Internal imports 10 | from . import Worker 11 | from qless import logger, util 12 | from .serial import SerialWorker 13 | 14 | try: 15 | NUM_CPUS = multiprocessing.cpu_count() 16 | except NotImplementedError: 17 | NUM_CPUS = 1 18 | 19 | 20 | class ForkingWorker(Worker): 21 | '''A worker that forks child processes''' 22 | def __init__(self, *args, **kwargs): 23 | Worker.__init__(self, *args, **kwargs) 24 | # Worker class to use 25 | self.klass = self.kwargs.pop('klass', SerialWorker) 26 | # How many children to launch 27 | self.count = self.kwargs.pop('workers', 0) or NUM_CPUS 28 | # A dictionary of child pids to information about them 29 | self.sandboxes = {} 30 | # Whether or not we're supposed to shutdown 31 | self.shutdown = False 32 | 33 | def stop(self, sig=signal.SIGINT): 34 | '''Stop all the workers, and then wait for them''' 35 | for cpid in self.sandboxes: 36 | logger.warn('Stopping %i...' % cpid) 37 | try: 38 | os.kill(cpid, sig) 39 | except OSError: # pragma: no cover 40 | logger.exception('Error stopping %s...' % cpid) 41 | 42 | # While we still have children running, wait for them 43 | # We edit the dictionary during the loop, so we need to copy its keys 44 | for cpid in list(self.sandboxes): 45 | try: 46 | logger.info('Waiting for %i...' % cpid) 47 | pid, status = os.waitpid(cpid, 0) 48 | logger.warn('%i stopped with status %i' % (pid, status >> 8)) 49 | except OSError: # pragma: no cover 50 | logger.exception('Error waiting for %i...' % cpid) 51 | finally: 52 | self.sandboxes.pop(cpid, None) 53 | 54 | def spawn(self, **kwargs): 55 | '''Return a new worker for a child process''' 56 | copy = dict(self.kwargs) 57 | copy.update(kwargs) 58 | # Apparently there's an issue with importing gevent in the parent 59 | # process and then using it int he child. This is meant to relieve that 60 | # problem by allowing `klass` to be specified as a string. 61 | if isinstance(self.klass, string_types): 62 | self.klass = util.import_class(self.klass) 63 | return self.klass(self.queues, self.client, **copy) 64 | 65 | def run(self): 66 | '''Run this worker''' 67 | self.signals(('TERM', 'INT', 'QUIT')) 68 | # Divide up the jobs that we have to divy up between the workers. This 69 | # produces evenly-sized groups of jobs 70 | resume = self.divide(self.resume, self.count) 71 | for index in range(self.count): 72 | # The sandbox for the child worker 73 | sandbox = os.path.join( 74 | os.getcwd(), 'qless-py-workers', 'sandbox-%s' % index) 75 | cpid = os.fork() 76 | if cpid: 77 | logger.info('Spawned worker %i' % cpid) 78 | self.sandboxes[cpid] = sandbox 79 | else: # pragma: no cover 80 | # Move to the sandbox as the current working directory 81 | with Worker.sandbox(sandbox): 82 | os.chdir(sandbox) 83 | try: 84 | self.spawn(resume=resume[index], sandbox=sandbox).run() 85 | except: 86 | logger.exception('Exception in spawned worker') 87 | finally: 88 | os._exit(0) 89 | 90 | try: 91 | while not self.shutdown: 92 | pid, status = os.wait() 93 | logger.warn('Worker %i died with status %i from signal %i' % ( 94 | pid, status >> 8, status & 0xff)) 95 | sandbox = self.sandboxes.pop(pid) 96 | cpid = os.fork() 97 | if cpid: 98 | logger.info('Spawned replacement worker %i' % cpid) 99 | self.sandboxes[cpid] = sandbox 100 | else: # pragma: no cover 101 | with Worker.sandbox(sandbox): 102 | os.chdir(sandbox) 103 | try: 104 | self.spawn(sandbox=sandbox).run() 105 | except: 106 | logger.exception('Exception in spawned worker') 107 | finally: 108 | os._exit(0) 109 | finally: 110 | self.stop(signal.SIGKILL) 111 | 112 | def handler(self, signum, frame): # pragma: no cover 113 | '''Signal handler for this process''' 114 | if signum in (signal.SIGTERM, signal.SIGINT, signal.SIGQUIT): 115 | self.stop(signum) 116 | os._exit(0) 117 | -------------------------------------------------------------------------------- /qless/workers/greenlet.py: -------------------------------------------------------------------------------- 1 | '''A Gevent-based worker''' 2 | 3 | import os 4 | import gevent 5 | import gevent.pool 6 | from six import next 7 | 8 | from . import Worker 9 | from qless import logger 10 | 11 | 12 | class GeventWorker(Worker): 13 | '''A Gevent-based worker''' 14 | def __init__(self, *args, **kwargs): 15 | Worker.__init__(self, *args, **kwargs) 16 | # Should we shut down after this? 17 | self.shutdown = False 18 | # A mapping of jids to the greenlets handling them 19 | self.greenlets = {} 20 | count = kwargs.pop('greenlets', 10) 21 | self.pool = gevent.pool.Pool(count) 22 | # A list of the sandboxes that we'll use 23 | self.sandbox = kwargs.pop( 24 | 'sandbox', os.path.join(os.getcwd(), 'qless-py-workers')) 25 | self.sandboxes = [ 26 | os.path.join(self.sandbox, 'greenlet-%i' % i) for i in range(count)] 27 | 28 | def process(self, job): 29 | '''Process a job''' 30 | sandbox = self.sandboxes.pop(0) 31 | try: 32 | with Worker.sandbox(sandbox): 33 | job.sandbox = sandbox 34 | job.process() 35 | finally: 36 | # Delete its entry from our greenlets mapping 37 | self.greenlets.pop(job.jid, None) 38 | self.sandboxes.append(sandbox) 39 | 40 | def kill(self, jid): 41 | '''Stop the greenlet processing the provided jid''' 42 | greenlet = self.greenlets.get(jid) 43 | if greenlet is not None: 44 | logger.warn('Lost ownership of %s' % jid) 45 | greenlet.kill() 46 | 47 | def run(self): 48 | '''Work on jobs''' 49 | # Register signal handlers 50 | self.signals() 51 | 52 | # Start listening 53 | with self.listener(): 54 | try: 55 | generator = self.jobs() 56 | while not self.shutdown: 57 | self.pool.wait_available() 58 | job = next(generator) 59 | if job: 60 | # For whatever reason, doing imports within a greenlet 61 | # (there's one implicitly invoked in job.process), was 62 | # throwing exceptions. The hacky way to get around this 63 | # is to force the import to happen before the greenlet 64 | # is spawned. 65 | job.klass 66 | greenlet = gevent.Greenlet(self.process, job) 67 | self.greenlets[job.jid] = greenlet 68 | self.pool.start(greenlet) 69 | else: 70 | logger.debug('Sleeping for %fs' % self.interval) 71 | gevent.sleep(self.interval) 72 | except StopIteration: 73 | logger.info('Exhausted jobs') 74 | finally: 75 | logger.info('Waiting for greenlets to finish') 76 | self.pool.join() 77 | -------------------------------------------------------------------------------- /qless/workers/serial.py: -------------------------------------------------------------------------------- 1 | '''A worker that serially pops and complete jobs''' 2 | 3 | import os 4 | import time 5 | 6 | from . import Worker 7 | 8 | 9 | class SerialWorker(Worker): 10 | '''A worker that just does serial work''' 11 | def __init__(self, *args, **kwargs): 12 | Worker.__init__(self, *args, **kwargs) 13 | # The jid that we're working on at the moment 14 | self.jid = None 15 | # This is the sandbox we use 16 | self.sandbox = kwargs.pop( 17 | 'sandbox', os.path.join(os.getcwd(), 'qless-py-workers')) 18 | 19 | def kill(self, jid): 20 | '''The best way to do this is to fall on our sword''' 21 | if jid == self.jid: 22 | exit(1) 23 | 24 | def run(self): 25 | '''Run jobs, popping one after another''' 26 | # Register our signal handlers 27 | self.signals() 28 | 29 | with self.listener(): 30 | for job in self.jobs(): 31 | # If there was no job to be had, we should sleep a little bit 32 | if not job: 33 | self.jid = None 34 | self.title('Sleeping for %fs' % self.interval) 35 | time.sleep(self.interval) 36 | else: 37 | self.jid = job.jid 38 | self.title('Working on %s (%s)' % (job.jid, job.klass_name)) 39 | with Worker.sandbox(self.sandbox): 40 | job.sandbox = self.sandbox 41 | job.process() 42 | if self.shutdown: 43 | break 44 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | coverage==4.0.3 2 | decorator==3.4.0 3 | funcsigs==0.4 4 | gevent==1.2.2 5 | greenlet==0.4.9 6 | hiredis==0.1.0 7 | mock==1.3.0 8 | nose==1.3.7 9 | pbr==1.8.1 10 | redis==2.7.5 11 | rednose==1.1.1 12 | setproctitle==1.1.5 13 | simplejson==3.3.0 14 | six==1.10.0 15 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [nosetests] 2 | verbosity=2 3 | rednose=1 4 | exe=1 5 | cover-package=qless 6 | cover-branches=1 7 | cover-min-percentage=98 8 | cover-erase=1 9 | logging-clear-handlers=1 10 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | from setuptools import setup 4 | 5 | setup( 6 | name='qless-py', 7 | version='0.11.4', 8 | description='Redis-based Queue Management', 9 | long_description=''' 10 | Redis-based queue management, with heartbeating, job tracking, 11 | stats, notifications, and a whole lot more.''', 12 | url='http://github.com/seomoz/qless-py', 13 | author='Dan Lecocq', 14 | author_email='dan@moz.com', 15 | license="MIT License", 16 | keywords='redis, qless, job', 17 | packages=[ 18 | 'qless', 19 | 'qless.workers' 20 | ], 21 | package_dir={ 22 | 'qless': 'qless', 23 | 'qless.workers': 'qless/workers' 24 | }, 25 | package_data={ 26 | 'qless': [ 27 | 'qless-core/*.lua' 28 | ] 29 | }, 30 | include_package_data=True, 31 | scripts=[ 32 | 'bin/qless-py-worker' 33 | ], 34 | extras_require={ 35 | 'ps': [ 36 | 'setproctitle' 37 | ] 38 | }, 39 | install_requires=[ 40 | 'argparse', 41 | 'decorator', 42 | 'hiredis', 43 | 'redis', 44 | 'six', 45 | 'simplejson' 46 | ], 47 | tests_requires=[ 48 | 'coverage', 49 | 'mock', 50 | 'nose', 51 | 'rednose', 52 | 'setuptools>=17.1' 53 | ], 54 | classifiers=[ 55 | 'License :: OSI Approved :: MIT License', 56 | 'Programming Language :: Python :: 2', 57 | 'Programming Language :: Python :: 2.7', 58 | 'Programming Language :: Python :: 3', 59 | 'Programming Language :: Python :: 3.3', 60 | 'Programming Language :: Python :: 3.4', 61 | 'Programming Language :: Python :: 3.5', 62 | 'Intended Audience :: Developers', 63 | 'Operating System :: OS Independent' 64 | ] 65 | ) 66 | -------------------------------------------------------------------------------- /test/common.py: -------------------------------------------------------------------------------- 1 | '''A base class for all of our common tests''' 2 | 3 | import redis 4 | import unittest 5 | 6 | # Qless stuff 7 | import qless 8 | import logging 9 | 10 | 11 | class TestQless(unittest.TestCase): 12 | '''Base class for all of our tests''' 13 | @classmethod 14 | def setUpClass(cls): 15 | qless.logger.setLevel(logging.CRITICAL) 16 | cls.redis = redis.Redis() 17 | # Clear the script cache, and nuke everything 18 | cls.redis.execute_command('script', 'flush') 19 | 20 | def setUp(self): 21 | assert(len(self.redis.keys('*')) == 0) 22 | # The qless client we're using 23 | self.client = qless.Client() 24 | self.worker = qless.Client() 25 | self.worker.worker_name = 'worker' 26 | 27 | def tearDown(self): 28 | # Ensure that we leave no keys behind, and that we've unfrozen time 29 | self.redis.flushdb() 30 | -------------------------------------------------------------------------------- /test/test_client.py: -------------------------------------------------------------------------------- 1 | '''Basic tests about the client''' 2 | 3 | from common import TestQless 4 | 5 | 6 | class TestClient(TestQless): 7 | '''Test the client''' 8 | def test_track(self): 9 | '''Gives us access to track and untrack jobs''' 10 | self.client.queues['foo'].put('Foo', {}, jid='jid') 11 | self.client.track('jid') 12 | self.assertEqual(self.client.jobs.tracked()['jobs'][0].jid, 'jid') 13 | self.client.untrack('jid') 14 | self.assertEqual(self.client.jobs.tracked(), 15 | {'jobs': [], 'expired': {}}) 16 | 17 | def test_attribute_error(self): 18 | '''Throws AttributeError for non-attributes''' 19 | self.assertRaises(AttributeError, lambda: self.client.foo) 20 | 21 | def test_tags(self): 22 | '''Provides access to top tags''' 23 | self.assertEqual(self.client.tags(), {}) 24 | for _ in range(10): 25 | self.client.queues['foo'].put('Foo', {}, tags=['foo']) 26 | self.assertEqual(self.client.tags(), ['foo']) 27 | 28 | def test_unfail(self): 29 | '''Provides access to unfail''' 30 | jids = map(str, range(10)) 31 | for jid in jids: 32 | self.client.queues['foo'].put('Foo', {}, jid=jid) 33 | self.client.queues['foo'].pop().fail('foo', 'bar') 34 | for jid in jids: 35 | self.assertEqual(self.client.jobs[jid].state, 'failed') 36 | self.client.unfail('foo', 'foo') 37 | for jid in jids: 38 | self.assertEqual(self.client.jobs[jid].state, 'waiting') 39 | 40 | 41 | class TestJobs(TestQless): 42 | '''Test the Jobs class''' 43 | def test_basic(self): 44 | '''Can give us access to jobs''' 45 | self.assertEqual(self.client.jobs['jid'], None) 46 | self.client.queues['foo'].put('Foo', {}, jid='jid') 47 | self.assertNotEqual(self.client.jobs['jid'], None) 48 | 49 | def test_recurring(self): 50 | '''Can give us access to recurring jobs''' 51 | self.assertEqual(self.client.jobs['jid'], None) 52 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 53 | self.assertNotEqual(self.client.jobs['jid'], None) 54 | 55 | def test_complete(self): 56 | '''Can give us access to complete jobs''' 57 | self.assertEqual(self.client.jobs.complete(), []) 58 | self.client.queues['foo'].put('Foo', {}, jid='jid') 59 | self.client.queues['foo'].pop().complete() 60 | self.assertEqual(self.client.jobs.complete(), ['jid']) 61 | 62 | def test_tracked(self): 63 | '''Gives us access to tracked jobs''' 64 | self.assertEqual(self.client.jobs.tracked(), 65 | {'jobs': [], 'expired': {}}) 66 | self.client.queues['foo'].put('Foo', {}, jid='jid') 67 | self.client.track('jid') 68 | self.assertEqual(self.client.jobs.tracked()['jobs'][0].jid, 'jid') 69 | 70 | def test_tagged(self): 71 | '''Gives us access to tagged jobs''' 72 | self.assertEqual(self.client.jobs.tagged('foo'), 73 | {'total': 0, 'jobs': {}}) 74 | self.client.queues['foo'].put('Foo', {}, jid='jid', tags=['foo']) 75 | self.assertEqual(self.client.jobs.tagged('foo')['jobs'][0], 'jid') 76 | 77 | def test_failed(self): 78 | '''Gives us access to failed jobs''' 79 | self.assertEqual(self.client.jobs.failed('foo'), 80 | {'total': 0, 'jobs': []}) 81 | self.client.queues['foo'].put('Foo', {}, jid='jid') 82 | self.client.queues['foo'].pop().fail('foo', 'bar') 83 | self.assertEqual(self.client.jobs.failed('foo')['jobs'][0].jid, 'jid') 84 | 85 | def test_failures(self): 86 | '''Gives us access to failure types''' 87 | self.assertEqual(self.client.jobs.failed(), {}) 88 | self.client.queues['foo'].put('Foo', {}, jid='jid') 89 | self.client.queues['foo'].pop().fail('foo', 'bar') 90 | self.assertEqual(self.client.jobs.failed(), {'foo': 1}) 91 | 92 | 93 | class TestQueues(TestQless): 94 | '''Test the Queues class''' 95 | def test_basic(self): 96 | '''Gives us access to queues''' 97 | self.assertNotEqual(self.client.queues['foo'], None) 98 | 99 | def test_counts(self): 100 | '''Gives us access to counts''' 101 | self.assertEqual(self.client.queues.counts, {}) 102 | self.client.queues['foo'].put('Foo', {}) 103 | self.assertEqual(self.client.queues.counts, [{ 104 | 'scheduled': 0, 105 | 'name': 'foo', 106 | 'paused': False, 107 | 'waiting': 1, 108 | 'depends': 0, 109 | 'running': 0, 110 | 'stalled': 0, 111 | 'recurring': 0 112 | }]) 113 | 114 | def test_attribute_error(self): 115 | '''Raises AttributeErrors for non-attributes''' 116 | self.assertRaises(AttributeError, lambda: self.client.queues.foo) 117 | 118 | 119 | class TestWorkers(TestQless): 120 | '''Test the Workers class''' 121 | def test_individual(self): 122 | '''Gives us access to individual workers''' 123 | self.client.queues['foo'].put('Foo', {}, jid='jid') 124 | self.assertEqual(self.client.workers['worker'], 125 | {'jobs': [], 'stalled': []}) 126 | self.worker.queues['foo'].pop() 127 | self.assertEqual(self.client.workers['worker'], 128 | {'jobs': ['jid'], 'stalled': []}) 129 | 130 | def test_counts(self): 131 | '''Gives us access to worker counts''' 132 | self.client.queues['foo'].put('Foo', {}, jid='jid') 133 | self.assertEqual(self.client.workers.counts, {}) 134 | self.worker.queues['foo'].pop() 135 | self.assertEqual(self.client.workers.counts, 136 | [{'jobs': 1, 'name': 'worker', 'stalled': 0}]) 137 | 138 | def test_attribute_error(self): 139 | '''Raises AttributeErrors for non-attributes''' 140 | self.assertRaises(AttributeError, lambda: self.client.workers.foo) 141 | 142 | 143 | # This is used for TestRetry 144 | class Foo(object): 145 | from qless import retry 146 | 147 | @staticmethod 148 | @retry(ValueError) 149 | def process(job): 150 | '''This is supposed to raise an Exception''' 151 | if 'valueerror' in job.tags: 152 | raise ValueError('Foo') 153 | else: 154 | raise Exception('Foo') 155 | 156 | 157 | class TestRetry(TestQless): 158 | '''Test the retry decorator''' 159 | def test_basic(self): 160 | '''Ensure the retry decorator works''' 161 | # The first time, it should just be retries automatically 162 | self.client.queues['foo'].put(Foo, {}, tags=['valueerror'], jid='jid') 163 | self.client.queues['foo'].pop().process() 164 | # Not remove the tag so it should fail 165 | self.client.jobs['jid'].untag('valueerror') 166 | self.client.queues['foo'].pop().process() 167 | self.assertEqual(self.client.jobs['jid'].state, 'failed') 168 | 169 | def test_docstring(self): 170 | '''Retry decorator should preserve docstring''' 171 | self.assertEqual(Foo.process.__doc__, 172 | 'This is supposed to raise an Exception') 173 | -------------------------------------------------------------------------------- /test/test_config.py: -------------------------------------------------------------------------------- 1 | '''Tests about the config class''' 2 | 3 | from common import TestQless 4 | 5 | 6 | class TestConfig(TestQless): 7 | '''Test the config class''' 8 | def test_set_get_unset(self): 9 | '''Basic set/get/unset''' 10 | self.assertEqual(self.client.config['foo'], None) 11 | self.client.config['foo'] = 5 12 | self.assertEqual(self.client.config['foo'], 5) 13 | del self.client.config['foo'] 14 | self.assertEqual(self.client.config['foo'], None) 15 | 16 | def test_get_all(self): 17 | '''Ensure we can get all the configuration''' 18 | self.assertEqual(self.client.config.all, { 19 | 'application': 'qless', 20 | 'grace-period': 10, 21 | 'heartbeat': 60, 22 | 'histogram-history': 7, 23 | 'jobs-history': 604800, 24 | 'jobs-history-count': 50000, 25 | 'stats-history': 30 26 | }) 27 | 28 | def test_clear(self): 29 | '''Can unset all keys''' 30 | original = dict(self.client.config.items()) 31 | for key in self.client.config.keys(): 32 | self.client.config[key] = 1 33 | for value in self.client.config.values(): 34 | self.assertEqual(value, '1') 35 | self.client.config.clear() 36 | self.assertEqual(self.client.config.all, original) 37 | 38 | def test_attribute_error(self): 39 | '''Only has the 'all' attribute''' 40 | self.assertRaises(AttributeError, lambda: self.client.config.foo) 41 | 42 | def test_len(self): 43 | '''We can see how many items are in the config''' 44 | self.assertEqual(len(self.client.config), 7) 45 | 46 | def test_contains(self): 47 | '''We can use the 'in' syntax''' 48 | self.assertFalse('foo' in self.client.config) 49 | self.client.config['foo'] = 5 50 | self.assertTrue('foo' in self.client.config) 51 | 52 | def test_iter(self): 53 | '''We can iterate over the config''' 54 | self.assertEqual( 55 | set([key for key in self.client.config]), set(self.client.config.keys())) 56 | 57 | def test_get(self): 58 | '''We can use dictionary-style get''' 59 | self.assertFalse('foo' in self.client.config) 60 | self.assertEqual(self.client.config.get('foo', 5), 5) 61 | 62 | def test_pop(self): 63 | '''We can use dictionary-style pop''' 64 | self.assertFalse('foo' in self.client.config) 65 | self.client.config['foo'] = 5 66 | self.assertEqual(self.client.config.pop('foo'), 5) 67 | self.assertFalse('foo' in self.client.config) 68 | 69 | def test_update(self): 70 | '''We can use dictionary-style update''' 71 | updated = dict((key, '1') for key in self.client.config) 72 | self.assertNotEqual(self.client.config.all, updated) 73 | self.client.config.update(updated) 74 | self.assertEqual(self.client.config.all, updated) 75 | 76 | def test_default_config(self): 77 | '''We can get default config values.''' 78 | self.assertEqual(self.client.config['heartbeat'], 60) 79 | -------------------------------------------------------------------------------- /test/test_events.py: -------------------------------------------------------------------------------- 1 | '''Tests about events''' 2 | 3 | from common import TestQless 4 | 5 | 6 | class TestEvents(TestQless): 7 | '''Tests about events''' 8 | def setUp(self): 9 | TestQless.setUp(self) 10 | self.client.queues['foo'].put('Foo', {}, jid='jid') 11 | self.client.jobs['jid'].track() 12 | 13 | def test_basic(self): 14 | '''Ensure we can get a basic event''' 15 | def func(_): 16 | '''No docstring''' 17 | func.count += 1 18 | 19 | func.count = 0 20 | self.client.events.on('popped', func) 21 | with self.client.events.thread(): 22 | self.client.queues['foo'].pop() 23 | self.assertEqual(func.count, 1) 24 | 25 | def test_off(self): 26 | '''Ensure we can turn off callbacks''' 27 | def popped(_): 28 | '''No docstring''' 29 | popped.count += 1 30 | 31 | def completed(_): 32 | '''No docstring''' 33 | completed.count += 1 34 | 35 | popped.count = 0 36 | completed.count = 0 37 | self.client.events.on('popped', popped) 38 | self.client.events.on('completed', completed) 39 | self.client.events.off('popped') 40 | with self.client.events.thread(): 41 | self.client.queues['foo'].pop().complete() 42 | self.assertEqual(popped.count, 0) 43 | self.assertEqual(completed.count, 1) 44 | 45 | def test_not_implemented(self): 46 | '''Ensure missing events throw errors''' 47 | self.assertRaises( 48 | NotImplementedError, self.client.events.on, 'foo', int) 49 | -------------------------------------------------------------------------------- /test/test_forking.py: -------------------------------------------------------------------------------- 1 | '''Test the forking worker''' 2 | 3 | # Internal imports 4 | from common import TestQless 5 | 6 | import os 7 | import time 8 | import signal 9 | import threading 10 | 11 | # The stuff we're actually testing 12 | import qless 13 | from qless.workers import Worker 14 | from qless.workers.forking import ForkingWorker 15 | 16 | 17 | class Foo(object): 18 | '''Dummy class''' 19 | @staticmethod 20 | def foo(job): 21 | '''Fall on your sword!''' 22 | os.kill(os.getpid(), signal.SIGKILL) 23 | 24 | 25 | class CWD(object): 26 | '''Completes with our current working directory''' 27 | @staticmethod 28 | def foo(job): 29 | '''Puts your current working directory in the job data''' 30 | job.data['cwd'] = os.getcwd() 31 | job.complete() 32 | os.kill(os.getpid(), signal.SIGKILL) 33 | 34 | 35 | class PatchedForkingWorker(ForkingWorker): 36 | '''A forking worker that doesn't register signal handlers''' 37 | def signals(self, signals=()): 38 | '''Do not actually register signal handlers''' 39 | pass 40 | 41 | 42 | class TestWorker(TestQless): 43 | '''Test the worker''' 44 | def setUp(self): 45 | TestQless.setUp(self) 46 | self.worker = PatchedForkingWorker( 47 | ['foo'], self.client, workers=1, interval=1) 48 | self.queue = self.client.queues['foo'] 49 | self.thread = None 50 | 51 | def tearDown(self): 52 | if self.thread: 53 | self.thread.join() 54 | TestQless.tearDown(self) 55 | 56 | def test_respawn(self): 57 | '''It respawns workers as needed''' 58 | self.thread = threading.Thread(target=self.worker.run) 59 | self.thread.start() 60 | time.sleep(0.1) 61 | self.worker.shutdown = True 62 | self.queue.put(Foo, {}) 63 | self.thread.join(1) 64 | self.assertFalse(self.thread.is_alive()) 65 | 66 | def test_cwd(self): 67 | '''Should set the child's cwd appropriately''' 68 | self.thread = threading.Thread(target=self.worker.run) 69 | self.thread.start() 70 | time.sleep(0.1) 71 | self.worker.shutdown = True 72 | jid = self.queue.put(CWD, {}) 73 | self.thread.join(1) 74 | self.assertFalse(self.thread.is_alive()) 75 | expected = os.path.join(os.getcwd(), 'qless-py-workers/sandbox-0') 76 | self.assertEqual(self.client.jobs[jid]['cwd'], expected) 77 | 78 | def test_spawn_klass_string(self): 79 | '''Should be able to import by class string''' 80 | worker = PatchedForkingWorker(['foo'], self.client) 81 | worker.klass = 'qless.workers.serial.SerialWorker' 82 | self.assertIsInstance(worker.spawn(), Worker) 83 | 84 | def test_spawn(self): 85 | '''It gives us back a worker instance''' 86 | self.assertIsInstance(self.worker.spawn(), Worker) 87 | -------------------------------------------------------------------------------- /test/test_greenlet.py: -------------------------------------------------------------------------------- 1 | '''Test the serial worker''' 2 | 3 | # Internal imports 4 | from common import TestQless 5 | 6 | import time 7 | import gevent 8 | from six import next 9 | 10 | # The stuff we're actually testing 11 | from qless.workers.greenlet import GeventWorker 12 | 13 | 14 | class GeventJob(object): 15 | '''Dummy class''' 16 | @staticmethod 17 | def foo(job): 18 | '''Dummy job''' 19 | job.data['sandbox'] = job.sandbox 20 | job.complete() 21 | 22 | 23 | class PatchedGeventWorker(GeventWorker): 24 | '''A worker that limits the number of jobs it runs''' 25 | @classmethod 26 | def patch(cls): 27 | '''Don't monkey-patch anything''' 28 | pass 29 | 30 | def jobs(self): 31 | '''Yield only a few jobs''' 32 | generator = GeventWorker.jobs(self) 33 | for _ in range(5): 34 | yield next(generator) 35 | 36 | def listen(self, _): 37 | '''Don't actually listen for pubsub events''' 38 | pass 39 | 40 | 41 | class TestWorker(TestQless): 42 | '''Test the worker''' 43 | def setUp(self): 44 | TestQless.setUp(self) 45 | self.worker = PatchedGeventWorker( 46 | ['foo'], self.client, greenlets=1, interval=0.2) 47 | self.queue = self.client.queues['foo'] 48 | self.thread = None 49 | 50 | def tearDown(self): 51 | if self.thread: 52 | self.thread.join() 53 | TestQless.tearDown(self) 54 | 55 | def test_basic(self): 56 | '''Can complete jobs in a basic way''' 57 | jids = [self.queue.put(GeventJob, {}) for _ in range(5)] 58 | self.worker.run() 59 | states = [self.client.jobs[jid].state for jid in jids] 60 | self.assertEqual(states, ['complete'] * 5) 61 | sandboxes = [self.client.jobs[jid].data['sandbox'] for jid in jids] 62 | for sandbox in sandboxes: 63 | self.assertIn('qless-py-workers/greenlet-0', sandbox) 64 | 65 | def test_sleeps(self): 66 | '''Make sure the client sleeps if there aren't jobs to be had''' 67 | for _ in range(4): 68 | self.queue.put(GeventJob, {}) 69 | before = time.time() 70 | self.worker.run() 71 | self.assertGreater(time.time() - before, 0.2) 72 | 73 | def test_kill(self): 74 | '''Can kill greenlets when it loses its lock''' 75 | worker = PatchedGeventWorker(['foo'], self.client) 76 | greenlet = gevent.spawn(gevent.sleep, 1) 77 | worker.greenlets['foo'] = greenlet 78 | worker.kill('foo') 79 | greenlet.join() 80 | self.assertIsInstance(greenlet.value, gevent.GreenletExit) 81 | 82 | def test_kill_dead(self): 83 | '''Does not panic if the greenlet handling a job is no longer around''' 84 | # This test succeeds if it finishes without an exception 85 | self.worker.kill('foo') 86 | -------------------------------------------------------------------------------- /test/test_job.py: -------------------------------------------------------------------------------- 1 | '''Basic tests about the Job class''' 2 | 3 | import sys 4 | from six import PY3 5 | import mock 6 | 7 | from common import TestQless 8 | from qless.job import Job, BaseJob 9 | 10 | 11 | class Foo(object): 12 | '''A dummy job''' 13 | @staticmethod 14 | def bar(job): 15 | '''A dummy method''' 16 | job['foo'] = 'bar' 17 | job.complete() 18 | 19 | def nonstatic(self, job): 20 | '''A nonstatic method''' 21 | pass 22 | 23 | 24 | class TestJob(TestQless): 25 | '''Test the Job class''' 26 | def test_attributes(self): 27 | '''Has all the basic attributes we'd expect''' 28 | self.client.queues['foo'].put('Foo', {'whiz': 'bang'}, jid='jid', 29 | tags=['foo'], retries=3) 30 | job = self.client.jobs['jid'] 31 | atts = ['data', 'jid', 'priority', 'klass_name', 'queue_name', 'tags', 32 | 'expires_at', 'original_retries', 'retries_left', 'worker_name', 33 | 'dependents', 'dependencies'] 34 | values = [getattr(job, att) for att in atts] 35 | self.assertEqual(dict(zip(atts, values)), { 36 | 'data': {'whiz': 'bang'}, 37 | 'dependencies': [], 38 | 'dependents': [], 39 | 'expires_at': 0, 40 | 'jid': 'jid', 41 | 'klass_name': 'Foo', 42 | 'original_retries': 3, 43 | 'priority': 0, 44 | 'queue_name': 'foo', 45 | 'retries_left': 3, 46 | 'tags': ['foo'], 47 | 'worker_name': u'' 48 | }) 49 | 50 | def test_set_priority(self): 51 | '''We can set a job's priority''' 52 | self.client.queues['foo'].put('Foo', {}, jid='jid', priority=0) 53 | self.assertEqual(self.client.jobs['jid'].priority, 0) 54 | self.client.jobs['jid'].priority = 10 55 | self.assertEqual(self.client.jobs['jid'].priority, 10) 56 | 57 | def test_queue(self): 58 | '''Exposes a queue object''' 59 | self.client.queues['foo'].put('Foo', {}, jid='jid') 60 | self.assertEqual(self.client.jobs['jid'].queue.name, 'foo') 61 | 62 | def test_klass(self): 63 | '''Exposes the class for a job''' 64 | self.client.queues['foo'].put(Job, {}, jid='jid') 65 | self.assertEqual(self.client.jobs['jid'].klass, Job) 66 | 67 | def test_ttl(self): 68 | '''Exposes the ttl for a job''' 69 | self.client.config['heartbeat'] = 10 70 | self.client.queues['foo'].put(Job, {}, jid='jid') 71 | self.client.queues['foo'].pop() 72 | self.assertTrue(self.client.jobs['jid'].ttl < 10) 73 | self.assertTrue(self.client.jobs['jid'].ttl > 9) 74 | 75 | def test_attribute_error(self): 76 | '''Raises an attribute error for nonexistent attributes''' 77 | self.client.queues['foo'].put(Job, {}, jid='jid') 78 | self.assertRaises(AttributeError, lambda: self.client.jobs['jid'].foo) 79 | 80 | def test_cancel(self): 81 | '''Exposes the cancel method''' 82 | self.client.queues['foo'].put('Foo', {}, jid='jid') 83 | self.client.jobs['jid'].cancel() 84 | self.assertEqual(self.client.jobs['jid'], None) 85 | 86 | def test_tag_untag(self): 87 | '''Exposes a way to tag and untag a job''' 88 | self.client.queues['foo'].put('Foo', {}, jid='jid') 89 | self.client.jobs['jid'].tag('foo') 90 | self.assertEqual(self.client.jobs['jid'].tags, ['foo']) 91 | self.client.jobs['jid'].untag('foo') 92 | self.assertEqual(self.client.jobs['jid'].tags, []) 93 | 94 | def test_getitem(self): 95 | '''Exposes job data through []''' 96 | self.client.queues['foo'].put('Foo', {'foo': 'bar'}, jid='jid') 97 | self.assertEqual(self.client.jobs['jid']['foo'], 'bar') 98 | 99 | def test_setitem(self): 100 | '''Sets jobs data through []''' 101 | self.client.queues['foo'].put('Foo', {}, jid='jid') 102 | job = self.client.jobs['jid'] 103 | job['foo'] = 'bar' 104 | self.assertEqual(job['foo'], 'bar') 105 | 106 | def test_move(self): 107 | '''Able to move jobs through the move method''' 108 | self.client.queues['foo'].put('Foo', {}, jid='jid') 109 | self.client.jobs['jid'].move('bar') 110 | self.assertEqual(self.client.jobs['jid'].queue.name, 'bar') 111 | 112 | def test_complete(self): 113 | '''Able to complete a job''' 114 | self.client.queues['foo'].put('Foo', {}, jid='jid') 115 | self.client.queues['foo'].pop().complete() 116 | self.assertEqual(self.client.jobs['jid'].state, 'complete') 117 | 118 | def test_advance(self): 119 | '''Able to advance a job to another queue''' 120 | self.client.queues['foo'].put('Foo', {}, jid='jid') 121 | self.client.queues['foo'].pop().complete('bar') 122 | self.assertEqual(self.client.jobs['jid'].state, 'waiting') 123 | 124 | def test_heartbeat(self): 125 | '''Provides access to heartbeat''' 126 | self.client.config['heartbeat'] = 10 127 | self.client.queues['foo'].put('Foo', {}, jid='jid') 128 | job = self.client.queues['foo'].pop() 129 | before = job.ttl 130 | self.client.config['heartbeat'] = 20 131 | job.heartbeat() 132 | self.assertTrue(job.ttl > before) 133 | 134 | def test_heartbeat_fail(self): 135 | '''Failed heartbeats raise an error''' 136 | from qless.exceptions import LostLockException 137 | self.client.queues['foo'].put('Foo', {}, jid='jid') 138 | self.assertRaises(LostLockException, self.client.jobs['jid'].heartbeat) 139 | 140 | def test_track_untrack(self): 141 | '''Exposes a track, untrack method''' 142 | self.client.queues['foo'].put('Foo', {}, jid='jid') 143 | self.assertFalse(self.client.jobs['jid'].tracked) 144 | self.client.jobs['jid'].track() 145 | self.assertTrue(self.client.jobs['jid'].tracked) 146 | self.client.jobs['jid'].untrack() 147 | self.assertFalse(self.client.jobs['jid'].tracked) 148 | 149 | def test_depend_undepend(self): 150 | '''Exposes a depend, undepend methods''' 151 | self.client.queues['foo'].put('Foo', {}, jid='a') 152 | self.client.queues['foo'].put('Foo', {}, jid='b') 153 | self.client.queues['foo'].put('Foo', {}, jid='c', depends=['a']) 154 | self.assertEqual(self.client.jobs['c'].dependencies, ['a']) 155 | self.client.jobs['c'].depend('b') 156 | self.assertEqual(self.client.jobs['c'].dependencies, ['a', 'b']) 157 | self.client.jobs['c'].undepend('a') 158 | self.assertEqual(self.client.jobs['c'].dependencies, ['b']) 159 | self.client.jobs['c'].undepend(all=True) 160 | self.assertEqual(self.client.jobs['c'].dependencies, []) 161 | 162 | def test_retry_fail(self): 163 | '''Retry raises an error if retry fails''' 164 | from qless.exceptions import QlessException 165 | self.client.queues['foo'].put('Foo', {}, jid='jid') 166 | self.assertRaises(QlessException, self.client.jobs['jid'].retry) 167 | 168 | def test_retry_group_and_message(self): 169 | '''Can supply a group and message when retrying.''' 170 | self.client.queues['foo'].put('Foo', {}, jid='jid', retries=0) 171 | self.client.queues['foo'].pop().retry(group='group', message='message') 172 | self.assertEqual(self.client.jobs['jid'].failure['group'], 'group') 173 | self.assertEqual(self.client.jobs['jid'].failure['message'], 'message') 174 | 175 | def test_repr(self): 176 | '''Has a reasonable repr''' 177 | self.client.queues['foo'].put(Job, {}, jid='jid') 178 | self.assertEqual(repr(self.client.jobs['jid']), '') 179 | 180 | def test_no_method(self): 181 | '''Raises an error if the class doesn't have the method''' 182 | self.client.queues['foo'].put(Foo, {}, jid='jid') 183 | self.client.queues['foo'].pop().process() 184 | job = self.client.jobs['jid'] 185 | self.assertEqual(job.state, 'failed') 186 | self.assertEqual(job.failure['group'], 'foo-method-missing') 187 | 188 | def test_no_import(self): 189 | '''Raises an error if it can't import the class''' 190 | self.client.queues['foo'].put('foo.Foo', {}, jid='jid') 191 | self.client.queues['foo'].pop().process() 192 | job = self.client.jobs['jid'] 193 | self.assertEqual(job.state, 'failed') 194 | self.assertEqual(job.failure['group'], 'foo-ImportError') 195 | 196 | def test_nonstatic(self): 197 | '''Rasises an error if the relevant function's not static''' 198 | self.client.queues['nonstatic'].put(Foo, {}, jid='jid') 199 | self.client.queues['nonstatic'].pop().process() 200 | job = self.client.jobs['jid'] 201 | self.assertEqual(job.state, 'failed') 202 | if PY3: 203 | self.assertEqual(job.failure['group'], 'nonstatic-TypeError') 204 | else: 205 | self.assertEqual(job.failure['group'], 'nonstatic-method-type') 206 | 207 | def test_reload(self): 208 | '''Ensure that nothing blows up if we reload a class''' 209 | self.client.queues['foo'].put(Foo, {}, jid='jid') 210 | self.assertEqual(self.client.jobs['jid'].klass, Foo) 211 | Job.reload(self.client.jobs['jid'].klass_name) 212 | self.assertEqual(self.client.jobs['jid'].klass, Foo) 213 | 214 | def test_no_mtime(self): 215 | '''Don't blow up we cannot check the modification time of a module.''' 216 | exc = OSError('Could not stat file') 217 | with mock.patch('qless.job.os.stat', side_effect=exc): 218 | Job._import('test_job.Foo') 219 | Job._import('test_job.Foo') 220 | 221 | 222 | class TestRecurring(TestQless): 223 | def test_attributes(self): 224 | '''We can access all the recurring attributes''' 225 | self.client.queues['foo'].recur('Foo', {'whiz': 'bang'}, 60, jid='jid', 226 | tags=['foo'], retries=3) 227 | job = self.client.jobs['jid'] 228 | atts = ['data', 'jid', 'priority', 'klass_name', 'queue_name', 'tags', 229 | 'retries', 'interval', 'count'] 230 | values = [getattr(job, att) for att in atts] 231 | self.assertEqual(dict(zip(atts, values)), { 232 | 'count': 0, 233 | 'data': {'whiz': 'bang'}, 234 | 'interval': 60, 235 | 'jid': 'jid', 236 | 'klass_name': 'Foo', 237 | 'priority': 0, 238 | 'queue_name': 'foo', 239 | 'retries': 3, 240 | 'tags': ['foo'] 241 | }) 242 | 243 | def test_set_priority(self): 244 | '''We can set priority on recurring jobs''' 245 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid', priority=0) 246 | self.client.jobs['jid'].priority = 10 247 | self.assertEqual(self.client.jobs['jid'].priority, 10) 248 | 249 | def test_set_retries(self): 250 | '''We can set retries''' 251 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid', retries=2) 252 | self.client.jobs['jid'].retries = 2 253 | self.assertEqual(self.client.jobs['jid'].retries, 2) 254 | 255 | def test_set_interval(self): 256 | '''We can set the interval''' 257 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 258 | self.client.jobs['jid'].interval = 10 259 | self.assertEqual(self.client.jobs['jid'].interval, 10) 260 | 261 | def test_set_data(self): 262 | '''We can set the job data''' 263 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 264 | self.client.jobs['jid'].data = {'foo': 'bar'} 265 | self.assertEqual(self.client.jobs['jid'].data, {'foo': 'bar'}) 266 | 267 | def test_set_klass(self): 268 | '''We can set the klass''' 269 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 270 | self.client.jobs['jid'].klass = Foo 271 | self.assertEqual(self.client.jobs['jid'].klass, Foo) 272 | 273 | def test_get_next(self): 274 | '''Exposes the next time a job will run''' 275 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 276 | nxt = self.client.jobs['jid'].next 277 | self.client.queues['foo'].pop() 278 | self.assertTrue(abs(self.client.jobs['jid'].next - nxt - 60) < 1) 279 | 280 | def test_attribute_error(self): 281 | '''Raises attribute errors for non-attributes''' 282 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 283 | self.assertRaises(AttributeError, lambda: self.client.jobs['jid'].foo) 284 | 285 | def test_move(self): 286 | '''Exposes a way to move a job''' 287 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 288 | self.client.jobs['jid'].move('bar') 289 | self.assertEqual(self.client.jobs['jid'].queue.name, 'bar') 290 | 291 | def test_cancel(self): 292 | '''Exposes a way to cancel jobs''' 293 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 294 | self.client.jobs['jid'].cancel() 295 | self.assertEqual(self.client.jobs['jid'], None) 296 | 297 | def test_tag_untag(self): 298 | '''Exposes a way to tag jobs''' 299 | self.client.queues['foo'].recur('Foo', {}, 60, jid='jid') 300 | self.client.jobs['jid'].tag('foo') 301 | self.assertEqual(self.client.jobs['jid'].tags, ['foo']) 302 | self.client.jobs['jid'].untag('foo') 303 | self.assertEqual(self.client.jobs['jid'].tags, []) 304 | -------------------------------------------------------------------------------- /test/test_queue.py: -------------------------------------------------------------------------------- 1 | '''Basic tests about the Job class''' 2 | 3 | from common import TestQless 4 | 5 | 6 | class TestQueue(TestQless): 7 | '''Test the Job class''' 8 | def test_jobs(self): 9 | '''The queue.Jobs class provides access to job counts''' 10 | queue = self.client.queues['foo'] 11 | queue.put('Foo', {}) 12 | self.assertEqual(queue.jobs.depends(), []) 13 | self.assertEqual(queue.jobs.running(), []) 14 | self.assertEqual(queue.jobs.stalled(), []) 15 | self.assertEqual(queue.jobs.scheduled(), []) 16 | self.assertEqual(queue.jobs.recurring(), []) 17 | 18 | def test_counts(self): 19 | '''Provides access to job counts''' 20 | self.client.queues['foo'].put('Foo', {}) 21 | self.assertEqual(self.client.queues['foo'].counts, { 22 | 'depends': 0, 23 | 'name': 'foo', 24 | 'paused': False, 25 | 'recurring': 0, 26 | 'running': 0, 27 | 'scheduled': 0, 28 | 'stalled': 0, 29 | 'waiting': 1 30 | }) 31 | 32 | def test_pause(self): 33 | '''Pause/Unpause Queue''' 34 | queue = self.client.queues['foo'] 35 | 36 | queue.pause() 37 | self.assertTrue(queue.counts['paused']) 38 | 39 | queue.unpause() 40 | self.assertFalse(queue.counts['paused']) 41 | 42 | def test_heartbeat(self): 43 | '''Provided access to heartbeat configuration''' 44 | original = self.client.queues['foo'].heartbeat 45 | self.client.queues['foo'].heartbeat = 10 46 | self.assertNotEqual(original, self.client.queues['foo'].heartbeat) 47 | 48 | def test_attribute_error(self): 49 | '''Raises an attribute error if there is no attribute''' 50 | self.assertRaises(AttributeError, lambda: self.client.queues['foo'].foo) 51 | 52 | def test_multipop(self): 53 | '''Exposes multi-pop''' 54 | self.client.queues['foo'].put('Foo', {}) 55 | self.client.queues['foo'].put('Foo', {}) 56 | self.assertEqual(len(self.client.queues['foo'].pop(10)), 2) 57 | 58 | def test_peek(self): 59 | '''Exposes queue peeking''' 60 | self.client.queues['foo'].put('Foo', {}, jid='jid') 61 | self.assertEqual(self.client.queues['foo'].peek().jid, 'jid') 62 | 63 | def test_multipeek(self): 64 | '''Exposes multi-peek''' 65 | self.client.queues['foo'].put('Foo', {}) 66 | self.client.queues['foo'].put('Foo', {}) 67 | self.assertEqual(len(self.client.queues['foo'].peek(10)), 2) 68 | 69 | def test_stats(self): 70 | '''Exposes stats''' 71 | self.client.queues['foo'].stats() 72 | 73 | def test_len(self): 74 | '''Exposes the length of a queue''' 75 | self.client.queues['foo'].put('Foo', {}) 76 | self.assertEqual(len(self.client.queues['foo']), 1) 77 | -------------------------------------------------------------------------------- /test/test_serial.py: -------------------------------------------------------------------------------- 1 | '''Test the serial worker''' 2 | 3 | # Internal imports 4 | from common import TestQless 5 | 6 | import time 7 | from threading import Thread 8 | from six import next 9 | 10 | # The stuff we're actually testing 11 | from qless import logger 12 | from qless.workers.serial import SerialWorker 13 | 14 | 15 | class SerialJob(object): 16 | '''Dummy class''' 17 | @staticmethod 18 | def foo(job): 19 | '''Dummy job''' 20 | time.sleep(job.data.get('sleep', 0)) 21 | try: 22 | job.complete() 23 | except: 24 | logger.exception('Unable to complete job %s' % job.jid) 25 | 26 | 27 | class Worker(SerialWorker): 28 | '''A worker that limits the number of jobs it runs''' 29 | def jobs(self): 30 | '''Yield only a few jobs''' 31 | generator = SerialWorker.jobs(self) 32 | for _ in range(5): 33 | yield next(generator) 34 | 35 | def kill(self, jid): 36 | '''We'll push a message to redis instead of falling on our sword''' 37 | self.client.redis.rpush('foo', jid) 38 | raise KeyboardInterrupt() 39 | 40 | def signals(self): 41 | '''Do not set any signal handlers''' 42 | pass 43 | 44 | 45 | class NoListenWorker(Worker): 46 | '''A worker that just won't listen''' 47 | def listen(self, _): 48 | '''Don't listen for lost locks''' 49 | pass 50 | 51 | 52 | class TestWorker(TestQless): 53 | '''Test the worker''' 54 | def setUp(self): 55 | TestQless.setUp(self) 56 | self.queue = self.client.queues['foo'] 57 | self.thread = None 58 | 59 | def tearDown(self): 60 | if self.thread: 61 | self.thread.join() 62 | TestQless.tearDown(self) 63 | 64 | def test_basic(self): 65 | '''Can complete jobs in a basic way''' 66 | jids = [self.queue.put(SerialJob, {}) for _ in range(5)] 67 | NoListenWorker(['foo'], self.client, interval=0.2).run() 68 | states = [self.client.jobs[jid].state for jid in jids] 69 | self.assertEqual(states, ['complete'] * 5) 70 | 71 | def test_jobs(self): 72 | '''The jobs method yields None if there are no jobs''' 73 | worker = NoListenWorker(['foo'], self.client, interval=0.2) 74 | self.assertEqual(next(worker.jobs()), None) 75 | 76 | def test_sleeps(self): 77 | '''Make sure the client sleeps if there aren't jobs to be had''' 78 | for _ in range(4): 79 | self.queue.put(SerialJob, {}) 80 | before = time.time() 81 | NoListenWorker(['foo'], self.client, interval=0.2).run() 82 | self.assertGreater(time.time() - before, 0.2) 83 | 84 | def test_lost_locks(self): 85 | '''The worker should be able to stop processing if need be''' 86 | jid = [self.queue.put(SerialJob, {'sleep': 0.1}) for _ in range(5)][0] 87 | self.thread = Thread( 88 | target=Worker(['foo'], self.client, interval=0.2).run) 89 | self.thread.start() 90 | # Now, we'll timeout one of the jobs and ensure that kill is invoked 91 | while self.client.jobs[jid].state != 'running': 92 | time.sleep(0.01) 93 | self.client.jobs[jid].timeout() 94 | self.assertEqual(self.client.redis.brpop('foo', 1), ('foo', jid)) 95 | 96 | def test_kill(self): 97 | '''Should be able to fall on its sword if need be''' 98 | worker = SerialWorker([], self.client) 99 | worker.jid = 'foo' 100 | thread = Thread(target=worker.kill, args=(worker.jid,)) 101 | thread.start() 102 | thread.join() 103 | self.assertFalse(thread.is_alive()) 104 | 105 | def test_kill_dead(self): 106 | '''If we've moved on to another job, say so''' 107 | # If this tests runs to completion, it has succeeded 108 | worker = SerialWorker([], self.client) 109 | worker.kill('foo') 110 | 111 | def test_shutdown(self): 112 | '''We should be able to shutdown a serial worker''' 113 | # If this test finishes, it passes 114 | worker = SerialWorker([], self.client, interval=0.1) 115 | worker.stop() 116 | worker.run() 117 | -------------------------------------------------------------------------------- /test/test_worker.py: -------------------------------------------------------------------------------- 1 | '''Test worker''' 2 | 3 | # Internal imports 4 | from common import TestQless 5 | 6 | import qless 7 | from qless.workers import Worker 8 | 9 | # External dependencies 10 | import os 11 | import itertools 12 | from six import next 13 | 14 | 15 | class TestWorker(TestQless): 16 | '''Test the worker''' 17 | def setUp(self): 18 | TestQless.setUp(self) 19 | self.worker = Worker(['foo'], self.client) 20 | 21 | def test_proctitle(self): 22 | '''Make sure we can get / set the process title''' 23 | try: 24 | import setproctitle 25 | before = Worker.title() 26 | Worker.title('Foo') 27 | self.assertNotEqual(before, Worker.title()) 28 | except ImportError: 29 | self.skipTest('setproctitle not available') 30 | 31 | def test_kill(self): 32 | '''The base worker class' kill method should raise an exception''' 33 | self.assertRaises(NotImplementedError, self.worker.kill, 1) 34 | 35 | def test_clean(self): 36 | '''Should be able to clean a directory''' 37 | if not os.path.exists('test/tmp'): 38 | os.makedirs('test/tmp') 39 | self.assertEqual(os.listdir('test/tmp'), []) 40 | os.makedirs('test/tmp/foo/bar') 41 | with open('test/tmp/file.out', 'w+'): 42 | pass 43 | self.assertNotEqual(os.listdir('test/tmp'), []) 44 | Worker.clean('test/tmp') 45 | self.assertEqual(os.listdir('test/tmp'), []) 46 | 47 | def test_sandbox(self): 48 | '''The sandbox utility should work''' 49 | path = 'test/tmp/foo' 50 | self.assertFalse(os.path.exists(path)) 51 | try: 52 | with Worker.sandbox(path): 53 | self.assertTrue(os.path.exists(path)) 54 | for name in ['whiz', 'widget', 'bang']: 55 | with open(os.path.join(path, name), 'w+'): 56 | pass 57 | # Now raise an exception 58 | raise ValueError('foo') 59 | except ValueError: 60 | pass 61 | # Make sure the directory has been cleaned 62 | self.assertEqual(os.listdir(path), []) 63 | os.rmdir(path) 64 | 65 | def test_sandbox_exists(self): 66 | '''Sandbox creation should not throw an error if the path exists''' 67 | path = 'test/tmp' 68 | self.assertEqual(os.listdir(path), []) 69 | with Worker.sandbox(path): 70 | pass 71 | # If we get to this point, the test succeeds 72 | self.assertTrue(True) 73 | 74 | def test_dirty_sandbox(self): 75 | '''If a sandbox is dirty on arrival, clean it first''' 76 | path = 'test/tmp/foo' 77 | with Worker.sandbox(path): 78 | for name in ['whiz', 'widget', 'bang']: 79 | with open(os.path.join(path, name), 'w+'): 80 | pass 81 | # Now it's sullied. Clean it up 82 | self.assertNotEqual(os.listdir(path), []) 83 | with Worker.sandbox(path): 84 | self.assertEqual(os.listdir(path), []) 85 | os.rmdir(path) 86 | 87 | def test_resume(self): 88 | '''We should be able to resume jobs''' 89 | queue = self.worker.client.queues['foo'] 90 | queue.put('foo', {}) 91 | job = next(self.worker.jobs()) 92 | self.assertTrue(isinstance(job, qless.Job)) 93 | # Now, we'll create a new worker and make sure it gets that job first 94 | worker = Worker(['foo'], self.client, resume=[job]) 95 | self.assertEqual(next(worker.jobs()).jid, job.jid) 96 | 97 | def test_unresumable(self): 98 | '''If we can't heartbeat jobs, we should not try to resume it''' 99 | queue = self.worker.client.queues['foo'] 100 | queue.put('foo', {}) 101 | # Pop from another worker 102 | other = qless.Client(hostname='other') 103 | job = other.queues['foo'].pop() 104 | self.assertTrue(isinstance(job, qless.Job)) 105 | # Now, we'll create a new worker and make sure it gets that job first 106 | worker = Worker( 107 | ['foo'], self.client, resume=[self.client.jobs[job.jid]]) 108 | self.assertEqual(next(worker.jobs()), None) 109 | 110 | def test_resumable(self): 111 | '''We should be able to find all the jobs that can be resumed''' 112 | # We're going to put some jobs into some queues, and pop them. 113 | jid = self.client.queues['foo'].put('Foo', {}) 114 | self.client.queues['bar'].put('Foo', {}) 115 | self.client.queues['foo'].pop() 116 | self.client.queues['bar'].pop() 117 | 118 | # Now, we should be able to see a resumable job in 'foo', but we should 119 | # not see the job that we popped from 'bar' 120 | worker = Worker(['foo'], self.client, resume=True) 121 | jids = [job.jid for job in worker.resume] 122 | self.assertEqual(jids, [jid]) 123 | 124 | def test_divide(self): 125 | '''We should be able to divide resumable jobs evenly''' 126 | items = self.worker.divide(range(100), 7) 127 | # Make sure we have the same items as output as input 128 | self.assertEqual(sorted(itertools.chain(*items)), list(range(100))) 129 | lengths = [len(batch) for batch in items] 130 | self.assertLessEqual(max(lengths) - min(lengths), 1) 131 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27,py33,py34,py35 3 | 4 | [testenv] 5 | deps = 6 | nose 7 | rednose 8 | mock 9 | gevent 10 | setproctitle 11 | commands = 12 | nosetests -sv 13 | --------------------------------------------------------------------------------