├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── config.go ├── doc.go ├── encode.go ├── encode_test.go ├── job.go ├── job_status.go ├── job_status_test.go ├── job_test.go ├── job_type.go ├── job_type_test.go ├── pool.go ├── pool_test.go ├── redis_keys.go ├── redis_pool.go ├── scripts.go ├── scripts ├── add_job_to_set.lua ├── destroy_job.lua ├── generate.go ├── get_jobs_by_ids.lua ├── pop_next_jobs.lua ├── purge_stale_pool.lua ├── retry_or_fail_job.lua ├── scripts.go.tmpl ├── set_job_field.lua └── set_job_status.lua ├── scripts_test.go ├── test_utils.go ├── transaction.go ├── utils.go └── worker.go /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | Contributing 2 | ------------ 3 | 4 | Feedback, bug reports, and pull requests are greatly appreciated :) 5 | 6 | ### Issues 7 | 8 | The following are all great reasons to submit an issue: 9 | 10 | 1. You found a bug in the code. 11 | 2. Something is missing from the documentation or the existing documentation is unclear. 12 | 3. You have an idea for a new feature. 13 | 14 | If you are thinking about submitting an issue please remember to: 15 | 16 | 1. Describe the issue in detail. 17 | 2. If applicable, describe the steps to reproduce the error, which probably should include some example code. 18 | 3. Mention details about your platform: OS, version of Go and Redis, etc. 19 | 20 | ### Pull Requests 21 | 22 | Jobs uses semantic versioning and the [git branching model described here](http://nvie.com/posts/a-successful-git-branching-model/). 23 | If you plan on submitting a pull request, you should: 24 | 25 | 1. Fork the repository. 26 | 2. Create a new "feature branch" off of **develop** (not master) with a descriptive name (e.g. fix-database-error). 27 | 3. Make your changes in the feature branch. 28 | 4. Run the tests to make sure that they still pass. Updated the tests if needed. 29 | 5. Submit a pull request to merge your feature branch into the **develop** branch. Please do not request to merge directly into master. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | (The MIT License) 2 | 3 | Copyright (C) 2015, Alex Browne 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | 7 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Jobs 2 | ==== 3 | 4 | Development Status 5 | ------------------ 6 | 7 | Jobs is ***no longer*** being actively developed. I will still try my best to 8 | respond to issues and pull requests, but in general you should not expect much 9 | support. No new features will be added. Still, Jobs is reasonably 10 | well-tested, and it is probably fine to use it for low-traffic hobby sites. If 11 | you are looking for something for more serious, production use-cases, consider 12 | alternatives such as [RabbitMQ](https://github.com/streadway/amqp). 13 | 14 | Jobs follows semantic versioning but offers no guarantees of backwards compatibility until version 15 | 1.0. 16 | 17 | About 18 | ----- 19 | 20 | Jobs is a persistent and flexible background jobs library for go. 21 | 22 | [![Version](https://img.shields.io/badge/version-0.4.2-5272B4.svg)](https://github.com/albrow/jobs/releases) 23 | [![Circle CI](https://img.shields.io/circleci/project/albrow/jobs.svg)](https://circleci.com/gh/albrow/jobs) 24 | [![GoDoc](https://godoc.org/github.com/albrow/jobs?status.svg)](https://godoc.org/github.com/albrow/jobs) 25 | 26 | Jobs is powered by Redis and supports the following features: 27 | 28 | - A job can encapsulate any arbitrary functionality. A job can do anything 29 | which can be done in a go function. 30 | - A job can be one-off (only executed once) or recurring (scheduled to 31 | execute at a specific interval). 32 | - A job can be retried a specified number of times if it fails. 33 | - A job is persistent, with protections against power loss and other worst 34 | case scenarios. (See the [Guarantees](#guarantees) section below) 35 | - Work on jobs can be spread amongst any number of concurrent workers across any 36 | number of machines. 37 | - Provided it is persisted to disk, every job will be executed *at least* once, 38 | and in ideal network conditions will be executed *exactly* once. (See the 39 | [Guarantees](#guarantees) section below) 40 | - You can query the database to find out e.g. the number of jobs that are 41 | currently executing or how long a particular job took to execute. 42 | - Any job that permanently fails will have its error captured and stored. 43 | 44 | 45 | Why is it Useful? 46 | ----------------- 47 | 48 | Jobs is intended to be used in web applications. It is useful for cases where you need 49 | to execute some long-running code, but you don't want your users to wait for the code to 50 | execute before rendering a response. A good example is sending a welcome email to your users 51 | after they sign up. You can use Jobs to schedule the email to be sent asynchronously, and 52 | render a response to your user without waiting for the email to be sent. You could use a 53 | goroutine to accomplish the same thing, but in the event of a server restart or power loss, 54 | the email might never be sent. Jobs guarantees that the email will be sent at some time 55 | and allows you to spread the work between different machines. 56 | 57 | 58 | Installation 59 | ------------ 60 | 61 | Jobs requires Go version >= 1.2. If you do not already have it, follow these instructions: 62 | 63 | - [Install Go](http://golang.org/doc/install) 64 | - Follow the instructions for [setting up your go workspace](https://golang.org/doc/code.html) 65 | 66 | Jobs requires access to a Redis database. If you plan to have multiple worker pools spread 67 | out across different machines, they should all connect to the same Redis database. If you 68 | only want to run one worker pool, it is safe to install Redis locally and run it on the same 69 | machine. In either case, if you need to install Redis, follow these instructions: 70 | 71 | - [Install Redis](http://redis.io/download). 72 | - Follow the instructions in the section called 73 | [Installing Redis more properly](http://redis.io/topics/quickstart#installing-redis-more-properly). 74 | - Make sure you understand how [Redis Persistence](http://redis.io/topics/persistence) works and have 75 | edited your config file to get your desired persistence. We recommend using both RDB and AOF and setting 76 | fsync to either "always" or "everysec". 77 | 78 | After that, you can install Jobs like you would any other go package: `go get github.com/albrow/jobs`. 79 | If you want to update the package later, use `go get -u github.com/albrow/jobs`. Then you can import 80 | Jobs like you would any other go package by adding `import github.com/albrow/jobs` to your go source 81 | file. 82 | 83 | 84 | Quickstart Guide 85 | ---------------- 86 | 87 | ### Connecting to Redis 88 | 89 | You can configure the connection to Redis by editing Config.Db. Here are the options: 90 | 91 | - Address is the address of the redis database to connect to. Default is 92 | "localhost:6379". 93 | - Network is the type of network to use to connect to the redis database 94 | Default is "tcp". 95 | - Database is the redis database number to use for storing all data. Default 96 | is 0. 97 | - Password is a password to use for connecting to a redis database via the 98 | AUTH command. If empty, Jobs will not attempt to authenticate. Default is 99 | "" (an empty string). 100 | 101 | You should edit Config.Db during program initialization, before running Pool.Start 102 | or scheduling any jobs. Here's an example of how to configure Jobs to use databse #10 103 | and authenticate with the password "foobar": 104 | 105 | ``` go 106 | func main() { 107 | // Configure database options at the start of your application 108 | jobs.Config.Db.Database = 10 109 | jobs.Config.Db.Password = "foobar" 110 | } 111 | ``` 112 | 113 | ### Registering Job Types 114 | 115 | Jobs must be organized into discrete types. Here's an example of how to register a job 116 | which sends a welcome email to users: 117 | 118 | ``` go 119 | // We'll specify that we want the job to be retried 3 times before finally failing 120 | welcomeEmailJobs, err := jobs.RegisterType("welcomeEmail", 3, func(user *User) error { 121 | msg := fmt.Sprintf("Hello, %s! Thanks for signing up for foo.com.", user.Name) 122 | if err := emails.Send(user.EmailAddress, msg); err != nil { 123 | // The returned error will be captured by a worker, which will then log the error 124 | // in the database and trigger up to 3 retries. 125 | return err 126 | } 127 | }) 128 | ``` 129 | 130 | The final argument to the RegisterType function is a [HandlerFunc](http://godoc.org/github.com/albrow/jobs#HandlerFunc) 131 | which will be executed when the job runs. HandlerFunc must be a function which accepts either 132 | zero or one arguments and returns an error. 133 | 134 | ### Scheduling a Job 135 | 136 | After registering a job type, you can schedule a job using the Schedule or ScheduleRecurring 137 | methods like so: 138 | 139 | ``` go 140 | // The priority argument lets you choose how important the job is. Higher 141 | // priority jobs will be executed first. 142 | job, err := welcomeEmailJobs.Schedule(100, time.Now(), &User{EmailAddress: "foo@example.com"}) 143 | if err != nil { 144 | // Handle err 145 | } 146 | ``` 147 | 148 | You can use the [Job object](http://godoc.org/github.com/albrow/jobs#Job) returned by Schedule 149 | or ScheduleRecurring to check on the status of the job or cancel it manually. 150 | 151 | ### Starting and Configuring Worker Pools 152 | 153 | You can schedule any number of worker pools across any number of machines, provided every machine 154 | agrees on the definition of the job types. If you want, you can start a worker pool on the same 155 | machines that are scheduling jobs, or you can have each worker pool running on a designated machine. 156 | Since each pool is assigned an id based on a unique hardware identifier, you must only run one 157 | worker pool per machine. 158 | 159 | To create a new pool with the [default configuration](http://godoc.org/github.com/albrow/jobs#pkg-variables), 160 | just pass in nil: 161 | 162 | ``` go 163 | pool, err := jobs.NewPool(nil) 164 | if err != nil { 165 | // Handle err 166 | } 167 | ``` 168 | 169 | You can also specify a different configuration by passing in 170 | [*PoolConfig](http://godoc.org/github.com/albrow/jobs#PoolConfig). Any zero values in the config you pass 171 | in will fallback to the default values. So here's how you could start a pool with 10 workers and a batch 172 | size of 10, while letting the other options remain the default. 173 | 174 | ``` go 175 | pool, err := jobs.NewPool(&jobs.PoolConfig{ 176 | NumWorkers: 10, 177 | BatchSize: 10, 178 | }) 179 | if err != nil { 180 | // Handle err 181 | } 182 | ``` 183 | 184 | After you have created a pool, you can start it with the Start method. Once started, the pool will 185 | continuously query the database for new jobs and delegate those jobs to workers. Any program that calls 186 | Pool.Start() should also wait for the workers to finish before exiting. You can do so by wrapping Close and 187 | Wait in a defer statement. Typical usage looks something like this: 188 | 189 | ``` go 190 | func main() { 191 | pool, err := jobs.NewPool(nil) 192 | if err != nil { 193 | // Handle err 194 | } 195 | defer func() { 196 | pool.Close() 197 | if err := pool.Wait(); err != nil { 198 | // Handle err 199 | } 200 | }() 201 | if err := pool.Start(); err != nil { 202 | // Handle err 203 | } 204 | } 205 | ``` 206 | 207 | You can also call Close and Wait at any time to manually stop the pool from executing new jobs. In this 208 | case, any jobs that are currently being executed will still finish. 209 | 210 | 211 | Testing 212 | ------- 213 | 214 | To run the tests, make sure you have Redis running and accepting unix socket connections on the address 215 | /tmp/redis.sock. The tests will use database #14. **WARNING:** After each test is run, database #14 will be completely 216 | erased, so make sure you do not have any important data stored there. 217 | 218 | To run the tests just run `go test .` If anything fails, please report an issue and describe what happened. 219 | 220 | 221 | Contributing 222 | ------------ 223 | 224 | See [Contributing.md](https://github.com/albrow/jobs/blob/master/CONTRIBUTING.md) 225 | 226 | 227 | Guarantees 228 | ----------- 229 | 230 | ### Persistence 231 | 232 | Since jobs is powered by Redis, there is a chance that you can lose data with the default Redis configuration. 233 | To get the best persistence guarantees, you should set Redis to use both AOF and RDB persistence modes and set 234 | fsync to "always". With these settings, Redis is more or less 235 | [as persistent as a database like postgres](http://redis.io/topics/persistence#ok-so-what-should-i-use). If want 236 | better performance and are okay with a slightly greater chance of losing data (i.e. jobs not executing), you can 237 | set fsync to "everysec". 238 | 239 | [Read more about Redis persistence](http://redis.io/topics/persistence). 240 | 241 | ### Atomicity 242 | 243 | Jobs is carefully written using Redis transactions and lua scripting so that all database changes are atomic. 244 | If Redis crashes in the middle of a transaction or script execution, it is possible that your AOF file can become 245 | corrupted. If this happens, Redis will refuse to start until the AOF file is fixed. It is relatively easy to fix 246 | the problem with the redis-check-aof tool, which will remove the partial transaction from the AOF file. In effect, 247 | this guarantees that modifications of the database are atomic, even in the event of a power loss or hard reset, 248 | with the caveat that you may need to use the redis-check-aof tool in the worst case scenario. 249 | 250 | Read more about [Redis transactions](http://redis.io/topics/transactions) and 251 | [scripts](http://redis.io/commands#scripting). 252 | 253 | ### Job Execution 254 | 255 | Jobs guarantees that a job will be executed *at least* once, provided it has been persisted on disk. (See the section 256 | on [Persistence](#persistence) directly above). A job can only picked up by one pool at a time because a pool 257 | atomically pops (gets and immediately moves) the next available jobs from the database. A job can only be executed 258 | by one worker at a time because the jobs are delegated to workers via a shared channel. Each worker pool checks on 259 | the health of all the other pools when it starts. If a pool crashes or is otherwise disconnected, any jobs it had 260 | grabbed from the database that did not yet finish will be re-queued and picked up by a different pool. 261 | 262 | This is in no way an exhaustive list, but here are some known examples of scenarios that may cause a job to be 263 | executed more than once: 264 | 265 | 1. If there is a power failure or hard reset while a worker is in the middle of executing a job, the job may be 266 | stuck in a half-executed state. Since there is no way to know how much of the job was successfully completed, 267 | the job will be re-queued and picked up by a different pool, where it may be partially or fully executed 268 | more than once. 269 | 2. If a pool becomes disconnected, it will be considered stale and its jobs will be re-queued and reclaimed 270 | by a different pool. However, if the stale pool is able to partly or fully execute jobs without a reliable 271 | internet connection, any jobs belonging to the stale pool might be executed more than once. You can increase 272 | the [StaleTimeout](https://godoc.org/github.com/albrow/jobs#PoolConfig) parameter for a pool to make this 273 | scenario less likely. 274 | 275 | 276 | License 277 | ------- 278 | 279 | Jobs is licensed under the MIT License. See the LICENSE file for more information. 280 | -------------------------------------------------------------------------------- /config.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import () 8 | 9 | // configType holds different config variables 10 | type configType struct { 11 | Db databaseConfig 12 | } 13 | 14 | // databaseConfig holds config variables specific to the database 15 | type databaseConfig struct { 16 | Address string 17 | Network string 18 | Database int 19 | Password string 20 | } 21 | 22 | // Config is where all configuration variables are stored. You may modify Config 23 | // directly in order to change config variables, and should only do so at the start 24 | // of your program. 25 | var Config = configType{ 26 | Db: databaseConfig{ 27 | // Address is the address of the redis database to connect to. Default is 28 | // "localhost:6379". 29 | Address: "localhost:6379", 30 | // Network is the type of network to use to connect to the redis database 31 | // Default is "tcp". 32 | Network: "tcp", 33 | // Database is the redis database number to use for storing all data. Default 34 | // is 0. 35 | Database: 0, 36 | // Password is a password to use for connecting to a redis database via the 37 | // AUTH command. If empty, Jobs will not attempt to authenticate. Default is 38 | // "" (an empty string). 39 | Password: "", 40 | }, 41 | } 42 | -------------------------------------------------------------------------------- /doc.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | // Package jobs is a persistent and flexible background jobs library. 6 | // 7 | // Version: 0.4.2 8 | // 9 | // Jobs is powered by redis and supports the following features: 10 | // 11 | // - A job can encapsulate any arbitrary functionality. A job can do anything 12 | // which can be done in a go function. 13 | // - A job can be one-off (only executed once) or recurring (scheduled to 14 | // execute at a specific interval). 15 | // - A job can be retried a specified number of times if it fails. 16 | // - A job is persistent, with protections against power loss and other worst 17 | // case scenarios. 18 | // - Jobs can be executed by any number of concurrent workers accross any 19 | // number of machines. 20 | // - Provided it is persisted to disk, every job will be executed *at least* once, 21 | // and in ideal network conditions will be executed *exactly* once. 22 | // - You can query the database to find out e.g. the number of jobs that are 23 | // currently executing or how long a particular job took to execute. 24 | // - Any job that permanently fails will have its error captured and stored. 25 | // 26 | // Why is it Useful 27 | // 28 | // Jobs is intended to be used in web applications. It is useful for cases where you need 29 | // to execute some long-running code, but you don't want your users to wait for the code to 30 | // execute before rendering a response. A good example is sending a welcome email to your users 31 | // after they sign up. You can use Jobs to schedule the email to be sent asynchronously, and 32 | // render a response to your user without waiting for the email to be sent. You could use a 33 | // goroutine to accomplish the same thing, but in the event of a server restart or power loss, 34 | // the email might never be sent. Jobs guarantees that the email will be sent at some time, 35 | // and allows you to spread the work between different machines. 36 | // 37 | // 38 | // More Information 39 | // 40 | // Visit https://github.com/albrow/jobs for a Quickstart Guide, code examples, and more 41 | // information. 42 | // 43 | package jobs 44 | -------------------------------------------------------------------------------- /encode.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "bytes" 9 | "encoding/gob" 10 | "fmt" 11 | "reflect" 12 | ) 13 | 14 | // decode decodes a slice of bytes and scans the value into dest using the gob package. 15 | // All types are supported except recursive data structures and functions. 16 | func decode(reply []byte, dest interface{}) error { 17 | // Check the type of dest and make sure it is a pointer to something, 18 | // otherwise we can't set its value in any meaningful way. 19 | val := reflect.ValueOf(dest) 20 | if val.Kind() != reflect.Ptr { 21 | return fmt.Errorf("jobs: Argument to decode must be pointer. Got %T", dest) 22 | } 23 | 24 | // Use the gob package to decode the reply and write the result into 25 | // dest. 26 | buf := bytes.NewBuffer(reply) 27 | dec := gob.NewDecoder(buf) 28 | if err := dec.DecodeValue(val.Elem()); err != nil { 29 | return err 30 | } 31 | return nil 32 | } 33 | 34 | // encode encodes data into a slice of bytes using the gob package. 35 | // All types are supported except recursive data structures and functions. 36 | func encode(data interface{}) ([]byte, error) { 37 | if data == nil { 38 | return nil, nil 39 | } 40 | buf := bytes.NewBuffer([]byte{}) 41 | enc := gob.NewEncoder(buf) 42 | if err := enc.Encode(data); err != nil { 43 | return nil, err 44 | } 45 | return buf.Bytes(), nil 46 | } 47 | -------------------------------------------------------------------------------- /encode_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "reflect" 9 | "testing" 10 | ) 11 | 12 | // NOTE: 13 | // I know this code isn't very dry. Unfortunately, this is due to a restriction 14 | // in reflect and gob packages. We can't have a general test case or test function 15 | // that treats different types as interfaces, because then gob doesn't know what type 16 | // to attempt to decode into. So we have to test the different types manually. 17 | 18 | func TestConvertInt(t *testing.T) { 19 | v := int(7) 20 | // Encode v to a slice of bytes 21 | reply, err := encode(v) 22 | if err != nil { 23 | t.Errorf("Unexpected error in encode: %s", err.Error()) 24 | } 25 | // Decode reply and write results to the holder 26 | holder := int(0) 27 | if err := decode(reply, &holder); err != nil { 28 | t.Errorf("Unexpected error in decode: %s", err.Error()) 29 | } 30 | // Now the holder and the original should be equal. If they're not, 31 | // there was a problem 32 | expectEncodeDecodeEquals(t, v, holder) 33 | } 34 | 35 | func TestConvertString(t *testing.T) { 36 | v := "test" 37 | // Encode v to a slice of bytes 38 | reply, err := encode(v) 39 | if err != nil { 40 | t.Errorf("Unexpected error in encode: %s", err.Error()) 41 | } 42 | // Decode reply and write results to the holder 43 | holder := "" 44 | if err := decode(reply, &holder); err != nil { 45 | t.Errorf("Unexpected error in decode: %s", err.Error()) 46 | } 47 | // Now the holder and the original should be equal. If they're not, 48 | // there was a problem 49 | expectEncodeDecodeEquals(t, v, holder) 50 | } 51 | 52 | func TestConvertBool(t *testing.T) { 53 | v := true 54 | // Encode v to a slice of bytes 55 | reply, err := encode(v) 56 | if err != nil { 57 | t.Errorf("Unexpected error in encode: %s", err.Error()) 58 | } 59 | // Decode reply and write results to the holder 60 | holder := false 61 | if err := decode(reply, &holder); err != nil { 62 | t.Errorf("Unexpected error in decode: %s", err.Error()) 63 | } 64 | // Now the holder and the original should be equal. If they're not, 65 | // there was a problem 66 | expectEncodeDecodeEquals(t, v, holder) 67 | } 68 | 69 | func TestConvertStruct(t *testing.T) { 70 | v := struct { 71 | Name string 72 | Age int 73 | }{"test person", 23} 74 | // Encode v to a slice of bytes 75 | reply, err := encode(v) 76 | if err != nil { 77 | t.Errorf("Unexpected error in encode: %s", err.Error()) 78 | } 79 | // Decode reply and write results to the holder 80 | holder := struct { 81 | Name string 82 | Age int 83 | }{} 84 | if err := decode(reply, &holder); err != nil { 85 | t.Errorf("Unexpected error in decode: %s", err.Error()) 86 | } 87 | // Now the holder and the original should be equal. If they're not, 88 | // there was a problem 89 | expectEncodeDecodeEquals(t, v, holder) 90 | } 91 | 92 | func TestConvertSlice(t *testing.T) { 93 | v := []string{"a", "b", "c"} 94 | // Encode v to a slice of bytes 95 | reply, err := encode(v) 96 | if err != nil { 97 | t.Errorf("Unexpected error in encode: %s", err.Error()) 98 | } 99 | // Decode reply and write results to the holder 100 | holder := []string{} 101 | if err := decode(reply, &holder); err != nil { 102 | t.Errorf("Unexpected error in decode: %s", err.Error()) 103 | } 104 | // Now the holder and the original should be equal. If they're not, 105 | // there was a problem 106 | expectEncodeDecodeEquals(t, v, holder) 107 | } 108 | 109 | func expectEncodeDecodeEquals(t *testing.T, expected, got interface{}) { 110 | if !reflect.DeepEqual(expected, got) { 111 | t.Errorf("Error encoding/decoding type %T. Expected %v but got %v.", expected, expected, got) 112 | } 113 | } 114 | -------------------------------------------------------------------------------- /job.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "fmt" 9 | "github.com/garyburd/redigo/redis" 10 | "time" 11 | ) 12 | 13 | // Job represents a discrete piece of work to be done by a worker. 14 | type Job struct { 15 | id string 16 | data []byte 17 | typ *Type 18 | status Status 19 | time int64 20 | freq int64 21 | priority int 22 | err error 23 | retries uint 24 | started int64 25 | finished int64 26 | poolId string 27 | } 28 | 29 | // ErrorJobNotFound is returned whenever a specific job is not found, 30 | // e.g. from the FindById function. 31 | type ErrorJobNotFound struct { 32 | id string 33 | } 34 | 35 | func (e ErrorJobNotFound) Error() string { 36 | if e.id == "" { 37 | return fmt.Sprintf("jobs: Could not find job with the given criteria.") 38 | } 39 | return fmt.Sprintf("jobs: Could not find job with id: %s", e.id) 40 | } 41 | 42 | // Id returns the unique identifier used for the job. If the job has not yet 43 | // been saved to the database, it may return an empty string. 44 | func (j *Job) Id() string { 45 | return j.id 46 | } 47 | 48 | // Data returns the gob-encoded data of the job 49 | func (j *Job) Data() []byte { 50 | return j.data 51 | } 52 | 53 | // Status returns the status of the job. 54 | func (j *Job) Status() Status { 55 | return j.status 56 | } 57 | 58 | // Time returns the time at which the job should be executed in UTC UNIX 59 | // format with nanosecond precision. 60 | func (j *Job) Time() int64 { 61 | return j.time 62 | } 63 | 64 | // Freq returns the frequency at which the job should be executed. Specifically 65 | // it returns the number of nanoseconds between each scheduled execution. 66 | func (j *Job) Freq() int64 { 67 | return j.freq 68 | } 69 | 70 | // Priority returns the job's priority. 71 | func (j *Job) Priority() int { 72 | return j.priority 73 | } 74 | 75 | // Error returns the last error that arose during execution of the job. It is 76 | // only non-nil if the job has failed at some point. 77 | func (j *Job) Error() error { 78 | return j.err 79 | } 80 | 81 | // Retries returns the number of remaining retries for the job. 82 | func (j *Job) Retries() uint { 83 | return j.retries 84 | } 85 | 86 | // Started returns the time that the job started executing (in local time 87 | // with nanosecond precision) or the zero time if the job has not started 88 | // executing yet. 89 | func (j *Job) Started() time.Time { 90 | return time.Unix(0, j.started).Local() 91 | } 92 | 93 | // Finished returns the time that the job finished executing (in local 94 | // time with nanosecond precision) or the zero time if the job has not 95 | // finished executing yet. 96 | func (j *Job) Finished() time.Time { 97 | return time.Unix(0, j.finished).Local() 98 | } 99 | 100 | // PoolId returns the pool id of the job if it is currently being executed 101 | // or has been executed and at some point has been assigned to a specific pool. 102 | // Otherwise, it returns an empty string. 103 | func (j *Job) PoolId() string { 104 | return j.poolId 105 | } 106 | 107 | // Duration returns how long the job took to execute with nanosecond 108 | // precision. I.e. the difference between j.Finished() and j.Started(). 109 | // It returns a duration of zero if the job has not finished yet. 110 | func (j *Job) Duration() time.Duration { 111 | if j.Finished().IsZero() { 112 | return 0 * time.Second 113 | } 114 | return j.Finished().Sub(j.Started()) 115 | } 116 | 117 | // Key returns the key used for the hash in redis which stores all the 118 | // fields for this job. 119 | func (j *Job) Key() string { 120 | return "jobs:" + j.id 121 | } 122 | 123 | // IsRecurring returns true iff the job is recurring 124 | func (j *Job) IsRecurring() bool { 125 | return j.freq != 0 126 | } 127 | 128 | // NextTime returns the time (unix UTC with nanosecond precision) that the 129 | // job should execute next, if it is a recurring job, and 0 if it is not. 130 | func (j *Job) NextTime() int64 { 131 | if !j.IsRecurring() { 132 | return 0 133 | } 134 | // NOTE: is this the proper way to handle rescheduling? 135 | // What if we schedule jobs faster than they can be executed? 136 | // Should we just let them build up and expect the end user to 137 | // allocate more workers? Or should we schedule for time.Now at 138 | // the earliest to prevent buildup? 139 | return j.time + j.freq 140 | } 141 | 142 | // save writes the job to the database and adds it to the appropriate indexes and status 143 | // sets, but does not enqueue it. If you want to add it to the queue, use the enqueue method 144 | // after save. 145 | func (j *Job) save() error { 146 | t := newTransaction() 147 | t.saveJob(j) 148 | if err := t.exec(); err != nil { 149 | return err 150 | } 151 | return nil 152 | } 153 | 154 | // saveJob adds commands to the transaction to set all the fields for the main hash for the job, 155 | // add the job to the time index, move the job to the appropriate status set. It will 156 | // also mutate the job by 1) generating an id if the id is empty and 2) setting the status to 157 | // StatusSaved if the status is empty. 158 | func (t *transaction) saveJob(job *Job) { 159 | // Generate id if needed 160 | if job.id == "" { 161 | job.id = generateRandomId() 162 | } 163 | // Set status to saved if needed 164 | if job.status == "" { 165 | job.status = StatusSaved 166 | } 167 | // Add the job attributes to a hash 168 | t.command("HMSET", job.mainHashArgs(), nil) 169 | // Add the job to the appropriate status set 170 | t.setStatus(job, job.status) 171 | // Add the job to the time index 172 | t.addJobToTimeIndex(job) 173 | } 174 | 175 | // addJobToTimeIndex adds commands to the transaction which will, when executed, 176 | // add the job id to the time index with a score equal to the job's time field. 177 | // If the job has been destroyed, addJobToTimeIndex will have no effect. 178 | func (t *transaction) addJobToTimeIndex(job *Job) { 179 | t.addJobToSet(job, Keys.JobsTimeIndex, float64(job.time)) 180 | } 181 | 182 | // Refresh mutates the job by setting its fields to the most recent data 183 | // found in the database. It returns an error if there was a problem connecting 184 | // to the database or if the job was destroyed. 185 | func (j *Job) Refresh() error { 186 | t := newTransaction() 187 | t.scanJobById(j.id, j) 188 | if err := t.exec(); err != nil { 189 | return err 190 | } 191 | return nil 192 | } 193 | 194 | // enqueue adds the job to the queue and sets its status to StatusQueued. Queued jobs will 195 | // be completed by workers in order of priority. Attempting to enqueue a destroyed job 196 | // will have no effect. 197 | func (j *Job) enqueue() error { 198 | if err := j.setStatus(StatusQueued); err != nil { 199 | return err 200 | } 201 | return nil 202 | } 203 | 204 | // Reschedule reschedules the job with the given time. It can be used to reschedule 205 | // cancelled jobs. It may also be used to reschedule finished or failed jobs, however, 206 | // in most cases if you want to reschedule finished jobs you should use the ScheduleRecurring 207 | // method and if you want to reschedule failed jobs, you should set the number of retries > 0 208 | // when registering the job type. Attempting to reschedule a destroyed job will have no effect. 209 | // Reschedule returns an error if there was a problem connecting to the database. 210 | func (j *Job) Reschedule(time time.Time) error { 211 | t := newTransaction() 212 | unixNanoTime := time.UTC().UnixNano() 213 | t.setJobField(j, "time", unixNanoTime) 214 | t.setStatus(j, StatusQueued) 215 | j.time = unixNanoTime 216 | t.addJobToTimeIndex(j) 217 | if err := t.exec(); err != nil { 218 | return err 219 | } 220 | j.status = StatusQueued 221 | return nil 222 | } 223 | 224 | // Cancel cancels the job, but does not remove it from the database. It will be 225 | // added to a list of cancelled jobs. If you wish to remove it from the database, 226 | // use the Destroy method. Attempting to cancel a destroyed job will have no effect. 227 | func (j *Job) Cancel() error { 228 | if err := j.setStatus(StatusCancelled); err != nil { 229 | return err 230 | } 231 | return nil 232 | } 233 | 234 | // setError sets the err property of j and adds it to the set of jobs which had errors. 235 | // If the job has been destroyed, setError will have no effect. 236 | func (j *Job) setError(err error) error { 237 | j.err = err 238 | t := newTransaction() 239 | t.setJobField(j, "error", j.err.Error()) 240 | if err := t.exec(); err != nil { 241 | return err 242 | } 243 | return nil 244 | } 245 | 246 | // Destroy removes all traces of the job from the database. If the job is currently 247 | // being executed by a worker, the worker may still finish the job. Attempting to 248 | // destroy a job that has already been destroyed will have no effect, so it is safe 249 | // to call Destroy multiple times. 250 | func (j *Job) Destroy() error { 251 | if j.id == "" { 252 | return fmt.Errorf("jobs: Cannot destroy job that doesn't have an id.") 253 | } 254 | // Start a new transaction 255 | t := newTransaction() 256 | // Call the script to destroy the job 257 | t.destroyJob(j) 258 | // Execute the transaction 259 | if err := t.exec(); err != nil { 260 | return err 261 | } 262 | j.status = StatusDestroyed 263 | return nil 264 | } 265 | 266 | // setStatus updates the job's status in the database and moves it to the appropriate 267 | // status set. Attempting to set the status of a job which has been destroyed will have 268 | // no effect. 269 | func (j *Job) setStatus(status Status) error { 270 | if j.id == "" { 271 | return fmt.Errorf("jobs: Cannot set status to %s because job doesn't have an id.", status) 272 | } 273 | if j.status == StatusDestroyed { 274 | return fmt.Errorf("jobs: Cannot set job:%s status to %s because it was destroyed.", j.id, status) 275 | } 276 | // Use a transaction to move the job to the appropriate status set and set its status 277 | t := newTransaction() 278 | t.setStatus(j, status) 279 | if err := t.exec(); err != nil { 280 | return err 281 | } 282 | j.status = status 283 | return nil 284 | } 285 | 286 | // mainHashArgs returns the args for the hash which will store the job data 287 | func (j *Job) mainHashArgs() []interface{} { 288 | hashArgs := []interface{}{j.Key(), 289 | "data", string(j.data), 290 | "type", j.typ.name, 291 | "time", j.time, 292 | "freq", j.freq, 293 | "priority", j.priority, 294 | "retries", j.retries, 295 | "status", j.status, 296 | "started", j.started, 297 | "finished", j.finished, 298 | "poolId", j.poolId, 299 | } 300 | if j.err != nil { 301 | hashArgs = append(hashArgs, "error", j.err.Error()) 302 | } 303 | return hashArgs 304 | } 305 | 306 | // scanJob scans the values of reply into job. reply should be the 307 | // response of an HMGET or HGETALL query. 308 | func scanJob(reply interface{}, job *Job) error { 309 | fields, err := redis.Values(reply, nil) 310 | if err != nil { 311 | return err 312 | } else if len(fields) == 0 { 313 | return ErrorJobNotFound{} 314 | } else if len(fields)%2 != 0 { 315 | return fmt.Errorf("jobs: In scanJob: Expected length of fields to be even but got: %d", len(fields)) 316 | } 317 | for i := 0; i < len(fields)-1; i += 2 { 318 | fieldName, err := redis.String(fields[i], nil) 319 | if err != nil { 320 | return fmt.Errorf("jobs: In scanJob: Could not convert fieldName (fields[%d] = %v) of type %T to string.", i, fields[i], fields[i]) 321 | } 322 | fieldValue := fields[i+1] 323 | switch fieldName { 324 | case "id": 325 | if err := scanString(fieldValue, &(job.id)); err != nil { 326 | return err 327 | } 328 | case "data": 329 | if err := scanBytes(fieldValue, &(job.data)); err != nil { 330 | return err 331 | } 332 | case "type": 333 | typeName := "" 334 | if err := scanString(fieldValue, &typeName); err != nil { 335 | return err 336 | } 337 | Type, found := Types[typeName] 338 | if !found { 339 | return fmt.Errorf("jobs: In scanJob: Could not find Type with name = %s", typeName) 340 | } 341 | job.typ = Type 342 | case "time": 343 | if err := scanInt64(fieldValue, &(job.time)); err != nil { 344 | return err 345 | } 346 | case "freq": 347 | if err := scanInt64(fieldValue, &(job.freq)); err != nil { 348 | return err 349 | } 350 | case "priority": 351 | if err := scanInt(fieldValue, &(job.priority)); err != nil { 352 | return err 353 | } 354 | case "retries": 355 | if err := scanUint(fieldValue, &(job.retries)); err != nil { 356 | return err 357 | } 358 | case "status": 359 | status := "" 360 | if err := scanString(fieldValue, &status); err != nil { 361 | return err 362 | } 363 | job.status = Status(status) 364 | case "started": 365 | if err := scanInt64(fieldValue, &(job.started)); err != nil { 366 | return err 367 | } 368 | case "finished": 369 | if err := scanInt64(fieldValue, &(job.finished)); err != nil { 370 | return err 371 | } 372 | case "poolId": 373 | if err := scanString(fieldValue, &(job.poolId)); err != nil { 374 | return err 375 | } 376 | } 377 | } 378 | return nil 379 | } 380 | 381 | // scanInt converts a reply from redis into an int and scans the value into v. 382 | func scanInt(reply interface{}, v *int) error { 383 | if v == nil { 384 | return fmt.Errorf("jobs: In scanInt: argument v was nil") 385 | } 386 | val, err := redis.Int(reply, nil) 387 | if err != nil { 388 | return fmt.Errorf("jobs: In scanInt: Could not convert %v of type %T to int.", reply, reply) 389 | } 390 | (*v) = val 391 | return nil 392 | } 393 | 394 | // scanUint converts a reply from redis into a uint and scans the value into v. 395 | func scanUint(reply interface{}, v *uint) error { 396 | if v == nil { 397 | return fmt.Errorf("jobs: In scanUint: argument v was nil") 398 | } 399 | val, err := redis.Uint64(reply, nil) 400 | if err != nil { 401 | return fmt.Errorf("jobs: In scanUint: Could not convert %v of type %T to uint.", reply, reply) 402 | } 403 | (*v) = uint(val) 404 | return nil 405 | } 406 | 407 | // scanInt64 converts a reply from redis into an int64 and scans the value into v. 408 | func scanInt64(reply interface{}, v *int64) error { 409 | if v == nil { 410 | return fmt.Errorf("jobs: In scanInt64: argument v was nil") 411 | } 412 | val, err := redis.Int64(reply, nil) 413 | if err != nil { 414 | return fmt.Errorf("jobs: In scanInt64: Could not convert %v of type %T to int64.", reply, reply) 415 | } 416 | (*v) = val 417 | return nil 418 | } 419 | 420 | // scanString converts a reply from redis into a string and scans the value into v. 421 | func scanString(reply interface{}, v *string) error { 422 | if v == nil { 423 | return fmt.Errorf("jobs: In String: argument v was nil") 424 | } 425 | val, err := redis.String(reply, nil) 426 | if err != nil { 427 | return fmt.Errorf("jobs: In String: Could not convert %v of type %T to string.", reply, reply) 428 | } 429 | (*v) = val 430 | return nil 431 | } 432 | 433 | // scanBytes converts a reply from redis into a slice of bytes and scans the value into v. 434 | func scanBytes(reply interface{}, v *[]byte) error { 435 | if v == nil { 436 | return fmt.Errorf("jobs: In scanBytes: argument v was nil") 437 | } 438 | val, err := redis.Bytes(reply, nil) 439 | if err != nil { 440 | return fmt.Errorf("jobs: In scanBytes: Could not convert %v of type %T to []byte.", reply, reply) 441 | } 442 | (*v) = val 443 | return nil 444 | } 445 | 446 | // scanJobById adds commands and a reply handler to the transaction which, when run, 447 | // will scan the values of the job corresponding to id into job. It does not execute 448 | // the transaction. 449 | func (t *transaction) scanJobById(id string, job *Job) { 450 | job.id = id 451 | t.command("HGETALL", redis.Args{job.Key()}, newScanJobHandler(job)) 452 | } 453 | 454 | // FindById returns the job with the given id or an error if the job cannot be found 455 | // (in which case the error will have type ErrorJobNotFound) or there was a problem 456 | // connecting to the database. 457 | func FindById(id string) (*Job, error) { 458 | job := &Job{} 459 | t := newTransaction() 460 | t.scanJobById(id, job) 461 | if err := t.exec(); err != nil { 462 | switch e := err.(type) { 463 | case ErrorJobNotFound: 464 | // If the job was not found, add the id to the error 465 | // so that the caller can get a more useful error message. 466 | e.id = id 467 | return nil, e 468 | default: 469 | return nil, err 470 | } 471 | } 472 | return job, nil 473 | } 474 | -------------------------------------------------------------------------------- /job_status.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "github.com/garyburd/redigo/redis" 9 | ) 10 | 11 | // Status represents the different statuses a job can have. 12 | type Status string 13 | 14 | const ( 15 | // StatusSaved is the status of any job that has been saved into the database but not yet queued 16 | StatusSaved Status = "saved" 17 | // StatusQueued is the status of any job that has been queued for execution but not yet selected 18 | StatusQueued Status = "queued" 19 | // StatusExecuting is the status of any job that has been selected for execution and is being delegated 20 | // to some worker and any job that is currently being executed by some worker. 21 | StatusExecuting Status = "executing" 22 | // StatusFinished is the status of any job that has been successfully executed. 23 | StatusFinished Status = "finished" 24 | // StatusFailed is the status of any job that failed to execute and for which there are no remaining retries. 25 | StatusFailed Status = "failed" 26 | // StatusCancelled is the status of any job that was manually cancelled. 27 | StatusCancelled Status = "cancelled" 28 | // StatusDestroyed is the status of any job that has been destroyed, i.e. completely removed 29 | // from the database. 30 | StatusDestroyed Status = "destroyed" 31 | ) 32 | 33 | // key returns the key used for the sorted set in redis which will hold 34 | // all jobs with this status. 35 | func (status Status) Key() string { 36 | return "jobs:" + string(status) 37 | } 38 | 39 | // Count returns the number of jobs that currently have the given status 40 | // or an error if there was a problem connecting to the database. 41 | func (status Status) Count() (int, error) { 42 | conn := redisPool.Get() 43 | defer conn.Close() 44 | return redis.Int(conn.Do("ZCARD", status.Key())) 45 | } 46 | 47 | // JobIds returns the ids of all jobs that have the given status, ordered by 48 | // priority or an error if there was a problem connecting to the database. 49 | func (status Status) JobIds() ([]string, error) { 50 | conn := redisPool.Get() 51 | defer conn.Close() 52 | return redis.Strings(conn.Do("ZREVRANGE", status.Key(), 0, -1)) 53 | } 54 | 55 | // Jobs returns all jobs that have the given status, ordered by priority or 56 | // an error if there was a problem connecting to the database. 57 | func (status Status) Jobs() ([]*Job, error) { 58 | t := newTransaction() 59 | jobs := []*Job{} 60 | t.getJobsByIds(status.Key(), newScanJobsHandler(&jobs)) 61 | if err := t.exec(); err != nil { 62 | return nil, err 63 | } 64 | return jobs, nil 65 | } 66 | 67 | // possibleStatuses is simply an array of all the possible job statuses. 68 | var possibleStatuses = []Status{ 69 | StatusSaved, 70 | StatusQueued, 71 | StatusExecuting, 72 | StatusFinished, 73 | StatusFailed, 74 | StatusCancelled, 75 | StatusDestroyed, 76 | } 77 | -------------------------------------------------------------------------------- /job_status_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "reflect" 9 | "testing" 10 | ) 11 | 12 | func TestStatusCount(t *testing.T) { 13 | testingSetUp() 14 | defer testingTeardown() 15 | jobs, err := createAndSaveTestJobs(5) 16 | if err != nil { 17 | t.Fatalf("Unexpected error: %s", err) 18 | } 19 | for _, status := range possibleStatuses { 20 | if status == StatusDestroyed { 21 | // Skip this one, since destroying a job means erasing all records from the database 22 | continue 23 | } 24 | for _, job := range jobs { 25 | job.setStatus(status) 26 | } 27 | count, err := status.Count() 28 | if err != nil { 29 | t.Errorf("Unexpected error in status.Count(): %s", err.Error()) 30 | } 31 | if count != len(jobs) { 32 | t.Errorf("Expected %s.Count() to return %d after setting job statuses to %s, but got %d", status, len(jobs), status, count) 33 | } 34 | } 35 | } 36 | 37 | func TestStatusJobIds(t *testing.T) { 38 | testingSetUp() 39 | defer testingTeardown() 40 | jobs, err := createAndSaveTestJobs(5) 41 | if err != nil { 42 | t.Fatalf("Unexpected error: %s", err) 43 | } 44 | jobIds := make([]string, len(jobs)) 45 | for i, job := range jobs { 46 | jobIds[i] = job.id 47 | } 48 | for _, status := range possibleStatuses { 49 | if status == StatusDestroyed { 50 | // Skip this one, since destroying a job means erasing all records from the database 51 | continue 52 | } 53 | for _, job := range jobs { 54 | job.setStatus(status) 55 | } 56 | gotIds, err := status.JobIds() 57 | if err != nil { 58 | t.Errorf("Unexpected error in status.JobIds(): %s", err.Error()) 59 | } 60 | if len(gotIds) != len(jobIds) { 61 | t.Errorf("%s.JobIds() was incorrect. Expected slice of length %d but got %d", len(jobIds), len(gotIds)) 62 | } 63 | if !reflect.DeepEqual(jobIds, gotIds) { 64 | t.Errorf("%s.JobIds() was incorrect. Expected %v but got %v", status, jobIds, gotIds) 65 | } 66 | } 67 | } 68 | 69 | func TestStatusJobs(t *testing.T) { 70 | testingSetUp() 71 | defer testingTeardown() 72 | jobs, err := createAndSaveTestJobs(5) 73 | if err != nil { 74 | t.Fatalf("Unexpected error: %s", err) 75 | } 76 | for _, status := range possibleStatuses { 77 | if status == StatusDestroyed { 78 | // Skip this one, since destroying a job means erasing all records from the database 79 | continue 80 | } 81 | for _, job := range jobs { 82 | job.setStatus(status) 83 | } 84 | gotJobs, err := status.Jobs() 85 | if err != nil { 86 | t.Errorf("Unexpected error in status.Jobs(): %s", err.Error()) 87 | } 88 | if len(gotJobs) != len(jobs) { 89 | t.Errorf("%s.Jobs() was incorrect. Expected slice of length %d but got %d", len(jobs), len(gotJobs)) 90 | } 91 | if !reflect.DeepEqual(jobs, gotJobs) { 92 | t.Errorf("%s.Jobs() was incorrect. Expected %v but got %v", status, jobs, gotJobs) 93 | } 94 | } 95 | } 96 | -------------------------------------------------------------------------------- /job_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "errors" 9 | "reflect" 10 | "strings" 11 | "testing" 12 | "time" 13 | ) 14 | 15 | func TestJobSave(t *testing.T) { 16 | testingSetUp() 17 | defer testingTeardown() 18 | 19 | // Create and save a test job 20 | job, err := createTestJob() 21 | if err != nil { 22 | t.Fatal(err) 23 | } 24 | job.started = 1 25 | job.finished = 5 26 | job.freq = 10 27 | job.retries = 3 28 | job.poolId = "testPool" 29 | if err := job.save(); err != nil { 30 | t.Errorf("Unexpected error saving job: %s", err.Error()) 31 | } 32 | 33 | // Make sure the main hash was saved correctly 34 | expectJobFieldEquals(t, job, "data", job.data, nil) 35 | expectJobFieldEquals(t, job, "type", job.typ.name, stringConverter) 36 | expectJobFieldEquals(t, job, "time", job.time, int64Converter) 37 | expectJobFieldEquals(t, job, "freq", job.freq, int64Converter) 38 | expectJobFieldEquals(t, job, "priority", job.priority, intConverter) 39 | expectJobFieldEquals(t, job, "started", job.started, int64Converter) 40 | expectJobFieldEquals(t, job, "finished", job.finished, int64Converter) 41 | expectJobFieldEquals(t, job, "retries", job.retries, uintConverter) 42 | expectJobFieldEquals(t, job, "poolId", job.poolId, stringConverter) 43 | 44 | // Make sure the job status was correct 45 | expectStatusEquals(t, job, StatusSaved) 46 | 47 | // Make sure the job was indexed by its time correctly 48 | expectJobInTimeIndex(t, job) 49 | } 50 | 51 | func TestJobFindById(t *testing.T) { 52 | testingSetUp() 53 | defer testingTeardown() 54 | 55 | // Create and save a test job 56 | job, err := createTestJob() 57 | if err != nil { 58 | t.Fatal(err) 59 | } 60 | job.started = 1 61 | job.finished = 5 62 | job.freq = 10 63 | job.retries = 3 64 | job.poolId = "testPool" 65 | if err := job.save(); err != nil { 66 | t.Errorf("Unexpected error saving job: %s", err.Error()) 67 | } 68 | 69 | // Find the job in the database 70 | jobCopy, err := FindById(job.id) 71 | if err != nil { 72 | t.Errorf("Unexpected error in FindById: %s", err) 73 | } 74 | if !reflect.DeepEqual(jobCopy, job) { 75 | t.Errorf("Found job was not correct.\n\tExpected: %+v\n\tBut got: %+v", job, jobCopy) 76 | } 77 | 78 | // Attempting to find a job that doesn't exist should return an error 79 | fakeId := "foobar" 80 | if _, err := FindById(fakeId); err == nil { 81 | t.Error("Expected error when FindById was called with a fake id but got none.") 82 | } else if _, ok := err.(ErrorJobNotFound); !ok { 83 | t.Errorf("Expected error to have type ErrorJobNotFound, but got %T", err) 84 | } else if !strings.Contains(err.Error(), fakeId) { 85 | t.Error("Expected error message to contain the fake id but it did not.") 86 | } 87 | } 88 | 89 | func TestJobRefresh(t *testing.T) { 90 | testingSetUp() 91 | defer testingTeardown() 92 | 93 | // Create and save a job 94 | job, err := createAndSaveTestJob() 95 | if err != nil { 96 | t.Fatalf("Unexpected error: %s", err) 97 | } 98 | 99 | // Get a copy of that job directly from database 100 | jobCopy := &Job{} 101 | tx := newTransaction() 102 | tx.scanJobById(job.id, jobCopy) 103 | if err := tx.exec(); err != nil { 104 | t.Errorf("Unexpected error in tx.exec(): %s", err.Error()) 105 | } 106 | 107 | // Modify and save the copy 108 | newPriority := jobCopy.priority + 100 109 | jobCopy.priority = newPriority 110 | if err := jobCopy.save(); err != nil { 111 | t.Errorf("Unexpected error in jobCopy.save(): %s", err.Error()) 112 | } 113 | 114 | // Refresh the original job 115 | if err := job.Refresh(); err != nil { 116 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 117 | } 118 | 119 | // Now the original and the copy should b equal 120 | if !reflect.DeepEqual(job, jobCopy) { 121 | t.Errorf("Expected job to equal jobCopy but it did not.\n\tExpected %+v\n\tBut got %+v", jobCopy, job) 122 | } 123 | } 124 | 125 | func TestJobenqueue(t *testing.T) { 126 | testingSetUp() 127 | defer testingTeardown() 128 | 129 | // Run through a set of possible state paths and make sure the result is 130 | // always what we expect 131 | statePaths := []statePath{ 132 | { 133 | steps: []func(*Job) error{ 134 | // Just call enqueue after creating a new job 135 | enqueueJob, 136 | }, 137 | expected: StatusQueued, 138 | }, 139 | { 140 | steps: []func(*Job) error{ 141 | // Call enqueue, then Cancel, then enqueue again 142 | enqueueJob, 143 | cancelJob, 144 | enqueueJob, 145 | }, 146 | expected: StatusQueued, 147 | }, 148 | } 149 | testJobStatePaths(t, statePaths) 150 | } 151 | 152 | func TestJobCancel(t *testing.T) { 153 | testingSetUp() 154 | defer testingTeardown() 155 | 156 | // Run through a set of possible state paths and make sure the result is 157 | // always what we expect 158 | statePaths := []statePath{ 159 | { 160 | steps: []func(*Job) error{ 161 | // Just call Cancel after creating a new job 162 | cancelJob, 163 | }, 164 | expected: StatusCancelled, 165 | }, 166 | { 167 | steps: []func(*Job) error{ 168 | // Call Cancel, then enqueue, then Cancel again 169 | cancelJob, 170 | enqueueJob, 171 | cancelJob, 172 | }, 173 | expected: StatusCancelled, 174 | }, 175 | } 176 | testJobStatePaths(t, statePaths) 177 | } 178 | 179 | func TestJobReschedule(t *testing.T) { 180 | testingSetUp() 181 | defer testingTeardown() 182 | 183 | // Create and save a new job, then make sure that the time 184 | // parameter is set correctly when we call reschedule. 185 | job, err := createAndSaveTestJob() 186 | if err != nil { 187 | t.Fatalf("Unexpected error in createAndSaveTestJob(): %s", err.Error()) 188 | } 189 | currentTime := time.Now() 190 | unixNanoTime := currentTime.UTC().UnixNano() 191 | if err := job.Reschedule(currentTime); err != nil { 192 | t.Errorf("Unexpected error in job.Reschedule: %s", err.Error()) 193 | } 194 | expectJobFieldEquals(t, job, "time", unixNanoTime, int64Converter) 195 | expectJobInTimeIndex(t, job) 196 | 197 | // Run through a set of possible state paths and make sure the result is 198 | // always what we expect 199 | statePaths := []statePath{ 200 | { 201 | steps: []func(*Job) error{ 202 | // Just call Reschedule after creating a new job 203 | rescheduleJob, 204 | }, 205 | expected: StatusQueued, 206 | }, 207 | { 208 | steps: []func(*Job) error{ 209 | // Call Cancel, then reschedule 210 | cancelJob, 211 | rescheduleJob, 212 | }, 213 | expected: StatusQueued, 214 | }, 215 | } 216 | testJobStatePaths(t, statePaths) 217 | } 218 | 219 | func TestJobDestroy(t *testing.T) { 220 | testingSetUp() 221 | defer testingTeardown() 222 | 223 | // Run through a set of possible state paths and make sure the result is 224 | // always what we expect 225 | statePaths := []statePath{ 226 | { 227 | steps: []func(*Job) error{ 228 | // Just call Destroy after creating a new job 229 | destroyJob, 230 | }, 231 | expected: StatusDestroyed, 232 | }, 233 | { 234 | steps: []func(*Job) error{ 235 | // Call Destroy after cancel 236 | cancelJob, 237 | destroyJob, 238 | }, 239 | expected: StatusDestroyed, 240 | }, 241 | { 242 | steps: []func(*Job) error{ 243 | // Call Destroy after enqueue 244 | enqueueJob, 245 | destroyJob, 246 | }, 247 | expected: StatusDestroyed, 248 | }, 249 | { 250 | steps: []func(*Job) error{ 251 | // Call Destroy after enqueue then cancel 252 | enqueueJob, 253 | cancelJob, 254 | destroyJob, 255 | }, 256 | expected: StatusDestroyed, 257 | }, 258 | } 259 | testJobStatePaths(t, statePaths) 260 | } 261 | 262 | func TestJobSetError(t *testing.T) { 263 | testingSetUp() 264 | defer testingTeardown() 265 | 266 | job, err := createAndSaveTestJob() 267 | if err != nil { 268 | t.Fatalf("Unexpected error in createAndSaveTestJob(): %s", err.Error()) 269 | } 270 | testErr := errors.New("Test Error") 271 | if err := job.setError(testErr); err != nil { 272 | t.Errorf("Unexpected error in job.setError(): %s", err.Error()) 273 | } 274 | expectJobFieldEquals(t, job, "error", testErr.Error(), stringConverter) 275 | } 276 | 277 | // statePath represents a path through which a job can travel, where each step 278 | // potentially modifies its status. expected is what we expect the job status 279 | // to be after the last step. 280 | type statePath struct { 281 | steps []func(*Job) error 282 | expected Status 283 | } 284 | 285 | var ( 286 | // Some easy to use step functions 287 | enqueueJob = func(j *Job) error { 288 | return j.enqueue() 289 | } 290 | cancelJob = func(j *Job) error { 291 | return j.Cancel() 292 | } 293 | destroyJob = func(j *Job) error { 294 | return j.Destroy() 295 | } 296 | rescheduleJob = func(j *Job) error { 297 | return j.Reschedule(time.Now()) 298 | } 299 | ) 300 | 301 | // testJobStatePaths will for each statePath run through the steps, make sure 302 | // there were no errors at any step, and check that the status after the last 303 | // step is what we expect. 304 | func testJobStatePaths(t *testing.T, statePaths []statePath) { 305 | for _, statePath := range statePaths { 306 | testingSetUp() 307 | defer testingTeardown() 308 | // Create a new test job 309 | job, err := createAndSaveTestJob() 310 | if err != nil { 311 | t.Fatal(err) 312 | } 313 | // Run the job through each step 314 | for _, step := range statePath.steps { 315 | if err := step(job); err != nil { 316 | t.Errorf("Unexpected error in step %v: %s", step, err) 317 | } 318 | } 319 | expectStatusEquals(t, job, statePath.expected) 320 | } 321 | } 322 | 323 | func TestScanJob(t *testing.T) { 324 | testingSetUp() 325 | defer testingTeardown() 326 | job, err := createAndSaveTestJob() 327 | if err != nil { 328 | t.Fatalf("Unexpected error: %s", err) 329 | } 330 | conn := redisPool.Get() 331 | defer conn.Close() 332 | replies, err := conn.Do("HGETALL", job.Key()) 333 | if err != nil { 334 | t.Errorf("Unexpected error in HGETALL: %s", err.Error()) 335 | } 336 | jobCopy := &Job{id: job.id} 337 | if err := scanJob(replies, jobCopy); err != nil { 338 | t.Errorf("Unexpected error: %s", err) 339 | } 340 | if !reflect.DeepEqual(job, jobCopy) { 341 | t.Errorf("Result of scanJob was incorrect.\n\tExpected %+v\n\tbut got %+v", job, jobCopy) 342 | } 343 | } 344 | -------------------------------------------------------------------------------- /job_type.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "fmt" 9 | "reflect" 10 | "time" 11 | ) 12 | 13 | // Types is map of job type names to *Type 14 | var Types = map[string]*Type{} 15 | 16 | // Type represents a type of job that can be executed by workers 17 | type Type struct { 18 | name string 19 | handler interface{} 20 | retries uint 21 | dataType reflect.Type 22 | } 23 | 24 | // ErrorNameAlreadyRegistered is returned whenever RegisterType is called 25 | // with a name that has already been registered. 26 | type ErrorNameAlreadyRegistered struct { 27 | name string 28 | } 29 | 30 | // Error satisfies the error interface. 31 | func (e ErrorNameAlreadyRegistered) Error() string { 32 | return fmt.Sprintf("jobs: Cannot register job type because job type with name %s already exists", e.name) 33 | } 34 | 35 | // newErrorNameAlreadyRegistered returns an ErrorNameAlreadyRegistered with the given name. 36 | func newErrorNameAlreadyRegistered(name string) ErrorNameAlreadyRegistered { 37 | return ErrorNameAlreadyRegistered{name: name} 38 | } 39 | 40 | // A HandlerFunc is a function which accepts ether zero or one arguments and returns an error. 41 | // The function will be executed by a worker. If the function returns a non-nil error or causes 42 | // a panic, the worker will capture and log the error, and if applicable the job may be queued 43 | // for retry. 44 | type HandlerFunc interface{} 45 | 46 | // RegisterType registers a new type of job that can be executed by workers. 47 | // name should be a unique string identifier for the job. 48 | // retries is the number of times this type of job should be retried if it fails. 49 | // handler is a function that a worker will call in order to execute the job. 50 | // handler should be a function which accepts either 0 or 1 arguments of any type, 51 | // corresponding to the data for a job of this type. All jobs of this type must have 52 | // data with the same type as the first argument to handler, or nil if the handler 53 | // accepts no arguments. 54 | func RegisterType(name string, retries uint, handler HandlerFunc) (*Type, error) { 55 | // Make sure name is unique 56 | if _, found := Types[name]; found { 57 | return Types[name], newErrorNameAlreadyRegistered(name) 58 | } 59 | // Make sure handler is a function 60 | handlerType := reflect.TypeOf(handler) 61 | if handlerType.Kind() != reflect.Func { 62 | return nil, fmt.Errorf("jobs: in RegisterNewType, handler must be a function. Got %T", handler) 63 | } 64 | if handlerType.NumIn() > 1 { 65 | return nil, fmt.Errorf("jobs: in RegisterNewType, handler must accept 0 or 1 arguments. Got %d.", handlerType.NumIn()) 66 | } 67 | if handlerType.NumOut() != 1 { 68 | return nil, fmt.Errorf("jobs: in RegisterNewType, handler must have exactly one return value. Got %d.", handlerType.NumOut()) 69 | } 70 | if !typeIsError(handlerType.Out(0)) { 71 | return nil, fmt.Errorf("jobs: in RegisterNewType, handler must return an error. Got return value of type %s.", handlerType.Out(0).String()) 72 | } 73 | Type := &Type{ 74 | name: name, 75 | handler: handler, 76 | retries: retries, 77 | } 78 | if handlerType.NumIn() == 1 { 79 | Type.dataType = handlerType.In(0) 80 | } 81 | Types[name] = Type 82 | return Type, nil 83 | } 84 | 85 | var errorType = reflect.TypeOf(make([]error, 1)).Elem() 86 | 87 | func typeIsError(typ reflect.Type) bool { 88 | return typ.Implements(errorType) 89 | } 90 | 91 | // String satisfies the Stringer interface and returns the name of the Type. 92 | func (jt *Type) String() string { 93 | return jt.name 94 | } 95 | 96 | // Schedule schedules a on-off job of the given type with the given parameters. 97 | // Jobs with a higher priority will be executed first. The job will not be 98 | // executed until after time. data is the data associated with this particular 99 | // job and should have the same type as the first argument to the handler for this 100 | // Type. 101 | func (jt *Type) Schedule(priority int, time time.Time, data interface{}) (*Job, error) { 102 | // Encode the data 103 | encodedData, err := jt.encodeData(data) 104 | if err != nil { 105 | return nil, err 106 | } 107 | // Create and save the job 108 | job := &Job{ 109 | data: encodedData, 110 | typ: jt, 111 | time: time.UTC().UnixNano(), 112 | retries: jt.retries, 113 | priority: priority, 114 | } 115 | // Set the job's status to queued and save it in the database 116 | job.status = StatusQueued 117 | if err := job.save(); err != nil { 118 | return nil, err 119 | } 120 | return job, nil 121 | } 122 | 123 | // ScheduleRecurring schedules a recurring job of the given type with the given parameters. 124 | // Jobs with a higher priority will be executed first. The job will not be executed until after 125 | // time. After time, the job will be executed with a frequency specified by freq. data is the 126 | // data associated with this particular job and should have the same type as the first argument 127 | // to the handler for this Type. Every recurring execution of the job will use the 128 | // same data. 129 | func (jt *Type) ScheduleRecurring(priority int, time time.Time, freq time.Duration, data interface{}) (*Job, error) { 130 | // Encode the data 131 | encodedData, err := jt.encodeData(data) 132 | if err != nil { 133 | return nil, err 134 | } 135 | // Create and save the job 136 | job := &Job{ 137 | data: encodedData, 138 | typ: jt, 139 | time: time.UTC().UnixNano(), 140 | retries: jt.retries, 141 | freq: freq.Nanoseconds(), 142 | priority: priority, 143 | } 144 | // Set the job's status to queued and save it in the database 145 | job.status = StatusQueued 146 | if err := job.save(); err != nil { 147 | return nil, err 148 | } 149 | return job, nil 150 | } 151 | 152 | // encodeData checks that the type of data is what we expect based on the handler for the Type. 153 | // If it is, it encodes the data into a slice of bytes. 154 | func (jt *Type) encodeData(data interface{}) ([]byte, error) { 155 | // Check the type of data 156 | dataType := reflect.TypeOf(data) 157 | if dataType != jt.dataType { 158 | return nil, fmt.Errorf("jobs: provided data was not of the correct type.\nExpected %s for Type %s, but got %s", jt.dataType, jt, dataType) 159 | } 160 | // Encode the data 161 | encodedData, err := encode(data) 162 | if err != nil { 163 | return nil, fmt.Errorf("jobs: error encoding data: %s", err.Error()) 164 | } 165 | return encodedData, nil 166 | } 167 | -------------------------------------------------------------------------------- /job_type_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "testing" 9 | "time" 10 | ) 11 | 12 | func TestRegisterType(t *testing.T) { 13 | testingSetUp() 14 | defer testingTeardown() 15 | // Reset job types 16 | Types = map[string]*Type{} 17 | // Make sure we can register a job type without error 18 | testJobName := "testJob1" 19 | testJobRetries := uint(3) 20 | Type, err := RegisterType(testJobName, testJobRetries, noOpHandler) 21 | if err != nil { 22 | t.Fatalf("Unexpected err registering job type: %s", err.Error()) 23 | } 24 | // Make sure the name property is correct 25 | if Type.name != testJobName { 26 | t.Errorf("Got wrong name for job type. Expected %s but got %s", testJobName, Type.name) 27 | } 28 | // Make sure the retries property is correct 29 | if Type.retries != testJobRetries { 30 | t.Errorf("Got wrong number of retries for job type. Expected %d but got %d", testJobRetries, Type.retries) 31 | } 32 | // Make sure the Type was added to the global map 33 | if _, found := Types[testJobName]; !found { 34 | t.Errorf("Type was not added to the global map of job types.") 35 | } 36 | // Make sure we cannot register a job type with the same name 37 | if _, err := RegisterType(testJobName, 0, noOpHandler); err == nil { 38 | t.Errorf("Expected error when registering job with the same name but got none") 39 | } else if _, ok := err.(ErrorNameAlreadyRegistered); !ok { 40 | t.Errorf("Expected ErrorNameAlreadyRegistered but got error of type %T", err) 41 | } 42 | // Make sure we can register a job type with a handler function that has an argument 43 | if _, err := RegisterType("testJobWithArg", 0, func(s string) error { print(s); return nil }); err != nil { 44 | t.Errorf("Unexpected err registering job type with handler with one argument: %s", err) 45 | } 46 | // Make sure we cannot register a job type with an invalid handler 47 | invalidHandlers := []interface{}{ 48 | "notAFunc", 49 | func(a, b string) error { return nil }, 50 | } 51 | for _, handler := range invalidHandlers { 52 | if _, err := RegisterType("testJobWithInvalidHandler", 0, handler); err == nil { 53 | t.Errorf("Expected error when registering job with invalid handler type %T %v, but got none.", handler, handler) 54 | } 55 | } 56 | } 57 | 58 | func TestTypeSchedule(t *testing.T) { 59 | testingSetUp() 60 | defer testingTeardown() 61 | // Register a new job type 62 | testJobName := "testJob1" 63 | testJobPriority := 100 64 | testJobTime := time.Now() 65 | testJobData := "testData" 66 | encodedData, err := encode(testJobData) 67 | if err != nil { 68 | t.Errorf("Unexpected error encoding data: %s", err) 69 | } 70 | encodedTime := testJobTime.UTC().UnixNano() 71 | Type, err := RegisterType(testJobName, 0, func(string) error { return nil }) 72 | if err != nil { 73 | t.Fatalf("Unexpected error registering job type: %s", err) 74 | } 75 | // Call Schedule 76 | job, err := Type.Schedule(testJobPriority, testJobTime, testJobData) 77 | if err != nil { 78 | t.Errorf("Unexpected error in Type.Schedule(): %s", err) 79 | } 80 | // Make sure the job was saved in the database correctly 81 | if job.id == "" { 82 | t.Errorf("After Type.Schedule, job.id was empty.") 83 | } 84 | expectKeyExists(t, job.Key()) 85 | expectJobFieldEquals(t, job, "priority", testJobPriority, intConverter) 86 | expectJobFieldEquals(t, job, "time", encodedTime, int64Converter) 87 | expectJobFieldEquals(t, job, "data", encodedData, bytesConverter) 88 | expectStatusEquals(t, job, StatusQueued) 89 | // Make sure we get an error if the data is not the correct type 90 | if _, err := Type.Schedule(0, time.Now(), 0); err == nil { 91 | t.Errorf("Expected error when calling Type.Schedule with incorrect data type but got none") 92 | } 93 | } 94 | -------------------------------------------------------------------------------- /pool.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "fmt" 9 | "net" 10 | "runtime" 11 | "sync" 12 | "time" 13 | 14 | "github.com/garyburd/redigo/redis" 15 | ) 16 | 17 | // Pool is a pool of workers. Pool will query the database for queued jobs 18 | // and delegate those jobs to some number of workers. It will do this continuously 19 | // until the main program exits or you call Pool.Close(). 20 | type Pool struct { 21 | // config holds all config options for the pool 22 | config *PoolConfig 23 | // id is a unique identifier for each worker, which is generated whenver 24 | // Start() is called 25 | id string 26 | // workers is a slice of all workers 27 | workers []*worker 28 | // jobs is a channel through which jobs are delegated to workers 29 | jobs chan *Job 30 | // wg can be used after the jobs channel is closed to wait for all 31 | // workers to finish executing their current jobs. 32 | wg *sync.WaitGroup 33 | // exit is used to signal the pool to stop running the query loop 34 | // and close the jobs channel 35 | exit chan bool 36 | // afterFunc is a function that gets called after each job. 37 | afterFunc func(*Job) 38 | // RWMutex is only used during testing when we need to 39 | // change some of the fields for the pool after it was started. 40 | // NOTE: currently only used in one test (TestStalePoolsArePurged) 41 | // and might be removed if we refactor later. 42 | sync.RWMutex 43 | } 44 | 45 | // PoolConfig is a set of configuration options for pools. Setting any value 46 | // to the zero value will be interpretted as the default. 47 | type PoolConfig struct { 48 | // NumWorkers is the number of workers to run 49 | // Each worker will run inside its own goroutine 50 | // and execute jobs asynchronously. Default is 51 | // runtime.GOMAXPROCS. 52 | NumWorkers int 53 | // BatchSize is the number of jobs to send through 54 | // the jobs channel at once. Increasing BatchSize means 55 | // the worker pool will query the database less frequently, 56 | // so you would get higher performance. However this comes 57 | // at the cost that jobs with lower priority may sometimes be 58 | // executed before jobs with higher priority, because the jobs 59 | // with higher priority were not ready yet the last time the pool 60 | // queried the database. Decreasing BatchSize means more 61 | // frequent queries to the database and lower performance, but 62 | // greater likelihood of executing jobs in perfect order with regards 63 | // to priority. Setting BatchSize to 1 gaurantees that higher priority 64 | // jobs are always executed first as soon as they are ready. Default is 65 | // runtime.GOMAXPROCS. 66 | BatchSize int 67 | // MinWait is the minimum amount of time the pool will wait before checking 68 | // the database for queued jobs. The pool may take longer to query the database 69 | // if the jobs channel is blocking (i.e. if no workers are ready to execute new 70 | // jobs). Default is 200ms. 71 | MinWait time.Duration 72 | // StaleTimeout is the amount of time to wait for a pool to reply to a ping request 73 | // before considering it stale. Stale pools will be purged and if they have any 74 | // corresponding jobs in the executing set, those jobs will be requeued. Default 75 | // is 30 seconds. 76 | StaleTimeout time.Duration 77 | } 78 | 79 | // DefaultPoolConfig is the default config for pools. You can override any values 80 | // by passing in a *PoolConfig to NewPool. Any zero values in PoolConfig will be 81 | // interpreted as the default. 82 | var DefaultPoolConfig = &PoolConfig{ 83 | NumWorkers: runtime.GOMAXPROCS(0), 84 | BatchSize: runtime.GOMAXPROCS(0), 85 | MinWait: 200 * time.Millisecond, 86 | StaleTimeout: 30 * time.Second, 87 | } 88 | 89 | // NewPool creates and returns a new pool with the given configuration. You can 90 | // pass in nil to use the default values. Otherwise, any zero values in config will 91 | // be interpreted as the default value. 92 | func NewPool(config *PoolConfig) (*Pool, error) { 93 | finalConfig := getPoolConfig(config) 94 | hardwareId, err := getHardwareId() 95 | if err != nil { 96 | return nil, err 97 | } 98 | return &Pool{ 99 | config: finalConfig, 100 | id: hardwareId, 101 | wg: &sync.WaitGroup{}, 102 | exit: make(chan bool), 103 | workers: make([]*worker, finalConfig.NumWorkers), 104 | jobs: make(chan *Job, finalConfig.BatchSize), 105 | }, nil 106 | } 107 | 108 | // getPoolConfig replaces any zero values in passedConfig with the default values. 109 | // If passedConfig is nil, every value will be set to the default. 110 | func getPoolConfig(passedConfig *PoolConfig) *PoolConfig { 111 | if passedConfig == nil { 112 | return DefaultPoolConfig 113 | } 114 | finalConfig := &PoolConfig{} 115 | (*finalConfig) = (*passedConfig) 116 | if passedConfig.NumWorkers == 0 { 117 | finalConfig.NumWorkers = DefaultPoolConfig.NumWorkers 118 | } 119 | if passedConfig.BatchSize == 0 { 120 | finalConfig.BatchSize = DefaultPoolConfig.BatchSize 121 | } 122 | if passedConfig.MinWait == 0 { 123 | finalConfig.MinWait = DefaultPoolConfig.MinWait 124 | } 125 | if passedConfig.StaleTimeout == 0 { 126 | finalConfig.StaleTimeout = DefaultPoolConfig.StaleTimeout 127 | } 128 | return finalConfig 129 | } 130 | 131 | // addToPoolSet adds the id of the worker pool to a set of active pools 132 | // in the database. 133 | func (p *Pool) addToPoolSet() error { 134 | conn := redisPool.Get() 135 | defer conn.Close() 136 | p.RLock() 137 | thisId := p.id 138 | p.RUnlock() 139 | if _, err := conn.Do("SADD", Keys.ActivePools, thisId); err != nil { 140 | return err 141 | } 142 | return nil 143 | } 144 | 145 | // removeFromPoolSet removes the id of the worker pool from a set of active pools 146 | // in the database. 147 | func (p *Pool) removeFromPoolSet() error { 148 | conn := redisPool.Get() 149 | defer conn.Close() 150 | p.RLock() 151 | thisId := p.id 152 | p.RUnlock() 153 | if _, err := conn.Do("SREM", Keys.ActivePools, thisId); err != nil { 154 | return err 155 | } 156 | return nil 157 | } 158 | 159 | // getHardwareId returns a unique identifier for the current machine. It does this 160 | // by iterating through the network interfaces of the machine and picking the first 161 | // one that has a non-empty hardware (MAC) address. MAC Addresses are guaranteed by 162 | // IEEE to be unique, however, they are also sometimes spoofable. Spoofed MAC addresses 163 | // are fine as long as no two machines in the job pool have the same MAC address. 164 | func getHardwareId() (string, error) { 165 | inters, err := net.Interfaces() 166 | if err != nil { 167 | return "", fmt.Errorf("jobs: Unable to get network interfaces via net.Interfaces(). Does this machine have any network interfaces?\n%s", err.Error()) 168 | } 169 | address := "" 170 | for _, inter := range inters { 171 | if inter.HardwareAddr.String() != "" { 172 | address = inter.HardwareAddr.String() 173 | break 174 | } 175 | } 176 | if address == "" { 177 | return "", fmt.Errorf("jobs: Unable to find a network interface with a non-empty hardware (MAC) address. Does this machine have any valid network interfaces?\n%s", err.Error()) 178 | } 179 | return address, nil 180 | } 181 | 182 | // pingKey is the key for a pub/sub connection which allows a pool to ping, i.e. 183 | // check the status of, another pool. 184 | func (p *Pool) pingKey() string { 185 | p.RLock() 186 | thisId := p.id 187 | p.RUnlock() 188 | return "workers:" + thisId + ":ping" 189 | } 190 | 191 | // pongKey is the key for a pub/sub connection which allows a pool to respond to 192 | // pings with a pong, i.e. acknowledge that it is still alive and working. 193 | func (p *Pool) pongKey() string { 194 | p.RLock() 195 | thisId := p.id 196 | p.RUnlock() 197 | return "workers:" + thisId + ":pong" 198 | } 199 | 200 | // purgeStalePools will first get all the ids of pools from the activePools 201 | // set. All of these should be active, but if Pool.Wait was never called for 202 | // a pool (possibly because of power failure), some of them might not actually 203 | // be active. To find out for sure, purgeStalePools will ping each pool that is 204 | // supposed to be active and wait for a pong response. If it does not receive 205 | // a pong within some amount of time, the pool is considered stale (i.e. whatever 206 | // process that was running it was exited and it is no longer executing jobs). If 207 | // any stale pools are found, purgeStalePools will remove them from the set of 208 | // active pools and then moves any jobs associated with the stale pool from the 209 | // executing set to the queued set to be retried. 210 | func (p *Pool) purgeStalePools() error { 211 | conn := redisPool.Get() 212 | defer conn.Close() 213 | poolIds, err := redis.Strings(conn.Do("SMEMBERS", Keys.ActivePools)) 214 | if err != nil { 215 | return err 216 | } 217 | for _, poolId := range poolIds { 218 | p.RLock() 219 | thisId := p.id 220 | p.RUnlock() 221 | if poolId == thisId { 222 | // Don't ping self 223 | continue 224 | } 225 | pool := &Pool{id: poolId} 226 | go func(pool *Pool) { 227 | if err := p.pingAndPurgeIfNeeded(pool); err != nil { 228 | // TODO: send accross an err channel instead of panicking 229 | panic(err) 230 | } 231 | }(pool) 232 | } 233 | return nil 234 | } 235 | 236 | // pingAndPurgeIfNeeded pings other by publishing to others ping key. If it 237 | // does not receive a pong reply within some amount of time, it will 238 | // assume the pool is stale and purge it. 239 | func (p *Pool) pingAndPurgeIfNeeded(other *Pool) error { 240 | ping := redisPool.Get() 241 | pong := redis.PubSubConn{redisPool.Get()} 242 | // Listen for pongs by subscribing to the other pool's pong key 243 | pong.Subscribe(other.pongKey()) 244 | // Ping the other pool by publishing to its ping key 245 | ping.Do("PUBLISH", other.pingKey(), 1) 246 | // Use a select statement to either receive the pong or timeout 247 | pongChan := make(chan interface{}) 248 | errChan := make(chan error) 249 | go func() { 250 | defer func() { 251 | pong.Close() 252 | ping.Close() 253 | }() 254 | select { 255 | case <-p.exit: 256 | return 257 | default: 258 | } 259 | for { 260 | reply := pong.Receive() 261 | switch reply.(type) { 262 | case redis.Message: 263 | // The pong was received 264 | pongChan <- reply 265 | return 266 | case error: 267 | // There was some unexpected error 268 | err := reply.(error) 269 | errChan <- err 270 | return 271 | } 272 | } 273 | }() 274 | timeout := time.After(p.config.StaleTimeout) 275 | select { 276 | case <-pongChan: 277 | // The other pool responded with a pong 278 | return nil 279 | case err := <-errChan: 280 | // Received an error from the pubsub conn 281 | return err 282 | case <-timeout: 283 | // The pool is considered stale and should be purged 284 | t := newTransaction() 285 | other.RLock() 286 | otherId := other.id 287 | other.RUnlock() 288 | t.purgeStalePool(otherId) 289 | if err := t.exec(); err != nil { 290 | return err 291 | } 292 | } 293 | return nil 294 | } 295 | 296 | // respondToPings continuously listens for pings from other worker pools and 297 | // immediately responds with a pong. It will only return if there is an error. 298 | func (p *Pool) respondToPings() error { 299 | pong := redisPool.Get() 300 | ping := redis.PubSubConn{redisPool.Get()} 301 | defer func() { 302 | pong.Close() 303 | ping.Close() 304 | }() 305 | // Subscribe to the ping key for this pool to receive pings. 306 | if err := ping.Subscribe(p.pingKey()); err != nil { 307 | return err 308 | } 309 | for { 310 | // Whenever we recieve a ping, reply immediately with a pong by 311 | // publishing to the pong key for this pool. 312 | switch reply := ping.Receive().(type) { 313 | case redis.Message: 314 | if _, err := pong.Do("PUBLISH", p.pongKey(), 0); err != nil { 315 | return err 316 | } 317 | case error: 318 | err := reply.(error) 319 | return err 320 | } 321 | time.Sleep(1 * time.Millisecond) 322 | } 323 | } 324 | 325 | // removeStaleSelf will check if the current machine recently failed hard 326 | // (e.g. due to power failuer) by checking if p.id is in the set of active 327 | // pools. If p.id is still "active" according to the database, it means 328 | // there was a hard failure, and so removeStaleSelf then re-queues the 329 | // stale jobs. removeStaleSelf should only be run when the Pool is started. 330 | func (p *Pool) removeStaleSelf() error { 331 | t := newTransaction() 332 | t.purgeStalePool(p.id) 333 | if err := t.exec(); err != nil { 334 | return err 335 | } 336 | return nil 337 | } 338 | 339 | // SetAfterFunc will assign a function that will be executed each time 340 | // a job is finished. 341 | func (p *Pool) SetAfterFunc(f func(*Job)) { 342 | p.afterFunc = f 343 | } 344 | 345 | // Start starts the worker pool. This means the pool will initialize workers, 346 | // continuously query the database for queued jobs, and delegate those jobs 347 | // to the workers. 348 | func (p *Pool) Start() error { 349 | // Purge stale jobs belonging to this pool if there was a recent 350 | // hard failure 351 | if err := p.removeStaleSelf(); err != nil { 352 | return err 353 | } 354 | 355 | // Check on the status of other worker pools by pinging them and 356 | // start the process to repond to pings from other pools 357 | if err := p.addToPoolSet(); err != nil { 358 | return err 359 | } 360 | go func() { 361 | select { 362 | case <-p.exit: 363 | return 364 | default: 365 | } 366 | if err := p.respondToPings(); err != nil { 367 | // TODO: send the err accross a channel instead of panicking 368 | panic(err) 369 | } 370 | }() 371 | if err := p.purgeStalePools(); err != nil { 372 | return err 373 | } 374 | 375 | // Initialize workers 376 | for i := range p.workers { 377 | p.wg.Add(1) 378 | worker := &worker{ 379 | wg: p.wg, 380 | jobs: p.jobs, 381 | afterFunc: p.afterFunc, 382 | } 383 | p.workers[i] = worker 384 | worker.start() 385 | } 386 | go func() { 387 | if err := p.queryLoop(); err != nil { 388 | // TODO: send the err accross a channel instead of panicking 389 | panic(err) 390 | } 391 | }() 392 | return nil 393 | } 394 | 395 | // Close closes the worker pool and prevents it from delegating 396 | // any new jobs. However, any jobs that are currently being executed 397 | // will still be executed. Close returns immediately. If you want to 398 | // wait until all workers are done executing their current jobs, use the 399 | // Wait method. 400 | func (p *Pool) Close() { 401 | close(p.exit) 402 | } 403 | 404 | // Wait will return when all workers are done executing their jobs. 405 | // Wait can only possibly return after you have called Close. To prevent 406 | // errors due to partially-executed jobs, any go program which starts a 407 | // worker pool should call Wait (and Close before that if needed) before 408 | // exiting. 409 | func (p *Pool) Wait() error { 410 | // The shared waitgroup will only return after each worker is finished 411 | p.wg.Wait() 412 | // Remove the pool id from the set of active pools, only after we know 413 | // each worker finished executing. 414 | if err := p.removeFromPoolSet(); err != nil { 415 | return err 416 | } 417 | return nil 418 | } 419 | 420 | // queryLoop continuously queries the database for new jobs and, if 421 | // it finds any, sends them through the jobs channel for execution 422 | // by some worker. 423 | func (p *Pool) queryLoop() error { 424 | if err := p.sendNextJobs(p.config.BatchSize); err != nil { 425 | return err 426 | } 427 | for { 428 | minWait := time.After(p.config.MinWait) 429 | select { 430 | case <-p.exit: 431 | // Close the channel to tell workers to stop executing new jobs 432 | close(p.jobs) 433 | return nil 434 | case <-minWait: 435 | if err := p.sendNextJobs(p.config.BatchSize); err != nil { 436 | return err 437 | } 438 | } 439 | } 440 | return nil 441 | } 442 | 443 | // sendNextJobs queries the database to find the next n ready jobs, then 444 | // sends those jobs to the jobs channel, effectively delegating them to 445 | // a worker. 446 | func (p *Pool) sendNextJobs(n int) error { 447 | jobs, err := p.getNextJobs(p.config.BatchSize) 448 | if err != nil { 449 | return err 450 | } 451 | // Send the jobs across the channel, where they will be picked up 452 | // by exactly one worker 453 | for _, job := range jobs { 454 | p.jobs <- job 455 | } 456 | return nil 457 | } 458 | 459 | // getNextJobs queries the database and returns the next n ready jobs. 460 | func (p *Pool) getNextJobs(n int) ([]*Job, error) { 461 | p.RLock() 462 | thisId := p.id 463 | p.RUnlock() 464 | return getNextJobs(n, thisId) 465 | } 466 | 467 | // getNextJobs queries the database and returns the next n ready jobs. 468 | func getNextJobs(n int, poolId string) ([]*Job, error) { 469 | // Start a new transaction 470 | t := newTransaction() 471 | // Invoke a script to get all the jobs which are ready to execute based on their 472 | // time parameter and whether or not they are in the queued set. 473 | jobs := []*Job{} 474 | t.popNextJobs(n, poolId, newScanJobsHandler(&jobs)) 475 | 476 | // Execute the transaction 477 | if err := t.exec(); err != nil { 478 | return nil, err 479 | } 480 | return jobs, nil 481 | } 482 | -------------------------------------------------------------------------------- /pool_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "errors" 9 | "fmt" 10 | "reflect" 11 | "strconv" 12 | "sync" 13 | "testing" 14 | "time" 15 | 16 | "github.com/dustin/go-humanize" 17 | "github.com/garyburd/redigo/redis" 18 | ) 19 | 20 | // TestPoolIdSet tests that the pool id is set properly when a pool is started 21 | // and removed when it is closed 22 | func TestPoolIdSet(t *testing.T) { 23 | testingSetUp() 24 | defer testingTeardown() 25 | 26 | pool, err := NewPool(nil) 27 | if err != nil { 28 | t.Fatal(err) 29 | } 30 | if err := pool.Start(); err != nil { 31 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 32 | } 33 | expectSetContains(t, Keys.ActivePools, pool.id) 34 | pool.Close() 35 | if err := pool.Wait(); err != nil { 36 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 37 | } 38 | expectSetDoesNotContain(t, Keys.ActivePools, pool.id) 39 | } 40 | 41 | // TestGetNextJobs tests the getNextJobs function, which queries the database to find 42 | // the next queued jobs, in order of their priority. 43 | func TestGetNextJobs(t *testing.T) { 44 | testingSetUp() 45 | defer testingTeardown() 46 | 47 | // Create a test job with high priority 48 | highPriorityJob, err := createTestJob() 49 | if err != nil { 50 | t.Fatalf("Unexpected error creating test job: %s", err.Error()) 51 | } 52 | highPriorityJob.priority = 1000 53 | highPriorityJob.id = "highPriorityJob" 54 | if err := highPriorityJob.save(); err != nil { 55 | t.Fatalf("Unexpected error saving test job: %s", err.Error()) 56 | } 57 | if err := highPriorityJob.enqueue(); err != nil { 58 | t.Fatalf("Unexpected error enqueuing test job: %s", err.Error()) 59 | } 60 | 61 | // Create more tests with lower priorities 62 | for i := 0; i < 10; i++ { 63 | job, err := createTestJob() 64 | if err != nil { 65 | t.Fatalf("Unexpected error creating test job: %s", err.Error()) 66 | } 67 | job.priority = 100 68 | job.id = "lowPriorityJob" + strconv.Itoa(i) 69 | if err := job.save(); err != nil { 70 | t.Fatalf("Unexpected error saving test job: %s", err.Error()) 71 | } 72 | if err := job.enqueue(); err != nil { 73 | t.Fatalf("Unexpected error enqueuing test job: %s", err.Error()) 74 | } 75 | } 76 | 77 | // Call getNextJobs with n = 1. We expect the one job returned to be the 78 | // highpriority one, but the status should now be executing 79 | testPoolId := "testPool" 80 | jobs, err := getNextJobs(1, testPoolId) 81 | if err != nil { 82 | t.Errorf("Unexpected error from getNextJobs: %s", err.Error()) 83 | } 84 | if len(jobs) != 1 { 85 | t.Errorf("Length of jobs was incorrect. Expected 1 but got %d", len(jobs)) 86 | } else { 87 | gotJob := jobs[0] 88 | expectedJob := &Job{} 89 | (*expectedJob) = *highPriorityJob 90 | expectedJob.status = StatusExecuting 91 | expectedJob.poolId = testPoolId 92 | if !reflect.DeepEqual(expectedJob, gotJob) { 93 | t.Errorf("Job returned by getNextJobs was incorrect.\n\tExpected: %+v\n\tBut got: %+v", expectedJob, gotJob) 94 | } 95 | } 96 | } 97 | 98 | // TestStatusIsExecutingWhileExecuting tests that while a job is executing, its 99 | // status is set to StatusExecuting. 100 | func TestStatusIsExecutingWhileExecuting(t *testing.T) { 101 | testingSetUp() 102 | defer testingTeardown() 103 | 104 | // Create a pool with 4 workers 105 | pool, err := NewPool(&PoolConfig{ 106 | NumWorkers: 4, 107 | BatchSize: 4, 108 | MinWait: 1 * time.Millisecond, 109 | }) 110 | if err != nil { 111 | t.Fatal(err) 112 | } 113 | defer func() { 114 | // Close the pool and wait for workers to finish 115 | pool.Close() 116 | if err := pool.Wait(); err != nil { 117 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 118 | } 119 | }() 120 | 121 | // Register some jobs which will set the value of some string index, 122 | // signal the wait group, and then wait for an exit signal before closing. 123 | // waitForJobs is a wait group which will wait for each job to set their string 124 | waitForJobs := sync.WaitGroup{} 125 | // jobsCanExit signals all jobs to exit when closed 126 | jobsCanExit := make(chan bool) 127 | data := make([]string, 4) 128 | setStringJob, err := RegisterType("setString", 0, func(i int) error { 129 | data[i] = "ok" 130 | waitForJobs.Done() 131 | // Wait for the signal before returning from this function 132 | for range jobsCanExit { 133 | } 134 | return nil 135 | }) 136 | if err != nil { 137 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 138 | } 139 | 140 | // Queue up some jobs 141 | queuedJobs := make([]*Job, len(data)) 142 | for i := 0; i < len(data); i++ { 143 | waitForJobs.Add(1) 144 | job, err := setStringJob.Schedule(100, time.Now(), i) 145 | if err != nil { 146 | t.Errorf("Unexpected error in Schedule: %s", err.Error()) 147 | } 148 | queuedJobs[i] = job 149 | } 150 | 151 | // Start the pool 152 | if err := pool.Start(); err != nil { 153 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 154 | } 155 | 156 | // Wait for the jobs to finish setting their data 157 | waitForJobs.Wait() 158 | 159 | // At this point, we expect the status of all jobs to be executing. 160 | for _, job := range queuedJobs { 161 | // Refresh the job and make sure its status is correct 162 | if err := job.Refresh(); err != nil { 163 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 164 | } 165 | expectStatusEquals(t, job, StatusExecuting) 166 | } 167 | 168 | // Signal that the jobs can now exit 169 | close(jobsCanExit) 170 | } 171 | 172 | // TestExecuteJobWithNoArguments registers and executes a job without any 173 | // arguments and then checks that it executed correctly. 174 | func TestExecuteJobWithNoArguments(t *testing.T) { 175 | testingSetUp() 176 | // defer testingTeardown() 177 | 178 | // Register a job type with a handler that expects 0 arguments 179 | data := "" 180 | setOkayJob, err := RegisterType("setOkay", 0, func() error { 181 | data = "ok" 182 | return nil 183 | }) 184 | if err != nil { 185 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 186 | } 187 | 188 | // Queue up a single job 189 | if _, err := setOkayJob.Schedule(100, time.Now(), nil); err != nil { 190 | t.Errorf("Unexpected error in Schedule(): %s", err.Error()) 191 | } 192 | 193 | // Start the pool with 1 worker 194 | pool, err := NewPool(&PoolConfig{ 195 | NumWorkers: 1, 196 | BatchSize: 1, 197 | }) 198 | if err != nil { 199 | t.Fatal(err) 200 | } 201 | if err := pool.Start(); err != nil { 202 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 203 | } 204 | 205 | // Immediately close the pool and wait for workers to finish 206 | pool.Close() 207 | if err := pool.Wait(); err != nil { 208 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 209 | } 210 | 211 | // Make sure that data was set to "ok", indicating that the job executed 212 | // successfully. 213 | if data != "ok" { 214 | t.Errorf("Expected data to be \"ok\" but got \"%s\", indicating the job did not execute successfully.", data) 215 | } 216 | } 217 | 218 | // TestJobsWithHigherPriorityExecutedFirst creates two sets of jobs: one with lower priorities 219 | // and one with higher priorities. Then it starts the worker pool and runs for exactly one iteration. 220 | // Then it makes sure that the jobs with higher priorities were executed, and the lower priority ones 221 | // were not. 222 | func TestJobsWithHigherPriorityExecutedFirst(t *testing.T) { 223 | testingSetUp() 224 | defer testingTeardown() 225 | 226 | // Register some jobs which will simply set one of the values in data 227 | data := make([]string, 8) 228 | setStringJob, err := RegisterType("setString", 0, func(i int) error { 229 | data[i] = "ok" 230 | return nil 231 | }) 232 | if err != nil { 233 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 234 | } 235 | 236 | // Queue up some jobs 237 | queuedJobs := make([]*Job, len(data)) 238 | for i := 0; i < len(data); i++ { 239 | // Lower indexes have higher priority and should be completed first 240 | job, err := setStringJob.Schedule(8-i, time.Now(), i) 241 | if err != nil { 242 | t.Errorf("Unexpected error in Schedule: %s", err.Error()) 243 | } 244 | queuedJobs[i] = job 245 | } 246 | 247 | // Start the pool with 4 workers 248 | pool, err := NewPool(&PoolConfig{ 249 | NumWorkers: 4, 250 | BatchSize: 4, 251 | }) 252 | if err != nil { 253 | t.Fatal(err) 254 | } 255 | if err := pool.Start(); err != nil { 256 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 257 | } 258 | 259 | // Immediately stop the pool to stop the workers from doing more jobs 260 | pool.Close() 261 | 262 | // Wait for the workers to finish 263 | if err := pool.Wait(); err != nil { 264 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 265 | } 266 | 267 | // Check that the first 4 values of data were set to "ok" 268 | // This would mean that the first 4 jobs (in order of priority) 269 | // were successfully executed. 270 | expectTestDataOk(t, data[:4]) 271 | 272 | // Make sure all the other values of data are still blank 273 | expectTestDataBlank(t, data[4:]) 274 | 275 | // Make sure the first four jobs we queued are marked as finished 276 | for _, job := range queuedJobs[0:4] { 277 | // Refresh the job and make sure its status is correct 278 | if err := job.Refresh(); err != nil { 279 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 280 | } 281 | expectStatusEquals(t, job, StatusFinished) 282 | } 283 | 284 | // Make sure the next four jobs we queued are marked as queued 285 | for _, job := range queuedJobs[4:] { 286 | // Refresh the job and make sure its status is correct 287 | if err := job.Refresh(); err != nil { 288 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 289 | } 290 | expectStatusEquals(t, job, StatusQueued) 291 | } 292 | } 293 | 294 | // TestJobsOnlyExecutedOnce creates a few jobs that increment a counter (each job 295 | // has its own counter). Then it starts the pool and runs the query loop for at most two 296 | // iterations. Then it checks that each job was executed only once by observing the counters. 297 | func TestJobsOnlyExecutedOnce(t *testing.T) { 298 | testingSetUp() 299 | defer testingTeardown() 300 | 301 | // Register some jobs which will simply increment one of the values in data 302 | data := make([]int, 4) 303 | waitForJobs := sync.WaitGroup{} 304 | incrementJob, err := RegisterType("increment", 0, func(i int) error { 305 | data[i] += 1 306 | waitForJobs.Done() 307 | return nil 308 | }) 309 | if err != nil { 310 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 311 | } 312 | 313 | // Queue up some jobs 314 | for i := 0; i < len(data); i++ { 315 | waitForJobs.Add(1) 316 | if _, err := incrementJob.Schedule(100, time.Now(), i); err != nil { 317 | t.Errorf("Unexpected error in Schedule: %s", err.Error()) 318 | } 319 | } 320 | 321 | // Start the pool with 4 workers 322 | pool, err := NewPool(&PoolConfig{ 323 | NumWorkers: 4, 324 | BatchSize: 4, 325 | }) 326 | if err != nil { 327 | t.Fatal(err) 328 | } 329 | if err := pool.Start(); err != nil { 330 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 331 | } 332 | 333 | // Wait for the wait group, which tells us each job was executed at least once 334 | waitForJobs.Wait() 335 | // Close the pool, allowing for a max of one more iteration 336 | pool.Close() 337 | // Wait for the workers to finish 338 | if err := pool.Wait(); err != nil { 339 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 340 | } 341 | 342 | // Check that each value in data equals 1. 343 | // This would mean that each job was only executed once 344 | for i, datum := range data { 345 | if datum != 1 { 346 | t.Errorf(`Expected data[%d] to be 1 but got: %d`, i, datum) 347 | } 348 | } 349 | } 350 | 351 | // TestAllJobsExecuted creates many more jobs than workers. Then it starts 352 | // the pool and continuously checks if every job was executed, it which case 353 | // it exits successfully. If some of the jobs have not been executed after 1 354 | // second, it breaks and reports an error. 1 second should be plenty of time 355 | // to execute the jobs. 356 | func TestAllJobsExecuted(t *testing.T) { 357 | testingSetUp() 358 | defer testingTeardown() 359 | 360 | // Create a pool with 4 workers 361 | pool, err := NewPool(&PoolConfig{ 362 | NumWorkers: 4, 363 | BatchSize: 4, 364 | MinWait: 1 * time.Millisecond, 365 | }) 366 | if err != nil { 367 | t.Fatal(err) 368 | } 369 | defer func() { 370 | // Close the pool and wait for workers to finish 371 | pool.Close() 372 | if err := pool.Wait(); err != nil { 373 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 374 | } 375 | }() 376 | 377 | // Register some jobs which will simply set one of the elements in 378 | // data to "ok" 379 | dataMut := sync.Mutex{} 380 | data := make([]string, 100) 381 | setStringJob, err := RegisterType("setString", 0, func(i int) error { 382 | dataMut.Lock() 383 | data[i] = "ok" 384 | dataMut.Unlock() 385 | return nil 386 | }) 387 | if err != nil { 388 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 389 | } 390 | 391 | // Queue up some jobs 392 | for i := 0; i < len(data); i++ { 393 | if _, err := setStringJob.Schedule(100, time.Now(), i); err != nil { 394 | t.Errorf("Unexpected error in Schedule: %s", err.Error()) 395 | } 396 | } 397 | 398 | // Start the pool 399 | if err := pool.Start(); err != nil { 400 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 401 | } 402 | 403 | // Continuously check the data every 10 milliseconds. Eventually 404 | // we hope to see that everything was set to "ok". If 1 second has 405 | // passed, assume something went wrong. 406 | timeout := time.After(1 * time.Second) 407 | interval := time.Tick(10 * time.Millisecond) 408 | remainingJobs := len(data) 409 | for { 410 | select { 411 | case <-timeout: 412 | // More than 1 second has passed. Assume something went wrong. 413 | t.Errorf("1 second passed and %d jobs out of %d were not executed.", remainingJobs, len(data)) 414 | break 415 | case <-interval: 416 | // Count the number of elements in data that equal "ok". 417 | // Anything that doesn't equal ok represents a job that hasn't been executed yet 418 | remainingJobs = len(data) 419 | dataMut.Lock() 420 | for _, datum := range data { 421 | if datum == "ok" { 422 | remainingJobs -= 1 423 | } 424 | } 425 | dataMut.Unlock() 426 | if remainingJobs == 0 { 427 | // Each item in data was set to "ok", so all the jobs were executed correctly. 428 | return 429 | } 430 | } 431 | } 432 | } 433 | 434 | // TestJobsAreNotExecutedUntilTime sets up a few jobs with a time parameter in the future 435 | // Then it makes sure that those jobs are not executed until after that time. 436 | func TestJobsAreNotExecutedUntilTime(t *testing.T) { 437 | testingSetUp() 438 | defer testingTeardown() 439 | 440 | // Create a pool with 4 workers 441 | pool, err := NewPool(&PoolConfig{ 442 | NumWorkers: 4, 443 | BatchSize: 4, 444 | }) 445 | if err != nil { 446 | t.Fatal(err) 447 | } 448 | defer func() { 449 | // Close the pool and wait for workers to finish 450 | pool.Close() 451 | if err := pool.Wait(); err != nil { 452 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 453 | } 454 | }() 455 | 456 | // Register some jobs which will set one of the elements in data 457 | // For this test, we want to execute two jobs at a time, so we'll 458 | // use a waitgroup. 459 | data := make([]string, 4) 460 | dataMut := sync.Mutex{} 461 | setStringJob, err := RegisterType("setString", 0, func(i int) error { 462 | dataMut.Lock() 463 | data[i] = "ok" 464 | dataMut.Unlock() 465 | return nil 466 | }) 467 | if err != nil { 468 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 469 | } 470 | 471 | // Queue up some jobs with a time parameter in the future 472 | currentTime := time.Now() 473 | timeDiff := 200 * time.Millisecond 474 | futureTime := currentTime.Add(timeDiff) 475 | for i := 0; i < len(data); i++ { 476 | if _, err := setStringJob.Schedule(100, futureTime, i); err != nil { 477 | t.Errorf("Unexpected error in Schedule: %s", err.Error()) 478 | } 479 | } 480 | 481 | // Start the pool 482 | if err := pool.Start(); err != nil { 483 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 484 | } 485 | 486 | // Continuously check the data every 10 milliseconds. Eventually 487 | // we hope to see that everything was set to "ok". We will check that 488 | // this condition is only true after futureTime has been reached, since 489 | // the jobs should not be executed before then. 490 | timeout := time.After(1 * time.Second) 491 | interval := time.Tick(10 * time.Millisecond) 492 | remainingJobs := len(data) 493 | for { 494 | select { 495 | case <-timeout: 496 | // More than 1 second has passed. Assume something went wrong. 497 | t.Errorf("1 second passed and %d jobs were not executed.", remainingJobs) 498 | t.FailNow() 499 | case <-interval: 500 | // Count the number of elements in data that equal "ok". 501 | // Anything that doesn't equal ok represents a job that hasn't been executed yet 502 | dataMut.Lock() 503 | remainingJobs = len(data) 504 | for _, datum := range data { 505 | if datum == "ok" { 506 | remainingJobs -= 1 507 | } 508 | } 509 | dataMut.Unlock() 510 | if remainingJobs == 0 { 511 | // Each item in data was set to "ok", so all the jobs were executed correctly. 512 | // Check that this happend after futureTime 513 | if time.Now().Before(futureTime) { 514 | t.Errorf("jobs were executed before their time parameter was reached.") 515 | } 516 | return 517 | } 518 | } 519 | } 520 | } 521 | 522 | // TestJobTimestamps creates and executes a job, then tests that the started and finished 523 | // timestamps were correct. 524 | func TestJobTimestamps(t *testing.T) { 525 | testingSetUp() 526 | defer testingTeardown() 527 | 528 | // Register a job type which will do nothing but sleep for some duration 529 | sleepJob, err := RegisterType("sleep", 0, func(d time.Duration) error { 530 | time.Sleep(d) 531 | return nil 532 | }) 533 | if err != nil { 534 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 535 | } 536 | 537 | // Queue up a single job 538 | sleepDuration := 10 * time.Millisecond 539 | job, err := sleepJob.Schedule(100, time.Now(), sleepDuration) 540 | if err != nil { 541 | t.Errorf("Unexpected error in sleepJob.Schedule(): %s", err.Error()) 542 | } 543 | 544 | // Start a new pool with 1 worker 545 | pool, err := NewPool(&PoolConfig{ 546 | NumWorkers: 1, 547 | BatchSize: 1, 548 | }) 549 | if err != nil { 550 | t.Fatal(err) 551 | } 552 | poolStarted := time.Now() 553 | if err := pool.Start(); err != nil { 554 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 555 | } 556 | 557 | // Immediately stop the pool and wait for workers to finish 558 | pool.Close() 559 | if err := pool.Wait(); err != nil { 560 | t.Errorf("Unexpected error in Pool.Wait(): %s", err.Error()) 561 | } 562 | poolClosed := time.Now() 563 | 564 | // Update our copy of the job 565 | if err := job.Refresh(); err != nil { 566 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 567 | } 568 | 569 | // Make sure that the timestamps are correct 570 | expectTimeNotZero(t, job.Started()) 571 | expectTimeBetween(t, job.Started(), poolClosed, poolStarted) 572 | expectTimeNotZero(t, job.Finished()) 573 | expectTimeBetween(t, job.Finished(), poolClosed, poolStarted) 574 | expectDurationNotZero(t, job.Duration()) 575 | expectDurationBetween(t, job.Duration(), sleepDuration, poolClosed.Sub(poolStarted)) 576 | } 577 | 578 | // TestRecurringJob creates and executes a recurring job, then makes sure that the 579 | // job is actually executed with the expected frequency. 580 | func TestRecurringJob(t *testing.T) { 581 | testingSetUp() 582 | defer testingTeardown() 583 | 584 | // Create a new pool with 1 worker 585 | pool, err := NewPool(&PoolConfig{ 586 | NumWorkers: 1, 587 | BatchSize: 1, 588 | MinWait: 1 * time.Millisecond, 589 | }) 590 | if err != nil { 591 | t.Fatal(err) 592 | } 593 | defer func() { 594 | // Close the pool and wait for workers to finish 595 | pool.Close() 596 | if err := pool.Wait(); err != nil { 597 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 598 | } 599 | }() 600 | 601 | // Register a job type which will simply send through to a channel 602 | jobFinished := make(chan bool) 603 | signalJob, err := RegisterType("signalJob", 0, func() error { 604 | jobFinished <- true 605 | return nil 606 | }) 607 | if err != nil { 608 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 609 | } 610 | 611 | // Schedule a recurring signalJob 612 | const freq = 20 * time.Millisecond 613 | job, err := signalJob.ScheduleRecurring(100, time.Now(), freq, nil) 614 | if err != nil { 615 | t.Errorf("Unexpected error in ScheduleRecurring: %s", err.Error()) 616 | } 617 | 618 | // Start the pool 619 | if err := pool.Start(); err != nil { 620 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 621 | } 622 | 623 | // Wait for three successful scheduled executions at the specified 624 | // frequency, with some tolerance for variation due to execution overhead. 625 | expectedSuccesses := 5 626 | successCount := 0 627 | const tolerance = 0.1 628 | timeoutDur := time.Duration(int64(float64(freq.Nanoseconds()) * (1 + tolerance))) 629 | OuterLoop: 630 | for { 631 | timeout := time.Tick(timeoutDur) 632 | select { 633 | case <-jobFinished: 634 | // This means one more job was successfully executed 635 | successCount += 1 636 | lastTime := job.time 637 | nextTime := job.NextTime() 638 | if !(nextTime > lastTime) { 639 | t.Errorf("job.NextTime was calculated incorrectly. %d is not greater than %d", nextTime, lastTime) 640 | } else if nextTime-lastTime != freq.Nanoseconds() { 641 | t.Errorf("job.NextTime was calculated incorrectly. %d - %d is not the frequency (%d)", nextTime, lastTime, freq) 642 | } 643 | if err := job.Refresh(); err != nil { 644 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 645 | } 646 | // If we reached expectedSuccesses, we're done and the test passes! 647 | if successCount == expectedSuccesses { 648 | break OuterLoop 649 | } 650 | case <-timeout: 651 | t.Errorf("Expected %d jobs to execute within %v each, but only %d jobs executed successfully. There was a timeout for the %s job", expectedSuccesses, timeoutDur, successCount, humanize.Ordinal(successCount+1)) 652 | t.FailNow() 653 | } 654 | } 655 | } 656 | 657 | // TestJobFailError creates and executes a job that is guaranteed to fail by returning an error, 658 | // then tests that the error was captured and stored correctly and that the job status was 659 | // set to failed. 660 | func TestJobFailError(t *testing.T) { 661 | testingSetUp() 662 | defer testingTeardown() 663 | 664 | // Register a job type which will do nothing but sleep for some duration 665 | errorJob, err := RegisterType("errorJob", 0, func(msg string) error { 666 | return fmt.Errorf(msg) 667 | }) 668 | if err != nil { 669 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 670 | } 671 | testJobFail(t, errorJob) 672 | } 673 | 674 | // TestJobFailPanic creates and executes a job that is guaranteed to fail by panicking, 675 | // then tests that the error was captured and stored correctly and that the job status 676 | // was set to failed. 677 | func TestJobFailPanic(t *testing.T) { 678 | testingSetUp() 679 | defer testingTeardown() 680 | 681 | // Register a job type which immediately panic 682 | panicJob, err := RegisterType("panicJob", 0, func(msg string) error { 683 | panic(errors.New(msg)) 684 | }) 685 | if err != nil { 686 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 687 | } 688 | testJobFail(t, panicJob) 689 | } 690 | 691 | // testJobFail tests that jobs of the given jobType fail correctly. The given jobType must 692 | // have a HandlerFunc which accepts a string argument and then always fails. The string argument 693 | // should be the returned error value or the message sent to panic. 694 | func testJobFail(t *testing.T, jobType *Type) { 695 | // Queue up a single job 696 | failMsg := "Test Job Failed!" 697 | job, err := jobType.Schedule(100, time.Now(), failMsg) 698 | if err != nil { 699 | t.Errorf("Unexpected error in %s.Schedule(): %s", jobType.String(), err.Error()) 700 | } 701 | 702 | // Start a new pool with 1 worker 703 | pool, err := NewPool(&PoolConfig{ 704 | NumWorkers: 1, 705 | BatchSize: 1, 706 | }) 707 | if err != nil { 708 | t.Fatal(err) 709 | } 710 | if err := pool.Start(); err != nil { 711 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 712 | } 713 | 714 | // Immediately stop the pool and wait for workers to finish 715 | pool.Close() 716 | if err := pool.Wait(); err != nil { 717 | t.Errorf("Unexpected error in Pool.Wait(): %s", err.Error()) 718 | } 719 | 720 | // Update our copy of the job 721 | if err := job.Refresh(); err != nil { 722 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 723 | } 724 | 725 | // Make sure that the error field is correct and that the job was 726 | // moved to the failed set 727 | expectJobFieldEquals(t, job, "error", failMsg, stringConverter) 728 | expectStatusEquals(t, job, StatusFailed) 729 | } 730 | 731 | // TestRetryJob creates and executes a job that is guaranteed to fail, then tests that 732 | // the job is tried some number of times before finally failing. 733 | func TestRetryJob(t *testing.T) { 734 | testingSetUp() 735 | defer testingTeardown() 736 | 737 | // Create a new pool with 4 worker 738 | pool, err := NewPool(&PoolConfig{ 739 | NumWorkers: 4, 740 | BatchSize: 4, 741 | MinWait: 1 * time.Millisecond, 742 | }) 743 | if err != nil { 744 | t.Fatal(err) 745 | } 746 | defer func() { 747 | // Close the pool and wait for workers to finish 748 | pool.Close() 749 | if err := pool.Wait(); err != nil { 750 | t.Errorf("Unexpected error in pool.Wait(): %s", err.Error()) 751 | } 752 | }() 753 | 754 | // Register a job type which will increment a counter with the number of tries 755 | tries := uint(0) 756 | triesMut := sync.Mutex{} 757 | retries := uint(5) 758 | expectedTries := retries + 1 759 | jobFailed := make(chan bool) 760 | countTriesJob, err := RegisterType("countTriesJob", retries, func() error { 761 | triesMut.Lock() 762 | tries += 1 763 | done := tries == expectedTries 764 | triesMut.Unlock() 765 | if done { 766 | jobFailed <- true 767 | } 768 | msg := fmt.Sprintf("job failed on the %s try", humanize.Ordinal(int(tries))) 769 | panic(msg) 770 | }) 771 | if err != nil { 772 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 773 | } 774 | 775 | // Queue up a single job 776 | if _, err := countTriesJob.Schedule(100, time.Now(), nil); err != nil { 777 | t.Errorf("Unexpected error in countTriesJob.Schedule(): %s", err.Error()) 778 | } 779 | 780 | // Start the pool 781 | if err := pool.Start(); err != nil { 782 | t.Errorf("Unexpected error in pool.Start(): %s", err.Error()) 783 | } 784 | 785 | // Wait for the job failed signal, or timeout if we don't receive it within 1 second 786 | timeout := time.After(1 * time.Second) 787 | OuterLoop: 788 | for { 789 | select { 790 | case <-timeout: 791 | // More than 1 second has passed. Assume something went wrong. 792 | t.Errorf("1 second passed and the job never permanently failed. The job was tried %d times.", tries) 793 | t.FailNow() 794 | case <-jobFailed: 795 | if tries != expectedTries { 796 | t.Errorf("The job was not tried the right number of times. Expected %d but job was only tried %d times.", expectedTries, tries) 797 | } else { 798 | // The test should pass! 799 | break OuterLoop 800 | } 801 | } 802 | } 803 | } 804 | 805 | // TestStalePoolsArePurged tests that stale pools are properly purged when an active pool starts. 806 | // It does this by manually instantiating a pool, queueing some jobs in it, and then causing it to 807 | // go stale by changing its id (effectively preventing it from replying to pings). 808 | func TestStalePoolsArePurged(t *testing.T) { 809 | testingSetUp() 810 | defer testingTeardown() 811 | 812 | // Create and start a pool with one worker 813 | stalePool, err := NewPool(&PoolConfig{ 814 | NumWorkers: 1, 815 | BatchSize: 1, 816 | MinWait: 1 * time.Millisecond, 817 | StaleTimeout: 20 * time.Millisecond, 818 | }) 819 | if err != nil { 820 | t.Fatal(err) 821 | } 822 | stalePool.id = "stalePool" 823 | if err := stalePool.Start(); err != nil { 824 | t.Errorf("Unexpected error in stalePool.Start(): %s", err.Error()) 825 | } 826 | 827 | // Create another pool with similar config but don't 828 | // start it yet 829 | newPool, err := NewPool(&PoolConfig{ 830 | NumWorkers: 1, 831 | BatchSize: 1, 832 | MinWait: 1 * time.Millisecond, 833 | StaleTimeout: 20 * time.Millisecond, 834 | }) 835 | if err != nil { 836 | t.Fatal(err) 837 | } 838 | 839 | jobsCanFinish := make(chan bool) 840 | stalePoolNeedsClose := true 841 | defer func() { 842 | // Indicate that all outstanding jobs can finish by closing the channel 843 | close(jobsCanFinish) 844 | // Close both pools and wait for workers to finish 845 | newPool.Close() 846 | if err := newPool.Wait(); err != nil { 847 | t.Errorf("Unexpected error in newPool.Wait(): %s", err.Error()) 848 | } 849 | if stalePoolNeedsClose { 850 | stalePool.Close() 851 | } 852 | if err := stalePool.Wait(); err != nil { 853 | t.Errorf("Unexpected error in stalePool.Wait(): %s", err.Error()) 854 | } 855 | }() 856 | 857 | // Register a job type which will signal and then wait for a channel to close 858 | // before finishing 859 | jobStarted := make(chan bool) 860 | signalAndWaitJob, err := RegisterType("signalAndWaitJob", 0, func() error { 861 | jobStarted <- true 862 | for range jobsCanFinish { 863 | } 864 | return nil 865 | }) 866 | if err != nil { 867 | t.Fatalf("Unexpected error in RegisterType: %s", err.Error()) 868 | } 869 | 870 | // Queue up a job 871 | job, err := signalAndWaitJob.Schedule(100, time.Now(), nil) 872 | if err != nil { 873 | t.Errorf("Error in signalAndWaitJob.Schedule: %s", err.Error()) 874 | } 875 | 876 | // Wait for the job to start 877 | <-jobStarted 878 | 879 | // Now change the id of the stalePool so that it will no longer reply to pings properly 880 | oldId := stalePool.id 881 | oldPingKey := stalePool.pingKey() 882 | stalePool.Lock() 883 | stalePool.id = "invalidId" 884 | stalePool.Unlock() 885 | 886 | // Create a conn we can use to listen for the stale pool to be pinged 887 | ping := &redis.PubSubConn{Conn: redisPool.Get()} 888 | if err := ping.Subscribe(oldPingKey); err != nil { 889 | t.Errorf("Unexpected error in ping.Subscribe(): %s", err.Error()) 890 | } 891 | pingChan := make(chan interface{}) 892 | go func() { 893 | defer ping.Close() 894 | for { 895 | reply := ping.Receive() 896 | switch reply.(type) { 897 | case redis.Message: 898 | // The ping was received 899 | pingChan <- reply 900 | return 901 | case error: 902 | err := reply.(error) 903 | panic(err) 904 | } 905 | time.Sleep(1 * time.Millisecond) 906 | } 907 | }() 908 | 909 | // Start the new pool. We expect this to trigger a purge of the stale pool 910 | if err := newPool.Start(); err != nil { 911 | t.Errorf("Unexpected error in newPool.Start(): %s", err.Error()) 912 | } 913 | 914 | // Wait for the stale pool to be pinged or timeout after 1 second 915 | timeout := time.After(1 * time.Second) 916 | select { 917 | case <-pingChan: 918 | // If we received a ping, close the stale pool and continue with the test 919 | stalePool.Close() 920 | stalePoolNeedsClose = false 921 | case <-timeout: 922 | fmt.Println("timeout") 923 | t.Errorf("1 second passed but the stale pool was never pinged") 924 | t.FailNow() 925 | return 926 | } 927 | 928 | // If we've reached here, the stale pool was pinged. We should wait to receive 929 | // from the channel again to indicate that the job was requeued and picked up by 930 | // the new pool. 931 | timeout = time.After(1 * time.Second) 932 | select { 933 | case <-jobStarted: 934 | // If the job started again, continue with the test 935 | case <-timeout: 936 | fmt.Println("timeout") 937 | t.Errorf("1 second passed but the job was never started again.") 938 | t.FailNow() 939 | return 940 | } 941 | 942 | // At this point, the stale pool should have been fully purged. 943 | expectSetDoesNotContain(t, Keys.ActivePools, oldId) 944 | expectJobFieldEquals(t, job, "poolId", newPool.id, stringConverter) 945 | } 946 | 947 | // expectTestDataOk reports an error via t.Errorf if any elements in data do not equal "ok". It is only 948 | // used for tests in this file. Many of the tests use a slice of strings as data and queue up jobs to 949 | // set one of the elements to "ok", so this makes checking them easier. 950 | func expectTestDataOk(t *testing.T, data []string) { 951 | for i, datum := range data { 952 | if datum != "ok" { 953 | t.Errorf("Expected data[%d] to be \"ok\" but got: \"%s\"\ndata was: %v.", i, datum, data) 954 | } 955 | } 956 | } 957 | 958 | // expectTestDataBlank is like expectTestDataOk except it does the opposite. It reports an error if any 959 | // of the elements in data were not blank. 960 | func expectTestDataBlank(t *testing.T, data []string) { 961 | for i, datum := range data { 962 | if datum != "" { 963 | t.Errorf("Expected data[%d] to be \"\" but got: \"%s\"\ndata was: %v.", i, datum, data) 964 | } 965 | } 966 | } 967 | 968 | // expectTimeNotZero reports an error via t.Errorf if x is equal to the zero time. 969 | func expectTimeNotZero(t *testing.T, x time.Time) { 970 | if x.IsZero() { 971 | t.Errorf("Expected time x to be non-zero but got zero.") 972 | } 973 | } 974 | 975 | // expectTimeAfter reports an error via t.Errorf if x is not after the given time. 976 | func expectTimeAfter(t *testing.T, x, after time.Time) { 977 | if !x.After(after) { 978 | t.Errorf("time x was incorrect. Expected it to be after %v but got %v.", after, x) 979 | } 980 | } 981 | 982 | // expectTimeBefore reports an error via t.Errorf if x is not before the given time. 983 | func expectTimeBefore(t *testing.T, x, before time.Time) { 984 | if !x.Before(before) { 985 | t.Errorf("time x was incorrect. Expected it to be before %v but got %v.", before, x) 986 | } 987 | } 988 | 989 | // expectTimeBetween reports an error via t.Errorf if x is not before and after the given times. 990 | func expectTimeBetween(t *testing.T, x, before, after time.Time) { 991 | expectTimeBefore(t, x, before) 992 | expectTimeAfter(t, x, after) 993 | } 994 | 995 | // expectDurationNotZero reports an error via t.Errorf if d is equal to zero. 996 | func expectDurationNotZero(t *testing.T, d time.Duration) { 997 | if d.Nanoseconds() == 0 { 998 | t.Errorf("Expected duration d to be non-zero but got zero.") 999 | } 1000 | } 1001 | 1002 | // expectDurationBetween reports an error via t.Errorf if d is not more than min and less than max. 1003 | func expectDurationBetween(t *testing.T, d, min, max time.Duration) { 1004 | if !(d > min) { 1005 | t.Errorf("duration d was incorrect. Expected it to be more than %v but got %v.", min, d) 1006 | } 1007 | if !(d < max) { 1008 | t.Errorf("duration d was incorrect. Expected it to be less than %v but got %v.", max, d) 1009 | } 1010 | } 1011 | -------------------------------------------------------------------------------- /redis_keys.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | // keys stores any constant redis keys. By storing them all here, 8 | // we avoid using string literals which are prone to typos. 9 | var Keys = struct { 10 | // jobsTimeIndex is the key for a sorted set which keeps all outstanding 11 | // jobs sorted by their time field. 12 | JobsTimeIndex string 13 | // jobsTemp is the key for a temporary set which is created and then destroyed 14 | // during the process of getting the next jobs in the queue. 15 | JobsTemp string 16 | // activePools is the key for a set which holds the pool ids for all active 17 | // pools. 18 | ActivePools string 19 | }{ 20 | JobsTimeIndex: "jobs:time", 21 | JobsTemp: "jobs:temp", 22 | ActivePools: "pools:active", 23 | } 24 | -------------------------------------------------------------------------------- /redis_pool.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "github.com/garyburd/redigo/redis" 9 | "time" 10 | ) 11 | 12 | // redisPool is a thread-safe pool of redis connections. Use the Get method 13 | // to get a new connection. 14 | var redisPool = &redis.Pool{ 15 | MaxIdle: 10, 16 | MaxActive: 0, 17 | IdleTimeout: 240 * time.Second, 18 | Dial: func() (redis.Conn, error) { 19 | c, err := redis.Dial(Config.Db.Network, Config.Db.Address) 20 | if err != nil { 21 | return nil, err 22 | } 23 | // If a password was provided, use the AUTH command to authenticate 24 | if Config.Db.Password != "" { 25 | _, err = c.Do("AUTH", Config.Db.Password) 26 | if err != nil { 27 | return nil, err 28 | } 29 | } 30 | // Connect to the appropriate database with SELECT 31 | if _, err := c.Do("SELECT", Config.Db.Database); err != nil { 32 | c.Close() 33 | return nil, err 34 | } 35 | return c, nil 36 | }, 37 | } 38 | -------------------------------------------------------------------------------- /scripts.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | // File scripts.go contains code related to parsing 6 | // lua scripts in the scripts file. 7 | 8 | // This file has been automatically generated by go generate, 9 | // which calls scripts/generate.go. Do not edit it directly! 10 | 11 | package jobs 12 | 13 | import ( 14 | "github.com/garyburd/redigo/redis" 15 | ) 16 | 17 | var ( 18 | 19 | addJobToSetScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 20 | -- Use of this source code is governed by the MIT 21 | -- license, which can be found in the LICENSE file. 22 | 23 | -- add_job_to_set represents a lua script that takes the following arguments: 24 | -- 1) The id of the job 25 | -- 2) The name of a sorted set 26 | -- 3) The score the inserted job should have in the sorted set 27 | -- It first checks if the job exists in the database (has not been destroyed) 28 | -- and then adds it to the sorted set with the given score. 29 | 30 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 31 | 32 | local jobId = ARGV[1] 33 | local setName = ARGV[2] 34 | local score = ARGV[3] 35 | local jobKey = 'jobs:' .. jobId 36 | -- Make sure the job hasn't already been destroyed 37 | local exists = redis.call('EXISTS', jobKey) 38 | if exists ~= 1 then 39 | return 40 | end 41 | redis.call('ZADD', setName, score, jobId)`) 42 | destroyJobScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 43 | -- Use of this source code is governed by the MIT 44 | -- license, which can be found in the LICENSE file. 45 | 46 | -- destroy_job is a lua script that takes the following arguments: 47 | -- 1) The id of the job to destroy 48 | -- It then removes all traces of the job in the database by doing the following: 49 | -- 1) Removes the job from the status set (which it determines with an HGET call) 50 | -- 2) Removes the job from the time index 51 | -- 3) Removes the main hash for the job 52 | 53 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 54 | 55 | -- Assign args to variables for easy reference 56 | local jobId = ARGV[1] 57 | local jobKey = 'jobs:' .. jobId 58 | -- Remove the job from the status set 59 | local status = redis.call('HGET', jobKey, 'status') 60 | if status ~= '' then 61 | local statusSet = 'jobs:' .. status 62 | redis.call('ZREM', statusSet, jobId) 63 | end 64 | -- Remove the job from the time index 65 | redis.call('ZREM', 'jobs:time', jobId) 66 | -- Remove the main hash for the job 67 | redis.call('DEL', jobKey)`) 68 | getJobsByIdsScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 69 | -- Use of this source code is governed by the MIT 70 | -- license, which can be found in the LICENSE file. 71 | 72 | -- get_jobs_by_ids is a lua script that takes the following arguments: 73 | -- 1) The key of a sorted set of some job ids 74 | -- The script then gets all the data for those job ids from their respective 75 | -- hashes in the database. It returns an array of arrays where each element 76 | -- contains the fields for a particular job, and the jobs are sorted by 77 | -- priority. 78 | -- Here's an example response: 79 | -- [ 80 | -- [ 81 | -- "id", "afj9afjpa30", 82 | -- "data", [34, 67, 34, 23, 56, 67, 78, 79], 83 | -- "type", "emailJob", 84 | -- "time", 1234567, 85 | -- "freq", 0, 86 | -- "priority", 100, 87 | -- "retries", 0, 88 | -- "status", "executing", 89 | -- "started", 0, 90 | -- "finished", 0, 91 | -- ], 92 | -- [ 93 | -- "id", "E8v2ovkdaIw", 94 | -- "data", [46, 43, 12, 08, 34, 45, 57, 43], 95 | -- "type", "emailJob", 96 | -- "time", 1234568, 97 | -- "freq", 0, 98 | -- "priority", 95, 99 | -- "retries", 0, 100 | -- "status", "executing", 101 | -- "started", 0, 102 | -- "finished", 0, 103 | -- ] 104 | -- ] 105 | 106 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 107 | 108 | -- Assign keys to variables for easy access 109 | local setKey = ARGV[1] 110 | -- Get all the ids from the set name 111 | local jobIds = redis.call('ZREVRANGE', setKey, 0, -1) 112 | local allJobs = {} 113 | if #jobIds > 0 then 114 | -- Iterate over the ids and find each job 115 | for i, jobId in ipairs(jobIds) do 116 | local jobKey = 'jobs:' .. jobId 117 | local jobFields = redis.call('HGETALL', jobKey) 118 | -- Add the id itself to the fields 119 | jobFields[#jobFields+1] = 'id' 120 | jobFields[#jobFields+1] = jobId 121 | -- Add the field values to allJobs 122 | allJobs[#allJobs+1] = jobFields 123 | end 124 | end 125 | return allJobs`) 126 | popNextJobsScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 127 | -- Use of this source code is governed by the MIT 128 | -- license, which can be found in the LICENSE file. 129 | 130 | -- pop_next_jobs is a lua script that takes the following arguments: 131 | -- 1) The maximum number of jobs to pop and return 132 | -- 2) The current unix time UTC with nanosecond precision 133 | -- The script gets the next available jobs from the queued set which are 134 | -- ready based on their time parameter. Then it adds those jobs to the 135 | -- executing set, sets their status to executing, and removes them from the 136 | -- queued set. It returns an array of arrays where each element contains the 137 | -- fields for a particular job, and the jobs are sorted by priority. 138 | -- Here's an example response: 139 | -- [ 140 | -- [ 141 | -- "id", "afj9afjpa30", 142 | -- "data", [34, 67, 34, 23, 56, 67, 78, 79], 143 | -- "type", "emailJob", 144 | -- "time", 1234567, 145 | -- "freq", 0, 146 | -- "priority", 100, 147 | -- "retries", 0, 148 | -- "status", "executing", 149 | -- "started", 0, 150 | -- "finished", 0, 151 | -- ], 152 | -- [ 153 | -- "id", "E8v2ovkdaIw", 154 | -- "data", [46, 43, 12, 08, 34, 45, 57, 43], 155 | -- "type", "emailJob", 156 | -- "time", 1234568, 157 | -- "freq", 0, 158 | -- "priority", 95, 159 | -- "retries", 0, 160 | -- "status", "executing", 161 | -- "started", 0, 162 | -- "finished", 0, 163 | -- ] 164 | -- ] 165 | 166 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 167 | 168 | -- Assign args to variables for easy reference 169 | local n = ARGV[1] 170 | local currentTime = ARGV[2] 171 | local poolId = ARGV[3] 172 | -- Copy the time index set to a new temporary set 173 | redis.call('ZUNIONSTORE', 'jobs:temp', 1, 'jobs:time') 174 | -- Trim the new temporary set we just created to leave only the jobs which have a time 175 | -- parameter in the past 176 | redis.call('ZREMRANGEBYSCORE', 'jobs:temp', currentTime, '+inf') 177 | -- Intersect the jobs which are ready based on their time with those in the 178 | -- queued set. Use the weights parameter to set the scores entirely based on the 179 | -- queued set, effectively sorting the jobs by priority. Store the results in the 180 | -- temporary set. 181 | redis.call('ZINTERSTORE', 'jobs:temp', 2, 'jobs:queued', 'jobs:temp', 'WEIGHTS', 1, 0) 182 | -- Trim the temp set, so it contains only the first n jobs ordered by 183 | -- priority 184 | redis.call('ZREMRANGEBYRANK', 'jobs:temp', 0, -n - 1) 185 | -- Get all job ids from the temp set 186 | local jobIds = redis.call('ZREVRANGE', 'jobs:temp', 0, -1) 187 | local allJobs = {} 188 | if #jobIds > 0 then 189 | -- Add job ids to the executing set 190 | redis.call('ZUNIONSTORE', 'jobs:executing', 2, 'jobs:executing', 'jobs:temp') 191 | -- Now we are ready to construct our response. 192 | for i, jobId in ipairs(jobIds) do 193 | local jobKey = 'jobs:' .. jobId 194 | -- Remove the job from the queued set 195 | redis.call('ZREM', 'jobs:queued', jobId) 196 | -- Set the poolId field for the job 197 | redis.call('HSET', jobKey, 'poolId', poolId) 198 | -- Set the job status to executing 199 | redis.call('HSET', jobKey, 'status', 'executing') 200 | -- Get the fields from its main hash 201 | local jobFields = redis.call('HGETALL', jobKey) 202 | -- Add the id itself to the fields 203 | jobFields[#jobFields+1] = 'id' 204 | jobFields[#jobFields+1] = jobId 205 | -- Add the field values to allJobs 206 | allJobs[#allJobs+1] = jobFields 207 | end 208 | end 209 | -- Delete the temporary set 210 | redis.call('DEL', 'jobs:temp') 211 | -- Return all the fields for all the jobs 212 | return allJobs`) 213 | purgeStalePoolScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 214 | -- Use of this source code is governed by the MIT 215 | -- license, which can be found in the LICENSE file. 216 | 217 | -- purge_stale_pool is a lua script which takes the following arguments: 218 | -- 1) The id of the stale pool to purge 219 | -- It then does the following: 220 | -- 1) Removes the pool id from the set of active pools 221 | -- 2) Iterates through each job in the executing set and finds any jobs which 222 | -- have a poolId field equal to the id of the stale pool 223 | -- 3) If it finds any such jobs, it removes them from the executing set and 224 | -- adds them to the queued so that they will be retried 225 | 226 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 227 | 228 | -- Assign args to variables for easy reference 229 | local stalePoolId = ARGV[1] 230 | -- Check if the stale pool is in the set of active pools first 231 | local isActive = redis.call('SISMEMBER', 'pools:active', stalePoolId) 232 | if isActive then 233 | -- Remove the stale pool from the set of active pools 234 | redis.call('SREM', 'pools:active', stalePoolId) 235 | -- Get all the jobs in the executing set 236 | local jobIds = redis.call('ZRANGE', 'jobs:executing', 0, -1) 237 | for i, jobId in ipairs(jobIds) do 238 | local jobKey = 'jobs:' .. jobId 239 | -- Check the poolId field 240 | -- If the poolId is equal to the stale id, then this job is stuck 241 | -- in the executing set even though no worker is actually executing it 242 | local poolId = redis.call('HGET', jobKey, 'poolId') 243 | if poolId == stalePoolId then 244 | local jobPriority = redis.call('HGET', jobKey, 'priority') 245 | -- Move the job into the queued set 246 | redis.call('ZADD', 'jobs:queued', jobPriority, jobId) 247 | -- Remove the job from the executing set 248 | redis.call('ZREM', 'jobs:executing', jobId) 249 | -- Set the job status to queued and the pool id to blank 250 | redis.call('HMSET', jobKey, 'status', 'queued', 'poolId', '') 251 | end 252 | end 253 | end 254 | `) 255 | retryOrFailJobScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 256 | -- Use of this source code is governed by the MIT 257 | -- license, which can be found in the LICENSE file. 258 | 259 | -- retry_or_fail_job represents a lua script that takes the following arguments: 260 | -- 1) The id of the job to either retry or fail 261 | -- It first checks if the job has any retries remaining. If it does, 262 | -- then it: 263 | -- 1) Decrements the number of retries for the given job 264 | -- 2) Adds the job to the queued set 265 | -- 3) Removes the job from the executing set 266 | -- 4) Returns true 267 | -- If the job has no retries remaining then it: 268 | -- 1) Adds the job to the failed set 269 | -- 3) Removes the job from the executing set 270 | -- 2) Returns false 271 | 272 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 273 | 274 | -- Assign args to variables for easy reference 275 | local jobId = ARGV[1] 276 | local jobKey = 'jobs:' .. jobId 277 | -- Make sure the job hasn't already been destroyed 278 | local exists = redis.call('EXISTS', jobKey) 279 | if exists ~= 1 then 280 | return 0 281 | end 282 | -- Check how many retries remain 283 | local retries = redis.call('HGET', jobKey, 'retries') 284 | local newStatus = '' 285 | if retries == '0' then 286 | -- newStatus should be failed because there are no retries left 287 | newStatus = 'failed' 288 | else 289 | -- subtract 1 from the remaining retries 290 | redis.call('HINCRBY', jobKey, 'retries', -1) 291 | -- newStatus should be queued, so the job will be retried 292 | newStatus = 'queued' 293 | end 294 | -- Get the job priority (used as score) 295 | local jobPriority = redis.call('HGET', jobKey, 'priority') 296 | -- Add the job to the appropriate new set 297 | local newStatusSet = 'jobs:' .. newStatus 298 | redis.call('ZADD', newStatusSet, jobPriority, jobId) 299 | -- Remove the job from the old status set 300 | local oldStatus = redis.call('HGET', jobKey, 'status') 301 | if ((oldStatus ~= '') and (oldStatus ~= newStatus)) then 302 | local oldStatusSet = 'jobs:' .. oldStatus 303 | redis.call('ZREM', oldStatusSet, jobId) 304 | end 305 | -- Set the job status in the hash 306 | redis.call('HSET', jobKey, 'status', newStatus) 307 | if retries == '0' then 308 | -- Return false to indicate the job has not been queued for retry 309 | -- NOTE: 0 is used to represent false because apparently 310 | -- false gets converted to nil 311 | return 0 312 | else 313 | -- Return true to indicate the job has been queued for retry 314 | -- NOTE: 1 is used to represent true (for consistency) 315 | return 1 316 | end`) 317 | setJobFieldScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 318 | -- Use of this source code is governed by the MIT 319 | -- license, which can be found in the LICENSE file. 320 | 321 | -- set_job_field represents a lua script that takes the following arguments: 322 | -- 1) The id of the job 323 | -- 2) The name of the field 324 | -- 3) The value to set the field to 325 | -- It first checks if the job exists in the database (has not been destroyed) 326 | -- and then sets the given field to the given value. 327 | 328 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 329 | 330 | local jobId = ARGV[1] 331 | local fieldName = ARGV[2] 332 | local fieldVal = ARGV[3] 333 | local jobKey = 'jobs:' .. jobId 334 | -- Make sure the job hasn't already been destroyed 335 | local exists = redis.call('EXISTS', jobKey) 336 | if exists ~= 1 then 337 | return 338 | end 339 | redis.call('HSET', jobKey, fieldName, fieldVal)`) 340 | setJobStatusScript = redis.NewScript(0, `-- Copyright 2015 Alex Browne. All rights reserved. 341 | -- Use of this source code is governed by the MIT 342 | -- license, which can be found in the LICENSE file. 343 | 344 | -- set_job_status is a lua script that takes the following arguments: 345 | -- 1) The id of the job 346 | -- 2) The new status (e.g. "queued") 347 | -- It then does the following: 348 | -- 1) Adds the job to the new status set 349 | -- 2) Removes the job from the old status set (which it gets with an HGET call) 350 | -- 3) Sets the 'status' field in the main hash for the job 351 | 352 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 353 | 354 | -- Assign args to variables for easy reference 355 | local jobId = ARGV[1] 356 | local newStatus = ARGV[2] 357 | local jobKey = 'jobs:' .. jobId 358 | -- Make sure the job hasn't already been destroyed 359 | local exists = redis.call('EXISTS', jobKey) 360 | if exists ~= 1 then 361 | return 362 | end 363 | local newStatusSet = 'jobs:' .. newStatus 364 | -- Add the job to the new status set 365 | local jobPriority = redis.call('HGET', jobKey, 'priority') 366 | redis.call('ZADD', newStatusSet, jobPriority, jobId) 367 | -- Remove the job from the old status set 368 | local oldStatus = redis.call('HGET', jobKey, 'status') 369 | if ((oldStatus ~= '') and (oldStatus ~= newStatus)) then 370 | local oldStatusSet = 'jobs:' .. oldStatus 371 | redis.call('ZREM', oldStatusSet, jobId) 372 | end 373 | -- Set the status field 374 | redis.call('HSET', jobKey, 'status', newStatus)`) 375 | ) -------------------------------------------------------------------------------- /scripts/add_job_to_set.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- add_job_to_set represents a lua script that takes the following arguments: 6 | -- 1) The id of the job 7 | -- 2) The name of a sorted set 8 | -- 3) The score the inserted job should have in the sorted set 9 | -- It first checks if the job exists in the database (has not been destroyed) 10 | -- and then adds it to the sorted set with the given score. 11 | 12 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 13 | 14 | local jobId = ARGV[1] 15 | local setName = ARGV[2] 16 | local score = ARGV[3] 17 | local jobKey = 'jobs:' .. jobId 18 | -- Make sure the job hasn't already been destroyed 19 | local exists = redis.call('EXISTS', jobKey) 20 | if exists ~= 1 then 21 | return 22 | end 23 | redis.call('ZADD', setName, score, jobId) -------------------------------------------------------------------------------- /scripts/destroy_job.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- destroy_job is a lua script that takes the following arguments: 6 | -- 1) The id of the job to destroy 7 | -- It then removes all traces of the job in the database by doing the following: 8 | -- 1) Removes the job from the status set (which it determines with an HGET call) 9 | -- 2) Removes the job from the time index 10 | -- 3) Removes the main hash for the job 11 | 12 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 13 | 14 | -- Assign args to variables for easy reference 15 | local jobId = ARGV[1] 16 | local jobKey = 'jobs:' .. jobId 17 | -- Remove the job from the status set 18 | local status = redis.call('HGET', jobKey, 'status') 19 | if status ~= '' then 20 | local statusSet = 'jobs:' .. status 21 | redis.call('ZREM', statusSet, jobId) 22 | end 23 | -- Remove the job from the time index 24 | redis.call('ZREM', '{{.timeIndexSet}}', jobId) 25 | -- Remove the main hash for the job 26 | redis.call('DEL', jobKey) -------------------------------------------------------------------------------- /scripts/generate.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | // File main.go is intended to be used with go generate. 6 | // It reads the contents of any .lua file ins the scripts 7 | // directory, then it generates a go source file called 8 | // scritps.go which converts the file contents to a string 9 | // and assigns each script to a variable so they can be invoked. 10 | 11 | package main 12 | 13 | import ( 14 | "bytes" 15 | "github.com/albrow/jobs" 16 | "go/build" 17 | "io/ioutil" 18 | "os" 19 | "path/filepath" 20 | "strings" 21 | "text/template" 22 | ) 23 | 24 | var ( 25 | // scriptsPath is the path of the directory which holds lua scripts. 26 | scriptsPath string 27 | // destPath is the path to a file where the generated go code will be written. 28 | destPath string 29 | // genTmplPath is the path to a .tmpl file which will be used to generate go code. 30 | genTmplPath string 31 | ) 32 | 33 | var ( 34 | // scriptContext is a map which is passed in as the context to all lua script templates. 35 | // It holds the keys for all the different status sets, the names of the sets, and the keys 36 | // for other constant sets. 37 | scriptContext = map[string]string{ 38 | "statusSaved": string(jobs.StatusSaved), 39 | "statusQueued": string(jobs.StatusQueued), 40 | "statusExecuting": string(jobs.StatusExecuting), 41 | "statusFinished": string(jobs.StatusFinished), 42 | "statusFailed": string(jobs.StatusFailed), 43 | "statusCancelled": string(jobs.StatusCancelled), 44 | "statusDestroyed": string(jobs.StatusDestroyed), 45 | "savedSet": jobs.StatusSaved.Key(), 46 | "queuedSet": jobs.StatusQueued.Key(), 47 | "executingSet": jobs.StatusExecuting.Key(), 48 | "finishedSet": jobs.StatusFinished.Key(), 49 | "failedSet": jobs.StatusFailed.Key(), 50 | "cancelledSet": jobs.StatusCancelled.Key(), 51 | "destroyedSet": jobs.StatusDestroyed.Key(), 52 | "timeIndexSet": jobs.Keys.JobsTimeIndex, 53 | "jobsTempSet": jobs.Keys.JobsTemp, 54 | "activePoolsSet": jobs.Keys.ActivePools, 55 | } 56 | ) 57 | 58 | // script is a representation of a lua script file. 59 | type script struct { 60 | // VarName is the variable name that the script will be assigned to in the generated go code. 61 | VarName string 62 | // RawSrc is the contents of the original .lua file, which is a template. 63 | RawSrc string 64 | // Src is the the result of executing RawSrc as a template. 65 | Src string 66 | } 67 | 68 | func init() { 69 | // Use build to find the directory where this file lives. This always works as 70 | // long as you have go installed, even if you have multiple GOPATHs or are using 71 | // dependency management tools. 72 | pkg, err := build.Import("github.com/albrow/jobs", "", build.FindOnly) 73 | if err != nil { 74 | panic(err) 75 | } 76 | // Configure the required paths 77 | scriptsPath = filepath.Join(pkg.Dir, "scripts") 78 | destPath = filepath.Clean(filepath.Join(scriptsPath, "..", "scripts.go")) 79 | genTmplPath = filepath.Join(scriptsPath, "scripts.go.tmpl") 80 | } 81 | 82 | func main() { 83 | scripts, err := findScripts(scriptsPath) 84 | if err != nil { 85 | panic(err) 86 | } 87 | if err := generateFile(scripts, genTmplPath, destPath); err != nil { 88 | panic(err) 89 | } 90 | } 91 | 92 | // findScripts finds all the .lua script files in the given path 93 | // and creates a script object for each one. It returns a slice of 94 | // scripts or an error if there was a problem reading any of the files. 95 | func findScripts(path string) ([]*script, error) { 96 | filenames, err := filepath.Glob(filepath.Join(path, "*.lua")) 97 | if err != nil { 98 | return nil, err 99 | } 100 | scripts := []*script{} 101 | for _, filename := range filenames { 102 | script := script{ 103 | VarName: convertUnderscoresToCamelCase(strings.TrimSuffix(filepath.Base(filename), ".lua")) + "Script", 104 | } 105 | src, err := ioutil.ReadFile(filename) 106 | if err != nil { 107 | return nil, err 108 | } 109 | script.RawSrc = string(src) 110 | scripts = append(scripts, &script) 111 | } 112 | return scripts, nil 113 | } 114 | 115 | // convertUnderscoresToCamelCase converts a string of the form 116 | // foo_bar_baz to fooBarBaz. 117 | func convertUnderscoresToCamelCase(s string) string { 118 | if len(s) == 0 { 119 | return "" 120 | } 121 | result := "" 122 | shouldUpper := false 123 | for _, char := range s { 124 | if char == '_' { 125 | shouldUpper = true 126 | continue 127 | } 128 | if shouldUpper { 129 | result += strings.ToUpper(string(char)) 130 | } else { 131 | result += string(char) 132 | } 133 | shouldUpper = false 134 | } 135 | return result 136 | } 137 | 138 | // generateFile generates go source code and writes to 139 | // the source file located at dest (creating it if needed). 140 | // It executes the template located at tmplFile with scripts 141 | // as the context. 142 | func generateFile(scripts []*script, tmplFile string, dest string) error { 143 | // Treat the contents of the script file as a template and execute 144 | // it with scriptsContext (a map of constant keys) as the data. 145 | buf := bytes.NewBuffer([]byte{}) 146 | for _, script := range scripts { 147 | scriptTmpl, err := template.New("script").Parse(script.RawSrc) 148 | if err != nil { 149 | return err 150 | } 151 | buf.Reset() 152 | if err := scriptTmpl.Execute(buf, scriptContext); err != nil { 153 | return err 154 | } 155 | script.Src = buf.String() 156 | } 157 | // Now generate the go source code by executing genTmpl. The generated 158 | // code uses the Src property of the scripts as the argument to redis.NewScript. 159 | genTmpl, err := template.ParseFiles(tmplFile) 160 | if err != nil { 161 | return err 162 | } 163 | destFile, err := os.Create(dest) 164 | if err != nil { 165 | return err 166 | } 167 | return genTmpl.Execute(destFile, scripts) 168 | } 169 | -------------------------------------------------------------------------------- /scripts/get_jobs_by_ids.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- get_jobs_by_ids is a lua script that takes the following arguments: 6 | -- 1) The key of a sorted set of some job ids 7 | -- The script then gets all the data for those job ids from their respective 8 | -- hashes in the database. It returns an array of arrays where each element 9 | -- contains the fields for a particular job, and the jobs are sorted by 10 | -- priority. 11 | -- Here's an example response: 12 | -- [ 13 | -- [ 14 | -- "id", "afj9afjpa30", 15 | -- "data", [34, 67, 34, 23, 56, 67, 78, 79], 16 | -- "type", "emailJob", 17 | -- "time", 1234567, 18 | -- "freq", 0, 19 | -- "priority", 100, 20 | -- "retries", 0, 21 | -- "status", "executing", 22 | -- "started", 0, 23 | -- "finished", 0, 24 | -- ], 25 | -- [ 26 | -- "id", "E8v2ovkdaIw", 27 | -- "data", [46, 43, 12, 08, 34, 45, 57, 43], 28 | -- "type", "emailJob", 29 | -- "time", 1234568, 30 | -- "freq", 0, 31 | -- "priority", 95, 32 | -- "retries", 0, 33 | -- "status", "executing", 34 | -- "started", 0, 35 | -- "finished", 0, 36 | -- ] 37 | -- ] 38 | 39 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 40 | 41 | -- Assign keys to variables for easy access 42 | local setKey = ARGV[1] 43 | -- Get all the ids from the set name 44 | local jobIds = redis.call('ZREVRANGE', setKey, 0, -1) 45 | local allJobs = {} 46 | if #jobIds > 0 then 47 | -- Iterate over the ids and find each job 48 | for i, jobId in ipairs(jobIds) do 49 | local jobKey = 'jobs:' .. jobId 50 | local jobFields = redis.call('HGETALL', jobKey) 51 | -- Add the id itself to the fields 52 | jobFields[#jobFields+1] = 'id' 53 | jobFields[#jobFields+1] = jobId 54 | -- Add the field values to allJobs 55 | allJobs[#allJobs+1] = jobFields 56 | end 57 | end 58 | return allJobs -------------------------------------------------------------------------------- /scripts/pop_next_jobs.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- pop_next_jobs is a lua script that takes the following arguments: 6 | -- 1) The maximum number of jobs to pop and return 7 | -- 2) The current unix time UTC with nanosecond precision 8 | -- The script gets the next available jobs from the queued set which are 9 | -- ready based on their time parameter. Then it adds those jobs to the 10 | -- executing set, sets their status to executing, and removes them from the 11 | -- queued set. It returns an array of arrays where each element contains the 12 | -- fields for a particular job, and the jobs are sorted by priority. 13 | -- Here's an example response: 14 | -- [ 15 | -- [ 16 | -- "id", "afj9afjpa30", 17 | -- "data", [34, 67, 34, 23, 56, 67, 78, 79], 18 | -- "type", "emailJob", 19 | -- "time", 1234567, 20 | -- "freq", 0, 21 | -- "priority", 100, 22 | -- "retries", 0, 23 | -- "status", "executing", 24 | -- "started", 0, 25 | -- "finished", 0, 26 | -- ], 27 | -- [ 28 | -- "id", "E8v2ovkdaIw", 29 | -- "data", [46, 43, 12, 08, 34, 45, 57, 43], 30 | -- "type", "emailJob", 31 | -- "time", 1234568, 32 | -- "freq", 0, 33 | -- "priority", 95, 34 | -- "retries", 0, 35 | -- "status", "executing", 36 | -- "started", 0, 37 | -- "finished", 0, 38 | -- ] 39 | -- ] 40 | 41 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 42 | 43 | -- Assign args to variables for easy reference 44 | local n = ARGV[1] 45 | local currentTime = ARGV[2] 46 | local poolId = ARGV[3] 47 | -- Copy the time index set to a new temporary set 48 | redis.call('ZUNIONSTORE', '{{.jobsTempSet}}', 1, '{{.timeIndexSet}}') 49 | -- Trim the new temporary set we just created to leave only the jobs which have a time 50 | -- parameter in the past 51 | redis.call('ZREMRANGEBYSCORE', '{{.jobsTempSet}}', currentTime, '+inf') 52 | -- Intersect the jobs which are ready based on their time with those in the 53 | -- queued set. Use the weights parameter to set the scores entirely based on the 54 | -- queued set, effectively sorting the jobs by priority. Store the results in the 55 | -- temporary set. 56 | redis.call('ZINTERSTORE', '{{.jobsTempSet}}', 2, '{{.queuedSet}}', '{{.jobsTempSet}}', 'WEIGHTS', 1, 0) 57 | -- Trim the temp set, so it contains only the first n jobs ordered by 58 | -- priority 59 | redis.call('ZREMRANGEBYRANK', '{{.jobsTempSet}}', 0, -n - 1) 60 | -- Get all job ids from the temp set 61 | local jobIds = redis.call('ZREVRANGE', '{{.jobsTempSet}}', 0, -1) 62 | local allJobs = {} 63 | if #jobIds > 0 then 64 | -- Add job ids to the executing set 65 | redis.call('ZUNIONSTORE', '{{.executingSet}}', 2, '{{.executingSet}}', '{{.jobsTempSet}}') 66 | -- Now we are ready to construct our response. 67 | for i, jobId in ipairs(jobIds) do 68 | local jobKey = 'jobs:' .. jobId 69 | -- Remove the job from the queued set 70 | redis.call('ZREM', '{{.queuedSet}}', jobId) 71 | -- Set the poolId field for the job 72 | redis.call('HSET', jobKey, 'poolId', poolId) 73 | -- Set the job status to executing 74 | redis.call('HSET', jobKey, 'status', '{{.statusExecuting}}') 75 | -- Get the fields from its main hash 76 | local jobFields = redis.call('HGETALL', jobKey) 77 | -- Add the id itself to the fields 78 | jobFields[#jobFields+1] = 'id' 79 | jobFields[#jobFields+1] = jobId 80 | -- Add the field values to allJobs 81 | allJobs[#allJobs+1] = jobFields 82 | end 83 | end 84 | -- Delete the temporary set 85 | redis.call('DEL', '{{.jobsTempSet}}') 86 | -- Return all the fields for all the jobs 87 | return allJobs -------------------------------------------------------------------------------- /scripts/purge_stale_pool.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- purge_stale_pool is a lua script which takes the following arguments: 6 | -- 1) The id of the stale pool to purge 7 | -- It then does the following: 8 | -- 1) Removes the pool id from the set of active pools 9 | -- 2) Iterates through each job in the executing set and finds any jobs which 10 | -- have a poolId field equal to the id of the stale pool 11 | -- 3) If it finds any such jobs, it removes them from the executing set and 12 | -- adds them to the queued so that they will be retried 13 | 14 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 15 | 16 | -- Assign args to variables for easy reference 17 | local stalePoolId = ARGV[1] 18 | -- Check if the stale pool is in the set of active pools first 19 | local isActive = redis.call('SISMEMBER', '{{.activePoolsSet}}', stalePoolId) 20 | if isActive then 21 | -- Remove the stale pool from the set of active pools 22 | redis.call('SREM', '{{.activePoolsSet}}', stalePoolId) 23 | -- Get all the jobs in the executing set 24 | local jobIds = redis.call('ZRANGE', '{{.executingSet}}', 0, -1) 25 | for i, jobId in ipairs(jobIds) do 26 | local jobKey = 'jobs:' .. jobId 27 | -- Check the poolId field 28 | -- If the poolId is equal to the stale id, then this job is stuck 29 | -- in the executing set even though no worker is actually executing it 30 | local poolId = redis.call('HGET', jobKey, 'poolId') 31 | if poolId == stalePoolId then 32 | local jobPriority = redis.call('HGET', jobKey, 'priority') 33 | -- Move the job into the queued set 34 | redis.call('ZADD', '{{.queuedSet}}', jobPriority, jobId) 35 | -- Remove the job from the executing set 36 | redis.call('ZREM', '{{.executingSet}}', jobId) 37 | -- Set the job status to queued and the pool id to blank 38 | redis.call('HMSET', jobKey, 'status', '{{.statusQueued}}', 'poolId', '') 39 | end 40 | end 41 | end 42 | -------------------------------------------------------------------------------- /scripts/retry_or_fail_job.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- retry_or_fail_job represents a lua script that takes the following arguments: 6 | -- 1) The id of the job to either retry or fail 7 | -- It first checks if the job has any retries remaining. If it does, 8 | -- then it: 9 | -- 1) Decrements the number of retries for the given job 10 | -- 2) Adds the job to the queued set 11 | -- 3) Removes the job from the executing set 12 | -- 4) Returns true 13 | -- If the job has no retries remaining then it: 14 | -- 1) Adds the job to the failed set 15 | -- 3) Removes the job from the executing set 16 | -- 2) Returns false 17 | 18 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 19 | 20 | -- Assign args to variables for easy reference 21 | local jobId = ARGV[1] 22 | local jobKey = 'jobs:' .. jobId 23 | -- Make sure the job hasn't already been destroyed 24 | local exists = redis.call('EXISTS', jobKey) 25 | if exists ~= 1 then 26 | return 0 27 | end 28 | -- Check how many retries remain 29 | local retries = redis.call('HGET', jobKey, 'retries') 30 | local newStatus = '' 31 | if retries == '0' then 32 | -- newStatus should be failed because there are no retries left 33 | newStatus = '{{.statusFailed}}' 34 | else 35 | -- subtract 1 from the remaining retries 36 | redis.call('HINCRBY', jobKey, 'retries', -1) 37 | -- newStatus should be queued, so the job will be retried 38 | newStatus = '{{.statusQueued}}' 39 | end 40 | -- Get the job priority (used as score) 41 | local jobPriority = redis.call('HGET', jobKey, 'priority') 42 | -- Add the job to the appropriate new set 43 | local newStatusSet = 'jobs:' .. newStatus 44 | redis.call('ZADD', newStatusSet, jobPriority, jobId) 45 | -- Remove the job from the old status set 46 | local oldStatus = redis.call('HGET', jobKey, 'status') 47 | if ((oldStatus ~= '') and (oldStatus ~= newStatus)) then 48 | local oldStatusSet = 'jobs:' .. oldStatus 49 | redis.call('ZREM', oldStatusSet, jobId) 50 | end 51 | -- Set the job status in the hash 52 | redis.call('HSET', jobKey, 'status', newStatus) 53 | if retries == '0' then 54 | -- Return false to indicate the job has not been queued for retry 55 | -- NOTE: 0 is used to represent false because apparently 56 | -- false gets converted to nil 57 | return 0 58 | else 59 | -- Return true to indicate the job has been queued for retry 60 | -- NOTE: 1 is used to represent true (for consistency) 61 | return 1 62 | end -------------------------------------------------------------------------------- /scripts/scripts.go.tmpl: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | // File scripts.go contains code related to parsing 6 | // lua scripts in the scripts file. 7 | 8 | // This file has been automatically generated by go generate, 9 | // which calls scripts/generate.go. Do not edit it directly! 10 | 11 | package jobs 12 | 13 | import ( 14 | "github.com/garyburd/redigo/redis" 15 | ) 16 | 17 | var ( 18 | {{ range . }} 19 | {{ .VarName }} = redis.NewScript(0, `{{ .Src }}`){{ end }} 20 | ) -------------------------------------------------------------------------------- /scripts/set_job_field.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- set_job_field represents a lua script that takes the following arguments: 6 | -- 1) The id of the job 7 | -- 2) The name of the field 8 | -- 3) The value to set the field to 9 | -- It first checks if the job exists in the database (has not been destroyed) 10 | -- and then sets the given field to the given value. 11 | 12 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 13 | 14 | local jobId = ARGV[1] 15 | local fieldName = ARGV[2] 16 | local fieldVal = ARGV[3] 17 | local jobKey = 'jobs:' .. jobId 18 | -- Make sure the job hasn't already been destroyed 19 | local exists = redis.call('EXISTS', jobKey) 20 | if exists ~= 1 then 21 | return 22 | end 23 | redis.call('HSET', jobKey, fieldName, fieldVal) -------------------------------------------------------------------------------- /scripts/set_job_status.lua: -------------------------------------------------------------------------------- 1 | -- Copyright 2015 Alex Browne. All rights reserved. 2 | -- Use of this source code is governed by the MIT 3 | -- license, which can be found in the LICENSE file. 4 | 5 | -- set_job_status is a lua script that takes the following arguments: 6 | -- 1) The id of the job 7 | -- 2) The new status (e.g. "queued") 8 | -- It then does the following: 9 | -- 1) Adds the job to the new status set 10 | -- 2) Removes the job from the old status set (which it gets with an HGET call) 11 | -- 3) Sets the 'status' field in the main hash for the job 12 | 13 | -- IMPORTANT: If you edit this file, you must run go generate . to rewrite ../scripts.go 14 | 15 | -- Assign args to variables for easy reference 16 | local jobId = ARGV[1] 17 | local newStatus = ARGV[2] 18 | local jobKey = 'jobs:' .. jobId 19 | -- Make sure the job hasn't already been destroyed 20 | local exists = redis.call('EXISTS', jobKey) 21 | if exists ~= 1 then 22 | return 23 | end 24 | local newStatusSet = 'jobs:' .. newStatus 25 | -- Add the job to the new status set 26 | local jobPriority = redis.call('HGET', jobKey, 'priority') 27 | redis.call('ZADD', newStatusSet, jobPriority, jobId) 28 | -- Remove the job from the old status set 29 | local oldStatus = redis.call('HGET', jobKey, 'status') 30 | if ((oldStatus ~= '') and (oldStatus ~= newStatus)) then 31 | local oldStatusSet = 'jobs:' .. oldStatus 32 | redis.call('ZREM', oldStatusSet, jobId) 33 | end 34 | -- Set the status field 35 | redis.call('HSET', jobKey, 'status', newStatus) -------------------------------------------------------------------------------- /scripts_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "github.com/garyburd/redigo/redis" 9 | "reflect" 10 | "testing" 11 | "time" 12 | ) 13 | 14 | func TestPopNextJobsScript(t *testing.T) { 15 | testingSetUp() 16 | defer testingTeardown() 17 | 18 | // Set up some time parameters 19 | pastTime := time.Now().Add(-10 * time.Millisecond).UTC().UnixNano() 20 | 21 | // Set up the database 22 | tx0 := newTransaction() 23 | // One set will mimic the ready and sorted jobs 24 | tx0.command("ZADD", redis.Args{Keys.JobsTimeIndex, pastTime, "two", pastTime, "four"}, nil) 25 | // One set will mimic the queued set 26 | tx0.command("ZADD", redis.Args{StatusQueued.Key(), 1, "one", 2, "two", 3, "three", 4, "four"}, nil) 27 | // One set will mimic the executing set 28 | tx0.command("ZADD", redis.Args{StatusExecuting.Key(), 5, "five"}, nil) 29 | if err := tx0.exec(); err != nil { 30 | t.Errorf("Unexpected error executing transaction: %s", err.Error()) 31 | } 32 | 33 | // Start a new transaction and execute the script 34 | tx1 := newTransaction() 35 | gotJobs := []*Job{} 36 | testPoolId := "testPool" 37 | tx1.popNextJobs(2, testPoolId, newScanJobsHandler(&gotJobs)) 38 | if err := tx1.exec(); err != nil { 39 | t.Errorf("Unexpected error executing transaction: %s", err.Error()) 40 | } 41 | 42 | gotIds := []string{} 43 | for _, job := range gotJobs { 44 | gotIds = append(gotIds, job.id) 45 | } 46 | 47 | // Check the results 48 | expectedIds := []string{"four", "two"} 49 | if !reflect.DeepEqual(expectedIds, gotIds) { 50 | t.Errorf("Ids returned by script were incorrect.\n\tExpected: %v\n\tBut got: %v", expectedIds, gotIds) 51 | } 52 | conn := redisPool.Get() 53 | defer conn.Close() 54 | expectedExecuting := []string{"five", "four", "two"} 55 | gotExecuting, err := redis.Strings(conn.Do("ZREVRANGE", StatusExecuting.Key(), 0, -1)) 56 | if err != nil { 57 | t.Errorf("Unexpected error in ZREVRANGE: %s", err.Error()) 58 | } 59 | if !reflect.DeepEqual(expectedExecuting, gotExecuting) { 60 | t.Errorf("Ids in the executing set were incorrect.\n\tExpected: %v\n\tBut got: %v", expectedExecuting, gotExecuting) 61 | } 62 | expectedQueued := []string{"three", "one"} 63 | gotQueued, err := redis.Strings(conn.Do("ZREVRANGE", StatusQueued.Key(), 0, -1)) 64 | if err != nil { 65 | t.Errorf("Unexpected error in ZREVRANGE: %s", err.Error()) 66 | } 67 | if !reflect.DeepEqual(expectedQueued, gotQueued) { 68 | t.Errorf("Ids in the queued set were incorrect.\n\tExpected: %v\n\tBut got: %v", expectedQueued, gotQueued) 69 | } 70 | expectKeyNotExists(t, Keys.JobsTemp) 71 | } 72 | 73 | func TestRetryOrFailJobScript(t *testing.T) { 74 | testingSetUp() 75 | defer testingTeardown() 76 | 77 | testJob, err := RegisterType("testJob", 0, noOpHandler) 78 | if err != nil { 79 | t.Fatalf("Unexpected error registering job type: %s", err.Error()) 80 | } 81 | 82 | // We'll use table-driven tests here 83 | testCases := []struct { 84 | job *Job 85 | expectedReturn bool 86 | expectedRetries int 87 | }{ 88 | { 89 | // One job will start with 2 retries remaining 90 | job: &Job{typ: testJob, id: "retriesRemainingJob", retries: 2, status: StatusExecuting}, 91 | expectedReturn: true, 92 | expectedRetries: 1, 93 | }, 94 | { 95 | // One job will start with 0 retries remaining 96 | job: &Job{typ: testJob, id: "noRetriesJob", retries: 0, status: StatusExecuting}, 97 | expectedReturn: false, 98 | expectedRetries: 0, 99 | }, 100 | } 101 | 102 | // We can test all of the cases in a single transaction 103 | tx := newTransaction() 104 | gotReturns := make([]bool, len(testCases)) 105 | gotRetries := make([]int, len(testCases)) 106 | for i, tc := range testCases { 107 | // Save the job 108 | tx.saveJob(tc.job) 109 | // Run the script and save the return value in a slice 110 | tx.retryOrFailJob(tc.job, newScanBoolHandler(&(gotReturns[i]))) 111 | // Get the new number of retries from the database and save the value in a slice 112 | tx.command("HGET", redis.Args{tc.job.Key(), "retries"}, newScanIntHandler(&(gotRetries[i]))) 113 | } 114 | // Execute the transaction 115 | if err := tx.exec(); err != nil { 116 | t.Errorf("Unexpected error executing transaction: %s", err.Error()) 117 | } 118 | 119 | // Iterate through test cases again and check the results 120 | for i, tc := range testCases { 121 | if gotRetries[i] != tc.expectedRetries { 122 | t.Errorf("Number of retries after executing script was incorrect for test case %d (job:%s). Expected %v but got %v", i, tc.job.id, tc.expectedRetries, gotRetries[i]) 123 | } 124 | if gotReturns[i] != tc.expectedReturn { 125 | t.Errorf("Return value from script was incorrect for test case %d (job:%s). Expected %v but got %v", i, tc.job.id, tc.expectedReturn, gotReturns[i]) 126 | } 127 | // Make sure the job was removed from the executing set and placed in the correct set 128 | if err := tc.job.Refresh(); err != nil { 129 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 130 | } 131 | if tc.expectedReturn == false { 132 | // We expect the job to be in the failed set because it had no retries left 133 | expectStatusEquals(t, tc.job, StatusFailed) 134 | } else { 135 | // We expect the job to be in the queued set because it was queued for retry 136 | expectStatusEquals(t, tc.job, StatusQueued) 137 | } 138 | } 139 | } 140 | 141 | func TestSetStatusScript(t *testing.T) { 142 | testingSetUp() 143 | defer testingTeardown() 144 | 145 | job, err := createAndSaveTestJob() 146 | if err != nil { 147 | t.Fatalf("Unexpected error in createAndSaveTestJob(): %s", err.Error()) 148 | } 149 | 150 | // For all possible statuses, execute the script and check that the job status was set correctly 151 | for _, status := range possibleStatuses { 152 | if status == StatusDestroyed { 153 | continue 154 | } 155 | tx := newTransaction() 156 | tx.setStatus(job, status) 157 | if err := tx.exec(); err != nil { 158 | t.Errorf("Unexpected error in tx.exec(): %s", err.Error()) 159 | } 160 | if err := job.Refresh(); err != nil { 161 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 162 | } 163 | expectStatusEquals(t, job, status) 164 | } 165 | } 166 | 167 | func TestDestroyJobScript(t *testing.T) { 168 | testingSetUp() 169 | defer testingTeardown() 170 | 171 | job, err := createAndSaveTestJob() 172 | if err != nil { 173 | t.Fatalf("Unexpected error in createAndSaveTestJob(): %s", err.Error()) 174 | } 175 | 176 | // Execute the script to destroy the job 177 | tx := newTransaction() 178 | tx.destroyJob(job) 179 | if err := tx.exec(); err != nil { 180 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 181 | } 182 | 183 | // Make sure the job was destroyed 184 | job.status = StatusDestroyed 185 | expectStatusEquals(t, job, StatusDestroyed) 186 | } 187 | 188 | func TestPurgeStalePoolScript(t *testing.T) { 189 | testingSetUp() 190 | defer testingTeardown() 191 | 192 | testType, err := RegisterType("testType", 0, noOpHandler) 193 | if err != nil { 194 | t.Fatalf("Unexpected error in RegisterType(): %s", err.Error()) 195 | } 196 | 197 | // Set up the database. We'll put some jobs in the executing set with a stale poolId, 198 | // and some jobs with an active poolId. 199 | staleJobs := []*Job{} 200 | stalePoolId := "stalePool" 201 | for i := 0; i < 4; i++ { 202 | job := &Job{typ: testType, status: StatusExecuting, poolId: stalePoolId} 203 | if err := job.save(); err != nil { 204 | t.Errorf("Unexpected error in job.save(): %s", err.Error()) 205 | } 206 | staleJobs = append(staleJobs, job) 207 | } 208 | activeJobs := []*Job{} 209 | activePoolId := "activePool" 210 | for i := 0; i < 4; i++ { 211 | job := &Job{typ: testType, status: StatusExecuting, poolId: activePoolId} 212 | if err := job.save(); err != nil { 213 | t.Errorf("Unexpected error in job.save(): %s", err.Error()) 214 | } 215 | activeJobs = append(activeJobs, job) 216 | } 217 | 218 | // Add both pools to the set of active pools 219 | conn := redisPool.Get() 220 | defer conn.Close() 221 | if _, err := conn.Do("SADD", Keys.ActivePools, stalePoolId, activePoolId); err != nil { 222 | t.Errorf("Unexpected error adding pools to set: %s", err) 223 | } 224 | 225 | // Execute the script to purge the stale pool 226 | tx := newTransaction() 227 | tx.purgeStalePool(stalePoolId) 228 | if err := tx.exec(); err != nil { 229 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 230 | } 231 | 232 | // Check the result 233 | // The active pools set should contain only the activePoolId 234 | expectSetDoesNotContain(t, Keys.ActivePools, stalePoolId) 235 | expectSetContains(t, Keys.ActivePools, activePoolId) 236 | // All the active jobs should still be executing 237 | for _, job := range activeJobs { 238 | if err := job.Refresh(); err != nil { 239 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 240 | } 241 | expectStatusEquals(t, job, StatusExecuting) 242 | } 243 | // All the stale jobs should now be queued and have an empty poolId 244 | for _, job := range staleJobs { 245 | if err := job.Refresh(); err != nil { 246 | t.Errorf("Unexpected error in job.Refresh(): %s", err.Error()) 247 | } 248 | expectStatusEquals(t, job, StatusQueued) 249 | expectJobFieldEquals(t, job, "poolId", "", stringConverter) 250 | } 251 | } 252 | 253 | func TestGetJobsByIdsScript(t *testing.T) { 254 | testingSetUp() 255 | defer testingTeardown() 256 | 257 | // Create and save some jobs 258 | jobs, err := createAndSaveTestJobs(5) 259 | if err != nil { 260 | t.Fatalf("Unexpected error in createAndSaveTestJobs: %s", err.Error()) 261 | } 262 | 263 | // Execute the script to get the jobs we just created 264 | jobsCopy := []*Job{} 265 | tx := newTransaction() 266 | tx.getJobsByIds(StatusSaved.Key(), newScanJobsHandler(&jobsCopy)) 267 | if err := tx.exec(); err != nil { 268 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 269 | } 270 | 271 | // Check the result 272 | if len(jobsCopy) != len(jobs) { 273 | t.Errorf("getJobsByIds did not return the right number of jobs. Expected %d but got %d", len(jobs), len(jobsCopy)) 274 | } 275 | if !reflect.DeepEqual(jobs, jobsCopy) { 276 | t.Errorf("Result of getJobsByIds was incorrect.\n\tExpected: %v\n\tbut got: %v", jobs, jobsCopy) 277 | } 278 | } 279 | 280 | func TestSetJobFieldScript(t *testing.T) { 281 | testingSetUp() 282 | defer testingTeardown() 283 | 284 | // Create a test job 285 | jobs, err := createAndSaveTestJobs(1) 286 | if err != nil { 287 | t.Fatalf("Unexpected error in createAndSaveTestJobs: %s", err.Error()) 288 | } 289 | job := jobs[0] 290 | 291 | // Set the time to 7 days ago 292 | tx := newTransaction() 293 | expectedTime := time.Now().Add(-7 * 24 * time.Hour).UTC().UnixNano() 294 | tx.setJobField(job, "time", expectedTime) 295 | if err := tx.exec(); err != nil { 296 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 297 | } 298 | 299 | // Make sure the time field was set properly 300 | if err := job.Refresh(); err != nil { 301 | t.Errorf("Unexpected err in job.Refresh: %s", err.Error()) 302 | } 303 | if job.time != expectedTime { 304 | t.Errorf("time field was not set. Expected %d but got %d", job.time, expectedTime) 305 | } 306 | 307 | // Destroy the job and make sure the script does not set the field 308 | if err := job.Destroy(); err != nil { 309 | t.Errorf("Unexpected err in job.Destroy: %s", err.Error()) 310 | } 311 | tx = newTransaction() 312 | tx.setJobField(job, "foo", "bar") 313 | if err := tx.exec(); err != nil { 314 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 315 | } 316 | conn := redisPool.Get() 317 | defer conn.Close() 318 | exists, err := redis.Bool(conn.Do("EXISTS", job.Key())) 319 | if err != nil { 320 | t.Errorf("Unexpected err in EXISTS: %s", err.Error()) 321 | } 322 | if exists { 323 | t.Error("Expected job to not exist after being destroyed but it did.") 324 | } 325 | } 326 | 327 | func TestAddJobToSetScript(t *testing.T) { 328 | testingSetUp() 329 | defer testingTeardown() 330 | 331 | // Create a test job 332 | jobs, err := createAndSaveTestJobs(1) 333 | if err != nil { 334 | t.Fatalf("Unexpected error in createAndSaveTestJobs: %s", err.Error()) 335 | } 336 | job := jobs[0] 337 | 338 | // Add the job to the time index with a score of 7 days ago 339 | tx := newTransaction() 340 | expectedScore := float64(time.Now().Add(-7 * 24 * time.Hour).UTC().UnixNano()) 341 | tx.addJobToSet(job, Keys.JobsTimeIndex, expectedScore) 342 | if err := tx.exec(); err != nil { 343 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 344 | } 345 | 346 | // Make sure the job was added to the set properly 347 | conn := redisPool.Get() 348 | defer conn.Close() 349 | score, err := redis.Float64(conn.Do("ZSCORE", Keys.JobsTimeIndex, job.id)) 350 | if err != nil { 351 | t.Errorf("Unexpected error in ZSCORE: %s", err.Error()) 352 | } 353 | if score != expectedScore { 354 | t.Errorf("Score in time index set was incorrect. Expected %f but got %f", expectedScore, score) 355 | } 356 | 357 | // Destroy the job and make sure the script does not add it to a new set 358 | if err := job.Destroy(); err != nil { 359 | t.Errorf("Unexpected err in job.Destroy: %s", err.Error()) 360 | } 361 | tx = newTransaction() 362 | tx.addJobToSet(job, "fooSet", 42.0) 363 | if err := tx.exec(); err != nil { 364 | t.Errorf("Unexpected err in tx.exec(): %s", err.Error()) 365 | } 366 | exists, err := redis.Bool(conn.Do("EXISTS", "fooSet")) 367 | if err != nil { 368 | t.Errorf("Unexpected err in EXISTS: %s", err.Error()) 369 | } 370 | if exists { 371 | t.Error("Expected fooSet to not exist after the job was destroyed but it did.") 372 | } 373 | } 374 | -------------------------------------------------------------------------------- /test_utils.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "fmt" 9 | "reflect" 10 | "sync" 11 | "testing" 12 | "time" 13 | 14 | "github.com/garyburd/redigo/redis" 15 | ) 16 | 17 | // setUpOnce enforces that certain pieces of the set up process only occur once, 18 | // even after successive calls to testingSetUp. 19 | var setUpOnce = sync.Once{} 20 | 21 | // testingSetUp should be called at the beginning of any test that touches the database 22 | // or registers new job types. 23 | func testingSetUp() { 24 | setUpOnce.Do(func() { 25 | // Use database 14 and a unix socket connection for testing 26 | // TODO: allow this to be configured via command-line flags 27 | Config.Db.Database = 14 28 | }) 29 | // Clear out any old job types 30 | Types = map[string]*Type{} 31 | } 32 | 33 | // testingSetUp should be called at the end of any test that touches the database 34 | // or registers new job types, usually via defer. 35 | func testingTeardown() { 36 | // Flush the database 37 | flushdb() 38 | } 39 | 40 | // flushdb removes all keys from the current database. 41 | func flushdb() { 42 | conn := redisPool.Get() 43 | if _, err := conn.Do("FLUSHDB"); err != nil { 44 | panic(err) 45 | } 46 | } 47 | 48 | // noOpHandler is a HandlerFunc that simply does nothing 49 | var noOpHandler = func() error { return nil } 50 | 51 | // createTestJob creates and returns a job that can be used for testing. 52 | func createTestJob() (*Job, error) { 53 | // Register the "testType" 54 | TypeName := "testType" 55 | Type, err := RegisterType(TypeName, 0, noOpHandler) 56 | if err != nil { 57 | if _, ok := err.(ErrorNameAlreadyRegistered); !ok { 58 | // If the name was already registered, that's fine. 59 | // We should return any other type of error 60 | return nil, err 61 | } 62 | } 63 | // Create and return a test job 64 | j := &Job{ 65 | id: "testJob", 66 | data: []byte("testData"), 67 | typ: Type, 68 | time: time.Now().UTC().UnixNano(), 69 | priority: 100, 70 | } 71 | return j, nil 72 | } 73 | 74 | // createTestJobs creates and returns n jobs that can be used for testing. 75 | // Each job has a unique id and priority, and the jobs are returned in order 76 | // of decreasing priority. 77 | func createTestJobs(n int) ([]*Job, error) { 78 | // Register the "testType" 79 | TypeName := "testType" 80 | Type, err := RegisterType(TypeName, 0, noOpHandler) 81 | if err != nil { 82 | if _, ok := err.(ErrorNameAlreadyRegistered); !ok { 83 | // If the name was already registered, that's fine. 84 | // We should return any other type of error 85 | return nil, err 86 | } 87 | } 88 | jobs := make([]*Job, n) 89 | for i := 0; i < n; i++ { 90 | jobs[i] = &Job{ 91 | id: fmt.Sprintf("testJob%d", i), 92 | data: []byte("testData"), 93 | typ: Type, 94 | time: time.Now().UTC().UnixNano(), 95 | priority: (n - i) + 1, 96 | } 97 | } 98 | return jobs, nil 99 | } 100 | 101 | // createAndSaveTestJob creates, saves, and returns a job which can be used 102 | // for testing. Each job has a unique id and priority, and the jobs are 103 | // returned in order of decreasing priority. 104 | func createAndSaveTestJob() (*Job, error) { 105 | j, err := createTestJob() 106 | if err != nil { 107 | return nil, err 108 | } 109 | if err := j.save(); err != nil { 110 | return nil, fmt.Errorf("Unexpected error in j.save(): %s", err.Error()) 111 | } 112 | return j, nil 113 | } 114 | 115 | // createAndSaveTestJobs creates, saves, and returns n jobs which can be used 116 | // for testing. 117 | func createAndSaveTestJobs(n int) ([]*Job, error) { 118 | jobs, err := createTestJobs(n) 119 | if err != nil { 120 | return nil, err 121 | } 122 | // Save all the jobs in a single transaction 123 | t := newTransaction() 124 | for _, job := range jobs { 125 | t.saveJob(job) 126 | } 127 | if err := t.exec(); err != nil { 128 | return nil, err 129 | } 130 | return jobs, nil 131 | } 132 | 133 | // expectJobFieldEquals sets an error via t.Errorf if the the field identified by fieldName does 134 | // not equal expected according to the database. 135 | func expectJobFieldEquals(t *testing.T, job *Job, fieldName string, expected interface{}, converter replyConverter) { 136 | conn := redisPool.Get() 137 | defer conn.Close() 138 | got, err := conn.Do("HGET", job.Key(), fieldName) 139 | if err != nil { 140 | t.Errorf("Unexpected error: %s", err.Error()) 141 | } 142 | if converter != nil { 143 | got, err = converter(got) 144 | if err != nil { 145 | t.Errorf("Unexpected error in converter: %s", err.Error()) 146 | } 147 | } 148 | if !reflect.DeepEqual(expected, got) { 149 | t.Errorf("job.%s was not saved correctly.\n\tExpected: %v\n\tBut got: %v", fieldName, expected, got) 150 | } 151 | } 152 | 153 | // replyConverter represents a function which is capable of converting a redis 154 | // reply to some other type. 155 | type replyConverter func(interface{}) (interface{}, error) 156 | 157 | var ( 158 | int64Converter replyConverter = func(in interface{}) (interface{}, error) { 159 | return redis.Int64(in, nil) 160 | } 161 | intConverter replyConverter = func(in interface{}) (interface{}, error) { 162 | return redis.Int(in, nil) 163 | } 164 | uintConverter replyConverter = func(in interface{}) (interface{}, error) { 165 | v, err := redis.Uint64(in, nil) 166 | if err != nil { 167 | return nil, err 168 | } 169 | return uint(v), nil 170 | } 171 | stringConverter replyConverter = func(in interface{}) (interface{}, error) { 172 | return redis.String(in, nil) 173 | } 174 | bytesConverter replyConverter = func(in interface{}) (interface{}, error) { 175 | return redis.Bytes(in, nil) 176 | } 177 | ) 178 | 179 | // expectSetContains sets an error via t.Errorf if member is not in the set 180 | func expectSetContains(t *testing.T, setName string, member string) { 181 | conn := redisPool.Get() 182 | defer conn.Close() 183 | contains, err := redis.Bool(conn.Do("SISMEMBER", setName, member)) 184 | if err != nil { 185 | t.Errorf("Unexpected error: %s", err.Error()) 186 | } 187 | if !contains { 188 | t.Errorf("Expected set %s to contain %s but it did not.", setName, member) 189 | } 190 | } 191 | 192 | // expectSetDoesNotContain sets an error via t.Errorf if member is in the set 193 | func expectSetDoesNotContain(t *testing.T, setName string, member string) { 194 | conn := redisPool.Get() 195 | defer conn.Close() 196 | contains, err := redis.Bool(conn.Do("SISMEMBER", setName, member)) 197 | if err != nil { 198 | t.Errorf("Unexpected error: %s", err.Error()) 199 | } 200 | if contains { 201 | t.Errorf("Expected set %s to not contain %s but it did.", setName, member) 202 | } 203 | } 204 | 205 | // expectJobInStatusSet sets an error via t.Errorf if job is not in the status set 206 | // corresponding to status. 207 | func expectJobInStatusSet(t *testing.T, j *Job, status Status) { 208 | conn := redisPool.Get() 209 | defer conn.Close() 210 | gotIds, err := redis.Strings(conn.Do("ZRANGEBYSCORE", status.Key(), j.priority, j.priority)) 211 | if err != nil { 212 | t.Errorf("Unexpected error: %s", err.Error()) 213 | } 214 | for _, id := range gotIds { 215 | if id == j.id { 216 | // We found the job we were looking for 217 | return 218 | } 219 | } 220 | // If we reached here, we did not find the job we were looking for 221 | t.Errorf("job:%s was not found in set %s", j.id, status.Key()) 222 | } 223 | 224 | // expectJobInTimeIndex sets an error via t.Errorf if job is not in the time index 225 | // set. 226 | func expectJobInTimeIndex(t *testing.T, j *Job) { 227 | conn := redisPool.Get() 228 | defer conn.Close() 229 | gotIds, err := redis.Strings(conn.Do("ZRANGEBYSCORE", Keys.JobsTimeIndex, j.time, j.time)) 230 | if err != nil { 231 | t.Errorf("Unexpected error: %s", err.Error()) 232 | } 233 | for _, id := range gotIds { 234 | if id == j.id { 235 | // We found the job we were looking for 236 | return 237 | } 238 | } 239 | // If we reached here, we did not find the job we were looking for 240 | t.Errorf("job:%s was not found in set %s", j.id, Keys.JobsTimeIndex) 241 | } 242 | 243 | // expectJobNotInStatusSet sets an error via t.Errorf if job is in the status set 244 | // corresponding to status. 245 | func expectJobNotInStatusSet(t *testing.T, j *Job, status Status) { 246 | conn := redisPool.Get() 247 | defer conn.Close() 248 | gotIds, err := redis.Strings(conn.Do("ZRANGEBYSCORE", status.Key(), j.priority, j.priority)) 249 | if err != nil { 250 | t.Errorf("Unexpected error: %s", err.Error()) 251 | } 252 | for _, id := range gotIds { 253 | if id == j.id { 254 | // We found the job, but it wasn't supposed to be here! 255 | t.Errorf("job:%s was found in set %s but expected it to be removed", j.id, status.Key()) 256 | } 257 | } 258 | } 259 | 260 | // expectJobNotInTimeIndex sets an error via t.Errorf if job is in the time index 261 | // set. 262 | func expectJobNotInTimeIndex(t *testing.T, j *Job) { 263 | conn := redisPool.Get() 264 | defer conn.Close() 265 | gotIds, err := redis.Strings(conn.Do("ZRANGEBYSCORE", Keys.JobsTimeIndex, j.time, j.time)) 266 | if err != nil { 267 | t.Errorf("Unexpected error: %s", err.Error()) 268 | } 269 | for _, id := range gotIds { 270 | if id == j.id { 271 | // We found the job, but it wasn't supposed to be here! 272 | t.Errorf("job:%s was found in set %s but expected it to be removed", j.id, Keys.JobsTimeIndex) 273 | } 274 | } 275 | } 276 | 277 | // expectStatusEquals sets an error via t.Errorf if job.status does not equal expected, 278 | // if the status field for the job in the database does not equal expected, if the job is 279 | // not in the status set corresponding to expected, or if the job is in some other status 280 | // set. 281 | func expectStatusEquals(t *testing.T, job *Job, expected Status) { 282 | if job.status != expected { 283 | t.Errorf("Expected jobs:%s status to be %s but got %s", job.id, expected, job.status) 284 | } 285 | if expected == StatusDestroyed { 286 | // If the status is destroyed, we don't expect the job to be in the database 287 | // anymore. 288 | expectJobDestroyed(t, job) 289 | } else { 290 | // For every status other status, we expect the job to be in the database 291 | for _, status := range possibleStatuses { 292 | if status == expected { 293 | // Make sure the job hash has the correct status 294 | expectJobInStatusSet(t, job, status) 295 | // Make sure the job is in the correct set 296 | expectJobFieldEquals(t, job, "status", string(status), stringConverter) 297 | } else { 298 | // Make sure the job is not in any other set 299 | expectJobNotInStatusSet(t, job, status) 300 | } 301 | } 302 | } 303 | } 304 | 305 | // expectJobDestroyed sets an error via t.Errorf if job has not been destroyed. 306 | func expectJobDestroyed(t *testing.T, job *Job) { 307 | // Make sure the main hash is gone 308 | expectKeyNotExists(t, job.Key()) 309 | expectJobNotInTimeIndex(t, job) 310 | for _, status := range possibleStatuses { 311 | expectJobNotInStatusSet(t, job, status) 312 | } 313 | } 314 | 315 | // expectKeyExists sets an error via t.Errorf if key does not exist in the database. 316 | func expectKeyExists(t *testing.T, key string) { 317 | conn := redisPool.Get() 318 | defer conn.Close() 319 | if exists, err := redis.Bool(conn.Do("EXISTS", key)); err != nil { 320 | t.Errorf("Unexpected error in EXISTS: %s", err.Error()) 321 | } else if !exists { 322 | t.Errorf("Expected key %s to exist, but it did not.", key) 323 | } 324 | } 325 | 326 | // expectKeyNotExists sets an error via t.Errorf if key does exist in the database. 327 | func expectKeyNotExists(t *testing.T, key string) { 328 | conn := redisPool.Get() 329 | defer conn.Close() 330 | if exists, err := redis.Bool(conn.Do("EXISTS", key)); err != nil { 331 | t.Errorf("Unexpected error in EXISTS: %s", err.Error()) 332 | } else if exists { 333 | t.Errorf("Expected key %s to not exist, but it did exist.", key) 334 | } 335 | } 336 | -------------------------------------------------------------------------------- /transaction.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "fmt" 9 | "github.com/garyburd/redigo/redis" 10 | "time" 11 | ) 12 | 13 | // transaction is an abstraction layer around a redis transaction. 14 | // transactions feature delayed execution, so nothing touches the database 15 | // until exec is called. 16 | type transaction struct { 17 | conn redis.Conn 18 | actions []*action 19 | } 20 | 21 | // action is a single step in a transaction and must be either a command 22 | // or a script with optional arguments. 23 | type action struct { 24 | kind actionKind 25 | name string 26 | script *redis.Script 27 | args redis.Args 28 | handler replyHandler 29 | } 30 | 31 | // actionKind is either a command or a script 32 | type actionKind int 33 | 34 | const ( 35 | actionCommand = iota 36 | actionScript 37 | ) 38 | 39 | // replyHandler is a function which does something with the reply from a redis 40 | // command or script. 41 | type replyHandler func(interface{}) error 42 | 43 | // newTransaction instantiates and returns a new transaction. 44 | func newTransaction() *transaction { 45 | t := &transaction{ 46 | conn: redisPool.Get(), 47 | } 48 | return t 49 | } 50 | 51 | // command adds a command action to the transaction with the given args. 52 | // handler will be called with the reply from this specific command when 53 | // the transaction is executed. 54 | func (t *transaction) command(name string, args redis.Args, handler replyHandler) { 55 | t.actions = append(t.actions, &action{ 56 | kind: actionCommand, 57 | name: name, 58 | args: args, 59 | handler: handler, 60 | }) 61 | } 62 | 63 | // command adds a script action to the transaction with the given args. 64 | // handler will be called with the reply from this specific script when 65 | // the transaction is executed. 66 | func (t *transaction) script(script *redis.Script, args redis.Args, handler replyHandler) { 67 | t.actions = append(t.actions, &action{ 68 | kind: actionScript, 69 | script: script, 70 | args: args, 71 | handler: handler, 72 | }) 73 | } 74 | 75 | // sendAction writes a to a connection buffer using conn.Send() 76 | func (t *transaction) sendAction(a *action) error { 77 | switch a.kind { 78 | case actionCommand: 79 | return t.conn.Send(a.name, a.args...) 80 | case actionScript: 81 | return a.script.Send(t.conn, a.args...) 82 | } 83 | return nil 84 | } 85 | 86 | // doAction writes a to the connection buffer and then immediately 87 | // flushes the buffer and reads the reply via conn.Do() 88 | func (t *transaction) doAction(a *action) (interface{}, error) { 89 | switch a.kind { 90 | case actionCommand: 91 | return t.conn.Do(a.name, a.args...) 92 | case actionScript: 93 | return a.script.Do(t.conn, a.args...) 94 | } 95 | return nil, nil 96 | } 97 | 98 | // exec executes the transaction, sequentially sending each action and 99 | // calling all the action handlers with the corresponding replies. 100 | func (t *transaction) exec() error { 101 | // Return the connection to the pool when we are done 102 | defer t.conn.Close() 103 | 104 | if len(t.actions) == 1 { 105 | // If there is only one command, no need to use MULTI/EXEC 106 | a := t.actions[0] 107 | reply, err := t.doAction(a) 108 | if err != nil { 109 | return err 110 | } 111 | if a.handler != nil { 112 | if err := a.handler(reply); err != nil { 113 | return err 114 | } 115 | } 116 | } else { 117 | // Send all the commands and scripts at once using MULTI/EXEC 118 | t.conn.Send("MULTI") 119 | for _, a := range t.actions { 120 | if err := t.sendAction(a); err != nil { 121 | return err 122 | } 123 | } 124 | 125 | // Invoke redis driver to execute the transaction 126 | replies, err := redis.Values(t.conn.Do("EXEC")) 127 | if err != nil { 128 | return err 129 | } 130 | 131 | // Iterate through the replies, calling the corresponding handler functions 132 | for i, reply := range replies { 133 | a := t.actions[i] 134 | if a.handler != nil { 135 | if err := a.handler(reply); err != nil { 136 | return err 137 | } 138 | } 139 | } 140 | } 141 | return nil 142 | } 143 | 144 | // newScanJobHandler returns a replyHandler which, when run, will scan the values 145 | // of reply into job. 146 | func newScanJobHandler(job *Job) replyHandler { 147 | return func(reply interface{}) error { 148 | return scanJob(reply, job) 149 | } 150 | } 151 | 152 | // newScanJobsHandler returns a replyHandler which, when run, will scan the values 153 | // of reply into jobs. 154 | func newScanJobsHandler(jobs *[]*Job) replyHandler { 155 | return func(reply interface{}) error { 156 | values, err := redis.Values(reply, nil) 157 | if err != nil { 158 | return nil 159 | } 160 | for _, fields := range values { 161 | job := &Job{} 162 | if err := scanJob(fields, job); err != nil { 163 | return err 164 | } 165 | (*jobs) = append((*jobs), job) 166 | } 167 | return nil 168 | } 169 | } 170 | 171 | // debugSet simply prints out the value of the given set 172 | func (t *transaction) debugSet(setName string) { 173 | t.command("ZRANGE", redis.Args{setName, 0, -1, "WITHSCORES"}, func(reply interface{}) error { 174 | vals, err := redis.Strings(reply, nil) 175 | if err != nil { 176 | return err 177 | } 178 | fmt.Printf("%s: %v\n", setName, vals) 179 | return nil 180 | }) 181 | } 182 | 183 | // newScanStringsHandler returns a replyHandler which, when run, will scan the values 184 | // of reply into strings. 185 | func newScanStringsHandler(strings *[]string) replyHandler { 186 | return func(reply interface{}) error { 187 | if strings == nil { 188 | return fmt.Errorf("jobs: Error in newScanStringsHandler: expected strings arg to be a pointer to a slice of strings but it was nil") 189 | } 190 | var err error 191 | (*strings), err = redis.Strings(reply, nil) 192 | if err != nil { 193 | return fmt.Errorf("jobs: Error in newScanStringsHandler: %s", err.Error()) 194 | } 195 | return nil 196 | } 197 | } 198 | 199 | // newScanStringHandler returns a replyHandler which, when run, will convert reply to a 200 | // string and scan it into s. 201 | func newScanStringHandler(s *string) replyHandler { 202 | return func(reply interface{}) error { 203 | if s == nil { 204 | return fmt.Errorf("jobs: Error in newScanStringHandler: expected arg s to be a pointer to a string but it was nil") 205 | } 206 | var err error 207 | (*s), err = redis.String(reply, nil) 208 | if err != nil { 209 | return fmt.Errorf("jobs: Error in newScanStringHandler: %s", err.Error()) 210 | } 211 | return nil 212 | } 213 | } 214 | 215 | // newScanIntHandler returns a replyHandler which, when run, will convert reply to a 216 | // int and scan it into i. 217 | func newScanIntHandler(i *int) replyHandler { 218 | return func(reply interface{}) error { 219 | if i == nil { 220 | return fmt.Errorf("jobs: Error in newScanIntHandler: expected arg s to be a pointer to a string but it was nil") 221 | } 222 | var err error 223 | (*i), err = redis.Int(reply, nil) 224 | if err != nil { 225 | return fmt.Errorf("jobs: Error in newScanIntHandler: %s", err.Error()) 226 | } 227 | return nil 228 | } 229 | } 230 | 231 | // newScanBoolHandler returns a replyHandler which, when run, will convert reply to a 232 | // bool and scan it into b. 233 | func newScanBoolHandler(b *bool) replyHandler { 234 | return func(reply interface{}) error { 235 | if b == nil { 236 | return fmt.Errorf("jobs: Error in newScanBoolHandler: expected arg v to be a pointer to a bool but it was nil") 237 | } 238 | var err error 239 | (*b), err = redis.Bool(reply, nil) 240 | if err != nil { 241 | return fmt.Errorf("jobs: Error in newScanBoolHandler: %s", err.Error()) 242 | } 243 | return nil 244 | } 245 | } 246 | 247 | //go:generate go run scripts/generate.go 248 | 249 | // popNextJobs is a small function wrapper around getAndMovesJobToExecutingScript. 250 | // It offers some type safety and helps make sure the arguments you pass through to the are correct. 251 | // The script will get the next n jobs from the queue that are ready based on their time parameter. 252 | func (t *transaction) popNextJobs(n int, poolId string, handler replyHandler) { 253 | currentTime := time.Now().UTC().UnixNano() 254 | t.script(popNextJobsScript, redis.Args{n, currentTime, poolId}, handler) 255 | } 256 | 257 | // retryOrFailJob is a small function wrapper around retryOrFailJobScript. 258 | // It offers some type safety and helps make sure the arguments you pass through to the are correct. 259 | // The script will either mark the job as failed or queue it for retry depending on the number of 260 | // retries left. 261 | func (t *transaction) retryOrFailJob(job *Job, handler replyHandler) { 262 | t.script(retryOrFailJobScript, redis.Args{job.id}, handler) 263 | } 264 | 265 | // setStatus is a small function wrapper around setStatusScript. 266 | // It offers some type safety and helps make sure the arguments you pass through to the are correct. 267 | // The script will atomically update the status of the job, removing it from its old status set and 268 | // adding it to the new one. 269 | func (t *transaction) setStatus(job *Job, status Status) { 270 | t.script(setJobStatusScript, redis.Args{job.id, string(status)}, nil) 271 | } 272 | 273 | // destroyJob is a small function wrapper around destroyJobScript. 274 | // It offers some type safety and helps make sure the arguments you pass through to the are correct. 275 | // The script will remove all records associated with job from the database. 276 | func (t *transaction) destroyJob(job *Job) { 277 | t.script(destroyJobScript, redis.Args{job.id}, nil) 278 | } 279 | 280 | // purgeStalePool is a small function wrapper around purgeStalePoolScript. 281 | // It offers some type safety and helps make sure the arguments you pass through to the are correct. 282 | // The script will remove the stale pool from the active pools set, and then requeue any jobs associated 283 | // with the stale pool that are stuck in the executing set. 284 | func (t *transaction) purgeStalePool(poolId string) { 285 | t.script(purgeStalePoolScript, redis.Args{poolId}, nil) 286 | } 287 | 288 | // getJobsByIds is a small function wrapper around getJobsByIdsScript. 289 | // It offers some type safety and helps make sure the arguments you pass through to the are correct. 290 | // The script will return all the fields for jobs which are identified by ids in the given sorted set. 291 | // You can use the handler to scan the jobs into a slice of jobs. 292 | func (t *transaction) getJobsByIds(setKey string, handler replyHandler) { 293 | t.script(getJobsByIdsScript, redis.Args{setKey}, handler) 294 | } 295 | 296 | // setJobField is a small function wrapper around setJobFieldScript. 297 | // It offers some type safety and helps make sure the arguments you pass through are correct. 298 | // The script will set the given field to the given value iff the job exists and has not been 299 | // destroyed. 300 | func (t *transaction) setJobField(job *Job, fieldName string, fieldValue interface{}) { 301 | t.script(setJobFieldScript, redis.Args{job.id, fieldName, fieldValue}, nil) 302 | } 303 | 304 | // addJobToSet is a small function wrapper around addJobToSetScript. 305 | // It offers some type safety and helps make sure the arguments you pass through are correct. 306 | // The script will add the job to the given set with the given score iff the job exists 307 | // and has not been destroyed. 308 | func (t *transaction) addJobToSet(job *Job, setName string, score float64) { 309 | t.script(addJobToSetScript, redis.Args{job.id, setName, score}, nil) 310 | } 311 | -------------------------------------------------------------------------------- /utils.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "github.com/dchest/uniuri" 9 | "strconv" 10 | "time" 11 | ) 12 | 13 | // generateRandomId generates a random string that is more or less 14 | // garunteed to be unique. 15 | func generateRandomId() string { 16 | timeInt := time.Now().UnixNano() 17 | timeString := strconv.FormatInt(timeInt, 36) 18 | randomString := uniuri.NewLen(16) 19 | return randomString + timeString 20 | } 21 | -------------------------------------------------------------------------------- /worker.go: -------------------------------------------------------------------------------- 1 | // Copyright 2015 Alex Browne. All rights reserved. 2 | // Use of this source code is governed by the MIT 3 | // license, which can be found in the LICENSE file. 4 | 5 | package jobs 6 | 7 | import ( 8 | "fmt" 9 | "reflect" 10 | "sync" 11 | "time" 12 | ) 13 | 14 | // worker continuously executes jobs within its own goroutine. 15 | // The jobs chan is shared between all jobs. To stop the worker, 16 | // simply close the jobs channel. 17 | type worker struct { 18 | jobs chan *Job 19 | wg *sync.WaitGroup 20 | afterFunc func(*Job) 21 | } 22 | 23 | // start starts a goroutine in which the worker will continuously 24 | // execute jobs until the jobs channel is closed. 25 | func (w *worker) start() { 26 | go func() { 27 | for job := range w.jobs { 28 | w.doJob(job) 29 | } 30 | w.wg.Done() 31 | }() 32 | } 33 | 34 | // doJob executes the given job. It also sets the status and timestamps for 35 | // the job appropriately depending on the outcome of the execution. 36 | func (w *worker) doJob(job *Job) { 37 | if w.afterFunc != nil { 38 | defer w.afterFunc(job) 39 | } 40 | 41 | defer func() { 42 | if r := recover(); r != nil { 43 | // Get a reasonable error message from the panic 44 | msg := "" 45 | if err, ok := r.(error); ok { 46 | msg = err.Error() 47 | } else { 48 | msg = fmt.Sprint(r) 49 | } 50 | if err := setJobError(job, msg); err != nil { 51 | // Nothing left to do but panic 52 | panic(err) 53 | } 54 | } 55 | }() 56 | // Set the started field and save the job 57 | job.started = time.Now().UTC().UnixNano() 58 | t0 := newTransaction() 59 | t0.setJobField(job, "started", job.started) 60 | if err := t0.exec(); err != nil { 61 | if err := setJobError(job, err.Error()); err != nil { 62 | // NOTE: panics will be caught by the recover statment above 63 | panic(err) 64 | } 65 | return 66 | } 67 | // Use reflection to instantiate arguments for the handler 68 | handlerArgs := []reflect.Value{} 69 | if job.typ.dataType != nil { 70 | // Instantiate a new variable to hold this argument 71 | dataVal := reflect.New(job.typ.dataType) 72 | if err := decode(job.data, dataVal.Interface()); err != nil { 73 | if err := setJobError(job, err.Error()); err != nil { 74 | // NOTE: panics will be caught by the recover statment above 75 | panic(err) 76 | } 77 | return 78 | } 79 | handlerArgs = append(handlerArgs, dataVal.Elem()) 80 | } 81 | // Call the handler using the arguments we just instantiated 82 | handlerVal := reflect.ValueOf(job.typ.handler) 83 | returnVals := handlerVal.Call(handlerArgs) 84 | // Set the finished timestamp 85 | job.finished = time.Now().UTC().UnixNano() 86 | 87 | // Check if the error return value was nil 88 | if !returnVals[0].IsNil() { 89 | err := returnVals[0].Interface().(error) 90 | if err := setJobError(job, err.Error()); err != nil { 91 | // NOTE: panics will be caught by the recover statment above 92 | panic(err) 93 | } 94 | return 95 | } 96 | t1 := newTransaction() 97 | t1.setJobField(job, "finished", job.finished) 98 | if job.IsRecurring() { 99 | // If the job is recurring, reschedule and set status to queued 100 | job.time = job.NextTime() 101 | t1.setJobField(job, "time", job.time) 102 | t1.addJobToTimeIndex(job) 103 | t1.setStatus(job, StatusQueued) 104 | } else { 105 | // Otherwise, set status to finished 106 | t1.setStatus(job, StatusFinished) 107 | } 108 | if err := t1.exec(); err != nil { 109 | if err := setJobError(job, err.Error()); err != nil { 110 | // NOTE: panics will be caught by the recover statment above 111 | panic(err) 112 | } 113 | return 114 | } 115 | } 116 | 117 | func setJobError(job *Job, msg string) error { 118 | // Start a new transaction 119 | t := newTransaction() 120 | // Set the job error field 121 | t.setJobField(job, "error", msg) 122 | // Either queue the job for retry or mark it as failed depending 123 | // on how many retries the job has left 124 | t.retryOrFailJob(job, nil) 125 | if err := t.exec(); err != nil { 126 | return err 127 | } 128 | return nil 129 | } 130 | --------------------------------------------------------------------------------