├── .gitignore ├── README.md ├── commands ├── Base.js ├── Init.js ├── Job.js └── Work.js ├── package.json ├── providers ├── CommandProvider.js └── QueueProvider.js └── src ├── errors └── index.js ├── queue ├── Driver.js ├── JobMaker.js └── JobRegister.js ├── templates ├── Config.tmpl ├── consumer.tmpl ├── producer.tmpl ├── queue.tmpl └── queue_server.tmpl └── utils └── index.js /.gitignore: -------------------------------------------------------------------------------- 1 | node_modules/ 2 | package-lock.json 3 | npm-debug.log 4 | Adonis-Queue-Pro/* 5 | adonis-queue-pro-*.tgz 6 | bak -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Adonis Queue Pro 2 | Adonis queue pro is a worker-based queue library for [AdonisJS](https://github.com/adonisjs/adonis-framework), it is backed by [Kue](https://github.com/Automattic/kue) and [Kue-Scheduler](https://github.com/lykmapipo/kue-scheduler). 3 | 4 | **There has been a few breaking API changes since v1 (which supports adonis v3.2), please read the doc carefully!** 5 | 6 | ## Features 7 | - Ace commands for generating jobs and start workers 8 | - Use your existing adonis modules in your queue processor 9 | - (Close-to) Full kue/kue-scheduler API supported including future/repeat job scheduling 10 | - Multi-worker mode supported 11 | - Produce/Consumer model for structuring your job 12 | - Simple and Elegant API for scheduling and processing your jobs 13 | 14 | ## Notices 15 | This version only support **Adonis V4.0+**. For V3.2 support, please check the **v1** branch. 16 | 17 | ## Installation 18 | 19 | npm install --save adonis-queue-pro 20 | 21 | ## Configuration 22 | In your **star/app.js**, edit the following: 23 | 24 | - add `'adonis-queue-pro/providers/QueueProvider'` to your providers array. 25 | - add `'adonis-queue-pro/providers/CommandProvider'` to your aceProviders array. 26 | - add `Queue: 'Adonis/Addons/Queue'` to your aliases object 27 | - add the following commands to your commands array 28 | 29 | `'Adonis/Commands/Queue:Init'` 30 | 31 | `'Adonis/Commands/Queue:Job'` 32 | 33 | `'Adonis/Commands/Queue:Work'` 34 | 35 | ## Consumer/Producer model 36 | Instead of defining Job as a single entity, this library separates the responsibility of job into consumer and producer, here are the definitions: 37 | 38 | **Producer:** Define the static properties of the job, in kue's context, **supported** properties include **priority, attempts, backOff, delay, ttl and unique**. Documentations of each property can be found in Kue and Kue-scheduler's Github. 39 | 40 | **Consumer:** Define the runtime properties of the job, in kue's context, **supported** properties include **concurrency** and the process handler. 41 | 42 | Example of a basic producer/consumer pair can be found by generating a sample job using the ``./ace queue:job`` command. 43 | 44 | ## CLI API 45 | 46 | ### Initialize (Must be done first!) 47 | ```sh 48 | $ ./ace queue:init 49 | ``` 50 | This will create the queue configuration file and placed in the **config** directory, which is very similar to [kue-scheduler](https://github.com/lykmapipo/kue-scheduler)'s config file. 51 | 52 | This will also create the queue server adaptor in the **start** and root directory, which will be the entry point for the queue worker process. 53 | 54 | ### Manage jobs 55 | 56 | #### Create a job 57 | 58 | You can create a new job by running: 59 | 60 | ```sh 61 | $ ./ace queue:job SendEmail --jobId='send-email' 62 | ``` 63 | The option `jobId` is optional, if not provided, the kue type for the job will be a kebab-case of the argument. i.e. SendEmail -> send-email. 64 | 65 | This command will create job producers and consumers in designated directories, which are configurable in **config/queue.js** with **consumerPath** and **producerPath**; this defaults to **app/Jobs/{Consumers | Producers}**. 66 | 67 | The job consumers and producers will both run in Adonis framework's context, thus you can easily use any supported libraries within the job file. 68 | 69 | #### Remove a job 70 | 71 | Similarly, you can remove an existing job by running: 72 | 73 | ```sh 74 | $ ./ace queue:job SendEmail --remove 75 | ``` 76 | 77 | ### Run worker 78 | ```sh 79 | $ ./ace queue:work 4 80 | ``` 81 | The argument defines the number of workers to run simultaneously. Default to 1 if not provided. 82 | 83 | **Notice**: This command only supports a simple ``fork`` based process manager which is easy for local testing. It does not handle worker failure nor graceful restart. For production usage, you can use tools such as [Supervisor](https://github.com/Supervisor/supervisor) or [PM2](https://github.com/Unitech/pm2), and the command will be ``node start/queue.js`` in your app’s root directory. 84 | 85 | ## Job API 86 | 87 | The producer job file supports Kue job properties which are defined as an ES6 ``get`` property in the class, see example by running `./ace queue:job`. 88 | 89 | Refer to supported job properties above in the **Consumer/Producer Model** section. 90 | 91 | The consumer job file supports Kue job's **concurrecy** defined as an ES6 `static get` property in the class, see example by running `./ace queue:job`. 92 | 93 | The processing function is defined as an async function `async handle()` or `handle()` depending on whether your task is asynchronous. Within the task class, you can access constructor-injected payload with `this.data`. 94 | 95 | The producer job class also supports job events, listed below: 96 | ```js 97 | // with in producer class 98 | // job has been created and scheduled 99 | // useful for retrieving redis id for the job 100 | onInit(Kue/Job job) 101 | // See kue documentation for below 102 | onEnqueue(String jobType) 103 | onStart(String jobType) 104 | onPromotion(String jobType) 105 | onProgress(Float progress) 106 | // data returned from handle() method 107 | onComplete(Object data) 108 | onRemove(String jobType) 109 | // error caught in the handle() method 110 | onFailed(Error error) 111 | onFailedAttempts(Error error) 112 | ``` 113 | This producer job class itself is an Event Listener, thus you can export the data received from the job event to the outside world. 114 | 115 | A useful scenario is to remove job after it has been initialized: 116 | 117 | ```js 118 | // within job producer class 119 | onInit(job) { 120 | this.emit('init'); 121 | } 122 | // outside of the consumer 123 | // for queue.remove() see Queue API below 124 | job.on('init', () => Queue.remove(job).then(...)); 125 | ``` 126 | 127 | ## Queue API 128 | 129 | ### Access the queue 130 | ```js 131 | const Queue = use('Queue'); 132 | ``` 133 | ### Push job onto the queue 134 | ```js 135 | // optionally inject data into the job class using constructor 136 | // and access it in the consumer handler using this.data 137 | const ExampleJob = use('App/Jobs/Producer/ExampleJob'); 138 | Queue.dispatch(new ExampleJob({'data': 'whatever'})); 139 | ``` 140 | `Queue.dispatch()` has a second optional `String` argument default to 'now', which reflects the Kue-Scheduler API: 141 | ```js 142 | // schedule a job immediately 143 | Queue.dispatch(new ExampleJob, 'now'); 144 | // schedule a repeated job every 2 seconds 145 | // basically embeding the 'every' method into the string itself 146 | Queue.dispatch(new ExampleJob, 'every 2 seconds'); 147 | // schedule a single job in the future 148 | Queue.dispatch(new ExampleJob, '2 seconds from now'); 149 | ``` 150 | ### Remove jobs 151 | 152 | Remove a single job, the argument must be the **job instance** you created: 153 | 154 | ```js 155 | // asynchronous removal... 156 | Queue.remove(job).then(response => {}, error => {}); 157 | ``` 158 | 159 | Clear all jobs: 160 | 161 | ```js 162 | // also returns a promise 163 | Queue.clear().then(response => {}, error => {}); 164 | ``` 165 | 166 | Note: currently clear() will not trigger the remove event on the job. 167 | 168 | ### Run tests 169 | 170 | Please clone [the Adonis test app repo](https://github.com/ReactiveXYZ-Dev/adonis-queue-pro-test), install the dependencies, and run ``adonis test`` to run the spec tests. (Make sure redis is installed and configured properly as required by Kue). 171 | 172 | You can also contribute to the test repo by submitting issues and PRs. 173 | 174 | ## Development 175 | 176 | Contributions are welcome! This is a community project so please send a pull request whenever you feel like to! 177 | 178 | ### Todos 179 | - Expose API for graceful failure/restart 180 | - Improve efficiency 181 | - Squash bugs 182 | 183 | License 184 | ---- 185 | 186 | MIT 187 | 188 | -------------------------------------------------------------------------------- /commands/Base.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const dir = require('node-dir'); 4 | const { Command } = require('@adonisjs/ace'); 5 | const { dirExistsSync } = require('../src/utils'); 6 | 7 | /** 8 | * Convenient base class for all commands 9 | * 10 | * @version 2.0.0 11 | * @adonis-version 4.0+ 12 | */ 13 | class BaseCommand extends Command { 14 | 15 | /** 16 | * Inject adonis services into command 17 | * @param {Adonis/Src/Helpers} Helpers 18 | * @param {Adonis/Src/Config} Config 19 | */ 20 | constructor(Helpers, Config) { 21 | super(); 22 | this._helpers = Helpers; 23 | this._config = Config; 24 | } 25 | 26 | /** 27 | * injections from IoC container required to 28 | * be injected inside the contructor 29 | * 30 | * @return {Array} 31 | * 32 | * @public 33 | */ 34 | static get inject() { 35 | return ['Adonis/Src/Helpers', 'Adonis/Src/Config'] 36 | } 37 | 38 | /** 39 | * Check whether queue:init has run 40 | * @return {Boolean} 41 | */ 42 | hasInitialized() { 43 | return Boolean(this._config.get('queue')); 44 | } 45 | 46 | /** 47 | * Check whether the app has job files 48 | * @return {Boolean} 49 | */ 50 | hasJobs() { 51 | // load from defined job path 52 | const consumerPath = this._config.get('queue.consumerPath'); 53 | const producerPath = this._config.get('queue.producerPath'); 54 | 55 | if (!dirExistsSync(consumerPath) || !dirExistsSync(producerPath)) { 56 | return false; 57 | } 58 | 59 | let files = dir.files(consumerPath, { sync: true }); 60 | if (!files || files.length == 0) return false; 61 | 62 | files = dir.files(producerPath, { sync: true }); 63 | if (!files || files.length == 0) return false; 64 | 65 | return true; 66 | } 67 | 68 | } 69 | 70 | module.exports = BaseCommand; 71 | -------------------------------------------------------------------------------- /commands/Init.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const path = require('path') 4 | const BaseCommand = require('./Base'); 5 | const { copyFile } = require('../src/utils'); 6 | 7 | /** 8 | * Initialize all necessary boilerplates for the queue 9 | * 10 | * @version 2.0.0 11 | * @adonis-version 4.0+ 12 | */ 13 | class InitCommand extends BaseCommand { 14 | 15 | static get description() { 16 | return "Initialize queue configuration"; 17 | } 18 | 19 | static get signature() { 20 | return "queue:init"; 21 | } 22 | 23 | async handle() { 24 | 25 | try { 26 | // copy over sample configs and server files to respective directory 27 | const tmplPath = path.join(__dirname, '../src/templates'); 28 | await copyFile(path.join(tmplPath, 'config.tmpl'), 29 | this._helpers.appRoot() + "/config/queue.js"); 30 | 31 | await copyFile(path.join(tmplPath, 'queue.tmpl'), 32 | this._helpers.appRoot() + "/start/queue.js"); 33 | 34 | await copyFile(path.join(tmplPath, 'queue_server.tmpl'), 35 | this._helpers.appRoot() + "/queue_server.js"); 36 | 37 | this.success('Queue initialized successfully!'); 38 | 39 | } catch (e) { 40 | console.error(e); 41 | 42 | this.error('Failed to initialize queue with error: ' + e.message); 43 | } 44 | } 45 | 46 | } 47 | 48 | module.exports = InitCommand; 49 | -------------------------------------------------------------------------------- /commands/Job.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const path = require('path'); 4 | const mustache = require('mustache'); 5 | const { readFile, writeFile, dirExistsSync, createDir, deleteFile } = require('../src/utils'); 6 | const { paramCase } = require('change-case'); 7 | const BaseCommand = require('./Base'); 8 | 9 | /** 10 | * Generate producer/consumer pair for new jobs 11 | * 12 | * @version 2.0.0 13 | * @adonis-version 4.0+ 14 | */ 15 | 16 | class JobCommand extends BaseCommand { 17 | 18 | static get description() { 19 | return "Create a new job"; 20 | } 21 | 22 | static get signature() { 23 | return "queue:job {jobName:Name of job to process} {--jobId=@value} {--remove:remove this job}" 24 | } 25 | 26 | async handle({ jobName }, { jobId, remove }) { 27 | if (!this.hasInitialized()) { 28 | this.error("Please run queue:init before creating job!"); 29 | return; 30 | } 31 | 32 | if (!jobId) { 33 | jobId = paramCase(jobName); 34 | } 35 | 36 | try { 37 | // parse respective templates 38 | const producerTmpl = await readFile(path.join(__dirname, '../src/templates/producer.tmpl'), 'utf8'); 39 | const producerTask = mustache.render(producerTmpl, { 40 | jobName, jobId 41 | }); 42 | const consumerTmpl = await readFile(path.join(__dirname, '../src/templates/consumer.tmpl'), 'utf8'); 43 | const consumerTask = mustache.render(consumerTmpl, { 44 | jobName, jobId 45 | }); 46 | 47 | // save into selected directory 48 | const consumerPath = this._config.get('queue.consumerPath'); 49 | const producerPath = this._config.get('queue.producerPath'); 50 | 51 | if (!dirExistsSync(consumerPath)) { 52 | await createDir(consumerPath); 53 | } 54 | 55 | if (!dirExistsSync(producerPath)) { 56 | await createDir(producerPath); 57 | } 58 | 59 | if (remove) { 60 | await deleteFile(`${consumerPath}/${jobName}.js`); 61 | await deleteFile(`${producerPath}/${jobName}.js`); 62 | 63 | this.success("Job has been removed"); 64 | } else { 65 | await writeFile(`${consumerPath}/${jobName}.js`, consumerTask); 66 | await writeFile(`${producerPath}/${jobName}.js`, producerTask); 67 | 68 | this.success("Job has been created"); 69 | } 70 | 71 | } catch (e) { 72 | console.error(e); 73 | 74 | this.error("Failed to generate job classes with error " + e.message); 75 | } 76 | 77 | 78 | } 79 | 80 | } 81 | 82 | module.exports = JobCommand; 83 | -------------------------------------------------------------------------------- /commands/Work.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const { fork } = require('child_process'); 4 | const BaseCommand = require('./Base'); 5 | 6 | /** 7 | * Launch queue workers to start processing 8 | * 9 | * @version 2.0.0 10 | * @adonis-version 4.0+ 11 | */ 12 | 13 | class WorkCommand extends BaseCommand { 14 | 15 | static get description() { 16 | return "Start one or more workers"; 17 | } 18 | 19 | static get signature() { 20 | return "queue:work {numWorkers?:Number of workers to start with default of 1}" 21 | } 22 | 23 | async handle({ numWorkers }) { 24 | 25 | if (!this.hasInitialized()) { 26 | this.error("Please run queue:init before processing!"); 27 | return; 28 | } 29 | 30 | if (!this.hasJobs()) { 31 | this.error("No jobs to watch for. Please use queue:job to create jobs!"); 32 | return; 33 | } 34 | 35 | numWorkers = numWorkers ? parseInt(numWorkers) : 1; 36 | 37 | for (let i = 1; i <= numWorkers; ++i) { 38 | const worker = fork(this._helpers.appRoot() + "/queue_server.js", [], { 39 | silent: true 40 | }); 41 | console.log(`Started worker ${i}`); 42 | worker.stdout.on('data', message => { 43 | console.log(`Stdout from worker ${i}: ` + message.toString('utf8')); 44 | }); 45 | worker.stderr.on('data', message => { 46 | console.error(`Stderr from worker ${i}: ` + message.toString('utf8')); 47 | }); 48 | } 49 | 50 | this.success('Workers are running...'); 51 | 52 | // prevent the main process from exiting... 53 | setInterval(() => { }, 1000); 54 | } 55 | } 56 | 57 | module.exports = WorkCommand; 58 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "_from": "file:../../Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz", 3 | "_id": "adonis-queue-pro@2.0.0", 4 | "_inBundle": false, 5 | "_integrity": "sha1-PfMKR0plUULwcDJwb1M83YUm5Uo=", 6 | "_location": "/adonis-queue-pro", 7 | "_phantomChildren": { 8 | "graceful-fs": "4.1.15", 9 | "jsonfile": "4.0.0", 10 | "universalify": "0.1.2" 11 | }, 12 | "_requested": { 13 | "type": "file", 14 | "where": "/Users/jackiezhang/Projects/Web/Test/adonis", 15 | "raw": "../../Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz", 16 | "rawSpec": "../../Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz", 17 | "saveSpec": "file:../../Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz", 18 | "fetchSpec": "/Users/jackiezhang/Projects/Web/Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz" 19 | }, 20 | "_requiredBy": [ 21 | "#USER", 22 | "/" 23 | ], 24 | "_resolved": "/Users/jackiezhang/Projects/Web/Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz", 25 | "_spec": "../../Adonis-Queue-Pro/adonis-queue-pro-2.0.0.tgz", 26 | "_where": "/Users/jackiezhang/Projects/Web/Test/adonis", 27 | "author": { 28 | "name": "Jackie Zhang", 29 | "url": "ReactiveXYZ" 30 | }, 31 | "bugs": { 32 | "url": "https://github.com/ReactiveXYZ-Dev/Adonis-Queue-Pro/issues" 33 | }, 34 | "bundleDependencies": false, 35 | "dependencies": { 36 | "change-case": "^3.0.2", 37 | "fs-extra": "^5.0.0", 38 | "kue": "^0.11.6", 39 | "kue-scheduler": "^0.8.3", 40 | "mkdirp": "^0.5.1", 41 | "mustache": "^2.3.0", 42 | "node-dir": "^0.1.17", 43 | "randomstring": "^1.1.5" 44 | }, 45 | "deprecated": false, 46 | "description": "An easy-to-use job queue for AdonisJS framework backed by Kue and Kue-scheduler", 47 | "devDependencies": { 48 | "@adonisjs/ace": "^5.0.8", 49 | "@adonisjs/fold": "^4.0.9", 50 | "japa": "^2.0.10", 51 | "japa-cli": "^1.0.1" 52 | }, 53 | "homepage": "https://github.com/ReactiveXYZ-Dev/Adonis-Queue-Pro#readme", 54 | "keywords": [ 55 | "adonis", 56 | "queue", 57 | "kue", 58 | "job", 59 | "worker" 60 | ], 61 | "license": "MIT", 62 | "main": "index.js", 63 | "name": "adonis-queue-pro", 64 | "repository": { 65 | "type": "git", 66 | "url": "git+https://github.com/ReactiveXYZ-Dev/Adonis-Queue-Pro.git" 67 | }, 68 | "scripts": { 69 | "test": "echo \"Error: no test specified\" && exit 1" 70 | }, 71 | "version": "2.0.0" 72 | } 73 | -------------------------------------------------------------------------------- /providers/CommandProvider.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const { ServiceProvider } = require('@adonisjs/fold'); 4 | 5 | /** 6 | * Provider for injecting Ace commands into the app 7 | * 8 | * @version 2.0.0 9 | * @adonis-version 4.0+ 10 | */ 11 | class CommandProvider extends ServiceProvider { 12 | 13 | constructor(Ioc) { 14 | super(Ioc) 15 | // define types of commands 16 | // init: copy queue server boostrapper, sample configuration, and create default job path 17 | // job: create a producer consumer job pair with name 18 | // work: start queue listeners with optional number of instances 19 | this._commands = ['Init', 'Job', 'Work']; 20 | } 21 | 22 | /** 23 | * Register all commands 24 | */ 25 | register() { 26 | this._commands.forEach(command => { 27 | this.app.bind(`Adonis/Commands/Queue:${command}`, () => require(`../commands/${command}`)); 28 | }); 29 | } 30 | 31 | /** 32 | * Add commands to ace 33 | */ 34 | boot() { 35 | const ace = require('@adonisjs/ace') 36 | this._commands.forEach(command => { 37 | ace.addCommand(`Adonis/Commands/Queue:${command}`); 38 | }); 39 | } 40 | 41 | } 42 | 43 | module.exports = CommandProvider; -------------------------------------------------------------------------------- /providers/QueueProvider.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const { ServiceProvider } = require('@adonisjs/fold'); 4 | const Queue = require('../src/queue/Driver'); 5 | 6 | /** 7 | * Provider for inject Queue service into the Adonis Ioc 8 | * 9 | * @version 2.0.0 10 | * @adonis-version 4.0+ 11 | */ 12 | 13 | class QueueProvider extends ServiceProvider { 14 | 15 | /** 16 | * Register a Queue instance 17 | */ 18 | register() { 19 | this.app.singleton('Adonis/Addons/Queue', app => { 20 | return new Queue(app); 21 | }); 22 | } 23 | }; 24 | 25 | module.exports = QueueProvider; -------------------------------------------------------------------------------- /src/errors/index.js: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | /** 4 | * Miscellaneous custom errors 5 | */ 6 | 7 | class BaseError extends Error { 8 | constructor(message) { 9 | super(message); 10 | } 11 | 12 | setError(error) { 13 | this.error = error; 14 | return this; 15 | } 16 | 17 | getError() { 18 | return this.error; 19 | } 20 | 21 | updateMessage() { 22 | let message = this.message; 23 | if (this.error) message += ` (src err: ${this.error.message})`; 24 | this.message = message; 25 | return this; 26 | } 27 | } 28 | 29 | class JobDirectoryNotFoundError extends BaseError {} 30 | class JobProcessError extends BaseError {} 31 | class JobFetchError extends BaseError {} 32 | 33 | module.exports = { 34 | JobDirectoryNotFoundError, 35 | JobProcessError, 36 | JobFetchError 37 | } 38 | -------------------------------------------------------------------------------- /src/queue/Driver.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | const kue = require('kue-scheduler'); 4 | const JobMaker = require('./JobMaker'); 5 | const JobRegister = require('./JobRegister'); 6 | 7 | /** 8 | * Main queue driver 9 | * 10 | * @version 2.0.0 11 | * @adonis-version 4.0+ 12 | */ 13 | class Queue { 14 | 15 | /** 16 | * Construct the queue 17 | * @param {Adonis/App} app Adonis app/Ioc instance 18 | */ 19 | constructor(app) { 20 | // inject the app instance 21 | this._app = app; 22 | // initialize adonis logic 23 | this._config = this._app.use('Adonis/Src/Config'); 24 | this._helpers = this._app.use('Adonis/Src/Helpers'); 25 | // initialize kue queue 26 | this._queue = kue.createQueue(this._config.get('queue.connection')); 27 | // boost number of event listeners a queue instance can listen to 28 | this._queue.setMaxListeners(0); 29 | } 30 | 31 | /** 32 | * Register job event handlers 33 | * @return {Promise} 34 | */ 35 | processing() { 36 | const register = new JobRegister(this._config, this._helpers); 37 | return register.setQueue(this._queue).listenForAppJobs(); 38 | } 39 | 40 | 41 | /** 42 | * Dispatch a new job 43 | * @param {App/Jobs} job Job instances 44 | * @return {Void} 45 | */ 46 | dispatch(job, when = "now") { 47 | // create a job maker factory 48 | const maker = new JobMaker; 49 | 50 | // get the kue job converted from app job 51 | const kueJob = maker.setAppJob(job) 52 | .setQueue(this._queue) 53 | .process() 54 | .getFinalJob(); 55 | 56 | // schedule the job in the queue 57 | if (when == "now") { 58 | // immediate job 59 | this._queue.now(kueJob); 60 | } else if (when.includes("every") || when.includes("*")) { 61 | when = when.replace('every ', ''); 62 | // cron or repeating job 63 | this._queue.every(when, kueJob); 64 | } else { 65 | // schedule a job 66 | this._queue.schedule(when, kueJob); 67 | } 68 | } 69 | 70 | /** 71 | * Remove a job from queue 72 | * @param {App/Job} job Job producer 73 | * @return {Promise} 74 | */ 75 | remove(job) { 76 | return new Promise((resolve, reject) => { 77 | this._queue.remove(job._kueJob.id, (error, response) => { 78 | if (error) { 79 | reject(error); 80 | } else { 81 | // send the onRemove event 82 | if (job['onRemove']) { 83 | job['onRemove'](job.type); 84 | } 85 | resolve(response); 86 | } 87 | }); 88 | }); 89 | } 90 | 91 | /** 92 | * Clear all jobs within a queue for a clean start 93 | * @return {Promise} 94 | */ 95 | clear() { 96 | return new Promise((resolve, reject) => { 97 | this._queue.clear((error, response) => { 98 | if (error) { 99 | reject(error); 100 | } else { 101 | resolve(response); 102 | } 103 | }); 104 | }); 105 | } 106 | } 107 | 108 | 109 | module.exports = Queue; 110 | -------------------------------------------------------------------------------- /src/queue/JobMaker.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | const kue = require('kue'); 4 | const randomString = require("randomstring"); 5 | const { JobFetchError } = require("../errors"); 6 | 7 | /** 8 | * Parse producer job contents and generate kue job 9 | * 10 | * @version 2.0.0 11 | * @adonis-version 4.0+ 12 | */ 13 | class JobMaker { 14 | 15 | /** 16 | * Get the final made job 17 | * @return {Kue/Job} 18 | */ 19 | getFinalJob() { 20 | return this._kueJob; 21 | } 22 | 23 | /** 24 | * Inject the app job 25 | * @param {App/Job} job 26 | */ 27 | setAppJob(job) { 28 | this._job = job; 29 | return this; 30 | } 31 | 32 | /** 33 | * Inject the Kue queue 34 | * @param {Queue} queue 35 | */ 36 | setQueue(queue) { 37 | this._queue = queue; 38 | return this; 39 | } 40 | 41 | /** 42 | * Run through the making procedures 43 | * @return {this} 44 | */ 45 | process() { 46 | return this.initialize() 47 | .assignPriority() 48 | .assignFailureAttempts() 49 | .assignFailureBackoff() 50 | .assignFailureBackoff() 51 | .assignDelay() 52 | .assignTTL() 53 | .assignUnique() 54 | .assignEventListeners(); 55 | } 56 | 57 | /** 58 | * Initalize the Kue Job 59 | * @param {App/Job} job 60 | * @return {this} 61 | */ 62 | initialize() { 63 | // generate an UUID for this job along with its type 64 | this._job.data['__unique_id__'] = this._job.constructor.type + randomString.generate(15); 65 | this._kueJob = this._queue.createJob( 66 | this._job.constructor.type, 67 | this._job.data 68 | ); 69 | return this; 70 | } 71 | 72 | /** 73 | * Set priority for the job 74 | * @return {this} 75 | */ 76 | assignPriority() { 77 | if (this._job.priority) { 78 | this._kueJob.priority(this._job.priority); 79 | } 80 | return this; 81 | } 82 | 83 | /** 84 | * Set failure attempts for the job 85 | * @return {this} 86 | */ 87 | assignFailureAttempts() { 88 | if (this._job.attempts) { 89 | this._kueJob.attempts(this._job.attempts); 90 | } 91 | return this; 92 | } 93 | 94 | /** 95 | * Set failure backoff for the job 96 | * @return {this} 97 | */ 98 | assignFailureBackoff() { 99 | if (this._job.backoff) { 100 | this._kueJob.backoff(this._job.backoff); 101 | } 102 | return this; 103 | } 104 | 105 | /** 106 | * Set job delay 107 | * @return {this} 108 | */ 109 | assignDelay() { 110 | if (this._job.delay) { 111 | this._kueJob.delay(this,job.delay * 1000); 112 | } 113 | return this; 114 | } 115 | 116 | /** 117 | * Set Time to live for the job 118 | * @return {this} 119 | */ 120 | assignTTL() { 121 | if (this._job.ttl) { 122 | this._kueJob.ttl(this._job.ttl * 1000); 123 | } 124 | return this; 125 | } 126 | 127 | /** 128 | * Set uniqueness of this job 129 | * @return {this} 130 | */ 131 | assignUnique() { 132 | if (this._job.unique) { 133 | this._kueJob.unique(this._job.constructor.type); 134 | } 135 | return this; 136 | } 137 | 138 | /** 139 | * Assign event listeners for the job 140 | * @return {this} 141 | */ 142 | assignEventListeners() { 143 | let events = [ 144 | 'enqueue', 'start', 'promotion', 'progress', 145 | 'failed attempts', 'failed', 'complete' 146 | ]; 147 | events.forEach(event => { 148 | this._queue.on(`job ${event}`, (id, ...args) => { 149 | kue.Job.get(id, (err, kueJob) => { 150 | if (!err) { 151 | if (kueJob.data['__unique_id__'] === this._job.data['__unique_id__']) { 152 | const eventName = "on" + event.split(' ').map(word => { 153 | return word[0].toUpperCase() + word.slice(1); 154 | }).join(''); 155 | 156 | if (event === 'enqueue') { 157 | this._job.onInit(kueJob); 158 | // save kue job to job when enqueued 159 | this._job._kueJob = kueJob; 160 | } 161 | if (this._job[eventName]) { 162 | this._job[eventName](...args); 163 | } 164 | } 165 | } else if (this._job['onFailed']) { 166 | this._job.onFailed( 167 | new JobFetchError(`Failed to fetch job id ${id}, event ${event}`) 168 | .setError(err).updateMessage() 169 | ); 170 | } 171 | 172 | }); 173 | }); 174 | }); 175 | return this; 176 | } 177 | 178 | } 179 | 180 | module.exports = JobMaker; 181 | -------------------------------------------------------------------------------- /src/queue/JobRegister.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | const dir = require('node-dir'); 4 | const { JobDirectoryNotFoundError, JobProcessError } = require('../errors'); 5 | 6 | /** 7 | * Register and preload consumer processes 8 | * 9 | * @version 2.0.0 10 | * @adonis-version 4.0+ 11 | */ 12 | class JobRegister { 13 | 14 | /** 15 | * Inject adonis app for accessing dynamic data 16 | * @param {Adonis/Src/Config} config 17 | * @param {Adonis/Src/Helpers} helpers 18 | */ 19 | constructor(Config, Helpers) { 20 | this._config = Config; 21 | this._helpers = Helpers; 22 | } 23 | 24 | /** 25 | * Inject Kue Queue into the app 26 | * @param {Kue/Queue} queue 27 | */ 28 | setQueue(queue) { 29 | this.queue = queue; 30 | return this; 31 | } 32 | 33 | /** 34 | * Load all job classes aynchronously 35 | * @return {Promise} 36 | */ 37 | listenForAppJobs() { 38 | return this._jobFilePaths() 39 | .then(filePaths => { 40 | this._requireAndProcessJobs(filePaths); 41 | return Promise.resolve({ 42 | message: "Preprocessed jobs and started queue listener!" 43 | }); 44 | }, error => { 45 | const e = new JobDirectoryNotFoundError("Consumer/Producer directory not found. Please make sure to create job with ./ace queue:job") 46 | return Promise.reject(e.setError(error)); 47 | }); 48 | } 49 | 50 | /** 51 | * Get all job file paths 52 | * @return {Promise} File paths 53 | */ 54 | _jobFilePaths() { 55 | const consumerPath = this._config.get('queue.consumerPath'); 56 | return dir.promiseFiles(consumerPath ? consumerPath : this._helpers.appRoot() + '/app/Jobs/Consumers'); 57 | } 58 | 59 | /** 60 | * Require all available jobs and process them 61 | * @param {Array} filePaths Job class files 62 | * @return {Void} 63 | */ 64 | _requireAndProcessJobs(filePaths) { 65 | filePaths.forEach(path => { 66 | // disable getting index job 67 | if (path.includes('index')) { 68 | return; 69 | } 70 | 71 | const Job = require(path); 72 | const concurrency = Job.concurrency || 1; 73 | 74 | this.queue.process( 75 | Job.type, 76 | concurrency, 77 | (job, ctx, done) => { 78 | // recreate the job and apply the handle function 79 | const appJob = new Job(job.data); 80 | appJob.setContext(ctx); 81 | try { 82 | const res = appJob.handle.apply(appJob); 83 | if (res instanceof Promise) { 84 | res.then( 85 | success => done(null, success), 86 | error => { 87 | const e = new JobProcessError(`Failed to process job ${Job.name}!`); 88 | done(e.setError(error).updateMessage()); 89 | } 90 | ); 91 | } else { 92 | // just a regular call 93 | done(null, res); 94 | } 95 | } catch (error) { 96 | const e = new JobProcessError(`Failed to process job ${Job.name}!`); 97 | done(e.setError(error).updateMessage()); 98 | } 99 | }); 100 | }); 101 | } 102 | } 103 | 104 | module.exports = JobRegister; 105 | -------------------------------------------------------------------------------- /src/templates/Config.tmpl: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | const Helpers = use('Helpers') 4 | const Env = use('Env') 5 | 6 | 7 | /** 8 | * Sample configuration for queue, very similar to 9 | * https://github.com/lykmapipo/kue-scheduler 10 | * 11 | * @version 2.0.0 12 | * @adonis-version 4.0+ 13 | */ 14 | 15 | module.exports = { 16 | 17 | /** 18 | * Default directory for storing consumer/producer job handlers 19 | * 20 | * You may modify the consumer/producer job paths to wherever you want 21 | */ 22 | 'consumerPath' : Helpers.appRoot() + "/app/Jobs/Consumers", 23 | 'producerPath' : Helpers.appRoot() + "/app/Jobs/Producers", 24 | 25 | 'connection' : { 26 | 27 | /** 28 | * The prefix is for differentiating kue job names from 29 | * other redis-related tasks. Modify to your needs. 30 | * 31 | * @type {String} 32 | */ 33 | 'prefix' : 'adonis:queue', 34 | 35 | 'redis' : { 36 | 37 | host: '127.0.0.1', 38 | port: 6379, 39 | db: 0, 40 | options: {} 41 | 42 | }, 43 | 44 | 'restore' : false, 45 | 46 | 'worker' : true 47 | 48 | } 49 | 50 | }; -------------------------------------------------------------------------------- /src/templates/consumer.tmpl: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | 4 | /** 5 | * Sample job consumer class 6 | * 7 | * @version 2.0.0 8 | * @adonis-version 4.0+ 9 | */ 10 | 11 | class {{ jobName }} { 12 | 13 | /** 14 | * Concurrency for processing this job 15 | * @return {Int} Num of jobs processed at time 16 | */ 17 | static get concurrency() { 18 | return 1; 19 | } 20 | 21 | /** 22 | * UUID for this job class 23 | * Make sure consumer and producer are in sync 24 | * @return {String} 25 | */ 26 | static get type() { 27 | return '{{ jobId }}'; 28 | } 29 | 30 | /** 31 | * Inject custom payload into the job class 32 | * @param {Object} data 33 | * 34 | * DO NOT MODIFY! 35 | */ 36 | constructor(data) { 37 | this.data = data; 38 | } 39 | 40 | /** 41 | * Inject the kue ctx to the consumer, you can use it to 42 | * pause(), shutdown() or remove() handler actions. 43 | * See kue's doc for more details 44 | * @param {Object} data 45 | * 46 | * DO NOT MODIFY! 47 | */ 48 | setContext(ctx) { 49 | this.ctx = ctx; 50 | } 51 | 52 | /** 53 | * Handle the sending of email data 54 | * You can drop the async keyword if it is synchronous 55 | */ 56 | async handle() { 57 | // Execute your task here... 58 | } 59 | 60 | 61 | } 62 | 63 | module.exports = {{ jobName }}; 64 | -------------------------------------------------------------------------------- /src/templates/producer.tmpl: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | const EventEmitter = require('events'); 4 | 5 | /** 6 | * Sample job producer class 7 | * 8 | * @version 2.0.0 9 | * @adonis-version 4.0+ 10 | */ 11 | 12 | class {{ jobName }} extends EventEmitter { 13 | 14 | /** 15 | * UUID for this job class 16 | * Make sure consumer and producer are in sync 17 | * @return {String} 18 | */ 19 | static get type() { 20 | return '{{ jobId }}'; 21 | } 22 | 23 | /** 24 | * Inject custom payload into the job class 25 | * @param {Object} data 26 | * 27 | * DO NOT MODIFY! 28 | */ 29 | constructor(data) { 30 | super(); 31 | this.data = data; 32 | } 33 | 34 | /** 35 | * Priority for this job 36 | * @return {String|Int} 37 | */ 38 | get priority() { 39 | return 'normal'; 40 | } 41 | 42 | /** 43 | * Number of attempts after each failure 44 | * @return {Int} 45 | */ 46 | get attempts() { 47 | return 3; 48 | } 49 | 50 | /** 51 | * Whether this job will be unique 52 | * @return {Boolean} 53 | */ 54 | get unique() { 55 | return false; 56 | } 57 | 58 | /** 59 | * Event handlers 60 | */ 61 | /** 62 | * Job created and sent to queue 63 | * @param {Kue/Job} job Kue job, see https://github.com/Automattic/kue/blob/master/lib/queue/job.js 64 | * @return {Void} 65 | */ 66 | onInit(job) { 67 | console.log("Inited Job ID " + job.id); 68 | 69 | this.emit('init', job.id); // emit the id of the kue job 70 | } 71 | 72 | /** 73 | * Completed with optional data 74 | * @param {Object} data Data passed from consumer's handle() method 75 | * @return {Void} 76 | */ 77 | onComplete(data) { 78 | console.log("Completed!"); 79 | 80 | this.emit('complete', data); // emit the completion data 81 | } 82 | 83 | /** 84 | * Failed event 85 | * @param {String} e Error message 86 | * @return {Void} 87 | */ 88 | onFailed(e) { 89 | console.log(e); 90 | 91 | this.emit('failed', e); // emit the error 92 | } 93 | 94 | } 95 | 96 | module.exports = {{ jobName }}; 97 | -------------------------------------------------------------------------------- /src/templates/queue.tmpl: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | /** 4 | * Start the queue listener for the consumer process 5 | * a.k.a your main application 6 | * 7 | * @version 2.0.0 8 | * @adonis-version 4.0+ 9 | */ 10 | const Queue = use('Queue'); 11 | 12 | Queue.processing().then( 13 | success => { console.log(success.message) }, 14 | error => { 15 | console.error(error.message); 16 | process.exit(1); 17 | } 18 | ); 19 | -------------------------------------------------------------------------------- /src/templates/queue_server.tmpl: -------------------------------------------------------------------------------- 1 | 'use strict' 2 | 3 | /* 4 | |-------------------------------------------------------------------------- 5 | | Queue server 6 | |-------------------------------------------------------------------------- 7 | | 8 | | This file bootstrap Adonisjs to start the Queue server. 9 | | 10 | | It is based on the standard AdonisJs Http server configuration, but removed 11 | | the HTTP server overhead, leaving only a queue task processor 12 | */ 13 | 14 | const { Ignitor } = require('@adonisjs/ignitor') 15 | 16 | new Ignitor(require('@adonisjs/fold')) 17 | .appRoot(__dirname) 18 | .preLoad('start/queue') 19 | .fire() 20 | .catch(console.error) 21 | -------------------------------------------------------------------------------- /src/utils/index.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Utility functions 3 | * 4 | * @version 2.0.0 5 | * @adonis-version 4.0+ 6 | */ 7 | 8 | const fs = require('fs'); 9 | const fse = require('fs-extra'); 10 | const mkdirp = require('mkdirp'); 11 | 12 | module.exports = { 13 | 14 | dirExistsSync: path => { 15 | 16 | return fs.existsSync(path); 17 | 18 | }, 19 | 20 | createDir: path => { 21 | 22 | return new Promise((resolve, reject) => { 23 | 24 | mkdirp(path, err => { 25 | 26 | if (err) { 27 | reject(err); 28 | } else { 29 | resolve() 30 | } 31 | 32 | }); 33 | 34 | }); 35 | 36 | }, 37 | 38 | copyFile: (src, dest) => { 39 | 40 | return new Promise((resolve, reject) => { 41 | 42 | fse.copy(src, dest, err => { 43 | 44 | if (err) { 45 | reject(err); 46 | } else { 47 | resolve(); 48 | } 49 | 50 | }); 51 | 52 | }); 53 | 54 | }, 55 | 56 | readFile: (src, options) => { 57 | 58 | return new Promise((resolve, reject) => { 59 | 60 | fs.readFile(src, options, (err, data) => { 61 | 62 | if (err) { 63 | reject(err); 64 | } else { 65 | resolve(data); 66 | } 67 | 68 | }); 69 | 70 | }); 71 | 72 | }, 73 | 74 | writeFile: (dest, content, options) => { 75 | 76 | return new Promise((resolve, reject) => { 77 | 78 | fs.writeFile(dest, content, options, err => { 79 | 80 | if (err) { 81 | reject(err); 82 | } else { 83 | resolve(); 84 | } 85 | 86 | }); 87 | 88 | }); 89 | 90 | }, 91 | 92 | deleteFile: src => { 93 | 94 | return new Promise((resolve, reject) => { 95 | 96 | fs.unlink(src, err => { 97 | 98 | if (err) { 99 | reject(err); 100 | } else { 101 | resolve(); 102 | } 103 | 104 | }); 105 | 106 | }); 107 | 108 | } 109 | 110 | }; 111 | --------------------------------------------------------------------------------