├── .github └── FUNDING.yml ├── CHANGELOG.md ├── LICENSE ├── README.md ├── composer.json └── src └── Queue.php /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | github: clue 2 | custom: https://clue.engineering/support 3 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## 1.7.0 (2025-05-23) 4 | 5 | * Feature: Improve template types to support typed handler arguments. 6 | (#51 by @clue) 7 | 8 | * Improve documentation and examples. 9 | (#47 by @yadaiio and #44 @szepeviktor) 10 | 11 | * Improve test suite, run tests on PHP 8.3 + PHP 8.4 and update test environment. 12 | (#46 by @yadaiio and #50 by @PaulRotmann) 13 | 14 | ## 1.6.0 (2023-07-28) 15 | 16 | * Feature: Improve Promise v3 support and use template types. 17 | (#41 and #42 by @clue) 18 | 19 | * Feature: Improve PHP 8.2+ support by refactoring queuing logic. 20 | (#43 by @clue) 21 | 22 | * Improve test suite, ensure 100% code coverage and report failed assertions. 23 | (#37 and #39 by @clue) 24 | 25 | ## 1.5.0 (2022-09-30) 26 | 27 | * Feature: Forward compatibility with upcoming Promise v3. 28 | (#33 by @clue) 29 | 30 | * Update to use new reactphp/async package instead of clue/reactphp-block. 31 | (#34 by @SimonFrings) 32 | 33 | ## 1.4.0 (2021-11-15) 34 | 35 | * Feature: Support PHP 8.1, avoid deprecation warning concerning `\Countable::count(...)` return type. 36 | (#32 by @bartvanhoutte) 37 | 38 | * Improve documentation and simplify examples by updating to new [default loop](https://reactphp.org/event-loop/#loop). 39 | (#27 and #29 by @PaulRotmann and #30 by @SimonFrings) 40 | 41 | * Improve test suite to use GitHub actions for continuous integration (CI). 42 | (#28 by @SimonFrings) 43 | 44 | ## 1.3.0 (2020-10-16) 45 | 46 | * Enhanced documentation for ReactPHP's new HTTP client and 47 | add support / sponsorship info. 48 | (#21 and #24 by @clue) 49 | 50 | * Improve test suite and add `.gitattributes` to exclude dev files from exports. 51 | Prepare PHP 8 support, update to PHPUnit 9 and simplify test matrix. 52 | (#22, #23 and #25 by @SimonFrings) 53 | 54 | ## 1.2.0 (2019-12-05) 55 | 56 | * Feature: Add `any()` helper to await first successful fulfillment of operations. 57 | (#18 by @clue) 58 | 59 | ```php 60 | // new: limit concurrency while awaiting any operation to complete 61 | $promise = Queue::any(3, $urls, function ($url) use ($browser) { 62 | return $browser->get($url); 63 | }); 64 | 65 | $promise->then(function (ResponseInterface $response) { 66 | echo 'First successful: ' . $response->getStatusCode() . PHP_EOL; 67 | }); 68 | ``` 69 | 70 | * Minor documentation improvements (fix syntax issues and typos) and update examples. 71 | (#9 and #11 by @clue and #15 by @holtkamp) 72 | 73 | * Improve test suite to test against PHP 7.4 and PHP 7.3, drop legacy HHVM support, 74 | update distro on Travis and update project homepage. 75 | (#10 and #19 by @clue) 76 | 77 | ## 1.1.0 (2018-04-30) 78 | 79 | * Feature: Add `all()` helper to await successful fulfillment of all operations 80 | (#8 by @clue) 81 | 82 | ```php 83 | // new: limit concurrency while awaiting all operations to complete 84 | $promise = Queue::all(3, $urls, function ($url) use ($browser) { 85 | return $browser->get($url); 86 | }); 87 | 88 | $promise->then(function (array $responses) { 89 | echo 'All ' . count($responses) . ' successful!' . PHP_EOL; 90 | }); 91 | ``` 92 | 93 | * Fix: Implement cancellation forwarding for previously queued operations 94 | (#7 by @clue) 95 | 96 | ## 1.0.0 (2018-02-26) 97 | 98 | * First stable release, following SemVer 99 | 100 | I'd like to thank [Bergfreunde GmbH](https://www.bergfreunde.de/), a German 101 | online retailer for Outdoor Gear & Clothing, for sponsoring the first release! 🎉 102 | Thanks to sponsors like this, who understand the importance of open source 103 | development, I can justify spending time and focus on open source development 104 | instead of traditional paid work. 105 | 106 | > Did you know that I offer custom development services and issuing invoices for 107 | sponsorships of releases and for contributions? Contact me (@clue) for details. 108 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2018 Christian Lück 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is furnished 10 | to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # clue/reactphp-mq 2 | 3 | [![CI status](https://github.com/clue/reactphp-mq/actions/workflows/ci.yml/badge.svg)](https://github.com/clue/reactphp-mq/actions) 4 | [![code coverage](https://img.shields.io/badge/code%20coverage-100%25-success)](#tests) 5 | [![installs on Packagist](https://img.shields.io/packagist/dt/clue/mq-react?color=blue&label=installs%20on%20Packagist)](https://packagist.org/packages/clue/mq-react) 6 | 7 | Mini Queue, the lightweight in-memory message queue to concurrently do many (but not too many) things at once, 8 | built on top of [ReactPHP](https://reactphp.org/). 9 | 10 | Let's say you crawl a page and find that you need to send 100 HTTP requests to 11 | following pages which each takes `0.2s`. You can either send them all 12 | sequentially (taking around `20s`) or you can use 13 | [ReactPHP](https://reactphp.org/) to concurrently request all your pages at the 14 | same time. This works perfectly fine for a small number of operations, but 15 | sending an excessive number of requests can either take up all resources on your 16 | side or may get you banned by the remote side as it sees an unreasonable number 17 | of requests from your side. 18 | Instead, you can use this library to effectively rate limit your operations and 19 | queue excessives ones so that not too many operations are processed at once. 20 | This library provides a simple API that is easy to use in order to manage any 21 | kind of async operation without having to mess with most of the low-level details. 22 | You can use this to throttle multiple HTTP requests, database queries or pretty 23 | much any API that already uses Promises. 24 | 25 | * **Async execution of operations** - 26 | Process any number of async operations and choose how many should be handled 27 | concurrently and how many operations can be queued in-memory. Process their 28 | results as soon as responses come in. 29 | The Promise-based design provides a *sane* interface to working with out of order results. 30 | * **Lightweight, SOLID design** - 31 | Provides a thin abstraction that is [*just good enough*](https://en.wikipedia.org/wiki/Principle_of_good_enough) 32 | and does not get in your way. 33 | Builds on top of well-tested components and well-established concepts instead of reinventing the wheel. 34 | * **Good test coverage** - 35 | Comes with an automated tests suite and is regularly tested in the *real world*. 36 | 37 | **Table of contents** 38 | 39 | * [Support us](#support-us) 40 | * [Quickstart example](#quickstart-example) 41 | * [Usage](#usage) 42 | * [Queue](#queue) 43 | * [Promises](#promises) 44 | * [Cancellation](#cancellation) 45 | * [Timeout](#timeout) 46 | * [all()](#all) 47 | * [any()](#any) 48 | * [Blocking](#blocking) 49 | * [Install](#install) 50 | * [Tests](#tests) 51 | * [License](#license) 52 | 53 | ## Support us 54 | 55 | We invest a lot of time developing, maintaining and updating our awesome 56 | open-source projects. You can help us sustain this high-quality of our work by 57 | [becoming a sponsor on GitHub](https://github.com/sponsors/clue). Sponsors get 58 | numerous benefits in return, see our [sponsoring page](https://github.com/sponsors/clue) 59 | for details. 60 | 61 | Let's take these projects to the next level together! 🚀 62 | 63 | ## Quickstart example 64 | 65 | Once [installed](#install), you can use the following code to access an 66 | HTTP webserver and send a large number of HTTP GET requests: 67 | 68 | ```php 69 | get($url); 82 | }); 83 | 84 | foreach ($urls as $url) { 85 | $q($url)->then(function (Psr\Http\Message\ResponseInterface $response) use ($url) { 86 | echo $url . ': ' . $response->getBody()->getSize() . ' bytes' . PHP_EOL; 87 | }, function (Exception $e) { 88 | echo 'Error: ' . $e->getMessage() . PHP_EOL; 89 | }); 90 | } 91 | 92 | ``` 93 | 94 | See also the [examples](examples/). 95 | 96 | ## Usage 97 | 98 | ### Queue 99 | 100 | The `Queue` is responsible for managing your operations and ensuring not too 101 | many operations are executed at once. It's a very simple and lightweight 102 | in-memory implementation of the 103 | [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket#As_a_queue) algorithm. 104 | 105 | This means that you control how many operations can be executed concurrently. 106 | If you add a job to the queue and it still below the limit, it will be executed 107 | immediately. If you keep adding new jobs to the queue and its concurrency limit 108 | is reached, it will not start a new operation and instead queue this for future 109 | execution. Once one of the pending operations complete, it will pick the next 110 | job from the queue and execute this operation. 111 | 112 | The `new Queue(int $concurrency, ?int $limit, callable(mixed):PromiseInterface $handler)` call 113 | can be used to create a new queue instance. 114 | You can create any number of queues, for example when you want to apply 115 | different limits to different kinds of operations. 116 | 117 | The `$concurrency` parameter sets a new soft limit for the maximum number 118 | of jobs to handle concurrently. Finding a good concurrency limit depends 119 | on your particular use case. It's common to limit concurrency to a rather 120 | small value, as doing more than a dozen of things at once may easily 121 | overwhelm the receiving side. 122 | 123 | The `$limit` parameter sets a new hard limit on how many jobs may be 124 | outstanding (kept in memory) at once. Depending on your particular use 125 | case, it's usually safe to keep a few hundreds or thousands of jobs in 126 | memory. If you do not want to apply an upper limit, you can pass a `null` 127 | value which is semantically more meaningful than passing a big number. 128 | 129 | ```php 130 | // handle up to 10 jobs concurrently, but keep no more than 1000 in memory 131 | $q = new Queue(10, 1000, $handler); 132 | ``` 133 | 134 | ```php 135 | // handle up to 10 jobs concurrently, do not limit queue size 136 | $q = new Queue(10, null, $handler); 137 | ``` 138 | 139 | ```php 140 | // handle up to 10 jobs concurrently, reject all further jobs 141 | $q = new Queue(10, 10, $handler); 142 | ``` 143 | 144 | The `$handler` parameter must be a valid callable that accepts your job 145 | parameters, invokes the appropriate operation and returns a Promise as a 146 | placeholder for its future result. 147 | 148 | ```php 149 | // using a Closure as handler is usually recommended 150 | $q = new Queue(10, null, function ($url) use ($browser) { 151 | return $browser->get($url); 152 | }); 153 | ``` 154 | 155 | ```php 156 | // accepts any callable, so PHP's array notation is also supported 157 | $q = new Queue(10, null, array($browser, 'get')); 158 | ``` 159 | 160 | #### Promises 161 | 162 | This library works under the assumption that you want to concurrently handle 163 | async operations that use a [Promise](https://github.com/reactphp/promise)-based API. 164 | 165 | The demonstration purposes, the examples in this documentation use 166 | [ReactPHP's async HTTP client](https://github.com/reactphp/http#client-usage), but you 167 | may use any Promise-based API with this project. Its API can be used like this: 168 | 169 | ```php 170 | $browser = new React\Http\Browser(); 171 | 172 | $promise = $browser->get($url); 173 | ``` 174 | 175 | If you wrap this in a `Queue` instance as given above, this code will look 176 | like this: 177 | 178 | ```php 179 | $browser = new React\Http\Browser(); 180 | 181 | $q = new Queue(10, null, function ($url) use ($browser) { 182 | return $browser->get($url); 183 | }); 184 | 185 | $promise = $q($url); 186 | ``` 187 | 188 | The `$q` instance is invokable, so that invoking `$q(...$args)` will 189 | actually be forwarded as `$browser->get(...$args)` as given in the 190 | `$handler` argument when concurrency is still below limits. 191 | 192 | Each operation is expected to be async (non-blocking), so you may actually 193 | invoke multiple operations concurrently (send multiple requests in parallel). 194 | The `$handler` is responsible for responding to each request with a resolution 195 | value, the order is not guaranteed. 196 | These operations use a [Promise](https://github.com/reactphp/promise)-based 197 | interface that makes it easy to react to when an operation is completed (i.e. 198 | either successfully fulfilled or rejected with an error): 199 | 200 | ```php 201 | $promise->then( 202 | function ($result) { 203 | var_dump('Result received', $result); 204 | }, 205 | function (Exception $error) { 206 | var_dump('There was an error', $error->getMessage()); 207 | } 208 | ); 209 | ``` 210 | 211 | Each operation may take some time to complete, but due to its async nature you 212 | can actually start any number of (queued) operations. Once the concurrency limit 213 | is reached, this invocation will simply be queued and this will return a pending 214 | promise which will start the actual operation once another operation is 215 | completed. This means that this is handled entirely transparently and you do not 216 | need to worry about this concurrency limit yourself. 217 | 218 | If this looks strange to you, you can also use the more traditional 219 | [blocking API](#blocking). 220 | 221 | #### Cancellation 222 | 223 | The returned Promise is implemented in such a way that it can be cancelled 224 | when it is still pending. 225 | Cancelling a pending operation will invoke its cancellation handler which is 226 | responsible for rejecting its value with an Exception and cleaning up any 227 | underlying resources. 228 | 229 | ```php 230 | $promise = $q($url); 231 | 232 | Loop::addTimer(2.0, function () use ($promise) { 233 | $promise->cancel(); 234 | }); 235 | ``` 236 | 237 | Similarly, cancelling an operation that is queued and has not yet been started 238 | will be rejected without ever starting the operation. 239 | 240 | #### Timeout 241 | 242 | By default, this library does not limit how long a single operation can take, 243 | so that the resulting promise may stay pending for a long time. 244 | Many use cases involve some kind of "timeout" logic so that an operation is 245 | cancelled after a certain threshold is reached. 246 | 247 | You can simply use [cancellation](#cancellation) as in the previous chapter or 248 | you may want to look into using [react/promise-timer](https://github.com/reactphp/promise-timer) 249 | which helps taking care of this through a simple API. 250 | 251 | The resulting code with timeouts applied look something like this: 252 | 253 | ```php 254 | use React\Promise\Timer; 255 | 256 | $q = new Queue(10, null, function ($uri) use ($browser) { 257 | return Timer\timeout($browser->get($uri), 2.0); 258 | }); 259 | 260 | $promise = $q($uri); 261 | ``` 262 | 263 | The resulting promise can be consumed as usual and the above code will ensure 264 | that execution of this operation can not take longer than the given timeout 265 | (i.e. after it is actually started). 266 | In particular, note how this differs from applying a timeout to the resulting 267 | promise. The following code will ensure that the total time for queuing and 268 | executing this operation can not take longer than the given timeout: 269 | 270 | ```php 271 | // usually not recommended 272 | $promise = Timer\timeout($q($url), 2.0); 273 | ``` 274 | 275 | Please refer to [react/promise-timer](https://github.com/reactphp/promise-timer) 276 | for more details. 277 | 278 | #### all() 279 | 280 | The static `all(int $concurrency, array $jobs, callable(TIn):PromiseInterface $handler): PromiseInterface>` method can be used to 281 | concurrently process all given jobs through the given `$handler`. 282 | 283 | This is a convenience method which uses the `Queue` internally to 284 | schedule all jobs while limiting concurrency to ensure no more than 285 | `$concurrency` jobs ever run at once. It will return a promise which 286 | resolves with the results of all jobs on success. 287 | 288 | ```php 289 | $browser = new React\Http\Browser(); 290 | 291 | $promise = Queue::all(3, $urls, function ($url) use ($browser) { 292 | return $browser->get($url); 293 | }); 294 | 295 | $promise->then(function (array $responses) { 296 | echo 'All ' . count($responses) . ' successful!' . PHP_EOL; 297 | }, function (Exception $e) { 298 | echo 'Error: ' . $e->getMessage() . PHP_EOL; 299 | }); 300 | ``` 301 | 302 | If either of the jobs fail, it will reject the resulting promise and will 303 | try to cancel all outstanding jobs. Similarly, calling `cancel()` on the 304 | resulting promise will try to cancel all outstanding jobs. See 305 | [promises](#promises) and [cancellation](#cancellation) for details. 306 | 307 | The `$concurrency` parameter sets a new soft limit for the maximum number 308 | of jobs to handle concurrently. Finding a good concurrency limit depends 309 | on your particular use case. It's common to limit concurrency to a rather 310 | small value, as doing more than a dozen of things at once may easily 311 | overwhelm the receiving side. Using a `1` value will ensure that all jobs 312 | are processed one after another, effectively creating a "waterfall" of 313 | jobs. Using a value less than 1 will reject with an 314 | `InvalidArgumentException` without processing any jobs. 315 | 316 | ```php 317 | // handle up to 10 jobs concurrently 318 | $promise = Queue::all(10, $jobs, $handler); 319 | ``` 320 | 321 | ```php 322 | // handle each job after another without concurrency (waterfall) 323 | $promise = Queue::all(1, $jobs, $handler); 324 | ``` 325 | 326 | The `$jobs` parameter must be an array with all jobs to process. Each 327 | value in this array will be passed to the `$handler` to start one job. 328 | The array keys will be preserved in the resulting array, while the array 329 | values will be replaced with the job results as returned by the 330 | `$handler`. If this array is empty, this method will resolve with an 331 | empty array without processing any jobs. 332 | 333 | The `$handler` parameter must be a valid callable that accepts your job 334 | parameters, invokes the appropriate operation and returns a Promise as a 335 | placeholder for its future result. If the given argument is not a valid 336 | callable, this method will reject with an `InvalidArgumentException` 337 | without processing any jobs. 338 | 339 | ```php 340 | // using a Closure as handler is usually recommended 341 | $promise = Queue::all(10, $jobs, function ($url) use ($browser) { 342 | return $browser->get($url); 343 | }); 344 | ``` 345 | 346 | ```php 347 | // accepts any callable, so PHP's array notation is also supported 348 | $promise = Queue::all(10, $jobs, array($browser, 'get')); 349 | ``` 350 | 351 | > Keep in mind that returning an array of response messages means that 352 | the whole response body has to be kept in memory. 353 | 354 | #### any() 355 | 356 | The static `any(int $concurrency, array $jobs, callable(TIn):Promise $handler): PromiseInterface` method can be used to 357 | concurrently process the given jobs through the given `$handler` and 358 | resolve with first resolution value. 359 | 360 | This is a convenience method which uses the `Queue` internally to 361 | schedule all jobs while limiting concurrency to ensure no more than 362 | `$concurrency` jobs ever run at once. It will return a promise which 363 | resolves with the result of the first job on success and will then try 364 | to `cancel()` all outstanding jobs. 365 | 366 | ```php 367 | $browser = new React\Http\Browser(); 368 | 369 | $promise = Queue::any(3, $urls, function ($url) use ($browser) { 370 | return $browser->get($url); 371 | }); 372 | 373 | $promise->then(function (ResponseInterface $response) { 374 | echo 'First response: ' . $response->getBody() . PHP_EOL; 375 | }, function (Exception $e) { 376 | echo 'Error: ' . $e->getMessage() . PHP_EOL; 377 | }); 378 | ``` 379 | 380 | If all of the jobs fail, it will reject the resulting promise. Similarly, 381 | calling `cancel()` on the resulting promise will try to cancel all 382 | outstanding jobs. See [promises](#promises) and 383 | [cancellation](#cancellation) for details. 384 | 385 | The `$concurrency` parameter sets a new soft limit for the maximum number 386 | of jobs to handle concurrently. Finding a good concurrency limit depends 387 | on your particular use case. It's common to limit concurrency to a rather 388 | small value, as doing more than a dozen of things at once may easily 389 | overwhelm the receiving side. Using a `1` value will ensure that all jobs 390 | are processed one after another, effectively creating a "waterfall" of 391 | jobs. Using a value less than 1 will reject with an 392 | `InvalidArgumentException` without processing any jobs. 393 | 394 | ```php 395 | // handle up to 10 jobs concurrently 396 | $promise = Queue::any(10, $jobs, $handler); 397 | ``` 398 | 399 | ```php 400 | // handle each job after another without concurrency (waterfall) 401 | $promise = Queue::any(1, $jobs, $handler); 402 | ``` 403 | 404 | The `$jobs` parameter must be an array with all jobs to process. Each 405 | value in this array will be passed to the `$handler` to start one job. 406 | The array keys have no effect, the promise will simply resolve with the 407 | job results of the first successful job as returned by the `$handler`. 408 | If this array is empty, this method will reject without processing any 409 | jobs. 410 | 411 | The `$handler` parameter must be a valid callable that accepts your job 412 | parameters, invokes the appropriate operation and returns a Promise as a 413 | placeholder for its future result. If the given argument is not a valid 414 | callable, this method will reject with an `InvalidArgumentExceptionn` 415 | without processing any jobs. 416 | 417 | ```php 418 | // using a Closure as handler is usually recommended 419 | $promise = Queue::any(10, $jobs, function ($url) use ($browser) { 420 | return $browser->get($url); 421 | }); 422 | ``` 423 | 424 | ```php 425 | // accepts any callable, so PHP's array notation is also supported 426 | $promise = Queue::any(10, $jobs, array($browser, 'get')); 427 | ``` 428 | 429 | #### Blocking 430 | 431 | As stated above, this library provides you a powerful, async API by default. 432 | 433 | You can also integrate this into your traditional, blocking environment by using 434 | [reactphp/async](https://github.com/reactphp/async). This allows you to simply 435 | await async HTTP requests like this: 436 | 437 | ```php 438 | use function React\Async\await; 439 | 440 | $browser = new React\Http\Browser(); 441 | 442 | $promise = Queue::all(3, $urls, function ($url) use ($browser) { 443 | return $browser->get($url); 444 | }); 445 | 446 | try { 447 | $responses = await($promise); 448 | // responses successfully received 449 | } catch (Exception $e) { 450 | // an error occurred while performing the requests 451 | } 452 | ``` 453 | 454 | Similarly, you can also wrap this in a function to provide a simple API and hide 455 | all the async details from the outside: 456 | 457 | ```php 458 | use function React\Async\await; 459 | 460 | /** 461 | * Concurrently downloads all the given URIs 462 | * 463 | * @param string[] $uris list of URIs to download 464 | * @return ResponseInterface[] map with a response object for each URI 465 | * @throws Exception if any of the URIs can not be downloaded 466 | */ 467 | function download(array $uris) 468 | { 469 | $browser = new React\Http\Browser(); 470 | 471 | $promise = Queue::all(3, $uris, function ($uri) use ($browser) { 472 | return $browser->get($uri); 473 | }); 474 | 475 | return await($promise); 476 | } 477 | ``` 478 | 479 | This is made possible thanks to fibers available in PHP 8.1+ and our 480 | compatibility API that also works on all supported PHP versions. 481 | Please refer to [reactphp/async](https://github.com/reactphp/async#readme) for more details. 482 | 483 | > Keep in mind that returning an array of response messages means that the whole 484 | response body has to be kept in memory. 485 | 486 | ## Install 487 | 488 | The recommended way to install this library is [through Composer](https://getcomposer.org/). 489 | [New to Composer?](https://getcomposer.org/doc/00-intro.md) 490 | 491 | This project follows [SemVer](https://semver.org/). 492 | This will install the latest supported version: 493 | 494 | ```bash 495 | composer require clue/mq-react:^1.7 496 | ``` 497 | 498 | See also the [CHANGELOG](CHANGELOG.md) for details about version upgrades. 499 | 500 | This project aims to run on any platform and thus does not require any PHP 501 | extensions and supports running on legacy PHP 5.3 through current PHP 8+. 502 | It's *highly recommended to use the latest supported PHP version* for this project. 503 | 504 | ## Tests 505 | 506 | To run the test suite, you first need to clone this repo and then install all 507 | dependencies [through Composer](https://getcomposer.org/): 508 | 509 | ```bash 510 | composer install 511 | ``` 512 | 513 | To run the test suite, go to the project root and run: 514 | 515 | ```bash 516 | vendor/bin/phpunit 517 | ``` 518 | 519 | The test suite is set up to always ensure 100% code coverage across all 520 | supported environments. If you have the Xdebug extension installed, you can also 521 | generate a code coverage report locally like this: 522 | 523 | ```bash 524 | XDEBUG_MODE=coverage vendor/bin/phpunit --coverage-text 525 | ``` 526 | 527 | ## License 528 | 529 | This project is released under the permissive [MIT license](LICENSE). 530 | 531 | I'd like to thank [Bergfreunde GmbH](https://www.bergfreunde.de/), a German 532 | online retailer for Outdoor Gear & Clothing, for sponsoring the first release! 🎉 533 | Thanks to sponsors like this, who understand the importance of open source 534 | development, I can justify spending time and focus on open source development 535 | instead of traditional paid work. 536 | 537 | > Did you know that I offer custom development services and issuing invoices for 538 | sponsorships of releases and for contributions? Contact me (@clue) for details. 539 | -------------------------------------------------------------------------------- /composer.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "clue/mq-react", 3 | "description": "Mini Queue, the lightweight in-memory message queue to concurrently do many (but not too many) things at once, built on top of ReactPHP", 4 | "keywords": ["Message Queue", "Mini Queue", "job", "message", "worker", "queue", "rate limit", "throttle", "concurrency", "ReactPHP", "async"], 5 | "homepage": "https://github.com/clue/reactphp-mq", 6 | "license": "MIT", 7 | "authors": [ 8 | { 9 | "name": "Christian Lück", 10 | "email": "christian@clue.engineering" 11 | } 12 | ], 13 | "require": { 14 | "php": ">=5.3", 15 | "react/promise": "^3 || ^2.2.1 || ^1.2.1" 16 | }, 17 | "require-dev": { 18 | "phpunit/phpunit": "^9.6 || ^5.7 || ^4.8.36", 19 | "react/async": "^4 || ^3 || ^2", 20 | "react/event-loop": "^1.2", 21 | "react/http": "^1.8" 22 | }, 23 | "autoload": { 24 | "psr-4": { 25 | "Clue\\React\\Mq\\": "src/" 26 | } 27 | }, 28 | "autoload-dev": { 29 | "psr-4": { 30 | "Clue\\Tests\\React\\Mq\\": "tests/" 31 | } 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /src/Queue.php: -------------------------------------------------------------------------------- 1 | */ 31 | private $pending = 0; 32 | 33 | /** @var array */ 34 | private $queue = array(); 35 | 36 | /** 37 | * Concurrently process all given jobs through the given `$handler`. 38 | * 39 | * This is a convenience method which uses the `Queue` internally to 40 | * schedule all jobs while limiting concurrency to ensure no more than 41 | * `$concurrency` jobs ever run at once. It will return a promise which 42 | * resolves with the results of all jobs on success. 43 | * 44 | * ```php 45 | * $browser = new React\Http\Browser(); 46 | * 47 | * $promise = Queue::all(3, $urls, function ($url) use ($browser) { 48 | * return $browser->get($url); 49 | * }); 50 | * 51 | * $promise->then(function (array $responses) { 52 | * echo 'All ' . count($responses) . ' successful!' . PHP_EOL; 53 | * }); 54 | * ``` 55 | * 56 | * If either of the jobs fail, it will reject the resulting promise and will 57 | * try to cancel all outstanding jobs. Similarly, calling `cancel()` on the 58 | * resulting promise will try to cancel all outstanding jobs. See 59 | * [promises](#promises) and [cancellation](#cancellation) for details. 60 | * 61 | * The `$concurrency` parameter sets a new soft limit for the maximum number 62 | * of jobs to handle concurrently. Finding a good concurrency limit depends 63 | * on your particular use case. It's common to limit concurrency to a rather 64 | * small value, as doing more than a dozen of things at once may easily 65 | * overwhelm the receiving side. Using a `1` value will ensure that all jobs 66 | * are processed one after another, effectively creating a "waterfall" of 67 | * jobs. Using a value less than 1 will reject with an 68 | * `InvalidArgumentException` without processing any jobs. 69 | * 70 | * ```php 71 | * // handle up to 10 jobs concurrently 72 | * $promise = Queue::all(10, $jobs, $handler); 73 | * ``` 74 | * 75 | * ```php 76 | * // handle each job after another without concurrency (waterfall) 77 | * $promise = Queue::all(1, $jobs, $handler); 78 | * ``` 79 | * 80 | * The `$jobs` parameter must be an array with all jobs to process. Each 81 | * value in this array will be passed to the `$handler` to start one job. 82 | * The array keys will be preserved in the resulting array, while the array 83 | * values will be replaced with the job results as returned by the 84 | * `$handler`. If this array is empty, this method will resolve with an 85 | * empty array without processing any jobs. 86 | * 87 | * The `$handler` parameter must be a valid callable that accepts your job 88 | * parameters, invokes the appropriate operation and returns a Promise as a 89 | * placeholder for its future result. If the given argument is not a valid 90 | * callable, this method will reject with an `InvalidArgumentException` 91 | * without processing any jobs. 92 | * 93 | * ```php 94 | * // using a Closure as handler is usually recommended 95 | * $promise = Queue::all(10, $jobs, function ($url) use ($browser) { 96 | * return $browser->get($url); 97 | * }); 98 | * ``` 99 | * 100 | * ```php 101 | * // accepts any callable, so PHP's array notation is also supported 102 | * $promise = Queue::all(10, $jobs, array($browser, 'get')); 103 | * ``` 104 | * 105 | * > Keep in mind that returning an array of response messages means that 106 | * the whole response body has to be kept in memory. 107 | * 108 | * @template TKey 109 | * @template TIn 110 | * @template TOut 111 | * @param int $concurrency concurrency soft limit 112 | * @param array $jobs 113 | * @param callable(TIn):PromiseInterface $handler 114 | * @return PromiseInterface> Returns a Promise which resolves with an array of all resolution values 115 | * or rejects when any of the operations reject. 116 | */ 117 | public static function all($concurrency, array $jobs, $handler) 118 | { 119 | try { 120 | // limit number of concurrent operations 121 | $q = new self($concurrency, null, $handler); 122 | } catch (\InvalidArgumentException $e) { 123 | // reject if $concurrency or $handler is invalid 124 | return Promise\reject($e); 125 | } 126 | 127 | // try invoking all operations and automatically queue excessive ones 128 | $promises = array_map($q, $jobs); 129 | 130 | return new Promise\Promise(function ($resolve, $reject) use ($promises) { 131 | Promise\all($promises)->then($resolve, function ($e) use ($promises, $reject) { 132 | // cancel all pending promises if a single promise fails 133 | foreach (array_reverse($promises) as $promise) { 134 | if ($promise instanceof PromiseInterface && \method_exists($promise, 'cancel')) { 135 | $promise->cancel(); 136 | } 137 | } 138 | 139 | // reject with original rejection message 140 | $reject($e); 141 | }); 142 | }, function () use ($promises) { 143 | // cancel all pending promises on cancellation 144 | foreach (array_reverse($promises) as $promise) { 145 | if ($promise instanceof PromiseInterface && \method_exists($promise, 'cancel')) { 146 | $promise->cancel(); 147 | } 148 | } 149 | }); 150 | } 151 | 152 | /** 153 | * Concurrently process the given jobs through the given `$handler` and 154 | * resolve with first resolution value. 155 | * 156 | * This is a convenience method which uses the `Queue` internally to 157 | * schedule all jobs while limiting concurrency to ensure no more than 158 | * `$concurrency` jobs ever run at once. It will return a promise which 159 | * resolves with the result of the first job on success and will then try 160 | * to `cancel()` all outstanding jobs. 161 | * 162 | * ```php 163 | * $browser = new React\Http\Browser(); 164 | * 165 | * $promise = Queue::any(3, $urls, function ($url) use ($browser) { 166 | * return $browser->get($url); 167 | * }); 168 | * 169 | * $promise->then(function (ResponseInterface $response) { 170 | * echo 'First response: ' . $response->getBody() . PHP_EOL; 171 | * }); 172 | * ``` 173 | * 174 | * If all of the jobs fail, it will reject the resulting promise. Similarly, 175 | * calling `cancel()` on the resulting promise will try to cancel all 176 | * outstanding jobs. See [promises](#promises) and 177 | * [cancellation](#cancellation) for details. 178 | * 179 | * The `$concurrency` parameter sets a new soft limit for the maximum number 180 | * of jobs to handle concurrently. Finding a good concurrency limit depends 181 | * on your particular use case. It's common to limit concurrency to a rather 182 | * small value, as doing more than a dozen of things at once may easily 183 | * overwhelm the receiving side. Using a `1` value will ensure that all jobs 184 | * are processed one after another, effectively creating a "waterfall" of 185 | * jobs. Using a value less than 1 will reject with an 186 | * `InvalidArgumentException` without processing any jobs. 187 | * 188 | * ```php 189 | * // handle up to 10 jobs concurrently 190 | * $promise = Queue::any(10, $jobs, $handler); 191 | * ``` 192 | * 193 | * ```php 194 | * // handle each job after another without concurrency (waterfall) 195 | * $promise = Queue::any(1, $jobs, $handler); 196 | * ``` 197 | * 198 | * The `$jobs` parameter must be an array with all jobs to process. Each 199 | * value in this array will be passed to the `$handler` to start one job. 200 | * The array keys have no effect, the promise will simply resolve with the 201 | * job results of the first successful job as returned by the `$handler`. 202 | * If this array is empty, this method will reject without processing any 203 | * jobs. 204 | * 205 | * The `$handler` parameter must be a valid callable that accepts your job 206 | * parameters, invokes the appropriate operation and returns a Promise as a 207 | * placeholder for its future result. If the given argument is not a valid 208 | * callable, this method will reject with an `InvalidArgumentExceptionn` 209 | * without processing any jobs. 210 | * 211 | * ```php 212 | * // using a Closure as handler is usually recommended 213 | * $promise = Queue::any(10, $jobs, function ($url) use ($browser) { 214 | * return $browser->get($url); 215 | * }); 216 | * ``` 217 | * 218 | * ```php 219 | * // accepts any callable, so PHP's array notation is also supported 220 | * $promise = Queue::any(10, $jobs, array($browser, 'get')); 221 | * ``` 222 | * 223 | * @template TKey 224 | * @template TIn 225 | * @template TOut 226 | * @param int $concurrency concurrency soft limit 227 | * @param array $jobs 228 | * @param callable(TIn):PromiseInterface $handler 229 | * @return PromiseInterface Returns a Promise which resolves with a single resolution value 230 | * or rejects when all of the operations reject. 231 | */ 232 | public static function any($concurrency, array $jobs, $handler) 233 | { 234 | // explicitly reject with empty jobs (https://github.com/reactphp/promise/pull/34) 235 | if (!$jobs) { 236 | return Promise\reject(new \UnderflowException('No jobs given')); 237 | } 238 | 239 | try { 240 | // limit number of concurrent operations 241 | $q = new self($concurrency, null, $handler); 242 | } catch (\InvalidArgumentException $e) { 243 | // reject if $concurrency or $handler is invalid 244 | return Promise\reject($e); 245 | } 246 | 247 | // try invoking all operations and automatically queue excessive ones 248 | $promises = array_map($q, $jobs); 249 | 250 | return new Promise\Promise(function ($resolve, $reject) use ($promises) { 251 | Promise\any($promises)->then(function ($result) use ($promises, $resolve) { 252 | // cancel all pending promises if a single result is ready 253 | foreach (array_reverse($promises) as $promise) { 254 | if ($promise instanceof PromiseInterface && \method_exists($promise, 'cancel')) { 255 | $promise->cancel(); 256 | } 257 | } 258 | 259 | // resolve with original resolution value 260 | $resolve($result); 261 | }, $reject); 262 | }, function () use ($promises) { 263 | // cancel all pending promises on cancellation 264 | foreach (array_reverse($promises) as $promise) { 265 | if ($promise instanceof PromiseInterface && \method_exists($promise, 'cancel')) { 266 | $promise->cancel(); 267 | } 268 | } 269 | }); 270 | } 271 | 272 | /** 273 | * Instantiates a new queue object. 274 | * 275 | * You can create any number of queues, for example when you want to apply 276 | * different limits to different kind of operations. 277 | * 278 | * The `$concurrency` parameter sets a new soft limit for the maximum number 279 | * of jobs to handle concurrently. Finding a good concurrency limit depends 280 | * on your particular use case. It's common to limit concurrency to a rather 281 | * small value, as doing more than a dozen of things at once may easily 282 | * overwhelm the receiving side. 283 | * 284 | * The `$limit` parameter sets a new hard limit on how many jobs may be 285 | * outstanding (kept in memory) at once. Depending on your particular use 286 | * case, it's usually safe to keep a few hundreds or thousands of jobs in 287 | * memory. If you do not want to apply an upper limit, you can pass a `null` 288 | * value which is semantically more meaningful than passing a big number. 289 | * 290 | * ```php 291 | * // handle up to 10 jobs concurrently, but keep no more than 1000 in memory 292 | * $q = new Queue(10, 1000, $handler); 293 | * ``` 294 | * 295 | * ```php 296 | * // handle up to 10 jobs concurrently, do not limit queue size 297 | * $q = new Queue(10, null, $handler); 298 | * ``` 299 | * 300 | * ```php 301 | * // handle up to 10 jobs concurrently, reject all further jobs 302 | * $q = new Queue(10, 10, $handler); 303 | * ``` 304 | * 305 | * The `$handler` parameter must be a valid callable that accepts your job 306 | * parameters, invokes the appropriate operation and returns a Promise as a 307 | * placeholder for its future result. 308 | * 309 | * ```php 310 | * // using a Closure as handler is usually recommended 311 | * $q = new Queue(10, null, function ($url) use ($browser) { 312 | * return $browser->get($url); 313 | * }); 314 | * ``` 315 | * 316 | * ```php 317 | * // PHP's array callable as handler is also supported 318 | * $q = new Queue(10, null, array($browser, 'get')); 319 | * ``` 320 | * 321 | * @template A1 (any number of function arguments, see https://github.com/phpstan/phpstan/issues/8214) 322 | * @template A2 323 | * @template A3 324 | * @template A4 325 | * @template A5 326 | * @param int $concurrency concurrency soft limit 327 | * @param int|null $limit queue hard limit or NULL=unlimited 328 | * @param callable(A1,A2,A3,A4,A5):PromiseInterface $handler 329 | * @throws \InvalidArgumentException 330 | */ 331 | public function __construct($concurrency, $limit, $handler) 332 | { 333 | if ($concurrency < 1 || ($limit !== null && ($limit < 1 || $concurrency > $limit))) { 334 | throw new \InvalidArgumentException('Invalid limit given'); 335 | } 336 | if (!is_callable($handler)) { 337 | throw new \InvalidArgumentException('Invalid handler given'); 338 | } 339 | 340 | $this->concurrency = $concurrency; 341 | $this->limit = $limit; 342 | $this->handler = $handler; 343 | } 344 | 345 | /** 346 | * The Queue instance is invokable, so that invoking `$q(...$args)` will 347 | * actually be forwarded as `$handler(...$args)` as given in the 348 | * `$handler` argument when concurrency is still below limits. 349 | * 350 | * Each operation may take some time to complete, but due to its async nature you 351 | * can actually start any number of (queued) operations. Once the concurrency limit 352 | * is reached, this invocation will simply be queued and this will return a pending 353 | * promise which will start the actual operation once another operation is 354 | * completed. This means that this is handled entirely transparently and you do not 355 | * need to worry about this concurrency limit yourself. 356 | * 357 | * @return PromiseInterface 358 | */ 359 | public function __invoke() 360 | { 361 | // happy path: simply invoke handler if we're below concurrency limit 362 | if ($this->pending < $this->concurrency) { 363 | ++$this->pending; 364 | 365 | // invoke handler and await its resolution before invoking next queued job 366 | return $this->await( 367 | call_user_func_array($this->handler, func_get_args()) 368 | ); 369 | } 370 | 371 | // we're currently above concurrency limit, make sure we do not exceed maximum queue limit 372 | if ($this->limit !== null && $this->count() >= $this->limit) { 373 | return Promise\reject(new \OverflowException('Maximum queue limit of ' . $this->limit . ' exceeded')); 374 | } 375 | 376 | // if we reach this point, then this job will need to be queued 377 | // get next queue position 378 | $queue =& $this->queue; 379 | $queue[] = null; 380 | end($queue); 381 | $id = key($queue); 382 | assert(is_int($id)); 383 | 384 | /** @var ?PromiseInterface $pending */ 385 | $pending = null; 386 | 387 | $deferred = new Deferred(function ($_, $reject) use (&$queue, $id, &$pending) { 388 | // forward cancellation to pending operation if it is currently executing 389 | if ($pending instanceof PromiseInterface && \method_exists($pending, 'cancel')) { 390 | $pending->cancel(); 391 | } 392 | $pending = null; 393 | 394 | if (isset($queue[$id])) { 395 | // queued promise cancelled before its handler is invoked 396 | // remove from queue and reject explicitly 397 | unset($queue[$id]); 398 | $reject(new \RuntimeException('Cancelled queued job before processing started')); 399 | } 400 | }); 401 | 402 | // queue job to process if number of pending jobs is below concurrency limit again 403 | $handler = $this->handler; // PHP 5.4+ 404 | $args = func_get_args(); 405 | $that = $this; // PHP 5.4+ 406 | $queue[$id] = function () use ($handler, $args, $deferred, &$pending, $that) { 407 | $pending = \call_user_func_array($handler, $args); 408 | 409 | $that->await($pending)->then( 410 | function ($result) use ($deferred, &$pending) { 411 | $pending = null; 412 | $deferred->resolve($result); 413 | }, 414 | function ($e) use ($deferred, &$pending) { 415 | $pending = null; 416 | $deferred->reject($e); 417 | } 418 | ); 419 | }; 420 | 421 | return $deferred->promise(); 422 | } 423 | 424 | #[\ReturnTypeWillChange] 425 | public function count() 426 | { 427 | return $this->pending + count($this->queue); 428 | } 429 | 430 | /** 431 | * @internal 432 | * @param PromiseInterface $promise 433 | */ 434 | public function await(PromiseInterface $promise) 435 | { 436 | $that = $this; // PHP 5.4+ 437 | 438 | return $promise->then(function ($result) use ($that) { 439 | $that->processQueue(); 440 | 441 | return $result; 442 | }, function ($error) use ($that) { 443 | $that->processQueue(); 444 | 445 | return Promise\reject($error); 446 | }); 447 | } 448 | 449 | /** 450 | * @internal 451 | */ 452 | public function processQueue() 453 | { 454 | // skip if we're still above concurrency limit or there's no queued job waiting 455 | if (--$this->pending >= $this->concurrency || !$this->queue) { 456 | return; 457 | } 458 | 459 | $next = reset($this->queue); 460 | assert($next instanceof \Closure); 461 | unset($this->queue[key($this->queue)]); 462 | 463 | // once number of pending jobs is below concurrency limit again: 464 | // await this situation, invoke handler and await its resolution before invoking next queued job 465 | ++$this->pending; 466 | 467 | // invoke handler and await its resolution before invoking next queued job 468 | $next(); 469 | } 470 | } 471 | --------------------------------------------------------------------------------