├── LICENSE ├── README.md ├── bin ├── queueman_arm_untest ├── queueman_freebsd_untest ├── queueman_linux └── queueman_macos ├── build.sh ├── go.mod ├── go.sum ├── libs ├── aliyun │ └── aliyun.go ├── command │ └── command.go ├── config │ └── config.go ├── constant │ └── constant.go ├── ohttp │ └── ohttp.go ├── pidfile │ ├── pidfile.go │ ├── pidfile_darwin.go │ ├── pidfile_test.go │ ├── pidfile_unix.go │ └── pidfile_windows.go ├── queue │ ├── queue.go │ ├── rabbitmq │ │ ├── connection.go │ │ ├── dispatcher.go │ │ └── types.go │ ├── redis │ │ ├── connection.go │ │ ├── dispatcher.go │ │ └── types.go │ └── types │ │ └── types.go ├── request │ └── request.go ├── statistic │ ├── redis.go │ └── statistic.go └── utils │ └── utils.go ├── main.go ├── queueman.json ├── queueman.service └── supervisor.conf /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2019-2020 Marknown 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Queueman 2 | 3 | Queueman 是一个适用于 RabbitMQ、Redis 队列的高性能分发中间件。支持延时队列、并发控制、失败自动重试。 4 | 5 | 1. 简单的并发控制 6 | 2. 简单配置就可以自动失败后重试 7 | 3. 不用再写命令行代码就可以消费队列了 8 | 9 | **测试理论速度：单机 1-3 万条/秒** 10 | 11 | **配置用例1:** 12 | - 服务器: 阿里云 ecs.sn1ne.large 2 vCPU 4 GiB (I/O优化) CentOS 7.2 64位 13 | - Redis: 阿里云 Redis 4.0 1G集群版 最大连接数： 20,000 14 | - RabbitMQ: 阿里云 消息队列 AMQP 协议版本：0-9-1 TPS峰值流量：1000 15 | - 网络：阿里云内网连接 16 | 17 | 队列类型 | 数据量 | 处理时间（秒） | 处理速度 | 备注 18 | ------------ | ------------- | ------------- | ------------- | ------------- 19 | RabbitMQ | 1000000 | 76.785453 | 13023/s | 阿里云内网连接、开启 auto ack、不开启 DispatchURL 转发 20 | Redis | 1000000 | 31.523070 | 31722/s | 阿里云内网连接、不开启 DispatchURL 转发 21 | 同时跑 RabbitMQ & Redis 各一个队列| 各测试1000000 | 126.724109 49.977177 | 11318/s | 内网、开启 auto ack、不开启 DispatchURL 转发 22 | 23 | 24 | **配置用例2:** 25 | - 服务器: MacBook Pro (Retina, 15-inch, Mid 2015) 2.2 GHz 四核Intel Core i7 macOS Catalina 10.15.3 26 | - Redis: 阿里云 Redis 4.0 1G集群版 最大连接数： 20,000 27 | - RabbitMQ: 阿里云 消息队列 AMQP 协议版本：0-9-1 TPS峰值流量：1000 28 | - 网络：公网连接 29 | 30 | 队列类型 | 数据量 | 处理时间（秒） | 处理速度 | 备注 31 | ------------ | ------------- | ------------- | ------------- | ------------- 32 | 本机 | RabbitMQ | 1000000 | 79.617749254 | 12560/s | 公网连接、开启 auto ack、不开启 DispatchURL 转发 33 | 本机 | Redis | 1000000 | 78.276917 | 12775/s | 公网连接、不开启 DispatchURL 转发 34 | 本机 | 同时跑 RabbitMQ & Redis 各一个队列| 各测试1000000 | 97.412025 89.785312 | 10752/s | 公网连接、开启 auto ack、不开启 DispatchURL 转发 35 | 36 | ## 内容列表 37 | 38 | - [背景](#背景) 39 | - [如何安装](#如何安装) 40 | - [克隆到本地并预处理](#克隆到本地并预处理) 41 | - [各种环境运行](#各种环境运行) 42 | - [详细介绍](#详细介绍) 43 | - [完整流程](#完整流程) 44 | - [开发者做两件事就够了](#开发者做两件事就够了) 45 | - [开发者推送数据到队列](#开发者推送数据到队列) 46 | - [指定地址接收数据](#指定地址接收数据) 47 | - [指定地址的响应](#指定地址的响应) 48 | - [并发控制](#并发控制) 49 | - [失败后处理](#失败后处理) 50 | - [命令行工具](#命令行工具) 51 | - [查看统计信息](#查看统计信息) 52 | - [配置文件详解](#配置文件详解) 53 | - [维护者](#维护者) 54 | - [如何贡献](#如何贡献) 55 | - [使用许可](#使用许可) 56 | 57 | ## 背景 58 | 59 | 1. 队列越来越多，消费脚本也越来越多，通过多进程来消费队列，开销也比较大。 60 | 2. 程序员既要写服务端代码，也要写命令行代码，还要对命令行代码进行部署，容易出错。 61 | 3. 正常业务要延时处理，有没有比较简单的方式来实现自动延时，不用写正常业务代码，还要写延时业务代码。 62 | 63 | 于是，是否可以有一种有新的轻量模式来取代这种传统模式，让开发人员更关注实现业务本身？让开发人员方便快捷的完成如下流程： 64 | 65 | 1. 开发人员写 web 代码 push 数据到队列 66 | 2. 队列中间件取出数据，转发到指定 URL 地址 67 | 3. 开发人员写 web 代码接收并处理 68 | 69 | ## 如何安装 70 | 71 | ### 克隆到本地并预处理 72 | ```sh 73 | # git clone https://github.com/marknown/queueman.git 74 | # cd queueman 75 | # # 配置你的 queueman.json 文件 # 76 | # cp bin/queueman_linux /usr/local/bin/queueman_linux 77 | # cp queueman.json /etc/queueman.json 78 | # mkdir -p /var/log/queueman 79 | ``` 80 | 81 | ### 各种环境运行 82 | 83 | #### Linux 84 | ``` 85 | # /usr/bin/nohup /usr/local/bin/queueman_linux -c /etc/queueman.json > /var/log/queueman/error.log 2>&1 & 86 | ``` 87 | 88 | #### MacOS 89 | 90 | ``` sh 91 | # sudo ./bin/queueman_macos -c ./queueman.json & 92 | ``` 93 | 94 | #### Systemctl 95 | ```sh 96 | # cp queueman.service /usr/lib/systemd/system 97 | # systemctl enable queueman.service 98 | # systemctl start queueman.service 99 | ``` 100 | 101 | #### Supervisor 102 | 复制文件到 Supervisor 配制目录 103 | ``` 104 | cp supervisor.conf /etc/supervisor.d/queueman.conf 105 | ``` 106 | 107 | 配置文件示例 108 | ``` 109 | [program:queueman] 110 | process_name=%(program_name)s_%(process_num)02d 111 | directory = /usr/local/bin/queueman_linux 112 | command=/usr/local/bin/queueman_linux -c /etc/queueman.json 113 | autostart=true 114 | autorestart=true 115 | user=root 116 | numprocs=1 117 | redirect_stderr=true 118 | stdout_logfile_maxbytes = 100MB 119 | stdout_logfile_backups = 20 120 | stdout_logfile=/var/log/queueman/error.log 121 | ``` 122 | 123 | 1. 把 Supervisor 配置文件放入 supervisor.d 目录 124 | 2. 使用如下命令启动 125 | ``` 126 | supervisorctl update 127 | ``` 128 | 3. 状态查看 129 | ``` 130 | supervisorctl status queueman: 131 | ``` 132 | 1. 启动与停止 133 | ``` 134 | supervisorctl start queueman: 135 | supervisorctl stop queueman: 136 | ``` 137 | 138 | ## 详细介绍 139 | 140 | ### 完整流程 141 | 1. 开发者推送数据到指定队列 142 | 2. Queuman根据配置取出数据 143 | 3. 通过 http(s) 分发至指定 URL 地址 144 | 4. 开发者在指定 URL 地址进行业务处理，并返回成功或者失败信息 145 | 5. 开发者返回成功（流程结束） 146 | 6. 开发者返回失败，Queueman会接收到失败响应，然后推送数据到失败延时重试队列 147 | 7. 失败延时重试队列到指定时间点，继续走 2 到 6 步 148 | 149 | ### 开发者做两件事就够了 150 | 1. 推送数据到指定队列 151 | 2. 在指定 URL 地址写业务处理代码，返回成功或者失败 152 | 153 | ### 开发者推送数据到队列 154 | #### Redis 正常队列 155 | ``` 156 | LPUSH queue:test1 value1 157 | ``` 158 | 159 | #### Redis 延时队列 160 | 使用 zset 实现，要使用特定格式并 json encode 成字符串 161 | ``` 162 | ZADD queue:test2 NX 指定触发的Unix时间戳 {"uuid":"cada1c8d-503e-4a82-b463-2b7ce2d2816e","time":1585817621,"data":"redis delay data"} 163 | 164 | 示例 165 | ZADD queue:test2 NX 1585817681 "{\"uuid\":\"cada1c8d-503e-4a82-b463-2b7ce2d2816e\",\"time\":1585817621,\"data\":\"redis delay data\"}" 166 | ``` 167 | 168 | Redis 标准 DelayData 格式 169 | 名称 | 说明 | 示例 170 | ------------ | ------------- | ------------- 171 | uuid | 唯一id(zset消息不允许重复) | "cada1c8d-503e-4a82-b463-2b7ce2d2816e" 172 | time | 生成当前信息的 Unix 时间戳 | 1585817621 173 | data | 实际要处理的值，指定 URL 只会收到这个内容，不含 uuid、time | "redis delay data" 174 | 175 | #### RabbitMQ 正常队列 176 | ``` 177 | ch.Publish( 178 | exchangeName, // exchange 179 | routingKey, // routing key 180 | false, // mandatory 181 | false, // immediate 182 | amqp.Publishing{ 183 | ContentType: "text/plain", 184 | Body: []byte(body), 185 | DeliveryMode: deliveryMode, 186 | }) 187 | ``` 188 | 189 | #### RabbitMQ 延时队列 190 | ``` 191 | header := amqp.Table{"x-delay": 10 * i * 1000} 192 | if "aliyun" == strings.ToLower(sourceType) { 193 | header = amqp.Table{"delay": 10 * i * 1000} 194 | } 195 | 196 | ch.Publish( 197 | exchangeName, // exchange 198 | routingKey, // routing key 199 | false, // mandatory 200 | false, // immediate 201 | amqp.Publishing{ 202 | Headers: header, 203 | ContentType: "text/plain", 204 | Body: []byte(body), 205 | DeliveryMode: deliveryMode, 206 | }) 207 | ``` 208 | 209 | ### 指定地址接收数据 210 | 211 | Queueman 会使用 `POST` 方式把数据以 `application/x-www-form-urlencoded` 格式推送到 `queueman.json` 里每个队列指定的 `DispatchURL` 212 | 213 | 名称 | 说明 | 示例 214 | ------------ | ------------- | ------------- 215 | queueName | 队列名称 | queue:test 216 | delayName | 正常队列失败后的延时处理队列名称 | queue:test:delay 217 | queueData | 取出的队列数据 | {"name":"jhon", "age":"16"} 218 | 219 | `UserAgent` 是 `QueueMan 版本号` 示例 `QueueMan V1.0.0` 注意版本号会升级 220 | 221 | ### 指定地址的响应 222 | 名称 | 说明 | 示例 223 | ------------ | ------------- | ------------- 224 | code | if success `1` else `0` | 1 225 | message | if success `ok` else `fail` | ok 226 | 227 | #### 成功请返回如下格式 228 | ```json 229 | {"code":1,"message":"ok"} 230 | ``` 231 | 232 | #### 失败请返回如下格式 233 | ```json 234 | {"code":0,"message":"fail"} 235 | ``` 236 | 237 | ### 并发控制 238 | `Concurency` Queueman转发并发数，请根据服务器性能来设置 239 | `DelayConcurency` 失败后的延时队列转发并发数，请根据服务器性能来设置 240 | 241 | ### 失败后处理 242 | 如果第一次处理失败，如有 DelayOnFailure 设置且长度大于0，则按配置依次重试，直到成功或者最后一次尝试失败。最后一次尝试失败后，丢弃队列数据 243 | ``` 244 | "DelayOnFailure": [60, 300] 245 | ``` 246 | 如上配置表示第一次失败后，投递消息到失败延时处理队列，60秒（一分钟）后触发，如果再一次失败则`300-60=240`秒后触发（即第五分钟），再次失败则丢弃。 247 | 248 | ## 命令行工具 249 | `-c` 指定配置文件路径 250 | ``` 251 | ./queueman_linux -c ./queueman.json 252 | ``` 253 | 254 | `-t` 测试指定的配置文件是否正常（可以结合 `-c`一起使用） 255 | ``` 256 | ./queueman_linux -c ./queueman.json -t 257 | ``` 258 | 259 | `-s` 在命令行下显示统计信息（可以结合 `-c`一起使用） 260 | ``` 261 | ./queueman_linux -c ./queueman.json -s 262 | ``` 263 | 264 | `-h` 查看帮助信息 265 | ``` 266 | ./queueman_linux -h 267 | 268 | ---------------------------------------------- 269 | Welcome to Use QueueMan V0.0.1 270 | ---------------------------------------------- 271 | 272 | usage: 273 | -c string 274 | the configure file path (default "./queueman.json") 275 | -h show help information 276 | -s show statistics information 277 | -t test configure in "queueman.json" file 278 | ``` 279 | 280 | ## 查看统计信息 281 | ### 命令行下查看 282 | ``` 283 | ./queueman_linux -c ./queueman.json -s 284 | ``` 285 | 286 | ### Web方式查看 287 | 返回 html 格式 288 | ``` 289 | curl "http://127.0.0.1:8080/statistic?format=html" 290 | ``` 291 | 292 | 返回 json 格式 293 | ``` 294 | curl "http://127.0.0.1:8080/statistic?format=json" 295 | ``` 296 | 297 | ## 配置文件详解 298 | ``` 299 | { 300 | "App": { # Queueman app 级别配置 301 | "IsDebug" : false, # true （会输出 info 级别信息） false （只输出 warn 级别及以上信息） 302 | "PIDFile" : "/var/run/queueman.pid", # pid文件位置 303 | "LogFormatter": "json", # 日志格式 text json 304 | "LogDir" : "/var/log/queueman/" # 日志文件目录，为空表示输出到 stdout 305 | }, 306 | "Statistic": { # 统计配置 307 | "HTTPPort": 8080, # Web 查看端口 308 | "SourceType": "Redis", # 统计到 Redis 309 | "RedisSource" : { 310 | "Network" : "tcp", # Redis 超时时间 311 | "Host" : "127.0.0.1", # Redis 地址 312 | "Port" : 6379, # Redis 端口 313 | "Password" : "", # Redis 密码，没有设置请置空 314 | "DB" : 0, # Redis DB 值 315 | "Timeout" : 5, # Redis 超时时间 316 | "MaxActive" : 1000, 317 | "MaxIdle" : 200, 318 | "MaxIdleTimeout": 10, 319 | "Wait" : true 320 | } 321 | }, 322 | "Redis": [ 323 | { 324 | "Config" : { # 第一个 Redis 的连接配置，为同一节点下的 Queues 服务 325 | "Network" : "tcp", # Redis 超时时间 326 | "Host" : "127.0.0.1", # Redis 地址 327 | "Port" : 6379, # Redis 端口 328 | "Password" : "", # Redis 密码，没有设置请置空 329 | "DB" : 0, # Redis DB 值 330 | "Timeout" : 5, # Redis 超时时间 331 | "MaxActive" : 1000, 332 | "MaxIdle" : 200, 333 | "MaxIdleTimeout": 10, 334 | "Wait" : true 335 | }, 336 | "Queues": [ 337 | { 338 | "IsEnabled": true, # 是否启用，若为 false 则不 Queueman 不处理 339 | "IsDelayQueue": false, # 是否延时队列 true 是 false 否 340 | "IsDelayRaw": false, # 延时队列是否初次读取时返回原始值，只有当延时队列zset格式不是Redis 标准 DelayData 格式时（比如自定义格式时）true 是 false 否 341 | "QueueName": "queue:test1", # 队列名称 342 | "DispatchURL": "http://127.0.0.1/receive", # 接收队列数据的 http(s) 地址 343 | "DispatchTimeout": 30, # Queueman转发超时时间 344 | "Concurency": 5, # Queueman转发并发数，请根据服务器性能来设置 345 | "DelayConcurency": 3, # 失败后的延时队列转发并发数，请根据服务器性能来设置 346 | "DelayOnFailure": [60, 120] # 单位（秒）失败后第N秒尝试，多个值则尝试多次。第一次失败后的秒数 347 | }, 348 | { 349 | "IsEnabled": true, 350 | "IsDelayQueue": true, 351 | "IsDelayRaw": false, 352 | "QueueName": "queue:test2", 353 | "DispatchURL": "http://127.0.0.1/receive", 354 | "DispatchTimeout": 30, 355 | "Concurency": 5, 356 | "DelayConcurency": 3, 357 | "DelayOnFailure": [60, 120] 358 | } 359 | ] 360 | }, 361 | { 362 | "Config" : { # 第二个 Redis 的连接配置，为同一节点下的 Queues 服务 363 | "Network" : "tcp", 364 | "Host" : "127.0.0.1", 365 | "Port" : 6379, 366 | "Password" : "", 367 | "DB" : 0, 368 | "Timeout" : 5, 369 | "MaxActive" : 1000, 370 | "MaxIdle" : 200, 371 | "MaxIdleTimeout": 10, 372 | "Wait" : true 373 | }, 374 | "Queues": [ 375 | { 376 | "IsEnabled": true, 377 | "IsDelayQueue": false, 378 | "IsDelayRaw": false, 379 | "QueueName": "queue:test3", 380 | "DispatchURL": "http://127.0.0.1/receive", 381 | "DispatchTimeout": 30, 382 | "Concurency": 5, 383 | "DelayConcurency": 3, 384 | "DelayOnFailure": [60, 120] 385 | }, 386 | { 387 | "IsEnabled": true, 388 | "IsDelayQueue": true, 389 | "IsDelayRaw": false, 390 | "QueueName": "queue:test4", 391 | "DispatchURL": "http://127.0.0.1/receive", 392 | "DispatchTimeout": 30, 393 | "Concurency": 5, 394 | "DelayConcurency": 3, 395 | "DelayOnFailure": [60, 120] 396 | } 397 | ] 398 | } 399 | ], 400 | "RabbitMQ": [ 401 | { 402 | "Config" : { # 第一个 RabbitMQ 连接配置，为同一节点下的 Queues 服务 403 | "Scheme" : "amqp", # RabbitMQ协议 amqp、amqps 404 | "Host" : "Replace your Host", # RabbitMQ 地址 405 | "Port" : 5672, # RabbitMQ 端口 406 | "User" : "", # RabbitMQ 用户名（Type为 aliyun 里保持空） 407 | "Password" : "", # RabbitMQ 密码（Type为 aliyun 里保持空） 408 | "Vhost" : "Replace your Vhost", # RabbitMQ Vhost 409 | "Type" : "Aliyun", # RabbitMQ 类型 为 空 或者 Aliyun 410 | "AliyunParams" : { # RabbitMQ Aliyun 的参数配置，如非阿时云，这一行可以删除 411 | "AccessKey" : "Replace your AccessKey", # RabbitMQ Aliyun 的 AccessKey 412 | "AccessKeySecret": "Replace your AccessKeySecret", # RabbitMQ Aliyun 的 AccessKeySecret 413 | "ResourceOwnerId": 1000000000000000 # RabbitMQ Aliyun 的 ResourceOwnerId 414 | } 415 | }, 416 | "Queues" : [ 417 | { 418 | "IsEnabled": true, # 是否启用，若为 false 则不 Queueman 不处理 419 | "IsDelayQueue": false, # 是否延时队列 true 是 false 否 420 | "IsDurable": true, # 是否持久化 421 | "ExchangeName": "test.exchange.direct1", # 交换机名称 422 | "ExchangeType": "direct", # 交换机类型 423 | "QueueName": "test.queue.direct1", # 队列名称 424 | "RoutingKey": "test.route.direct1", # RoutingKey 425 | "ConsumerTag": "test.consumer.direct1", # 消费Tag 426 | "IsAutoAck": false, # 是否自动确认 true (自动) false (Queueman接收到成功响应后触发 427 | "DispatchURL": "http://127.0.0.1/receive", # 接收队列数据的 http(s) 地址 428 | "DispatchTimeout": 30, # Queueman转发超时时间 429 | "Concurency": 5, # Queueman转发并发数，请根据服务器性能来设置 430 | "DelayConcurency": 3, # 失败后的延时队列转发并发数，请根据服务器性能来设置 431 | "DelayOnFailure": [60, 120] # 单位（秒）失败后第N秒尝试，多个值则尝试多次。第一次失败后的秒数 432 | }, 433 | { 434 | "IsEnabled": true, 435 | "IsDelayQueue": true, 436 | "IsDurable": true, 437 | "ExchangeName": "test.exchange.direct2", 438 | "ExchangeType": "direct", 439 | "QueueName": "test.queue.direct2", 440 | "RoutingKey": "test.route.direct2", 441 | "ConsumerTag": "test.consumer.direct2", 442 | "IsAutoAck": false, 443 | "DispatchURL": "http://127.0.0.1/receive", 444 | "DispatchTimeout": 30, 445 | "Concurency": 5, 446 | "DelayConcurency": 3, 447 | "DelayOnFailure": [60, 120] 448 | } 449 | ] 450 | }, 451 | { 452 | "Config" : { # 第二个 RabbitMQ 标准连接配置，为同一节点下的 Queues 服务 453 | "Scheme" : "amqp", # RabbitMQ协议 amqp、amqps 454 | "Host" : "Replace your Host", # RabbitMQ 地址 455 | "Port" : 5672, # RabbitMQ 端口 456 | "User" : "Replace your User", # RabbitMQ 用户名（Type为 aliyun 里保持空） 457 | "Password" : "Replace your Password", # RabbitMQ 密码（Type为 aliyun 里保持空） 458 | "Vhost" : "Replace your Vhost", # RabbitMQ Vhost 459 | "Type" : "" # RabbitMQ 类型 为 空 或者 Aliyun 460 | }, 461 | "Queues" : [ 462 | { 463 | "IsEnabled": true, 464 | "IsDelayQueue": false, 465 | "IsDurable": true, 466 | "ExchangeName": "test.exchange.direct3", 467 | "ExchangeType": "direct", 468 | "QueueName": "test.queue.direct3", 469 | "RoutingKey": "test.route.direct3", 470 | "ConsumerTag": "test.consumer.direct3", 471 | "IsAutoAck": false, 472 | "DispatchURL": "http://127.0.0.1/receive", 473 | "DispatchTimeout": 30, 474 | "Concurency": 5, 475 | "DelayConcurency": 3, 476 | "DelayOnFailure": [60, 120] 477 | }, 478 | { 479 | "IsEnabled": true, 480 | "IsDelayQueue": true, 481 | "IsDurable": true, 482 | "ExchangeName": "test.exchange.direct4", 483 | "ExchangeType": "direct", 484 | "QueueName": "test.queue.direct4", 485 | "RoutingKey": "test.route.direct4", 486 | "ConsumerTag": "test.consumer.direct4", 487 | "IsAutoAck": false, 488 | "DispatchURL": "http://127.0.0.1/receive", 489 | "DispatchTimeout": 30, 490 | "Concurency": 5, 491 | "DelayConcurency": 3, 492 | "DelayOnFailure": [60, 120] 493 | } 494 | ] 495 | } 496 | ] 497 | } 498 | ``` 499 | 500 | ## 维护者 501 | 502 | [@marknown](https://github.com/marknown) 503 | 504 | ## 如何贡献 505 | 506 | 非常欢迎你的加入! [提一个Issue](https://github.com/marknown/queueman/issues/new) 或者提交一个 Pull Request. 507 | 508 | ## 使用许可 509 | 510 | [MIT](LICENSE) © marknown 511 | -------------------------------------------------------------------------------- /bin/queueman_arm_untest: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/marknown/queueman/bb59c9322631872c18bc7f6828ee892fa3699027/bin/queueman_arm_untest -------------------------------------------------------------------------------- /bin/queueman_freebsd_untest: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/marknown/queueman/bb59c9322631872c18bc7f6828ee892fa3699027/bin/queueman_freebsd_untest -------------------------------------------------------------------------------- /bin/queueman_linux: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/marknown/queueman/bb59c9322631872c18bc7f6828ee892fa3699027/bin/queueman_linux -------------------------------------------------------------------------------- /bin/queueman_macos: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/marknown/queueman/bb59c9322631872c18bc7f6828ee892fa3699027/bin/queueman_macos -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | binName="bin/queueman" 3 | 4 | GOOS=linux GOARCH=amd64 go build -o "$binName"_linux 5 | GOOS=darwin GOARCH=amd64 go build -o "$binName"_macos 6 | GOOS=freebsd GOARCH=amd64 go build -o "$binName"_freebsd_untest 7 | GOOS=linux GOARCH=arm go build -o "$binName"_arm_untest 8 | GOOS=windows GOARCH=amd64 go build -o "$binName"_win64_untest.exe 9 | 10 | echo "build working is done, see below binary files" 11 | 12 | ls "$binName"* 13 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module queueman 2 | 3 | require ( 4 | github.com/docker/docker v1.13.1 5 | github.com/docker/go-units v0.4.0 // indirect 6 | github.com/go-sql-driver/mysql v1.5.0 // indirect 7 | github.com/gomodule/redigo v2.0.0+incompatible 8 | github.com/isayme/go-amqp-reconnect v0.0.0-20180930040740-e71660afb5ca 9 | github.com/marknown/oconfig v1.0.0 10 | github.com/marknown/ohttp v1.0.3 11 | github.com/marknown/oredis v1.0.0 12 | github.com/marknown/queueman v1.0.7 13 | github.com/mdempsky/gocode v0.0.0-20191202075140-939b4a677f2f // indirect 14 | github.com/satori/go.uuid v1.2.0 15 | github.com/sirupsen/logrus v1.4.2 16 | github.com/streadway/amqp v0.0.0-20200108173154-1c71cc93ed71 17 | golang.org/x/tools v0.0.0-20200302225559-9b52d559c609 // indirect 18 | ) 19 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 2 | github.com/docker/docker v1.13.1/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= 3 | github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= 4 | github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w= 5 | github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= 6 | github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4= 7 | github.com/isayme/go-amqp-reconnect v0.0.0-20180930040740-e71660afb5ca h1:/k6bi3UzEon87YXwewlKXiMT+Qxu+OCaOtkn5ZGgCOs= 8 | github.com/isayme/go-amqp-reconnect v0.0.0-20180930040740-e71660afb5ca/go.mod h1:4IOu90sBxNtO7GtD9//Ybh2UjZ9Dl+Cd9yIMj9GPRHQ= 9 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= 10 | github.com/marknown/oconfig v1.0.0/go.mod h1:42e5MF+dMgl3NBoCiTO6UUyBpcClq6lo+R+djzPKLhc= 11 | github.com/marknown/ohttp v1.0.0/go.mod h1:AYXnqCfB0RagWu43w35d7j5nCIm/OjHZFaAGy0aCGKc= 12 | github.com/marknown/ohttp v1.0.3/go.mod h1:AYXnqCfB0RagWu43w35d7j5nCIm/OjHZFaAGy0aCGKc= 13 | github.com/marknown/olog v1.0.0/go.mod h1:GyVIWHYkV0c7+D02vSpJ8nJHedcAjyYuDZCC1JTDWog= 14 | github.com/marknown/oredis v1.0.0/go.mod h1:M9oZ+dyzsCZIQgxW3TH6VU/VBttSLkUT4Zifd46G7fk= 15 | github.com/marknown/queueman v1.0.7/go.mod h1:/L73FcgZPj0czkhsddcGBuYO6Fp4ePs3PeLEdp/3DW4= 16 | github.com/mdempsky/gocode v0.0.0-20191202075140-939b4a677f2f/go.mod h1:hltEC42XzfMNgg0S1v6JTywwra2Mu6F6cLR03debVQ8= 17 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 18 | github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0= 19 | github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= 20 | github.com/streadway/amqp v0.0.0-20200108173154-1c71cc93ed71/go.mod h1:AZpEONHx3DKn8O/DFsRAY58/XVQiIPMTMB1SddzLXVw= 21 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 22 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 23 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 24 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 25 | golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= 26 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 27 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 28 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 29 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 30 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 31 | golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 32 | golang.org/x/sys v0.0.0-20190618155005-516e3c20635f h1:dHNZYIYdq2QuU6w73vZ/DzesPbVlZVYZTtTZmrnsbQ8= 33 | golang.org/x/sys v0.0.0-20190618155005-516e3c20635f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 34 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 35 | golang.org/x/tools v0.0.0-20200302225559-9b52d559c609/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 36 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 37 | -------------------------------------------------------------------------------- /libs/aliyun/aliyun.go: -------------------------------------------------------------------------------- 1 | package aliyun 2 | 3 | import ( 4 | "bytes" 5 | "encoding/base64" 6 | "strconv" 7 | "strings" 8 | "time" 9 | "crypto/hmac" 10 | "crypto/sha1" 11 | "encoding/hex" 12 | ) 13 | 14 | const ( 15 | ACCESS_FROM_USER = 0 16 | COLON = ":" 17 | ) 18 | 19 | // Config aliyun configure 20 | type Config struct { 21 | AccessKey string 22 | AccessKeySecret string 23 | ResourceOwnerId uint64 24 | } 25 | 26 | // GetUserName Get username for amqp 27 | func (c *Config) GetUserName() string { 28 | var buffer bytes.Buffer 29 | buffer.WriteString(strconv.Itoa(ACCESS_FROM_USER)) 30 | buffer.WriteString(COLON) 31 | buffer.WriteString(strconv.FormatUint(c.ResourceOwnerId,10)) 32 | buffer.WriteString(COLON) 33 | buffer.WriteString(c.AccessKey) 34 | return base64.StdEncoding.EncodeToString(buffer.Bytes()) 35 | } 36 | 37 | // GetPassword Get password for amqp 38 | func (c *Config) GetPassword() string { 39 | now := time.Now() 40 | currentMillis := strconv.FormatInt(now.UnixNano()/1000000,10) 41 | var buffer bytes.Buffer 42 | buffer.WriteString(strings.ToUpper(HmacSha1(currentMillis,c.AccessKeySecret))) 43 | buffer.WriteString(COLON) 44 | buffer.WriteString(currentMillis) 45 | return base64.StdEncoding.EncodeToString(buffer.Bytes()) 46 | } 47 | 48 | // HmacSha1 encoder 49 | func HmacSha1(keyStr string, message string) string { 50 | key := []byte(keyStr) 51 | mac := hmac.New(sha1.New, key) 52 | mac.Write([]byte(message)) 53 | return hex.EncodeToString(mac.Sum(nil)) 54 | } -------------------------------------------------------------------------------- /libs/command/command.go: -------------------------------------------------------------------------------- 1 | package command 2 | 3 | import ( 4 | "encoding/json" 5 | "flag" 6 | "fmt" 7 | "os" 8 | "queueman/libs/config" 9 | "queueman/libs/constant" 10 | "queueman/libs/queue/redis" 11 | "queueman/libs/statistic" 12 | "strings" 13 | 14 | "github.com/marknown/oredis" 15 | ) 16 | 17 | // Args is command argv 18 | type Args struct { 19 | ConfigFile string 20 | Help bool 21 | Test bool 22 | Stats bool // show stats info 23 | } 24 | 25 | var appname string 26 | var appversion string 27 | 28 | // GetArgs Get args form command line 29 | func GetArgs() *Args { 30 | appname = constant.APPNAME 31 | appversion = constant.APPVERSION 32 | 33 | args := &Args{} 34 | flag.BoolVar(&args.Help, "h", false, "show help information") 35 | flag.StringVar(&args.ConfigFile, "c", "./queueman.json", "the configure file path") 36 | flag.BoolVar(&args.Test, "t", false, `test configure in "queueman.json" file`) 37 | flag.BoolVar(&args.Stats, "s", false, "show statistics information") 38 | flag.Usage = printUsage 39 | flag.Parse() 40 | 41 | if args.Help { 42 | printUsage() 43 | return nil 44 | } 45 | 46 | if args.Test { 47 | printTest(args, true) 48 | return nil 49 | } 50 | 51 | printTest(args, false) 52 | 53 | // when configure file is right 54 | if args.Stats { 55 | info := GetStats(args, "") 56 | fmt.Println(info) 57 | os.Exit(0) 58 | return nil 59 | } 60 | 61 | return args 62 | } 63 | 64 | // printUsage print the useage 65 | func printUsage() { 66 | message := `---------------------------------------------- 67 | Welcome to Use %s %s 68 | ---------------------------------------------- 69 | 70 | usage: 71 | ` 72 | fmt.Printf(message, appname, appversion) 73 | flag.PrintDefaults() 74 | 75 | os.Exit(0) 76 | } 77 | 78 | // printTest print test configure results 79 | func printTest(args *Args, isJustTest bool) { 80 | message := fmt.Sprintf(`%s %s configuration file %s test is `, appname, appversion, args.ConfigFile) 81 | 82 | defer func() { 83 | if err := recover(); err != nil { 84 | fmt.Printf("%s wrong !!!\n> %s\n", message, err) 85 | os.Exit(0) 86 | } 87 | }() 88 | 89 | // Get config file content to struct 90 | cfg := config.GetConfig(args.ConfigFile) 91 | 92 | // check statistic redis source 93 | if "redis" == strings.ToLower(cfg.Statistic.SourceType) { 94 | oredis.GetInstancePanic(cfg.Statistic.RedisSource) 95 | } 96 | 97 | enabledQueueCount := 0 98 | // sort the DelayOnFailure array 99 | for _, r := range cfg.Redis { 100 | oredis.GetInstancePanic(r.Config) 101 | for _, n := range r.Queues { 102 | if n.IsEnabled { 103 | enabledQueueCount++ 104 | } 105 | } 106 | } 107 | 108 | for _, r := range cfg.RabbitMQ { 109 | r.Config.GetConnectionPanic() 110 | for _, n := range r.Queues { 111 | if n.IsEnabled { 112 | enabledQueueCount++ 113 | } 114 | } 115 | } 116 | 117 | if enabledQueueCount < 1 { 118 | panic(`There has no enabled queue, please check configure file "IsEnabled" fields for every queue`) 119 | } 120 | 121 | // if -t 122 | if isJustTest { 123 | fmt.Printf("%s ok\n", message) 124 | os.Exit(0) 125 | } 126 | } 127 | 128 | // GetStats get stats information format can be "text", "html", "json" 129 | func GetStats(args *Args, format string) string { 130 | cfg := config.GetConfig(args.ConfigFile) 131 | // init statistic for record 132 | statistic.InitStatistic(cfg.Statistic) 133 | 134 | allQueueStatistic := []*statistic.QueueStatistic{} 135 | 136 | for _, cc := range cfg.Redis { 137 | for _, queueConfig := range cc.Queues { 138 | s := &statistic.QueueStatistic{ 139 | QueueName: queueConfig.QueueName, 140 | SourceType: "Redis", 141 | IsEnabled: queueConfig.IsEnabled, 142 | } 143 | 144 | qi := &redis.QueueInstance{ 145 | Source: cc.Config, 146 | Queue: queueConfig, 147 | } 148 | 149 | if queueConfig.IsDelayQueue { 150 | s.Normal, _ = qi.DelayLength(queueConfig.QueueName) 151 | } else { 152 | s.Normal, _ = qi.Length(queueConfig.QueueName) 153 | } 154 | 155 | if len(queueConfig.DelayOnFailure) > 0 { 156 | queueName := fmt.Sprintf("%s:delayed", queueConfig.QueueName) 157 | s.Delayed, _ = qi.DelayLength(queueName) 158 | } 159 | 160 | s.Success, _ = statistic.GetCounter(fmt.Sprintf("%s:success", queueConfig.QueueName)) 161 | s.Failure, _ = statistic.GetCounter(fmt.Sprintf("%s:failure", queueConfig.QueueName)) 162 | 163 | s.Total = s.Normal + s.Delayed + s.Success + s.Failure 164 | 165 | allQueueStatistic = append(allQueueStatistic, s) 166 | } 167 | } 168 | 169 | for _, cc := range cfg.RabbitMQ { 170 | for _, queueConfig := range cc.Queues { 171 | s := &statistic.QueueStatistic{ 172 | QueueName: queueConfig.QueueName, 173 | SourceType: "RabbitMQ", 174 | IsEnabled: queueConfig.IsEnabled, 175 | } 176 | 177 | // qi := &rabbitmq.QueueInstance{ 178 | // Source: cc.Config, 179 | // Queue: queueConfig, 180 | // } 181 | // todo get queue length 182 | 183 | s.Normal = 0 184 | s.Delayed = 0 185 | 186 | s.Success, _ = statistic.GetCounter(fmt.Sprintf("%s:success", queueConfig.QueueName)) 187 | s.Failure, _ = statistic.GetCounter(fmt.Sprintf("%s:failure", queueConfig.QueueName)) 188 | 189 | s.Total = s.Normal + s.Delayed + s.Success + s.Failure 190 | 191 | allQueueStatistic = append(allQueueStatistic, s) 192 | } 193 | } 194 | 195 | if "json" == format { 196 | output, err := json.Marshal(allQueueStatistic) 197 | 198 | if nil != err { 199 | return "" 200 | } 201 | 202 | return string(output) 203 | } 204 | 205 | output := fmt.Sprintf("%s %s statistics information\n\n", constant.APPNAME, constant.APPVERSION) 206 | for _, s := range allQueueStatistic { 207 | status := "disable" 208 | if s.IsEnabled { 209 | status = "enable" 210 | } 211 | output += fmt.Sprintf(" > Type: %-8s Status: %-8s Name: %s\n%10d Total\n%10d Normal\n%10d Delayed\n%10d Success\n%10d Failure\n\n", s.SourceType, status, s.QueueName, s.Total, s.Normal, s.Delayed, s.Success, s.Failure) 212 | } 213 | 214 | if "html" == format { 215 | strings.Replace(output, "\n", "
", -1) 216 | } 217 | 218 | return output 219 | } 220 | -------------------------------------------------------------------------------- /libs/config/config.go: -------------------------------------------------------------------------------- 1 | package config 2 | 3 | import ( 4 | "queueman/libs/queue/rabbitmq" 5 | "queueman/libs/queue/redis" 6 | "queueman/libs/statistic" 7 | "sort" 8 | "sync" 9 | 10 | "github.com/marknown/oconfig" 11 | ) 12 | 13 | // App configure for the app 14 | type App struct { 15 | IsDebug bool // is debug mode 16 | PIDFile string // PIDFile path 17 | LogFormatter string // Log formatter text or json 18 | LogDir string // Log directory 19 | } 20 | 21 | // Config for the file 22 | type Config struct { 23 | App App 24 | Statistic statistic.Config 25 | Redis []redis.CombineConfig 26 | RabbitMQ []rabbitmq.CombineConfig 27 | } 28 | 29 | var once = &sync.Once{} 30 | var lock = &sync.Mutex{} 31 | var packageConfigInstance *Config 32 | 33 | // GetConfig only init one time 34 | func GetConfig(configPath string) *Config { 35 | lock.Lock() 36 | defer lock.Unlock() 37 | 38 | if nil != packageConfigInstance { 39 | return packageConfigInstance 40 | } 41 | 42 | once.Do(func() { 43 | packageConfigInstance = &Config{} 44 | oconfig.GetConfig(configPath, packageConfigInstance) 45 | 46 | // sort the DelayOnFailure array 47 | for _, qc := range packageConfigInstance.Redis { 48 | for _, qcc := range qc.Queues { 49 | sort.Ints(qcc.DelayOnFailure) 50 | } 51 | } 52 | }) 53 | 54 | return packageConfigInstance 55 | } 56 | -------------------------------------------------------------------------------- /libs/constant/constant.go: -------------------------------------------------------------------------------- 1 | package constant 2 | 3 | const ( 4 | // APPNAME app name 5 | APPNAME = "QueueMan" 6 | // APPVERSION app version 7 | APPVERSION = "V1.0.9" 8 | ) 9 | -------------------------------------------------------------------------------- /libs/ohttp/ohttp.go: -------------------------------------------------------------------------------- 1 | // Package ohttp is own http client 2 | package ohttp 3 | 4 | import ( 5 | "io" 6 | "io/ioutil" 7 | "net" 8 | "net/http" 9 | "net/url" 10 | "strings" 11 | "time" 12 | ) 13 | 14 | // Request 请求 15 | type Request http.Request 16 | 17 | // Response 响应 18 | type Response http.Response 19 | 20 | // RequestSettings 请求设置结构体 21 | type RequestSettings struct { 22 | Timeout time.Duration // 请求超时时间 23 | IsRandomUserAgent bool // 是否随机UserAgent 24 | UserAgent string // UserAgent 如果 IsRandomUserAgent 设置了，则随机 25 | Referer string // 设置 Referer 26 | IsFollowLocation bool // 是否跟随跳转 27 | IsAajx bool // 是否是 ajax 请求 28 | ContentType string // 内容类型 29 | Cookies string // Cookies 字符串 30 | Headers [][2]string // 额外的请求头 31 | } 32 | 33 | // defaultSetting 默认设置 34 | var defaultSetting = RequestSettings{ 35 | Timeout: 0 * time.Second, 36 | IsRandomUserAgent: false, 37 | UserAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36", 38 | Referer: "", 39 | IsFollowLocation: true, 40 | IsAajx: false, 41 | ContentType: "application/x-www-form-urlencoded", 42 | Cookies: "", 43 | Headers: [][2]string{}, 44 | } 45 | 46 | // InitSetttings 重新初始化请求设置 47 | func InitSetttings() *RequestSettings { 48 | s := defaultSetting 49 | 50 | return &s 51 | } 52 | 53 | // Get 执行 Get 请求 54 | func (settings *RequestSettings) Get(url string) (string, *Response, error) { 55 | return settings.request("GET", url, nil) 56 | } 57 | 58 | // Post 执行 Post 请求 59 | func (settings *RequestSettings) Post(url string, params interface{}) (string, *Response, error) { 60 | return settings.request("POST", url, params) 61 | } 62 | 63 | func (settings *RequestSettings) request(method string, url string, params interface{}) (string, *Response, error) { 64 | req, err := settings.NewRequest(method, url, params) 65 | 66 | if err != nil { 67 | return "", nil, err 68 | } 69 | 70 | resp, err := settings.Do(req) 71 | 72 | if err != nil { 73 | return "", nil, err 74 | } 75 | 76 | content, err := resp.ContentString() 77 | 78 | if err != nil { 79 | return "", nil, err 80 | } 81 | 82 | return content, resp, err 83 | } 84 | 85 | // NewRequest 创建一个 Request 请求 86 | func (settings *RequestSettings) NewRequest(method string, url string, params interface{}) (*Request, error) { 87 | var req *http.Request 88 | var err error 89 | if _, ok := params.(string); ok { 90 | req, err = http.NewRequest(method, url, strings.NewReader(params.(string))) 91 | } 92 | if _, ok := params.(map[string]string); ok { 93 | req, err = http.NewRequest(method, url, BuildQueryReader(params.(map[string]string))) 94 | } 95 | if nil == params { 96 | req, err = http.NewRequest(method, url, nil) 97 | } 98 | 99 | if err != nil { 100 | return nil, err 101 | } 102 | 103 | if "" != settings.UserAgent { 104 | req.Header.Set("User-Agent", settings.UserAgent) 105 | } 106 | 107 | if "" != settings.Referer { 108 | req.Header.Set("Referer", settings.Referer) 109 | } 110 | 111 | if settings.IsAajx { 112 | req.Header.Set("X-Requested-With", "XMLHttpRequest") 113 | } 114 | 115 | if "" != settings.ContentType { 116 | req.Header.Set("Content-Type", settings.ContentType) // 如果是 POST 一定要有这一行，不然接收不到数据 117 | } 118 | 119 | for _, v := range settings.Headers { 120 | req.Header.Set(v[0], v[1]) 121 | } 122 | 123 | if "" != settings.Cookies { 124 | req.Header.Set("Cookie", settings.Cookies) 125 | } 126 | 127 | r := Request(*req) 128 | return &r, err 129 | } 130 | 131 | func dialTimeout(network, addr string) (net.Conn, error) { 132 | return net.DialTimeout(network, addr, 10*time.Second) 133 | } 134 | 135 | // Do 执行网络请求 136 | func (settings *RequestSettings) Do(req *Request) (*Response, error) { 137 | r := http.Request(*req) 138 | 139 | transport := http.Transport{ 140 | Dial: dialTimeout, 141 | DisableKeepAlives: true, 142 | } 143 | 144 | // 超时设置 145 | var c = &http.Client{ 146 | Transport: &transport, 147 | Timeout: settings.Timeout * time.Second, 148 | } 149 | 150 | resp, err := c.Do(&r) 151 | 152 | if err != nil { 153 | return nil, err 154 | } 155 | 156 | response := Response(*resp) 157 | return &response, err 158 | } 159 | 160 | // SetCookie 设置 cookie 161 | func (req *Request) SetCookie(cookies string) { 162 | req.Header.Set("Cookie", cookies) 163 | } 164 | 165 | // ContentString 获取 response 的内容 166 | func (resp *Response) ContentString() (string, error) { 167 | defer resp.Body.Close() 168 | 169 | body, err := ioutil.ReadAll(resp.Body) 170 | 171 | if err != nil { 172 | return "", err 173 | } 174 | 175 | return string(body), nil 176 | } 177 | 178 | // CookieString 获取 response 的内容 179 | func (resp *Response) CookieString() string { 180 | r := http.Response(*resp) 181 | Cookies := r.Cookies() 182 | cookieArray := []string{} 183 | 184 | for _, cookie := range Cookies { 185 | cookieArray = append(cookieArray, cookie.Name+"="+cookie.Value) 186 | } 187 | 188 | return strings.Join(cookieArray, "; ") 189 | } 190 | 191 | // HeaderString 获取 Response 的 header 192 | func (resp *Response) HeaderString() string { 193 | Headers := resp.Header 194 | 195 | HeaderString := "" 196 | 197 | for k, v := range Headers { 198 | for _, vv := range v { 199 | HeaderString += k + ": " + vv + "\n" 200 | } 201 | } 202 | 203 | return "Response Header:\n\n" + HeaderString + "\n" 204 | } 205 | 206 | // RequestCookieString 获取 request 的 cookie 207 | func (resp *Response) RequestCookieString() string { 208 | Cookies := resp.Request.Cookies() 209 | cookieArray := []string{} 210 | 211 | for _, cookie := range Cookies { 212 | cookieArray = append(cookieArray, cookie.Name+"="+cookie.Value) 213 | } 214 | 215 | return strings.Join(cookieArray, "; ") 216 | } 217 | 218 | // RequestHeaderString 获取 request 的 header 219 | func (resp *Response) RequestHeaderString() string { 220 | Headers := resp.Request.Header 221 | 222 | HeaderString := "" 223 | 224 | for k, v := range Headers { 225 | for _, vv := range v { 226 | HeaderString += k + ": " + vv + "\n" 227 | } 228 | } 229 | 230 | return "Request Header:\n\n" + HeaderString + "\n" 231 | } 232 | 233 | // 以下为公共函数 234 | 235 | // BuildQuery 构造 GET 或者 POST 请求参数 236 | func BuildQuery(params map[string]string) string { 237 | values := url.Values{} 238 | for k, v := range params { 239 | values[k] = []string{v} 240 | } 241 | result := values.Encode() 242 | 243 | // todo 以下代码针对淘宝秒杀做的限制 244 | result = strings.Replace(result, "%27", `'`, -1) // ' 转成 \' 不能转成 %27 245 | result = strings.Replace(result, "+", `%20`, -1) // + 转成 %20 246 | result = strings.Replace(result, "%28", `(`, -1) // ( 不转成 %28 247 | result = strings.Replace(result, "%29", `)`, -1) // ) 不转成 %29 248 | // tododel 249 | result = strings.Replace(result, "info%5C%5C%5C%22%3A%5B%5D", `info%5C%5C%5C%22%3A%7B%7D`, -1) // ) 不转成 %29 250 | 251 | return result 252 | } 253 | 254 | // BuildQueryReader 构造 GET 或者 POST 请求参数 io.Reader 形式 255 | func BuildQueryReader(params map[string]string) io.Reader { 256 | return strings.NewReader(BuildQuery(params)) 257 | } 258 | 259 | // URLValuesToStringMap 把 url.Values 转成 map[string]string 260 | func URLValuesToStringMap(values url.Values) map[string]string { 261 | m := make(map[string]string) 262 | for k, v := range values { 263 | if len(v) > 0 { 264 | m[k] = v[0] 265 | } else { 266 | m[k] = "" 267 | } 268 | } 269 | 270 | return m 271 | } 272 | 273 | // IsAjaxRequest 判断是否是 ajax 请求 274 | func IsAjaxRequest(r *http.Request) bool { 275 | return "XMLHttpRequest" == r.Header.Get("X-Requested-With") 276 | } 277 | 278 | // IsPostRequest 判断是否是 ajax 请求 279 | func IsPostRequest(r *http.Request) bool { 280 | return "POST" == r.Method 281 | } 282 | 283 | // IsGetRequest 判断是否是 ajax 请求 284 | func IsGetRequest(r *http.Request) bool { 285 | return "GET" == r.Method 286 | } 287 | 288 | // MapCookies 把字符串 cookie 转成 map 键值对 289 | func MapCookies(cookies string) map[string]string { 290 | cookieArray := strings.Split(cookies, ";") 291 | result := map[string]string{} 292 | 293 | for _, cookie := range cookieArray { 294 | kv := strings.Split(strings.TrimSpace(cookie), "=") 295 | if len(kv) >= 2 && "" != kv[0] && "" != kv[1] { 296 | result[kv[0]] = kv[1] 297 | } 298 | } 299 | 300 | return result 301 | } 302 | 303 | // MapCookiesToString 把 map 键值对字符串转成 字符串 cookie 304 | func MapCookiesToString(m map[string]string) string { 305 | cookieArray := []string{} 306 | 307 | for k, v := range m { 308 | cookieArray = append(cookieArray, k+"="+v) 309 | } 310 | 311 | return strings.Join(cookieArray, "; ") 312 | } 313 | 314 | // AppendCookies 在老的 cookie 字符串中添加新的 cookie 字符串，如果名称相同，则替换 315 | func AppendCookies(oldCookie string, appendCookie string) string { 316 | oldCookieMap := MapCookies(oldCookie) 317 | appendCookieMap := MapCookies(appendCookie) 318 | 319 | for k, v := range appendCookieMap { 320 | oldCookieMap[k] = v 321 | } 322 | 323 | return MapCookiesToString(oldCookieMap) 324 | } 325 | -------------------------------------------------------------------------------- /libs/pidfile/pidfile.go: -------------------------------------------------------------------------------- 1 | // Package pidfile provides structure and helper functions to create and remove 2 | // PID file. A PID file is usually a file used to store the process ID of a 3 | // running process. 4 | package pidfile // import "github.com/docker/docker/pkg/pidfile" 5 | 6 | import ( 7 | "fmt" 8 | "io/ioutil" 9 | "os" 10 | "strconv" 11 | "strings" 12 | ) 13 | 14 | // PIDFile is a file used to store the process ID of a running process. 15 | type PIDFile struct { 16 | path string 17 | } 18 | 19 | func checkPIDFileAlreadyExists(path string) error { 20 | if pidByte, err := ioutil.ReadFile(path); err == nil { 21 | pidString := strings.TrimSpace(string(pidByte)) 22 | if pid, err := strconv.Atoi(pidString); err == nil { 23 | if processExists(pid) { 24 | return fmt.Errorf("pid file found, ensure the pid %s is not running or delete", path) 25 | } 26 | } 27 | } 28 | return nil 29 | } 30 | 31 | // New creates a PIDfile using the specified path. 32 | func New(path string) (*PIDFile, error) { 33 | if err := checkPIDFileAlreadyExists(path); err != nil { 34 | return nil, err 35 | } 36 | 37 | if err := ioutil.WriteFile(path, []byte(fmt.Sprintf("%d", os.Getpid())), 0644); err != nil { 38 | return nil, err 39 | } 40 | 41 | return &PIDFile{path: path}, nil 42 | } 43 | 44 | // Remove removes the PIDFile. 45 | func (file PIDFile) Remove() error { 46 | return os.Remove(file.path) 47 | } 48 | -------------------------------------------------------------------------------- /libs/pidfile/pidfile_darwin.go: -------------------------------------------------------------------------------- 1 | // +build darwin 2 | 3 | package pidfile // import "github.com/docker/docker/pkg/pidfile" 4 | 5 | import ( 6 | "golang.org/x/sys/unix" 7 | ) 8 | 9 | func processExists(pid int) bool { 10 | // OS X does not have a proc filesystem. 11 | // Use kill -0 pid to judge if the process exists. 12 | err := unix.Kill(pid, 0) 13 | return err == nil 14 | } 15 | -------------------------------------------------------------------------------- /libs/pidfile/pidfile_test.go: -------------------------------------------------------------------------------- 1 | package pidfile // import "github.com/docker/docker/pkg/pidfile" 2 | 3 | import ( 4 | "io/ioutil" 5 | "os" 6 | "path/filepath" 7 | "testing" 8 | ) 9 | 10 | func TestNewAndRemove(t *testing.T) { 11 | dir, err := ioutil.TempDir(os.TempDir(), "test-pidfile") 12 | if err != nil { 13 | t.Fatal("Could not create test directory") 14 | } 15 | 16 | path := filepath.Join(dir, "testfile") 17 | file, err := New(path) 18 | if err != nil { 19 | t.Fatal("Could not create test file", err) 20 | } 21 | 22 | _, err = New(path) 23 | if err == nil { 24 | t.Fatal("Test file creation not blocked") 25 | } 26 | 27 | if err := file.Remove(); err != nil { 28 | t.Fatal("Could not delete created test file") 29 | } 30 | } 31 | 32 | func TestRemoveInvalidPath(t *testing.T) { 33 | file := PIDFile{path: filepath.Join("foo", "bar")} 34 | 35 | if err := file.Remove(); err == nil { 36 | t.Fatal("Non-existing file doesn't give an error on delete") 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /libs/pidfile/pidfile_unix.go: -------------------------------------------------------------------------------- 1 | // +build !windows,!darwin 2 | 3 | package pidfile // import "github.com/docker/docker/pkg/pidfile" 4 | 5 | import ( 6 | "os" 7 | "path/filepath" 8 | "strconv" 9 | ) 10 | 11 | func processExists(pid int) bool { 12 | if _, err := os.Stat(filepath.Join("/proc", strconv.Itoa(pid))); err == nil { 13 | return true 14 | } 15 | return false 16 | } 17 | -------------------------------------------------------------------------------- /libs/pidfile/pidfile_windows.go: -------------------------------------------------------------------------------- 1 | package pidfile // import "github.com/docker/docker/pkg/pidfile" 2 | 3 | import ( 4 | "golang.org/x/sys/windows" 5 | ) 6 | 7 | const ( 8 | processQueryLimitedInformation = 0x1000 9 | 10 | stillActive = 259 11 | ) 12 | 13 | func processExists(pid int) bool { 14 | h, err := windows.OpenProcess(processQueryLimitedInformation, false, uint32(pid)) 15 | if err != nil { 16 | return false 17 | } 18 | var c uint32 19 | err = windows.GetExitCodeProcess(h, &c) 20 | windows.Close(h) 21 | if err != nil { 22 | return c == stillActive 23 | } 24 | return true 25 | } 26 | -------------------------------------------------------------------------------- /libs/queue/queue.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "queueman/libs/queue/rabbitmq" 5 | "queueman/libs/queue/redis" 6 | ) 7 | 8 | // QInterface queue interface 9 | type QInterface interface { 10 | Dispatcher(queueConfig interface{}) 11 | } 12 | 13 | // QFactory queue factory 14 | func QFactory(queueType string) QInterface { 15 | switch queueType { 16 | case "RabbitMQ": 17 | return &rabbitmq.Queue{} 18 | case "Redis": 19 | return &redis.Queue{} 20 | } 21 | 22 | return nil 23 | } 24 | -------------------------------------------------------------------------------- /libs/queue/rabbitmq/connection.go: -------------------------------------------------------------------------------- 1 | package rabbitmq 2 | 3 | import ( 4 | "crypto/md5" 5 | "errors" 6 | "fmt" 7 | "queueman/libs/aliyun" 8 | "strings" 9 | "sync" 10 | 11 | amqpReconnect "github.com/isayme/go-amqp-reconnect/rabbitmq" 12 | ) 13 | 14 | // Config configure for rabbitmq 15 | type Config struct { 16 | Scheme string // amqp or amqps 17 | Host string 18 | Port int32 19 | User string 20 | Password string 21 | Vhost string 22 | Type string // default is standard,other options is aliyun 23 | AliyunParams aliyun.Config 24 | } 25 | 26 | // 包内变量，存储实例相关对象 27 | var packageOnce = map[string]*sync.Once{} 28 | var packageInstance = map[string]*amqpReconnect.Connection{} 29 | var packageMutex = &sync.Mutex{} 30 | 31 | // URL get rabbitmq amqp url 32 | func (config *Config) URL() string { 33 | if "aliyun" == strings.ToLower(config.Type) { 34 | config.User = config.AliyunParams.GetUserName() 35 | config.Password = config.AliyunParams.GetPassword() 36 | } 37 | 38 | return fmt.Sprintf("%s://%s:%s@%s:%d/%s", config.Scheme, config.User, config.Password, config.Host, config.Port, config.Vhost) 39 | } 40 | 41 | // GetConnection get a rabbitmq Connection 42 | func (config *Config) GetConnection() (*amqpReconnect.Connection, error) { 43 | // TODODEL 44 | // amqpReconnect.Debug = true 45 | packageMutex.Lock() 46 | defer packageMutex.Unlock() 47 | 48 | md5byte := md5.Sum([]byte(fmt.Sprintf("%s%s%d%s%s%s", config.Scheme, config.Host, config.Port, config.User, config.Password, config.Vhost))) 49 | md5key := fmt.Sprintf("%x", md5byte) 50 | 51 | // 如果有值直接返回 52 | if v, ok := packageInstance[md5key]; ok { 53 | // fmt.Println("direct") 54 | return v, nil 55 | } 56 | 57 | // 如果once 不存在 58 | if _, ok := packageOnce[md5key]; !ok { 59 | var once = &sync.Once{} 60 | var conn *amqpReconnect.Connection 61 | var err error 62 | // var err error 63 | once.Do(func() { 64 | conn, err = amqpReconnect.Dial(config.URL()) 65 | 66 | if nil == err { 67 | packageInstance[md5key] = conn 68 | packageOnce[md5key] = once 69 | } 70 | }) 71 | 72 | if nil != err { 73 | return nil, err 74 | } 75 | 76 | return conn, nil 77 | } 78 | 79 | return nil, errors.New("RabbitMQ get connection error") 80 | } 81 | 82 | // GetConnectionPanic get a rabbitmq Connection and panic when error occurred 83 | func (config *Config) GetConnectionPanic() *amqpReconnect.Connection { 84 | conn, err := config.GetConnection() 85 | 86 | if nil != err { 87 | panic("RabbitMQ " + err.Error()) 88 | } 89 | 90 | return conn 91 | } 92 | -------------------------------------------------------------------------------- /libs/queue/rabbitmq/dispatcher.go: -------------------------------------------------------------------------------- 1 | package rabbitmq 2 | 3 | import ( 4 | "fmt" 5 | "queueman/libs/queue/types" 6 | "queueman/libs/request" 7 | "queueman/libs/statistic" 8 | "strings" 9 | "time" 10 | 11 | amqpReconnect "github.com/isayme/go-amqp-reconnect/rabbitmq" 12 | log "github.com/sirupsen/logrus" 13 | "github.com/streadway/amqp" 14 | ) 15 | 16 | // Dispatcher for Queue 17 | func (queue *Queue) Dispatcher(combineConfig interface{}) { 18 | if cc, ok := combineConfig.(CombineConfig); ok { 19 | for _, queueConfig := range cc.Queues { 20 | if queueConfig.IsEnabled { 21 | queueConfig.SourceType = "RabbitMQ" 22 | // enabledTotal++ todo 23 | qi := &QueueInstance{ 24 | Source: cc.Config, 25 | Queue: queueConfig, 26 | } 27 | go qi.QueueHandle() 28 | } 29 | } 30 | } else { 31 | log.WithFields(log.Fields{ 32 | "queueConfig": combineConfig, 33 | }).Warn("Not correct rabbitmq config") 34 | return 35 | } 36 | } 37 | 38 | // GetConnection get a connect to source 39 | func (qi *QueueInstance) GetConnection() (*amqpReconnect.Connection, error) { 40 | conn, err := qi.Source.GetConnection() 41 | if nil != err { 42 | log.WithFields(log.Fields{ 43 | "error": err, 44 | }).Warn("Can not connect to rabbitmq") 45 | return nil, err 46 | } 47 | 48 | return conn, nil 49 | } 50 | 51 | // QueueHandle handle Normal or Delay queue 52 | func (qi *QueueInstance) QueueHandle() { 53 | if qi.Queue.Concurency < 1 { 54 | log.WithFields(log.Fields{ 55 | "queueName": qi.Queue.QueueName, 56 | }).Warn("configure file of queue must have Concurency configure !!!") 57 | return 58 | } 59 | 60 | // process main queue 61 | // get the delay time 62 | delayTime := 0 63 | if len(qi.Queue.DelayOnFailure) > 0 { 64 | delayTime = qi.Queue.DelayOnFailure[0] 65 | } 66 | 67 | // first process delay queue 68 | totalDelays := len(qi.Queue.DelayOnFailure) 69 | if totalDelays > 0 { 70 | go qi.ProcessDelay("retry") 71 | } 72 | 73 | // wait ProcessDelay finished (to finished delay queue bind exchange queue) 74 | time.Sleep(1 * time.Second) 75 | 76 | if false == qi.Queue.IsDelayQueue { 77 | go qi.ProcessNormal(delayTime) 78 | } else { 79 | go qi.ProcessDelay("first") 80 | } 81 | 82 | } 83 | 84 | // ProcessNormal get a data from queue and dispatch 85 | func (qi *QueueInstance) ProcessNormal(delayTime int) { 86 | conn, err := qi.GetConnection() 87 | defer conn.Close() 88 | 89 | ch, err := conn.Channel() 90 | failOnError(err, "Failed to open a channel") 91 | defer ch.Close() 92 | 93 | // ExchangeDeclare(name, kind string, durable, autoDelete, internal, noWait bool, args Table) error 94 | err = ch.ExchangeDeclare(qi.Queue.ExchangeName, qi.Queue.ExchangeType, qi.Queue.IsDurable, false, false, false, nil) 95 | failOnError(err, "Failed to Declare a exchange") 96 | 97 | // QueueDeclare(name string, durable, autoDelete, exclusive, noWait bool, args Table) (Queue, error) 98 | q, err := ch.QueueDeclare( 99 | qi.Queue.QueueName, // name 100 | qi.Queue.IsDurable, // durable 101 | false, // delete when unused 102 | false, // exclusive 103 | false, // no-wait 104 | nil, // arguments 105 | ) 106 | failOnError(err, "Failed to declare a queue"+q.Name) 107 | 108 | // QueueBind(name, key, exchange string, noWait bool, args Table) error 109 | err = ch.QueueBind(qi.Queue.QueueName, qi.Queue.RoutingKey, qi.Queue.ExchangeName, false, nil) 110 | failOnError(err, "Failed to bind a queue") 111 | 112 | // auto-ack exclusive 113 | // Consume(queue, consumer string, autoAck, exclusive, noLocal, noWait bool, args Table) (<-chan Delivery, error) 114 | msgs, err := ch.Consume( 115 | qi.Queue.QueueName, // queue 116 | qi.Queue.ConsumerTag, // consumer 117 | false, // auto-ack 118 | false, // exclusive 119 | false, // no-local 120 | false, // no-wait 121 | nil, // args 122 | ) 123 | failOnError(err, "Failed to register a consumer") 124 | 125 | log.Printf("Start consume ( %s %s) queue (%s) from Exchange (%s) Vhost (%s)", qi.Queue.SourceType, qi.Source.Type, qi.Queue.QueueName, qi.Queue.ExchangeType, conn.Config.Vhost) 126 | 127 | // control the concurency 128 | concurency := make(chan bool, qi.Queue.Concurency) 129 | 130 | for delivery := range msgs { 131 | // control the concurency 132 | concurency <- true 133 | 134 | // dispatch to URLs 135 | go func(d amqp.Delivery) { 136 | queueData := string(d.Body) 137 | 138 | queueRequest := &request.QueueRequest{ 139 | QueueName: qi.Queue.QueueName, 140 | DispatchURL: qi.Queue.DispatchURL, 141 | DispatchTimeout: qi.Queue.DispatchTimeout, 142 | QueueData: queueData, 143 | } 144 | 145 | result, err := queueRequest.Post() 146 | 147 | status := "" 148 | if qi.Queue.IsAutoAck { 149 | d.Ack(false) 150 | status = "Auto acked" 151 | statistic.IncrSuccessCounter(qi.Queue.QueueName) 152 | } else { 153 | // failure 154 | if 1 != result.Code { 155 | log.WithFields(log.Fields{ 156 | "err": err, 157 | "Message": result.Message, 158 | "queueName": qi.Queue.QueueName, 159 | "delayTime": delayTime, 160 | }).Warn("Request data result") 161 | 162 | nextDelayTime := delayTime 163 | if nextDelayTime > 0 { 164 | serializeDelayQueueData, err := types.SerializeDelayQueueData(queueData, nextDelayTime) 165 | 166 | if nil != err { 167 | log.WithFields(log.Fields{ 168 | "error": err, 169 | "Message": result.Message, 170 | }).Warn("Serialize DelayQueueData error") 171 | 172 | d.Ack(false) // when a error occur acked 173 | statistic.IncrFailureCounter(qi.Queue.QueueName) 174 | <-concurency // remove control 175 | return 176 | } 177 | 178 | queueNameDelay := fmt.Sprintf("%s.delayed", qi.Queue.QueueName) 179 | routingKeyDelay := fmt.Sprintf("%s.delayed", qi.Queue.RoutingKey) 180 | exchangeNameDelay := fmt.Sprintf("%s.delayed", qi.Queue.ExchangeName) 181 | 182 | // when fail push queue data to first delay queue 183 | header := amqp.Table{"x-delay": nextDelayTime * 1000} 184 | if "aliyun" == strings.ToLower(qi.Source.Type) { 185 | header = amqp.Table{"delay": nextDelayTime * 1000} 186 | } 187 | 188 | var deliveryMode uint8 = 1 // Transient (0 or 1) or Persistent (2) 189 | if qi.Queue.IsDurable { 190 | deliveryMode = 2 191 | } 192 | err = ch.Publish( 193 | exchangeNameDelay, // exchange 194 | routingKeyDelay, // routing key 195 | false, // mandatory 196 | false, // immediate 197 | amqp.Publishing{ 198 | Headers: header, 199 | ContentType: "text/plain", 200 | Body: serializeDelayQueueData, 201 | DeliveryMode: deliveryMode, 202 | }) 203 | 204 | if nil != err { 205 | log.WithFields(log.Fields{ 206 | "error": err, 207 | "queueName": queueNameDelay, 208 | "exchangeName": exchangeNameDelay, 209 | }).Warn("Publish to rabbitmq failure") 210 | } 211 | status = "Normal Delayed" 212 | log.WithFields(log.Fields{ 213 | "status": status, 214 | "queueName": queueNameDelay, 215 | "exchangeName": exchangeNameDelay, 216 | "routingKeyDelay": routingKeyDelay, 217 | "trigglerTime": time.Now().Add(time.Duration(nextDelayTime) * time.Second), 218 | }).Info("Delayed to queue") 219 | } else { 220 | status = "Normal Failure" 221 | statistic.IncrFailureCounter(qi.Queue.QueueName) 222 | } 223 | 224 | d.Ack(false) 225 | } else { 226 | status = "Normal Acked" 227 | statistic.IncrSuccessCounter(qi.Queue.QueueName) 228 | d.Ack(false) 229 | } 230 | } 231 | 232 | log.WithFields(log.Fields{ 233 | "status": status, 234 | "queueName": qi.Queue.QueueName, 235 | "queueData": queueData, 236 | }).Info("Messages from queue") 237 | 238 | <-concurency // remove control 239 | }(delivery) 240 | } 241 | } 242 | 243 | // ProcessDelay to deal with delay queue 244 | func (qi *QueueInstance) ProcessDelay(runMode string) { 245 | queueName := "" 246 | queueNameDelay := "" 247 | routingKey := "" 248 | routingKeyDelay := "" 249 | exchangeName := "" 250 | exchangeNameDelay := "" 251 | if "retry" == runMode { 252 | queueName = fmt.Sprintf("%s.delayed", qi.Queue.QueueName) 253 | routingKey = fmt.Sprintf("%s.delayed", qi.Queue.RoutingKey) 254 | exchangeName = fmt.Sprintf("%s.delayed", qi.Queue.ExchangeName) 255 | queueNameDelay = queueName 256 | routingKeyDelay = routingKey 257 | exchangeNameDelay = exchangeName 258 | 259 | } else { 260 | // if runMode is first means it is the main delay queue 261 | queueName = qi.Queue.QueueName 262 | routingKey = qi.Queue.RoutingKey 263 | exchangeName = qi.Queue.ExchangeName 264 | queueNameDelay = fmt.Sprintf("%s.delayed", qi.Queue.QueueName) 265 | routingKeyDelay = fmt.Sprintf("%s.delayed", qi.Queue.RoutingKey) 266 | exchangeNameDelay = fmt.Sprintf("%s.delayed", qi.Queue.ExchangeName) 267 | } 268 | 269 | conn, err := qi.GetConnection() 270 | defer conn.Close() 271 | 272 | ch, err := conn.Channel() 273 | failOnError(err, "Failed to open a channel") 274 | defer ch.Close() 275 | 276 | // all queue is delay below ProcessDelay function 277 | // ExchangeDeclare(name, kind string, durable, autoDelete, internal, noWait bool, args Table) error 278 | if "aliyun" != strings.ToLower(qi.Source.Type) { 279 | err = ch.ExchangeDeclare(exchangeName, "x-delayed-message", qi.Queue.IsDurable, false, false, false, amqp.Table{"x-delayed-type": "direct"}) 280 | } else { 281 | err = ch.ExchangeDeclare(exchangeName, qi.Queue.ExchangeType, qi.Queue.IsDurable, false, false, false, nil) 282 | } 283 | failOnError(err, "Failed to Declare a exchange") 284 | 285 | // QueueDeclare(name string, durable, autoDelete, exclusive, noWait bool, args Table) (Queue, error) 286 | q, err := ch.QueueDeclare( 287 | queueName, // name 288 | qi.Queue.IsDurable, // durable 289 | false, // delete when unused 290 | false, // exclusive 291 | false, // no-wait 292 | nil, // arguments 293 | ) 294 | failOnError(err, "Failed to declare a queue"+q.Name) 295 | 296 | // QueueBind(name, key, exchange string, noWait bool, args Table) error 297 | err = ch.QueueBind(queueName, routingKey, exchangeName, false, nil) 298 | failOnError(err, "Failed to bind a queue") 299 | 300 | // auto-ack exclusive 301 | // Consume(queue, consumer string, autoAck, exclusive, noLocal, noWait bool, args Table) (<-chan Delivery, error) 302 | msgs, err := ch.Consume( 303 | queueName, // queue 304 | qi.Queue.ConsumerTag, // consumer 305 | false, // auto-ack 306 | false, // exclusive 307 | false, // no-local 308 | false, // no-wait 309 | nil, // args 310 | ) 311 | failOnError(err, "Failed to register a consumer") 312 | 313 | log.Printf("Start consume ( %s %s) queue (%s) from Exchange (%s) Vhost (%s)", qi.Queue.SourceType, qi.Source.Type, queueName, qi.Queue.ExchangeType, conn.Config.Vhost) 314 | 315 | // control the concurency 316 | concurency := make(chan bool, qi.Queue.Concurency) 317 | 318 | for delivery := range msgs { 319 | // control the concurency 320 | concurency <- true 321 | 322 | // dispatch to URLs 323 | go func(d amqp.Delivery) { 324 | queueData, nextDelayTime, delayTime, err := types.UnserializeDelayQueueData(runMode, string(d.Body), qi.Queue.DelayOnFailure) 325 | if nil != err { 326 | log.WithFields(log.Fields{ 327 | "error": err, 328 | "queueName": queueName, 329 | "exchangeName": exchangeName, 330 | }).Warn("DelayPop Unmarshal error") 331 | 332 | <-concurency // remove control 333 | return 334 | } 335 | 336 | log.WithFields(log.Fields{ 337 | "queueName": queueName, 338 | "queueData": queueData, 339 | }).Info("Queue data") 340 | 341 | queueRequest := &request.QueueRequest{ 342 | QueueName: qi.Queue.QueueName, 343 | DelayQueueName: queueName, 344 | DispatchURL: qi.Queue.DispatchURL, 345 | DispatchTimeout: qi.Queue.DispatchTimeout, 346 | QueueData: queueData, 347 | } 348 | 349 | result, err := queueRequest.Post() 350 | 351 | status := "" 352 | if qi.Queue.IsAutoAck { 353 | d.Ack(false) 354 | status = "Auto acked" 355 | statistic.IncrSuccessCounter(qi.Queue.QueueName) 356 | } else { 357 | // failure 358 | if 1 != result.Code { 359 | log.WithFields(log.Fields{ 360 | "err": err, 361 | "Message": result.Message, 362 | "queueName": queueName, 363 | "nextDelayTime": nextDelayTime, 364 | "delayTime": delayTime, 365 | }).Warn("Request data result") 366 | 367 | if nextDelayTime > 0 { 368 | serializeDelayQueueData, err := types.SerializeDelayQueueData(queueData, nextDelayTime) 369 | 370 | if nil != err { 371 | log.WithFields(log.Fields{ 372 | "error": err, 373 | "Message": result.Message, 374 | }).Warn("Serialize DelayQueueData error") 375 | 376 | d.Ack(false) // when a error occur acked 377 | statistic.IncrFailureCounter(qi.Queue.QueueName) 378 | <-concurency // remove control 379 | return 380 | } 381 | 382 | // when fail push queue data to first delay queue 383 | header := amqp.Table{"x-delay": (nextDelayTime - delayTime) * 1000} 384 | if "aliyun" == strings.ToLower(qi.Source.Type) { 385 | header = amqp.Table{"delay": (nextDelayTime - delayTime) * 1000} 386 | } 387 | 388 | var deliveryMode uint8 = 1 // Transient (0 or 1) or Persistent (2) 389 | if qi.Queue.IsDurable { 390 | deliveryMode = 2 391 | } 392 | err = ch.Publish( 393 | exchangeNameDelay, // exchange 394 | routingKeyDelay, // routing key 395 | false, // mandatory 396 | false, // immediate 397 | amqp.Publishing{ 398 | Headers: header, 399 | ContentType: "text/plain", 400 | Body: serializeDelayQueueData, 401 | DeliveryMode: deliveryMode, 402 | }) 403 | 404 | if nil != err { 405 | log.WithFields(log.Fields{ 406 | "error": err, 407 | "queueName": queueNameDelay, 408 | "exchangeName": exchangeNameDelay, 409 | }).Warn("Publish to rabbitmq failure") 410 | } 411 | 412 | status = "Delayed" 413 | 414 | log.WithFields(log.Fields{ 415 | "status": status, 416 | "queueName": queueNameDelay, 417 | "exchangeName": exchangeNameDelay, 418 | "routingKeyDelay": routingKeyDelay, 419 | "trigglerTime": time.Now().Add(time.Duration(nextDelayTime-delayTime) * time.Second), 420 | }).Info("Delayed to queue") 421 | } else { 422 | status = "Failure" 423 | statistic.IncrFailureCounter(qi.Queue.QueueName) 424 | } 425 | 426 | d.Ack(false) // we alrealy pushed failure message to delay queue, so mark as acked 427 | } else { 428 | status = "Success" 429 | statistic.IncrSuccessCounter(qi.Queue.QueueName) 430 | d.Ack(false) 431 | } 432 | } 433 | 434 | log.WithFields(log.Fields{ 435 | "status": status, 436 | "queueName": queueName, 437 | "queueData": queueData, 438 | }).Info("Messages from queue") 439 | <-concurency // remove control 440 | }(delivery) 441 | } 442 | } 443 | 444 | func failOnError(err error, msg string) { 445 | if err != nil { 446 | log.Fatalf("%s: %s", msg, err) 447 | } 448 | } 449 | -------------------------------------------------------------------------------- /libs/queue/rabbitmq/types.go: -------------------------------------------------------------------------------- 1 | package rabbitmq 2 | 3 | // Queue to bind functions 4 | type Queue struct{} 5 | 6 | // CombineConfig for the RabbitMQ 7 | type CombineConfig struct { 8 | Config Config 9 | Queues []QueueConfig 10 | } 11 | 12 | // QueueConfig for RabbitMQ queue 13 | type QueueConfig struct { 14 | SourceType string // queue type example "RabbitMQ" 15 | IsEnabled bool // this config is enabled 16 | IsDelayQueue bool // the queue is a delay queue 17 | IsDurable bool // true Durable false not 18 | ExchangeName string // name of the exchange 19 | ExchangeType string // type of the exchange 20 | QueueName string // name of the queue 21 | RoutingKey string // name of the routing key 22 | ConsumerTag string // name of the consumer tag 23 | IsAutoAck bool // true for auto ack 24 | DispatchURL string // URL to dispatch 25 | DispatchTimeout int // timeout for dispatch. 0 is unlimited 26 | Concurency int // dispatch concurency number 27 | DelayConcurency int // delay queue dispatch concurency number 28 | DelayOnFailure []int // when failed delay seconds to retry 29 | } 30 | 31 | // QueueInstance a sigle queue configure with 32 | type QueueInstance struct { 33 | Source Config 34 | Queue QueueConfig 35 | } 36 | -------------------------------------------------------------------------------- /libs/queue/redis/connection.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "encoding/json" 5 | "errors" 6 | "time" 7 | 8 | "github.com/gomodule/redigo/redis" 9 | uuid "github.com/satori/go.uuid" 10 | log "github.com/sirupsen/logrus" 11 | ) 12 | 13 | // DelayData struct 14 | type DelayData struct { 15 | UUID string `json:"uuid"` // UUID for delay value 16 | Time int64 `json:"time"` // the unix timestamp to trigger 17 | Data string `json:"data"` // the queue origin data 18 | } 19 | 20 | // Consume a list queue 21 | func (qi *QueueInstance) Consume(queueName string) (<-chan string, error) { 22 | deliveries := make(chan string, 200) 23 | concurency := make(chan bool, 200) 24 | 25 | go func() { 26 | for { 27 | concurency <- true 28 | go func() { 29 | defer func() { 30 | <-concurency 31 | }() 32 | 33 | d, err := qi.Pop(queueName) 34 | if err != nil { 35 | if nil != err && "redigo: nil returned" != err.Error() { 36 | log.WithFields(log.Fields{ 37 | "queueName": queueName, 38 | "Process handler has a error": err.Error(), 39 | }).Warn("!!! ProcessDelay handler has a error") 40 | } 41 | 42 | // if any error occured sleep a while 43 | time.Sleep(1 * time.Second) 44 | return 45 | } 46 | 47 | deliveries <- d 48 | }() 49 | } 50 | }() 51 | 52 | return deliveries, nil 53 | } 54 | 55 | // Pop a element 56 | func (qi *QueueInstance) Pop(queueName string) (string, error) { 57 | conn, _ := qi.GetConnection() 58 | defer conn.Close() 59 | 60 | reply, err := conn.Do("RPOP", queueName) 61 | 62 | if nil != err { 63 | return "", err 64 | } 65 | replyStr, err := redis.String(reply, err) 66 | if nil != err { 67 | return "", err 68 | } 69 | 70 | return replyStr, nil 71 | } 72 | 73 | // Push a element 74 | func (qi *QueueInstance) Push(queueName string, value string) (bool, error) { 75 | conn, _ := qi.GetConnection() 76 | defer conn.Close() 77 | 78 | reply, err := conn.Do("LPUSH", queueName, value) 79 | 80 | if nil != err { 81 | return false, err 82 | } 83 | replyInt, err := redis.Int(reply, err) 84 | if nil != err { 85 | return false, err 86 | } 87 | 88 | if replyInt < 1 { 89 | return false, errors.New("Push 0 element to queue") 90 | } 91 | 92 | return true, nil 93 | } 94 | 95 | // Length a queue's length 96 | func (qi *QueueInstance) Length(queueName string) (int64, error) { 97 | conn, _ := qi.GetConnection() 98 | defer conn.Close() 99 | 100 | reply, err := conn.Do("LLen", queueName) 101 | 102 | if nil != err { 103 | return 0, err 104 | } 105 | replyInt, err := redis.Int64(reply, err) 106 | if nil != err { 107 | return 0, err 108 | } 109 | 110 | return replyInt, nil 111 | } 112 | 113 | // DelayPop pop data from delay queue 114 | func (qi *QueueInstance) DelayPop(queueName string, isReturnRaw bool) ([]string, error) { 115 | conn, _ := qi.GetConnection() 116 | defer conn.Close() 117 | 118 | min := 0 119 | max := time.Now().Unix() 120 | // todo when number is to large, how to deal with 121 | // zrangebyscore test 0 2 LIMIT 2 4 122 | conn.Send("MULTI") 123 | conn.Send("ZRANGEBYSCORE", queueName, min, max) 124 | conn.Send("ZREMRANGEBYSCORE", queueName, min, max) 125 | reply, err := conn.Do("EXEC") 126 | 127 | result := []string{} 128 | replys, err := redis.MultiBulk(reply, err) 129 | if nil != err { 130 | return result, err 131 | } 132 | 133 | // not resut, return empty result 134 | if len(replys) <= 0 || replys[1].(int64) <= 0 { 135 | return result, nil 136 | } 137 | 138 | // has result but length less than 0 139 | replys2, err := redis.MultiBulk(replys[0], err) 140 | if len(replys2) <= 0 { 141 | return result, nil 142 | } 143 | 144 | for _, v := range replys2 { 145 | if isReturnRaw { 146 | result = append(result, string(v.([]byte))) 147 | } else { 148 | vv := &DelayData{} 149 | err := json.Unmarshal(v.([]byte), vv) 150 | if nil != err { 151 | continue 152 | } 153 | 154 | result = append(result, vv.Data) 155 | } 156 | } 157 | 158 | return result, err 159 | } 160 | 161 | // DelayPush push a element to a delayqueue 162 | func (qi *QueueInstance) DelayPush(queueName string, value string, delayUnixTime int64) (bool, error) { 163 | conn, _ := qi.GetConnection() 164 | defer conn.Close() 165 | 166 | uniqueID := uuid.NewV4().String() 167 | 168 | // Json encode 169 | delayData := &DelayData{ 170 | UUID: uniqueID, 171 | Time: delayUnixTime, 172 | Data: value, 173 | } 174 | 175 | jsonStr, err := json.Marshal(delayData) 176 | 177 | if nil != err { 178 | return false, err 179 | } 180 | 181 | reply, err := conn.Do("ZADD", queueName, "NX", delayUnixTime, jsonStr) 182 | 183 | if nil != err { 184 | return false, err 185 | } 186 | replyInt, err := redis.Int(reply, err) 187 | if nil != err { 188 | return false, err 189 | } 190 | 191 | if replyInt < 1 { 192 | return false, errors.New("Push 0 element to delay queue") 193 | } 194 | 195 | return true, nil 196 | } 197 | 198 | // DelayLength a delay queue's length 199 | func (qi *QueueInstance) DelayLength(queueName string) (int64, error) { 200 | conn, _ := qi.GetConnection() 201 | defer conn.Close() 202 | 203 | reply, err := conn.Do("ZCOUNT", queueName, "-INF", "+INF") 204 | 205 | if nil != err { 206 | return 0, err 207 | } 208 | replyInt, err := redis.Int64(reply, err) 209 | if nil != err { 210 | return 0, err 211 | } 212 | 213 | return replyInt, nil 214 | } 215 | -------------------------------------------------------------------------------- /libs/queue/redis/dispatcher.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "fmt" 5 | "queueman/libs/queue/types" 6 | "queueman/libs/request" 7 | "queueman/libs/statistic" 8 | "time" 9 | 10 | "github.com/gomodule/redigo/redis" 11 | "github.com/marknown/oredis" 12 | log "github.com/sirupsen/logrus" 13 | ) 14 | 15 | // Dispatcher for Queue 16 | func (queue *Queue) Dispatcher(combineConfig interface{}) { 17 | if cc, ok := combineConfig.(CombineConfig); ok { 18 | for _, queueConfig := range cc.Queues { 19 | if queueConfig.IsEnabled { 20 | queueConfig.SourceType = "Redis" 21 | // enabledTotal++ todo 22 | qi := &QueueInstance{ 23 | Source: cc.Config, 24 | Queue: queueConfig, 25 | } 26 | go qi.QueueHandle() 27 | } 28 | } 29 | } else { 30 | log.WithFields(log.Fields{ 31 | "queueConfig": combineConfig, 32 | }).Warn("Not correct redis config") 33 | 34 | return 35 | } 36 | } 37 | 38 | // GetConnection get a connect to source 39 | func (qi *QueueInstance) GetConnection() (redis.Conn, error) { 40 | conn := oredis.GetInstance(qi.Source) 41 | 42 | return conn, nil 43 | } 44 | 45 | // QueueHandle handle Normal or Delay queue 46 | func (qi *QueueInstance) QueueHandle() { 47 | if qi.Queue.Concurency < 1 { 48 | log.WithFields(log.Fields{ 49 | "queueName": qi.Queue.QueueName, 50 | }).Warn("Configure file of queue must have Concurency configure !!!") 51 | return 52 | } 53 | 54 | // process main queue 55 | // get the delay time 56 | delayTime := 0 57 | if len(qi.Queue.DelayOnFailure) > 0 { 58 | delayTime = qi.Queue.DelayOnFailure[0] 59 | } 60 | 61 | // first process delay queue 62 | totalDelays := len(qi.Queue.DelayOnFailure) 63 | if totalDelays > 0 { 64 | go qi.ProcessDelay("retry") 65 | } 66 | 67 | // wait ProcessDelay finished 68 | time.Sleep(1 * time.Second) 69 | 70 | if false == qi.Queue.IsDelayQueue { 71 | go qi.ProcessNormal(delayTime) 72 | } else { 73 | go qi.ProcessDelay("first") 74 | } 75 | } 76 | 77 | // ProcessNormal get a data from queue and dispatch 78 | func (qi *QueueInstance) ProcessNormal(delayTime int) { 79 | // log.WithFields(log.Fields{ 80 | // "queueName": qi.Queue.QueueName, 81 | // }).Info("Queue process begin") 82 | log.Printf("Start consume ( %s ) queue (%s)", qi.Queue.SourceType, qi.Queue.QueueName) 83 | 84 | // control the concurency 85 | concurency := make(chan bool, qi.Queue.Concurency) 86 | 87 | deliveries, _ := qi.Consume(qi.Queue.QueueName) 88 | 89 | for queueData := range deliveries { 90 | // control the concurency 91 | concurency <- true 92 | 93 | log.WithFields(log.Fields{ 94 | "queueData": queueData, 95 | }).Info("Get data from queue") 96 | 97 | // dispatch to URLs 98 | go func(queueData string) { 99 | queueRequest := &request.QueueRequest{ 100 | QueueName: qi.Queue.QueueName, 101 | DispatchURL: qi.Queue.DispatchURL, 102 | DispatchTimeout: qi.Queue.DispatchTimeout, 103 | QueueData: queueData, 104 | } 105 | 106 | result, err := queueRequest.Post() 107 | 108 | status := "" 109 | // failure 110 | if 1 != result.Code { 111 | log.WithFields(log.Fields{ 112 | "err": err, 113 | "Message": result.Message, 114 | "queueName": qi.Queue.QueueName, 115 | "delayTime": delayTime, 116 | }).Warn("Request data result") 117 | 118 | if delayTime > 0 { 119 | // when fail push queue data to first delay queue 120 | serializeDelayQueueData, err := types.SerializeDelayQueueData(queueData, delayTime) 121 | 122 | if nil != err { 123 | log.WithFields(log.Fields{ 124 | "error": err, 125 | "Message": result.Message, 126 | }).Warn("Serialize DelayQueueData error") 127 | 128 | statistic.IncrFailureCounter(qi.Queue.QueueName) 129 | <-concurency // remove control 130 | return 131 | } 132 | 133 | qi.DelayPush(fmt.Sprintf("%s:delayed", qi.Queue.QueueName), string(serializeDelayQueueData), time.Now().Unix()+int64(delayTime)) 134 | 135 | status = "Normal Delayed" 136 | log.WithFields(log.Fields{ 137 | "status": status, 138 | "queueName": qi.Queue.QueueName, 139 | "trigglerTime": time.Now().Add(time.Duration(delayTime) * time.Second), 140 | }).Info("Delayed to queue") 141 | } else { 142 | // finally also failure 143 | status = "Normal Failure" 144 | statistic.IncrFailureCounter(qi.Queue.QueueName) 145 | } 146 | } else { 147 | status = "Normal Acked" 148 | statistic.IncrSuccessCounter(qi.Queue.QueueName) 149 | } 150 | 151 | log.WithFields(log.Fields{ 152 | "status": status, 153 | "queueName": qi.Queue.QueueName, 154 | "queueData": queueData, 155 | }).Info("Messages from queue") 156 | 157 | <-concurency // remove control 158 | }(queueData) 159 | } 160 | } 161 | 162 | // ProcessDelay to deal with delay queue 163 | func (qi *QueueInstance) ProcessDelay(runMode string) { 164 | queueName := "" 165 | queueNameDelay := "" 166 | if "retry" == runMode { 167 | queueName = fmt.Sprintf("%s:delayed", qi.Queue.QueueName) 168 | queueNameDelay = queueName 169 | } else { 170 | // if runMode is first means it is the main delay queue 171 | queueName = qi.Queue.QueueName 172 | queueNameDelay = fmt.Sprintf("%s:delayed", qi.Queue.QueueName) 173 | } 174 | 175 | // log.WithFields(log.Fields{ 176 | // "queueName": queueName, 177 | // }).Info("Queue process begin") 178 | log.Printf("Start consume ( %s ) queue (%s)", qi.Queue.SourceType, queueName) 179 | 180 | // control the concurency 181 | concurency := make(chan bool, qi.Queue.DelayConcurency) 182 | 183 | for { 184 | // control the concurency 185 | concurency <- true 186 | 187 | // runMode is "first" and is main queue and configure file's IsDelayRaw is true 188 | isReturnRaw := false 189 | if "first" == runMode && qi.Queue.IsDelayRaw { 190 | isReturnRaw = true 191 | } 192 | queueDatas, err := qi.DelayPop(queueName, isReturnRaw) 193 | 194 | if nil != err || len(queueDatas) < 1 { 195 | <-concurency // remove control 196 | 197 | if nil != err && "redigo: nil returned" != err.Error() { 198 | log.WithFields(log.Fields{ 199 | "queueName": queueName, 200 | "Process handler has a error": err.Error(), 201 | }).Warn("!!! ProcessDelay handler has a error") 202 | } 203 | 204 | time.Sleep(1 * time.Second) 205 | continue 206 | } 207 | 208 | <-concurency // remove control 209 | 210 | log.WithFields(log.Fields{ 211 | "queueName": queueName, 212 | "length": len(queueDatas), 213 | }).Info("Get datas from delay queue") 214 | 215 | for _, queueData := range queueDatas { 216 | // control the concurency 217 | concurency <- true 218 | 219 | // dispatch to URLs 220 | go func(d string) { 221 | queueData, nextDelayTime, delayTime, err := types.UnserializeDelayQueueData(runMode, string(d), qi.Queue.DelayOnFailure) 222 | if nil != err { 223 | log.WithFields(log.Fields{ 224 | "error": err, 225 | "queueName": queueName, 226 | }).Warn("DelayPop Unmarshal error") 227 | return 228 | } 229 | 230 | log.WithFields(log.Fields{ 231 | "queueName": queueName, 232 | "queueData": queueData, 233 | }).Info("Queue data") 234 | 235 | queueRequest := &request.QueueRequest{ 236 | QueueName: qi.Queue.QueueName, 237 | DelayQueueName: queueName, 238 | DispatchURL: qi.Queue.DispatchURL, 239 | DispatchTimeout: qi.Queue.DispatchTimeout, 240 | QueueData: queueData, 241 | } 242 | 243 | result, err := queueRequest.Post() 244 | 245 | status := "" 246 | // failure 247 | if 1 != result.Code { 248 | log.WithFields(log.Fields{ 249 | "err": err, 250 | "Message": result.Message, 251 | "queueName": queueName, 252 | "nextDelayTime": nextDelayTime, 253 | "delayTime": delayTime, 254 | }).Warn("Request data result") 255 | 256 | if nextDelayTime > 0 { 257 | serializeDelayQueueData, err := types.SerializeDelayQueueData(queueData, nextDelayTime) 258 | 259 | if nil != err { 260 | log.WithFields(log.Fields{ 261 | "error": err, 262 | "Message": result.Message, 263 | }).Warn("Serialize DelayQueueData error") 264 | 265 | statistic.IncrFailureCounter(qi.Queue.QueueName) 266 | <-concurency // remove control 267 | return 268 | } 269 | 270 | // when fail push queue data to first delay queue 271 | qi.DelayPush(queueNameDelay, string(serializeDelayQueueData), time.Now().Unix()+int64(nextDelayTime)-int64(delayTime)) 272 | 273 | status = "Delayed" 274 | log.WithFields(log.Fields{ 275 | "status": status, 276 | "queueName": queueNameDelay, 277 | "trigglerTime": time.Now().Add(time.Duration(nextDelayTime-delayTime) * time.Second), 278 | }).Info("Delayed to queue") 279 | } else { 280 | // finally also failure 281 | status = "Failure" 282 | statistic.IncrFailureCounter(qi.Queue.QueueName) 283 | } 284 | } else { 285 | status = "Success" 286 | statistic.IncrSuccessCounter(qi.Queue.QueueName) 287 | } 288 | 289 | log.WithFields(log.Fields{ 290 | "status": status, 291 | "queueName": queueName, 292 | "queueData": queueData, 293 | }).Info("Messages from queue") 294 | 295 | <-concurency // remove control 296 | }(queueData) 297 | } 298 | } 299 | } 300 | -------------------------------------------------------------------------------- /libs/queue/redis/types.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "github.com/marknown/oredis" 5 | ) 6 | 7 | // Queue to bind functions 8 | type Queue struct{} 9 | 10 | // CombineConfig for the redis 11 | type CombineConfig struct { 12 | Config oredis.Config 13 | Queues []QueueConfig 14 | } 15 | 16 | // QueueConfig for redis queue 17 | type QueueConfig struct { 18 | SourceType string // queue type example "Redis" 19 | IsEnabled bool // this config is enabled 20 | IsDelayQueue bool // the queue is a delay queue 21 | IsDelayRaw bool // your format for redis zset delay queue, not the standard's DelayData 22 | QueueName string // name of the queue 23 | DispatchURL string // URL to dispatch 24 | DispatchTimeout int // timeout for dispatch. 0 is unlimited 25 | Concurency int // dispatch concurency number 26 | DelayConcurency int // delay queue dispatch concurency number 27 | DelayOnFailure []int // when failed delay seconds to retry 28 | } 29 | 30 | // QueueInstance a sigle queue configure with 31 | type QueueInstance struct { 32 | Source oredis.Config 33 | Queue QueueConfig 34 | } 35 | -------------------------------------------------------------------------------- /libs/queue/types/types.go: -------------------------------------------------------------------------------- 1 | package types 2 | 3 | import ( 4 | "encoding/json" 5 | "time" 6 | ) 7 | 8 | // DelayQueueData struct 9 | type DelayQueueData struct { 10 | Data string `json:"data"` // the queue origin data 11 | DelayTime int `json:"delaytime"` // the delay time, unit is second 12 | TriggerTime time.Time `json:"triggertime"` // the unix timestamp to trigger 13 | } 14 | 15 | // SerializeDelayQueueData serialize delay queue data for a auto run delay queue 16 | func SerializeDelayQueueData(data string, delayTime int) ([]byte, error) { 17 | // Json encode 18 | delayQueueData := &DelayQueueData{ 19 | Data: data, 20 | DelayTime: delayTime, 21 | TriggerTime: time.Now().Add(time.Duration(delayTime) * time.Second), 22 | } 23 | 24 | jsonStr, err := json.Marshal(delayQueueData) 25 | 26 | if nil != err { 27 | return []byte{}, err 28 | } 29 | 30 | return jsonStr, nil 31 | } 32 | 33 | // UnserializeDelayQueueData unserialize delay queue data for a auto run delay queue 34 | func UnserializeDelayQueueData(runMode string, data string, delayOnFailure []int) (string, int, int, error) { 35 | queueData := "" 36 | nextDelayTime := 0 37 | delayTime := 0 38 | 39 | // log.Printf("runMode %s ", runMode) 40 | // log.Printf("data %s ", data) 41 | // log.Printf("delayOnFailure %v ", delayOnFailure) 42 | 43 | if "retry" == runMode { 44 | delayQueueData := &DelayQueueData{} 45 | err := json.Unmarshal([]byte(data), delayQueueData) 46 | if nil != err { 47 | return "", 0, 0, err 48 | } 49 | 50 | queueData = delayQueueData.Data 51 | 52 | if len(delayOnFailure) > 0 { 53 | for _, dt := range delayOnFailure { 54 | if delayQueueData.DelayTime < dt { 55 | nextDelayTime = dt 56 | break 57 | } 58 | } 59 | } 60 | delayTime = delayQueueData.DelayTime 61 | } else { 62 | queueData = data 63 | if len(delayOnFailure) > 0 { 64 | nextDelayTime = delayOnFailure[0] 65 | } 66 | delayTime = 0 67 | } 68 | // log.Printf("delayTime %d ", delayTime) 69 | // log.Printf("nextDelayTime %d \n", nextDelayTime) 70 | 71 | return queueData, nextDelayTime, delayTime, nil 72 | } 73 | -------------------------------------------------------------------------------- /libs/request/request.go: -------------------------------------------------------------------------------- 1 | package request 2 | 3 | import ( 4 | "encoding/json" 5 | "queueman/libs/constant" 6 | "queueman/libs/ohttp" 7 | "time" 8 | 9 | log "github.com/sirupsen/logrus" 10 | ) 11 | 12 | // QueueResponse for queue request 13 | type QueueResponse struct { 14 | Code int // return 1 success, and 0 failure 15 | Message string // when code is 1 then message is "ok" otherwise message can be other info 16 | HTTPCode int // the HTTPCode for request 17 | } 18 | 19 | // QueueRequest for queue dispather 20 | type QueueRequest struct { 21 | QueueName string // name of the queue 22 | DelayQueueName string // name of the delay queue 23 | DispatchURL string // URL to dispatch 24 | DispatchTimeout int // timeout for dispatch. 0 is unlimited 25 | QueueData string // data for one element form queue 26 | UserAgent string // user agent for request 27 | } 28 | 29 | // Post for QueueRequest struct 30 | func (req *QueueRequest) Post() (*QueueResponse, error) { 31 | settings := ohttp.InitSetttings() 32 | settings.Timeout = time.Duration(req.DispatchTimeout) * time.Second 33 | settings.UserAgent = constant.APPNAME + " " + constant.APPVERSION 34 | 35 | params := map[string]string{ 36 | "queueName": req.QueueName, 37 | "delayName": req.DelayQueueName, 38 | "queueData": req.QueueData, 39 | } 40 | 41 | content, response, err := settings.Post(req.DispatchURL, params) 42 | 43 | result := &QueueResponse{ 44 | HTTPCode: 0, 45 | } 46 | 47 | // fixed a bug. when err is occour the response is nil, response.StatusCode will panic 48 | if err != nil { 49 | log.WithFields(log.Fields{ 50 | "err": err, 51 | "queueData": req.QueueData, 52 | "Response": content, 53 | }).Warn("Request post error") 54 | 55 | result.Code = 0 56 | result.Message = content 57 | return result, err 58 | } 59 | 60 | // init HTTPCode 61 | result.HTTPCode = response.StatusCode 62 | 63 | err = json.Unmarshal([]byte(content), &result) 64 | 65 | if nil != err { 66 | log.WithFields(log.Fields{ 67 | "err": err, 68 | "queueData": req.QueueData, 69 | "Response": content, 70 | }).Warn("Request unmarshal error") 71 | 72 | result.Code = 0 73 | result.Message = content 74 | return result, err 75 | } 76 | 77 | return result, nil 78 | } 79 | -------------------------------------------------------------------------------- /libs/statistic/redis.go: -------------------------------------------------------------------------------- 1 | package statistic 2 | 3 | import ( 4 | "github.com/gomodule/redigo/redis" 5 | "github.com/marknown/oredis" 6 | ) 7 | 8 | type redisConfig struct { 9 | RedisSource oredis.Config 10 | } 11 | 12 | // IncrCounter increase counter number 13 | func (c *redisConfig) IncrCounter(keyName string) (int64, error) { 14 | conn := oredis.GetInstance(c.RedisSource) 15 | defer conn.Close() 16 | 17 | reply, err := conn.Do("INCR", keyName) 18 | 19 | if nil != err { 20 | return 0, err 21 | } 22 | replyInt, err := redis.Int64(reply, err) 23 | if nil != err { 24 | return 0, err 25 | } 26 | 27 | return replyInt, nil 28 | } 29 | 30 | // GetCounter get counter number 31 | func (c *redisConfig) GetCounter(keyName string) (int64, error) { 32 | conn := oredis.GetInstance(c.RedisSource) 33 | defer conn.Close() 34 | 35 | reply, err := conn.Do("GET", keyName) 36 | 37 | if nil != err { 38 | return 0, err 39 | } 40 | replyInt, err := redis.Int64(reply, err) 41 | if nil != err { 42 | return 0, err 43 | } 44 | 45 | return replyInt, nil 46 | } 47 | -------------------------------------------------------------------------------- /libs/statistic/statistic.go: -------------------------------------------------------------------------------- 1 | package statistic 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "strings" 7 | 8 | "github.com/marknown/oredis" 9 | ) 10 | 11 | var packageConfig Config 12 | 13 | // Config configure for statistic 14 | type Config struct { 15 | HTTPPort int 16 | SourceType string 17 | RedisSource oredis.Config 18 | } 19 | 20 | // QueueStatistic queue statistic 21 | type QueueStatistic struct { 22 | QueueName string // 队列名称 23 | SourceType string // 队列类型 24 | IsEnabled bool // 队列是否启用 25 | Normal int64 // 主队列未处理的总数 26 | Delayed int64 // 延时队列未处理的总数 27 | Success int64 // 成功数量 28 | Failure int64 // 失败数量 29 | Total int64 // 以上总数合计 30 | } 31 | 32 | // InitStatistic init statistic configure 33 | func InitStatistic(c Config) { 34 | packageConfig = c 35 | } 36 | 37 | // IncrCounter Incr the special counter number 38 | func IncrCounter(queueName string) (int64, error) { 39 | if "redis" != strings.ToLower(packageConfig.SourceType) { 40 | return 0, nil 41 | } 42 | 43 | if "redis" == strings.ToLower(packageConfig.SourceType) { 44 | obj := &redisConfig{ 45 | RedisSource: packageConfig.RedisSource, 46 | } 47 | 48 | return obj.IncrCounter(queueName) 49 | } 50 | 51 | return 0, errors.New("Unsupport statistic source type") 52 | } 53 | 54 | // IncrSuccessCounter Incr the special counter number 55 | func IncrSuccessCounter(queueName string) (int64, error) { 56 | return IncrCounter(fmt.Sprintf("%s:success", queueName)) 57 | } 58 | 59 | // IncrFailureCounter Incr the special counter number 60 | func IncrFailureCounter(queueName string) (int64, error) { 61 | return IncrCounter(fmt.Sprintf("%s:failure", queueName)) 62 | } 63 | 64 | // GetCounter get the special counter number 65 | func GetCounter(queueName string) (int64, error) { 66 | if "redis" == strings.ToLower(packageConfig.SourceType) { 67 | obj := &redisConfig{ 68 | RedisSource: packageConfig.RedisSource, 69 | } 70 | 71 | return obj.GetCounter(queueName) 72 | } 73 | 74 | return 0, errors.New("Unsupport statistic source type") 75 | } 76 | 77 | // GetSuccessCounter get the special counter number 78 | func GetSuccessCounter(queueName string) (int64, error) { 79 | return GetCounter(fmt.Sprintf("%s:success", queueName)) 80 | } 81 | 82 | // GetFailureCounter get the special counter number 83 | func GetFailureCounter(queueName string) (int64, error) { 84 | return GetCounter(fmt.Sprintf("%s:failure", queueName)) 85 | } 86 | -------------------------------------------------------------------------------- /libs/utils/utils.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "os" 5 | "time" 6 | ) 7 | 8 | // NowTimeStringCN get now time and return china time format 9 | func NowTimeStringCN() string { 10 | var cstZone = time.FixedZone("CST", 8*3600) // UTC/GMT +08:00 11 | t := time.Now() 12 | return t.In(cstZone).Format("2006-01-02 15:04:05") 13 | } 14 | 15 | // NowDateStringCN get now date and return china date format 16 | func NowDateStringCN() string { 17 | var cstZone = time.FixedZone("CST", 8*3600) // UTC/GMT +08:00 18 | t := time.Now() 19 | return t.In(cstZone).Format("2006-01-02") 20 | } 21 | 22 | // Exists 判断所给路径文件/文件夹是否存在 23 | func Exists(path string) bool { 24 | _, err := os.Stat(path) //os.Stat获取文件信息 25 | if err != nil { 26 | if os.IsExist(err) { 27 | return true 28 | } 29 | return false 30 | } 31 | return true 32 | } 33 | 34 | // IsDir 判断所给路径是否为文件夹 35 | func IsDir(path string) bool { 36 | s, err := os.Stat(path) 37 | if err != nil { 38 | return false 39 | } 40 | return s.IsDir() 41 | } 42 | 43 | // IsFile 判断所给路径是否为文件 44 | func IsFile(path string) bool { 45 | return !IsDir(path) 46 | } 47 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "net/http" 7 | "os" 8 | "queueman/libs/command" 9 | "queueman/libs/config" 10 | "queueman/libs/constant" 11 | "queueman/libs/queue" 12 | "queueman/libs/statistic" 13 | "queueman/libs/utils" 14 | "runtime/debug" 15 | "time" 16 | 17 | "github.com/docker/docker/pkg/pidfile" 18 | log "github.com/sirupsen/logrus" 19 | ) 20 | 21 | func initLog() { 22 | log.SetOutput(os.Stdout) 23 | log.SetLevel(log.InfoLevel) 24 | } 25 | 26 | func setLogFormatter(formatter string) { 27 | if "json" == formatter { 28 | customFormatter := new(log.JSONFormatter) 29 | customFormatter.TimestampFormat = "2006-01-02 15:04:05" 30 | log.SetFormatter(customFormatter) 31 | } else { 32 | customFormatter := new(log.TextFormatter) 33 | customFormatter.TimestampFormat = "2006-01-02 15:04:05" 34 | customFormatter.FullTimestamp = true 35 | log.SetFormatter(customFormatter) 36 | } 37 | } 38 | 39 | func setLogFile(logDir string) { 40 | fileName := fmt.Sprintf("%s/queueman.%s.log", logDir, utils.NowDateStringCN()) 41 | var file, err = os.OpenFile(fileName, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666) 42 | if err != nil { 43 | log.WithFields(log.Fields{ 44 | "error": err, 45 | "fileName": fileName, 46 | }).Fatal(`Could Not Open Log File`) 47 | } else { 48 | log.SetOutput(file) 49 | } 50 | } 51 | 52 | func main() { 53 | // init logger 54 | initLog() 55 | 56 | // Get command args 57 | args := command.GetArgs() 58 | 59 | // Get config file content to struct 60 | cfg := config.GetConfig(args.ConfigFile) 61 | // init statistic for record 62 | statistic.InitStatistic(cfg.Statistic) 63 | 64 | if !cfg.App.IsDebug { 65 | log.SetLevel(log.WarnLevel) 66 | } 67 | 68 | if "" == cfg.App.LogFormatter { 69 | cfg.App.LogFormatter = "text" 70 | } 71 | setLogFormatter(cfg.App.LogFormatter) 72 | 73 | // write log to file 74 | if "" != cfg.App.LogDir { 75 | if !utils.IsDir(cfg.App.LogDir) { 76 | log.Warn("Log file dir is not exist or not a direcotry, please check it in your configure file.") 77 | } else { 78 | setLogFile(cfg.App.LogDir) 79 | go func() { 80 | for { 81 | select { 82 | case <-time.After(60 * time.Second): 83 | setLogFile(cfg.App.LogDir) 84 | } 85 | } 86 | }() 87 | } 88 | } 89 | 90 | // add the pidfile 91 | if "" == cfg.App.PIDFile { 92 | log.WithFields(log.Fields{ 93 | "error": errors.New("PIDFile option not configure in the configure file"), 94 | }).Fatal(`Please check the "PIDFile" option in configure file`) 95 | } 96 | 97 | // check & put the pid to the pid file 98 | pidHandle, err := pidfile.New(cfg.App.PIDFile) 99 | if nil != err { 100 | log.WithFields(log.Fields{ 101 | "error": err, 102 | }).Fatal("A pidfile error has occurred") 103 | } 104 | 105 | // catch the error 106 | defer func() { 107 | if err := recover(); err != nil { 108 | log.WithFields(log.Fields{ 109 | "error": err, 110 | }).Error("A error has occurred!") 111 | debug.PrintStack() 112 | 113 | if nil != pidHandle { 114 | err1 := pidHandle.Remove() 115 | if nil != err1 { 116 | log.WithFields(log.Fields{ 117 | "error": err1, 118 | }).Error("The pid file can not be deleted!") 119 | } 120 | } 121 | 122 | os.Exit(1) 123 | } 124 | }() 125 | 126 | // out put every time started 127 | log.Warnf("%s %s started at %s.", constant.APPNAME, constant.APPVERSION, utils.NowTimeStringCN()) 128 | 129 | for _, config := range cfg.Redis { 130 | go queue.QFactory("Redis").Dispatcher(config) 131 | } 132 | 133 | for _, config := range cfg.RabbitMQ { 134 | go queue.QFactory("RabbitMQ").Dispatcher(config) 135 | } 136 | 137 | if cfg.Statistic.HTTPPort > 0 { 138 | http.HandleFunc("/statistic", func(w http.ResponseWriter, r *http.Request) { 139 | format := r.FormValue("format") 140 | if "json" != format { 141 | format = "html" 142 | } 143 | 144 | fmt.Fprint(w, command.GetStats(args, format)) 145 | return 146 | }) 147 | 148 | err = http.ListenAndServe(fmt.Sprintf(":%d", cfg.Statistic.HTTPPort), nil) 149 | 150 | if err != nil { 151 | log.WithFields(log.Fields{ 152 | "error": err, 153 | }).Fatal("%s can not listening on port %d", constant.APPNAME, cfg.Statistic.HTTPPort) 154 | } 155 | } 156 | 157 | for { 158 | time.Sleep(5 * time.Second) 159 | } 160 | } 161 | -------------------------------------------------------------------------------- /queueman.json: -------------------------------------------------------------------------------- 1 | { 2 | "App": { 3 | "IsDebug" : false, 4 | "PIDFile" : "/var/run/queueman.pid", 5 | "LogFormatter": "text", 6 | "LogDir" : "/var/log/queueman/" 7 | }, 8 | "Statistic": { 9 | "HTTPPort": 8080, 10 | "SourceType": "Redis", 11 | "RedisSource" : { 12 | "Network" : "tcp", 13 | "Host" : "127.0.0.1", 14 | "Port" : 6379, 15 | "Password" : "", 16 | "DB" : 0, 17 | "Timeout" : 5, 18 | "MaxActive" : 1000, 19 | "MaxIdle" : 200, 20 | "MaxIdleTimeout": 10, 21 | "Wait" : true 22 | } 23 | }, 24 | "Redis": [ 25 | { 26 | "Config" : { 27 | "Network" : "tcp", 28 | "Host" : "127.0.0.1", 29 | "Port" : 6379, 30 | "Password" : "", 31 | "DB" : 0, 32 | "Timeout" : 5, 33 | "MaxActive" : 1000, 34 | "MaxIdle" : 200, 35 | "MaxIdleTimeout": 10, 36 | "Wait" : true 37 | }, 38 | "Queues": [ 39 | { 40 | "IsEnabled": true, 41 | "IsDelayQueue": false, 42 | "IsDelayRaw": false, 43 | "QueueName": "queue:test1", 44 | "DispatchURL": "http://127.0.0.1/examples/receive_from_queue", 45 | "DispatchTimeout": 30, 46 | "Concurency": 5, 47 | "DelayConcurency": 3, 48 | "DelayOnFailure": [60, 120] 49 | }, 50 | { 51 | "IsEnabled": true, 52 | "IsDelayQueue": true, 53 | "IsDelayRaw": false, 54 | "QueueName": "queue:test2", 55 | "DispatchURL": "http://127.0.0.1/examples/receive_from_queue", 56 | "DispatchTimeout": 30, 57 | "Concurency": 5, 58 | "DelayConcurency": 3, 59 | "DelayOnFailure": [60, 120] 60 | } 61 | ] 62 | } 63 | ], 64 | "RabbitMQ": [ 65 | { 66 | "Config" : { 67 | "Scheme" : "amqp", 68 | "Host" : "Replace your Host", 69 | "Port" : 5672, 70 | "User" : "", 71 | "Password" : "", 72 | "Vhost" : "Replace your Vhost", 73 | "Type" : "Aliyun", 74 | "AliyunParams" : { 75 | "AccessKey" : "Replace your AccessKey", 76 | "AccessKeySecret": "Replace your AccessKeySecret", 77 | "ResourceOwnerId": 1000000000000000 78 | } 79 | }, 80 | "Queues" : [ 81 | { 82 | "IsEnabled": true, 83 | "IsDelayQueue": false, 84 | "IsDurable": true, 85 | "ExchangeName": "test.exchange.direct1", 86 | "ExchangeType": "direct", 87 | "QueueName": "test.queue.direct1", 88 | "RoutingKey": "test.route.direct1", 89 | "ConsumerTag": "test.consumer.direct1", 90 | "IsAutoAck": false, 91 | "DispatchURL": "http://127.0.0.1/examples/receive_from_queue", 92 | "DispatchTimeout": 30, 93 | "Concurency": 5, 94 | "DelayConcurency": 3, 95 | "DelayOnFailure": [60, 120] 96 | }, 97 | { 98 | "IsEnabled": true, 99 | "IsDelayQueue": true, 100 | "IsDurable": true, 101 | "ExchangeName": "test.exchange.direct2", 102 | "ExchangeType": "direct", 103 | "QueueName": "test.queue.direct2", 104 | "RoutingKey": "test.route.direct2", 105 | "ConsumerTag": "test.consumer.direct2", 106 | "IsAutoAck": false, 107 | "DispatchURL": "http://127.0.0.1/examples/receive_from_queue", 108 | "DispatchTimeout": 30, 109 | "Concurency": 5, 110 | "DelayConcurency": 3, 111 | "DelayOnFailure": [60, 120] 112 | } 113 | ] 114 | }, 115 | { 116 | "Config" : { 117 | "Scheme" : "amqp", 118 | "Host" : "Replace your Host", 119 | "Port" : 5672, 120 | "User" : "Replace your User", 121 | "Password" : "Replace your Password", 122 | "Vhost" : "/", 123 | "Type" : "" 124 | }, 125 | "Queues" : [ 126 | { 127 | "IsEnabled": true, 128 | "IsDelayQueue": false, 129 | "IsDurable": true, 130 | "ExchangeName": "test.exchange.direct3", 131 | "ExchangeType": "direct", 132 | "QueueName": "test.queue.direct3", 133 | "RoutingKey": "test.route.direct3", 134 | "ConsumerTag": "test.consumer.direct3", 135 | "IsAutoAck": false, 136 | "DispatchURL": "http://127.0.0.1/examples/receive_from_queue", 137 | "DispatchTimeout": 30, 138 | "Concurency": 5, 139 | "DelayConcurency": 3, 140 | "DelayOnFailure": [60, 120] 141 | }, 142 | { 143 | "IsEnabled": true, 144 | "IsDelayQueue": true, 145 | "IsDurable": true, 146 | "ExchangeName": "test.exchange.direct4", 147 | "ExchangeType": "direct", 148 | "QueueName": "test.queue.direct4", 149 | "RoutingKey": "test.route.direct4", 150 | "ConsumerTag": "test.consumer.direct4", 151 | "IsAutoAck": false, 152 | "DispatchURL": "http://127.0.0.1/examples/receive_from_queue", 153 | "DispatchTimeout": 30, 154 | "Concurency": 5, 155 | "DelayConcurency": 3, 156 | "DelayOnFailure": [60, 120] 157 | } 158 | ] 159 | } 160 | ] 161 | } -------------------------------------------------------------------------------- /queueman.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=Queueman can dispatch queue data to your receiver URL via http(s) protocol 3 | Documentation=https://github.com/marknown/queueman 4 | After=network.target 5 | 6 | [Service] 7 | Type=simple 8 | PIDFile=/var/run/queueman.pid 9 | ExecStart=/usr/local/bin/queueman_linux -c /etc/queueman.json 10 | ExecStop=/bin/kill -s QUIT $MAINPID 11 | Restart=always 12 | RestartSec=1 13 | StartLimitInterval=0 14 | PrivateTmp=true 15 | #systemctl version great than 329 16 | #StandardOutput=append:/var/log/queueman/error.log 17 | #StandardError=append:/var/log/queueman/error.log 18 | 19 | [Install] 20 | WantedBy=multi-user.target -------------------------------------------------------------------------------- /supervisor.conf: -------------------------------------------------------------------------------- 1 | [program:queueman] 2 | process_name=%(program_name)s_%(process_num)02d 3 | directory = /usr/local/bin/ 4 | command=/usr/local/bin/queueman_linux -c /etc/queueman.json 5 | autostart=true 6 | autorestart=true 7 | user=root 8 | numprocs=1 9 | redirect_stderr=true 10 | stdout_logfile_maxbytes = 100MB 11 | stdout_logfile_backups = 20 12 | stdout_logfile=/var/log/queueman/error.log --------------------------------------------------------------------------------