├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── admin ├── admin.go ├── uAdmin.go └── uAdmin_test.go ├── entry ├── entry.go ├── httpEntry.go ├── httpEntry_test.go ├── mcEntry.go ├── mcEntry_test.go ├── mcResponse.go ├── redisCommand.go ├── redisEntry.go ├── redisEntry_test.go ├── redisQueue.go ├── redisReply.go ├── redisSession.go └── redisValidate.go ├── queue ├── fakeQ.go ├── queue.go ├── uEtcd.go ├── uLine.go ├── uQueue.go ├── uQueue_test.go ├── uStat.go ├── uTopic.go ├── uq.pb.go └── uq.proto ├── store ├── goleveldbStore.go ├── goleveldbStore_test.go ├── memStore.go ├── memStore_test.go ├── rocksdbStore.go └── storage.go ├── test ├── ab-post.sh ├── bench_reader.go ├── bench_writer.go ├── foo.dat ├── genHttpJsonReq.go ├── mc_bench.go ├── mc_bench_sc.go ├── post.lua ├── redis_bench.go ├── test.goconvey ├── testAcati.go └── wrk-post.sh ├── uq.go ├── uq_test.go ├── utils ├── error.go ├── error_test.go ├── httpUtils.go ├── limitedBufReader.go ├── limitedBufReader_test.go ├── stopListener.go ├── strings.go └── strings_test.go └── wercker.yml /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | data/ 3 | data1/ 4 | data2/ 5 | data3/ 6 | uq 7 | test/bench_reader 8 | test/bench_writer 9 | test/mc_bench 10 | test/mc_bench_sc 11 | test/redis_bench 12 | test/testAcati 13 | *.log 14 | *.out 15 | *.swp 16 | 17 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | 3 | go: 4 | - 1.5.3 5 | 6 | before_install: 7 | - go get -v github.com/axw/gocov/gocov 8 | - go get -v github.com/mattn/goveralls 9 | - go get -v github.com/golang/lint/golint 10 | 11 | install: 12 | - go get -d -t -v ./... 13 | - go build -v 14 | 15 | script: 16 | - go test -v ./admin 17 | - go test -v ./entry 18 | - go test -v ./queue 19 | - go test -v ./store 20 | - go test -v ./utils 21 | - go test -v . 22 | - go vet ./... 23 | - $HOME/gopath/bin/golint ./... 24 | - $HOME/gopath/bin/goveralls -service=travis-ci 25 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, 招牌疯子 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * Neither the name of uq nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | 29 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## UQ 2 | 3 | [![Build Status](https://travis-ci.org/buaazp/uq.svg)](https://travis-ci.org/buaazp/uq) [![wercker status](https://app.wercker.com/status/723a7608e0671075fed88d8f6489602a/s "wercker status")](https://app.wercker.com/project/bykey/723a7608e0671075fed88d8f6489602a) [![Build Status](https://semaphoreci.com/api/v1/projects/e157fc88-5f1a-4114-9cf5-d102913386c2/467082/shields_badge.svg)](https://semaphoreci.com/buaazp/uq) [![Coverage Status](https://coveralls.io/repos/buaazp/uq/badge.svg)](https://coveralls.io/r/buaazp/uq) 4 | 5 | Uq (also named `United Queue`, `Unique`) is another distributed message queue. It is written in Go and has many useful features including: 6 | 7 | - [multi client APIs support](#client-api) 8 | - [consume messages from multi lines of one topic](#topic-and-line) 9 | - [message confirmation and recycle](#message-confirmation-and-recycle) 10 | - [distributed cluster](#distributed-cluster) 11 | - [RESTful admin methods](#admin-api) 12 | - [multi ways of message persistence](#message-persistence) 13 | 14 | Messages in uq is very safe. It exists until a consumer confirms clearly. Consumer lines of a topic are independent. So one independent workflow can use a line to consume the messages in topic. A group of uq can make up a cluster. Cluster information is stored in [etcd](https://github.com/coreos/etcd). Clients can get the information from etcd and then connect to one or some uq servers in the cluster. Or you can use [libuq](https://github.com/buaazp/libuq) to communicate with uq cluster. 15 | 16 | Author: [@招牌疯子](http://weibo.com/819880808) 17 | Contact me: zp@buaa.us 18 | 19 | 20 | ### Getting Started 21 | 22 | Get uq with go get like below: 23 | 24 | ``` 25 | go get -u github.com/buaazp/uq 26 | uq -h 27 | Usage of ./uq: 28 | -admin-port=8809: admin listen port 29 | -cluster=“uq”: cluster name in etcd 30 | -db=“goleveldb”: backend storage type [goleveldb/memdb] 31 | -dir=“./data”: backend storage path 32 | -etcd=“”: etcd service location 33 | -host=“0.0.0.0”: listen ip 34 | -ip=“127.0.0.1”: self ip/host address 35 | -log=“”: uq log path 36 | -port=8808: listen port 37 | -protocol=“redis”: frontend interface type [redis/mc/http] 38 | ``` 39 | 40 | ### Concepts in UQ 41 | 42 | #### topic and line 43 | 44 | A topic is a class to store messages. Producers push messages into a topic. A line is a class to output messages for a clear purpose. A topic can have many lines for different purposes. For example: 45 | 46 | 1. Users upload pictures to [weibo.com](http://weibo.com). All the upload messages pushed into a topic named `wb_img_upload`. 47 | 2. Compress service pops the message from a line named `img_to_compress`, then compresses the picture and confirms the message in this line. 48 | 3. Analysis service pops the message from a line named `img_to_analysis`, then analyzes the picture and confirm the message in this line. 49 | 4. Any other services can pop and confirm messages from its own line. The status of different lines are independent. 50 | 51 | #### message confirmation and recycle 52 | 53 | Messages popped from a line should be confirmed after disposing. If a consumer pops a message but fails to dispose it, this message will be pushed back into the line after a recycle time which is set when creating the line. 54 | 55 | If a line is created with no recycle time. The line will degrade to a classical message queue, which means if a message is popped, it is lost. 56 | 57 | #### queue methods 58 | 59 | Uq defines a list of queue methods: 60 | 61 | - add tname = create a topic 62 | - add tname/lname 10s = create a line with the recycle time 63 | - push tname value = push a message into the topic 64 | - pop tname/lname = pop the latest message of the line 65 | - del tname/lname/mID = confirm the message according to the message ID 66 | 67 | Different protocols implement the queue methods above in its own way. But they are similar. 68 | 69 | #### admin methods 70 | 71 | And uq has some admin methods to manage the queue: 72 | 73 | - stat tname = get the topic’s status 74 | - stat tname/lname = get the line’s status 75 | - empty tname/lname = empty all the messages in a line 76 | - empty tname = empty all the messages in the topic and its lines 77 | - rm tname/lname = remove a line from the topic 78 | - rm tname = remove all lines of the topic and itself 79 | 80 | ### Client API 81 | 82 | Uq supports many client APIs like memcached, redis and http RESTful api. Choose the protocol you are most familiar with. 83 | 84 | #### memcached api 85 | 86 | Uq is designed to be an alternative to [memcacheQ](http://memcachedb.org/memcacheq/). And the author of mcq [@stvchu](http://stvchu.github.io/) provides a lot of advice for designing uq. Thank him a lot for his help. 87 | 88 | Start uq with mc protocol and connect uq with telnet: 89 | 90 | ``` 91 | uq -protocol mc 92 | telnet 127.0.0.1 8808 93 | ``` 94 | 95 | Send memcached requests like these: 96 | 97 | ``` 98 | // create a topic 99 | add foo 0 0 0 100 | 101 | STORED 102 | 103 | // create a line with 10s recycle time 104 | add foo/x 0 0 3 105 | 10s 106 | STORED 107 | 108 | // push a message into the topic 109 | set foo 0 0 3 110 | bar 111 | STORED 112 | 113 | // pop a message from the line 114 | get foo/x id 115 | VALUE foo/x 0 3 116 | bar 117 | VALUE id 0 7 118 | foo/x/0 119 | END 120 | 121 | // confirm a message 122 | delete foo/x/0 123 | DELETED 124 | 125 | ``` 126 | 127 | #### redis api 128 | 129 | Uq also supports redis protocol. And using redis protocol is easier than memcached. Start uq with redis protocol and use redis-cli to connect: 130 | 131 | ``` 132 | uq -protocol redis 133 | redis-cli -p 8808 134 | ``` 135 | 136 | Send redis requests like these: 137 | 138 | ``` 139 | // create a topic 140 | 127.0.0.1:8808> add foo 141 | OK 142 | 143 | // create a line with 10s recycle time 144 | 127.0.0.1:8808> add foo/x 10s 145 | OK 146 | 147 | // push a message into the topic 148 | 127.0.0.1:8808> set foo bar 149 | OK 150 | 151 | // pop a message from the line 152 | 127.0.0.1:8808> get foo/x 153 | 1) “bar” 154 | 2) “foo/x/0” 155 | 156 | // confirm a message 157 | 127.0.0.1:8808> del foo/x/0 158 | OK 159 | 160 | ``` 161 | 162 | #### http RESTful api 163 | 164 | If you don’t like to use any of memcached or redis client library, you can use http RESTful api which is simple and lightweight. Start uq with http protocol: 165 | 166 | ``` 167 | uq -protocol http 168 | ``` 169 | 170 | Send http requests like these: 171 | 172 | ``` 173 | // create a topic 174 | curl -XPUT -i localhost:8808/v1/queues -d topic=foo 175 | HTTP/1.1 201 Created 176 | Date: Sat, 18 Apr 2015 09:17:31 GMT 177 | Content-Length: 0 178 | Content-Type: text/plain; charset=utf-8 179 | 180 | // create a line with 10s recycle time 181 | curl -XPUT -i localhost:8808/v1/queues -d “topic=foo&line=x&recycle=10s” 182 | HTTP/1.1 201 Created 183 | Date: Sat, 18 Apr 2015 09:17:57 GMT 184 | Content-Length: 0 185 | Content-Type: text/plain; charset=utf-8 186 | 187 | // push a message into the topic 188 | curl -XPOST -i localhost:8808/v1/queues/foo -d “value=bar” 189 | HTTP/1.1 204 No Content 190 | Date: Sat, 18 Apr 2015 09:18:28 GMT 191 | 192 | // pop a message from the line 193 | curl -i localhost:8808/v1/queues/foo/x 194 | HTTP/1.1 200 OK 195 | Content-Type: text/plain 196 | X-Uq-Id: foo/x/0 197 | Date: Sat, 18 Apr 2015 09:16:38 GMT 198 | Content-Length: 3 199 | 200 | bar 201 | 202 | // confirm a message 203 | curl -XDELETE -i localhost:8808/v1/queues/foo/x/0 204 | HTTP/1.1 204 No Content 205 | Date: Sat, 18 Apr 2015 09:19:08 GMT 206 | 207 | ``` 208 | 209 | #### admin api 210 | 211 | An admin http server starts when uq is started. It uses another port (default is 8809) to listen for http requests. 212 | 213 | ``` 214 | // get stat of a line 215 | curl -i localhost:8809/v1/admin/stat/foo/x 216 | HTTP/1.1 200 OK 217 | Content-Type: application/json 218 | Date: Sat, 18 Apr 2015 10:55:03 GMT 219 | Content-Length: 84 220 | 221 | {“name”:”foo/x”,”type”:”line”,”recycle”:”10s”,”head”:1,”ihead”:1,”tail”:2,”count”:1} 222 | 223 | // get stat of a topic 224 | curl -i localhost:8809/v1/admin/stat/foo 225 | HTTP/1.1 200 OK 226 | Content-Type: application/json 227 | Date: Sat, 18 Apr 2015 10:55:42 GMT 228 | Content-Length: 162 229 | 230 | {“name”:”foo”,”type”:”topic”,”lines”:[{“name”:”foo/x”,”type”:”line”,”recycle”:”10s”,”head”:1,”ihead”:1,”tail”:2,”count”:1}],”head”:1,”ihead”:0,”tail”:2,”count”:1} 231 | 232 | // empty a line 233 | curl -XDELETE -i localhost:8809/v1/admin/empty/foo/x 234 | HTTP/1.1 204 No Content 235 | Date: Sat, 18 Apr 2015 10:56:42 GMT 236 | 237 | // empty a topic 238 | curl -XDELETE -i localhost:8809/v1/admin/empty/foo 239 | HTTP/1.1 204 No Content 240 | Date: Sat, 18 Apr 2015 10:57:02 GMT 241 | 242 | // remove a line 243 | curl -XDELETE -i localhost:8809/v1/admin/rm/foo/x 244 | HTTP/1.1 204 No Content 245 | Date: Sat, 18 Apr 2015 10:57:20 GMT 246 | 247 | // remove a topic 248 | curl -XDELETE -i localhost:8809/v1/admin/rm/foo 249 | HTTP/1.1 204 No Content 250 | Date: Sat, 18 Apr 2015 10:57:33 GMT 251 | 252 | ``` 253 | 254 | STAT method is also supported in memcached and redis protocol: 255 | 256 | ``` 257 | // in memcached protocol 258 | stats foo 259 | STAT name:foo 260 | STAT head:0 261 | STAT tail:2 262 | STAT count:2 263 | 264 | STAT name:foo/x 265 | STAT recycle:0 266 | STAT head:1 267 | STAT ihead:0 268 | STAT tail:2 269 | STAT count:1 270 | 271 | // in redis protocol 272 | 127.0.0.1:8808> info foo 273 | name:foo 274 | head:1 275 | tail:2 276 | count:1 277 | 278 | name:foo/x 279 | recycle:0 280 | head:1 281 | ihead:0 282 | tail:2 283 | count:1 284 | ``` 285 | 286 | #### api compatibility 287 | 288 | The compatibility of different protocols can be found below: 289 | 290 | | Method | Redis | Mc | Http | Description | 291 | | :----: |:---:|:---:|:---:|:---| 292 | | add | √ | √ | √ | create a topic/line | 293 | | push | √ | √ | √ | push a message into the topic | 294 | | pop | √ | √ | √ | pop the latest message of the line | 295 | | del | √ | √ | √ | confirm the message according to the message ID | 296 | | stat | √ | √ | √ | get the topic’s/line’s status | 297 | | empty | √ | × | √ | empty all the messages in a topic/line | 298 | | rm | × | × | √ | remove a topic/line | 299 | 300 | ### Distributed Cluster 301 | 302 | Uq cluster is based on etcd. You need to install and start etcd first. Then set etcd servers’ url when starting uq instances. 303 | 304 | ``` 305 | // start instance 1 306 | uq -port 8708 -admin-port 8709 -dir ./uq1 -etcd http://localhost:4001 -cluster uq 307 | // start instance 2 308 | uq -port 8808 -admin-port 8809 -dir ./uq2 -etcd http://localhost:4001 -cluster uq 309 | // start instance 3 310 | uq -port 8908 -admin-port 8909 -dir ./uq3 -etcd http://localhost:4001 -cluster uq 311 | ``` 312 | 313 | Then these uq instances make up a uq cluster. All instances in a cluster have same topics and lines. But the message in them is independent. In other words, a message can be only pushed into one instance in a cluster. Queue workflow in uq cluster is like below: 314 | 315 | 1. Client A adds topic [foo] in instance 1. All instances has topic named [foo]. 316 | 2. Client B adds line [foo/x] in instance 2. All instances has line [foo/x]. 317 | 3. Client C pushes a message [bar] to instacne 3. C can pop this message from instance 3. 318 | 4. A and B cannot pop any message from instance 1/2 because they have no message. 319 | 5. A, B, C continually push messages to topic [foo] in the instace they connect to. 320 | 6. Consumer D can pop [foo/x] to get a message from any instance in the cluster. All the messages in different instances are belong to line [foo/x]. 321 | 7. Consumer can only confirm a message in the instance which popped the message. 322 | 323 | #### using libuq 324 | 325 | Maybe you are in trouble with using the api of etcd and consideration of the connection pool. You can use [libuq](https://github.com/buaazp/libuq) to write simple codes. Libuq is designed for uq cluster. Now only Golang is supported. You can find more information about libuq in its github repository. 326 | 327 | ### Message Persistence 328 | 329 | The default storage of uq is goleveldb. It stores all the data in disk. So the messages are persistent. If the uq server broken down, the queue will recover after uq restarts. 330 | 331 | If you need a faster uq, you can use memory to store the messages. But if uq is shut down, the messages will be lost. 332 | 333 | Other storage like rocksdb, leveldb will be supported in the future. 334 | 335 | ### Unit Test 336 | 337 | Uq’s main funtions in package amdin/entry/queue/store/utils have been tested. You can test it by yourself after installing goconvey: 338 | 339 | ``` 340 | go get github.com/smartystreets/goconvey 341 | cd $GOPATH/src/github.com/buaazp/uq 342 | goconvey 343 | ``` 344 | 345 | ![Unit Tests Result](http://ww4.sinaimg.cn/large/4c422e03jw1era17icm96j212q0oowl3.jpg) 346 | 347 | ### Benchmark Test 348 | 349 | This benchmark test is between uq and memcacheQ v0.2.0 in my 13" Macbook Pro early 2011 with SSD. Uq is started with: 350 | 351 | ``` 352 | ./uq -protocol mc 353 | ``` 354 | 355 | MemcacheQ is started with: 356 | 357 | ``` 358 | memcacheq -p 8808 -r -c 1024 -H ./mcq_data/ -N -R -L 1024 -A 65536 -T 15 -B 8192 -m 256 -E 4096 359 | ``` 360 | 361 | The test tool is `test/mc_bench.go`. Build and start with: 362 | 363 | ``` 364 | go build mc_bench.go 365 | ./mc_bench -c 10 -n 500000 -m push 366 | ``` 367 | 368 | The result of benchmark test: 369 | 370 | | Program | QPS | Throughput | 371 | | :-------- |:--------------:| ----------:| 372 | | memcacheQ | 4169.952 msg/s | 0.795 MB/s | 373 | | UQ | 4319.417 msg/s | 0.824 MB/s | 374 | 375 | Uq performs as well as memcachedQ now. With the snappy compression of goleveldb, uq's data size is only 105MB while memcachedQ generated 4.5GB data after this test. You can use NoCompression option for goleveldb if you want. 376 | 377 | Anyway, we need more optimization measures to improve performance of uq. 378 | 379 | ### Feedback: 380 | 381 | If you have any question, please submit comment in my [BLOG](http://blog.buaa.us/) or mention me on [Weibo](http://weibo.com/819880808), [twitter](https://twitter.com/buaazp). 382 | Technical issues are also welcome to be submitted on [GitHub Issues](https://github.com/buaazp/uq/issues). 383 | 384 | -------------------------------------------------------------------------------- /admin/admin.go: -------------------------------------------------------------------------------- 1 | package admin 2 | 3 | // Administrator is the admin interface of uq 4 | type Administrator interface { 5 | ListenAndServe() error 6 | Stop() 7 | } 8 | -------------------------------------------------------------------------------- /admin/uAdmin.go: -------------------------------------------------------------------------------- 1 | package admin 2 | 3 | import ( 4 | "log" 5 | "net" 6 | "net/http" 7 | httpprof "net/http/pprof" 8 | "strings" 9 | 10 | "github.com/buaazp/uq/queue" 11 | "github.com/buaazp/uq/utils" 12 | ) 13 | 14 | const ( 15 | queuePrefixV1 = "/v1/queues" 16 | adminPrefixV1 = "/v1/admin" 17 | pprofPrefixCmd = "/debug/pprof/cmdline" 18 | pprofPrefixProfile = "/debug/pprof/profile" 19 | pprofPrefixSymbol = "/debug/pprof/symbol" 20 | pprofPrefixIndex = "/debug/pprof" 21 | ) 22 | 23 | // UnitedAdmin is the HTTP admin server of uq 24 | type UnitedAdmin struct { 25 | host string 26 | port int 27 | adminMux map[string]func(http.ResponseWriter, *http.Request, string) 28 | server *http.Server 29 | stopListener *utils.StopListener 30 | messageQueue queue.MessageQueue 31 | } 32 | 33 | // NewUnitedAdmin returns a UnitedAdmin 34 | func NewUnitedAdmin(host string, port int, messageQueue queue.MessageQueue) (*UnitedAdmin, error) { 35 | s := new(UnitedAdmin) 36 | 37 | s.adminMux = map[string]func(http.ResponseWriter, *http.Request, string){ 38 | "/stat": s.statHandler, 39 | "/empty": s.emptyHandler, 40 | "/rm": s.rmHandler, 41 | } 42 | 43 | addr := utils.Addrcat(host, port) 44 | server := new(http.Server) 45 | server.Addr = addr 46 | server.Handler = s 47 | 48 | s.host = host 49 | s.port = port 50 | s.server = server 51 | s.messageQueue = messageQueue 52 | 53 | return s, nil 54 | } 55 | 56 | func (s *UnitedAdmin) ServeHTTP(w http.ResponseWriter, req *http.Request) { 57 | if !utils.AllowMethod(w, req.Method, "HEAD", "GET", "POST", "PUT", "DELETE") { 58 | return 59 | } 60 | 61 | if strings.HasPrefix(req.URL.Path, queuePrefixV1) { 62 | key := req.URL.Path[len(queuePrefixV1):] 63 | s.queueHandler(w, req, key) 64 | return 65 | } else if strings.HasPrefix(req.URL.Path, adminPrefixV1) { 66 | key := req.URL.Path[len(adminPrefixV1):] 67 | s.adminHandler(w, req, key) 68 | return 69 | } else if strings.HasPrefix(req.URL.Path, pprofPrefixCmd) { 70 | httpprof.Cmdline(w, req) 71 | return 72 | } else if strings.HasPrefix(req.URL.Path, pprofPrefixProfile) { 73 | httpprof.Profile(w, req) 74 | return 75 | } else if strings.HasPrefix(req.URL.Path, pprofPrefixSymbol) { 76 | httpprof.Symbol(w, req) 77 | return 78 | } else if strings.HasPrefix(req.URL.Path, pprofPrefixIndex) { 79 | httpprof.Index(w, req) 80 | return 81 | } 82 | 83 | http.Error(w, "404 Not Found!", http.StatusNotFound) 84 | return 85 | } 86 | 87 | func (s *UnitedAdmin) queueHandler(w http.ResponseWriter, req *http.Request, key string) { 88 | switch req.Method { 89 | case "PUT": 90 | s.addHandler(w, req, key) 91 | case "POST": 92 | s.pushHandler(w, req, key) 93 | case "GET": 94 | s.popHandler(w, req, key) 95 | case "DELETE": 96 | s.delHandler(w, req, key) 97 | default: 98 | http.Error(w, "405 Method Not Allowed!", http.StatusMethodNotAllowed) 99 | } 100 | return 101 | } 102 | 103 | func (s *UnitedAdmin) adminHandler(w http.ResponseWriter, req *http.Request, key string) { 104 | for prefix, handler := range s.adminMux { 105 | if strings.HasPrefix(key, prefix) { 106 | key = key[len(prefix):] 107 | handler(w, req, key) 108 | return 109 | } 110 | } 111 | 112 | http.Error(w, "404 Not Found!", http.StatusNotFound) 113 | return 114 | } 115 | 116 | func writeErrorHTTP(w http.ResponseWriter, err error) { 117 | if err == nil { 118 | return 119 | } 120 | switch e := err.(type) { 121 | case *utils.Error: 122 | e.WriteTo(w) 123 | default: 124 | // log.Printf("unexpected error: %v", err) 125 | http.Error(w, "500 Internal Error!\r\n"+err.Error(), http.StatusInternalServerError) 126 | } 127 | } 128 | 129 | func (s *UnitedAdmin) addHandler(w http.ResponseWriter, req *http.Request, key string) { 130 | err := req.ParseForm() 131 | if err != nil { 132 | writeErrorHTTP(w, utils.NewError( 133 | utils.ErrInternalError, 134 | err.Error(), 135 | )) 136 | return 137 | } 138 | 139 | topicName := req.FormValue("topic") 140 | lineName := req.FormValue("line") 141 | key = topicName + "/" + lineName 142 | recycle := req.FormValue("recycle") 143 | 144 | // log.Printf("creating... %s %s", key, recycle) 145 | err = s.messageQueue.Create(key, recycle) 146 | if err != nil { 147 | writeErrorHTTP(w, err) 148 | return 149 | } 150 | w.WriteHeader(http.StatusCreated) 151 | } 152 | 153 | func (s *UnitedAdmin) pushHandler(w http.ResponseWriter, req *http.Request, key string) { 154 | err := req.ParseForm() 155 | if err != nil { 156 | writeErrorHTTP(w, utils.NewError( 157 | utils.ErrInternalError, 158 | err.Error(), 159 | )) 160 | return 161 | } 162 | 163 | data := []byte(req.FormValue("value")) 164 | err = s.messageQueue.Push(key, data) 165 | if err != nil { 166 | writeErrorHTTP(w, err) 167 | return 168 | } 169 | w.WriteHeader(http.StatusNoContent) 170 | } 171 | 172 | func (s *UnitedAdmin) popHandler(w http.ResponseWriter, req *http.Request, key string) { 173 | id, data, err := s.messageQueue.Pop(key) 174 | if err != nil { 175 | writeErrorHTTP(w, err) 176 | return 177 | } 178 | 179 | w.Header().Set("Content-Type", "text/plain") 180 | w.Header().Set("X-UQ-ID", id) 181 | w.WriteHeader(http.StatusOK) 182 | w.Write(data) 183 | } 184 | 185 | func (s *UnitedAdmin) delHandler(w http.ResponseWriter, req *http.Request, key string) { 186 | err := s.messageQueue.Confirm(key) 187 | if err != nil { 188 | writeErrorHTTP(w, err) 189 | return 190 | } 191 | w.WriteHeader(http.StatusNoContent) 192 | } 193 | 194 | func (s *UnitedAdmin) statHandler(w http.ResponseWriter, req *http.Request, key string) { 195 | if req.Method != "GET" { 196 | http.Error(w, "405 Method Not Allowed!", http.StatusMethodNotAllowed) 197 | return 198 | } 199 | 200 | qs, err := s.messageQueue.Stat(key) 201 | if err != nil { 202 | writeErrorHTTP(w, err) 203 | return 204 | } 205 | 206 | // log.Printf("qs: %v", qs) 207 | data, err := qs.ToJSON() 208 | if err != nil { 209 | writeErrorHTTP(w, utils.NewError( 210 | utils.ErrInternalError, 211 | err.Error(), 212 | )) 213 | return 214 | } 215 | 216 | w.Header().Set("Content-Type", "application/json") 217 | w.WriteHeader(http.StatusOK) 218 | w.Write(data) 219 | } 220 | 221 | func (s *UnitedAdmin) emptyHandler(w http.ResponseWriter, req *http.Request, key string) { 222 | if req.Method != "DELETE" { 223 | http.Error(w, "405 Method Not Allowed!", http.StatusMethodNotAllowed) 224 | return 225 | } 226 | 227 | err := s.messageQueue.Empty(key) 228 | if err != nil { 229 | writeErrorHTTP(w, err) 230 | return 231 | } 232 | w.WriteHeader(http.StatusNoContent) 233 | } 234 | 235 | func (s *UnitedAdmin) rmHandler(w http.ResponseWriter, req *http.Request, key string) { 236 | if req.Method != "DELETE" { 237 | http.Error(w, "405 Method Not Allowed!", http.StatusMethodNotAllowed) 238 | return 239 | } 240 | 241 | err := s.messageQueue.Remove(key) 242 | if err != nil { 243 | writeErrorHTTP(w, err) 244 | return 245 | } 246 | w.WriteHeader(http.StatusNoContent) 247 | } 248 | 249 | // ListenAndServe implements the ListenAndServe interface 250 | func (s *UnitedAdmin) ListenAndServe() error { 251 | addr := utils.Addrcat(s.host, s.port) 252 | l, err := net.Listen("tcp", addr) 253 | if err != nil { 254 | return err 255 | } 256 | 257 | stopListener, err := utils.NewStopListener(l) 258 | if err != nil { 259 | return err 260 | } 261 | s.stopListener = stopListener 262 | 263 | log.Printf("admin server serving at %s...", addr) 264 | return s.server.Serve(s.stopListener) 265 | } 266 | 267 | // Stop implements the Stop interface 268 | func (s *UnitedAdmin) Stop() { 269 | log.Printf("admin server stoping...") 270 | s.stopListener.Stop() 271 | } 272 | -------------------------------------------------------------------------------- /admin/uAdmin_test.go: -------------------------------------------------------------------------------- 1 | package admin 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "io/ioutil" 7 | "net/http" 8 | "testing" 9 | 10 | "github.com/buaazp/uq/queue" 11 | "github.com/buaazp/uq/store" 12 | . "github.com/smartystreets/goconvey/convey" 13 | ) 14 | 15 | var ( 16 | storage store.Storage 17 | messageQueue queue.MessageQueue 18 | adminServer Administrator 19 | client *http.Client 20 | ) 21 | 22 | func init() { 23 | client = new(http.Client) 24 | } 25 | 26 | func TestNewAdmin(t *testing.T) { 27 | Convey("Test New Admin", t, func() { 28 | var err error 29 | storage, err = store.NewMemStore() 30 | So(err, ShouldBeNil) 31 | So(storage, ShouldNotBeNil) 32 | messageQueue, err = queue.NewUnitedQueue(storage, "127.0.0.1", 8800, nil, "uq") 33 | So(err, ShouldBeNil) 34 | So(messageQueue, ShouldNotBeNil) 35 | 36 | adminServer, err = NewUnitedAdmin("0.0.0.0", 8800, messageQueue) 37 | So(err, ShouldBeNil) 38 | So(adminServer, ShouldNotBeNil) 39 | 40 | go func() { 41 | adminServer.ListenAndServe() 42 | }() 43 | }) 44 | } 45 | 46 | func TestAdminAdd(t *testing.T) { 47 | Convey("Test Admin Add Api", t, func() { 48 | bf := bytes.NewBufferString("topic=foo") 49 | body := ioutil.NopCloser(bf) 50 | req, err := http.NewRequest( 51 | "PUT", 52 | "http://127.0.0.1:8800/v1/queues", 53 | body, 54 | ) 55 | So(err, ShouldBeNil) 56 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 57 | 58 | resp, err := client.Do(req) 59 | So(err, ShouldBeNil) 60 | So(resp.StatusCode, ShouldEqual, http.StatusCreated) 61 | 62 | bf = bytes.NewBufferString("topic=foo&line=x&recycle=10s") 63 | body = ioutil.NopCloser(bf) 64 | req, err = http.NewRequest( 65 | "PUT", 66 | "http://127.0.0.1:8800/v1/queues", 67 | body, 68 | ) 69 | So(err, ShouldBeNil) 70 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 71 | 72 | resp, err = client.Do(req) 73 | So(err, ShouldBeNil) 74 | So(resp.StatusCode, ShouldEqual, http.StatusCreated) 75 | }) 76 | } 77 | 78 | func TestAdminPush(t *testing.T) { 79 | Convey("Test Admin Push Api", t, func() { 80 | bf := bytes.NewBufferString("value=1") 81 | body := ioutil.NopCloser(bf) 82 | req, err := http.NewRequest( 83 | "POST", 84 | "http://127.0.0.1:8800/v1/queues/foo", 85 | body, 86 | ) 87 | So(err, ShouldBeNil) 88 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 89 | 90 | resp, err := client.Do(req) 91 | So(err, ShouldBeNil) 92 | So(resp.StatusCode, ShouldEqual, http.StatusNoContent) 93 | }) 94 | } 95 | 96 | func TestAdminPop(t *testing.T) { 97 | Convey("Test Admin Pop Api", t, func() { 98 | req, err := http.NewRequest( 99 | "GET", 100 | "http://127.0.0.1:8800/v1/queues/foo/x", 101 | nil, 102 | ) 103 | So(err, ShouldBeNil) 104 | 105 | resp, err := client.Do(req) 106 | So(err, ShouldBeNil) 107 | So(resp.StatusCode, ShouldEqual, http.StatusOK) 108 | 109 | body, err := ioutil.ReadAll(resp.Body) 110 | So(err, ShouldBeNil) 111 | id := resp.Header.Get("X-UQ-ID") 112 | So(id, ShouldEqual, "foo/x/0") 113 | msg := string(body) 114 | So(msg, ShouldEqual, "1") 115 | }) 116 | } 117 | 118 | func TestAdminConfirm(t *testing.T) { 119 | Convey("Test Admin Confirm Api", t, func() { 120 | req, err := http.NewRequest( 121 | "DELETE", 122 | "http://127.0.0.1:8800/v1/queues/foo/x/0", 123 | nil, 124 | ) 125 | So(err, ShouldBeNil) 126 | resp, err := client.Do(req) 127 | So(err, ShouldBeNil) 128 | So(resp.StatusCode, ShouldEqual, http.StatusNoContent) 129 | }) 130 | } 131 | 132 | func TestAdminStat(t *testing.T) { 133 | Convey("Test Admin Stat Api", t, func() { 134 | req, err := http.NewRequest( 135 | "GET", 136 | "http://127.0.0.1:8800/v1/admin/stat/foo/x", 137 | nil, 138 | ) 139 | So(err, ShouldBeNil) 140 | 141 | resp, err := client.Do(req) 142 | So(err, ShouldBeNil) 143 | So(resp.StatusCode, ShouldEqual, http.StatusOK) 144 | 145 | body, err := ioutil.ReadAll(resp.Body) 146 | So(err, ShouldBeNil) 147 | var qs queue.Stat 148 | err = json.Unmarshal(body, &qs) 149 | So(err, ShouldBeNil) 150 | So(qs.Name, ShouldEqual, "foo/x") 151 | }) 152 | } 153 | 154 | func TestAdminEmpty(t *testing.T) { 155 | Convey("Test Admin Empty Api", t, func() { 156 | req, err := http.NewRequest( 157 | "DELETE", 158 | "http://127.0.0.1:8800/v1/admin/empty/foo/x", 159 | nil, 160 | ) 161 | So(err, ShouldBeNil) 162 | 163 | resp, err := client.Do(req) 164 | So(err, ShouldBeNil) 165 | So(resp.StatusCode, ShouldEqual, http.StatusNoContent) 166 | }) 167 | } 168 | 169 | func TestAdminRemove(t *testing.T) { 170 | Convey("Test Admin Remove Api", t, func() { 171 | req, err := http.NewRequest( 172 | "DELETE", 173 | "http://127.0.0.1:8800/v1/admin/rm/foo/x", 174 | nil, 175 | ) 176 | So(err, ShouldBeNil) 177 | 178 | resp, err := client.Do(req) 179 | So(err, ShouldBeNil) 180 | So(resp.StatusCode, ShouldEqual, http.StatusNoContent) 181 | }) 182 | } 183 | 184 | func TestCloseAdmin(t *testing.T) { 185 | Convey("Test Close Admin", t, func() { 186 | adminServer.Stop() 187 | messageQueue.Close() 188 | messageQueue = nil 189 | storage = nil 190 | }) 191 | } 192 | -------------------------------------------------------------------------------- /entry/entry.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | const ( 4 | // MaxKeyLength is the max length of a key 5 | MaxKeyLength int = 512 6 | // MaxBodyLength is the max body length of a request 7 | MaxBodyLength int = 10 * 1024 * 1024 8 | ) 9 | 10 | // Entrance is the interface of uq's entrance 11 | type Entrance interface { 12 | ListenAndServe() error 13 | Stop() 14 | } 15 | -------------------------------------------------------------------------------- /entry/httpEntry.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "log" 5 | "net" 6 | "net/http" 7 | "strings" 8 | 9 | "github.com/buaazp/uq/queue" 10 | "github.com/buaazp/uq/utils" 11 | ) 12 | 13 | const ( 14 | queuePrefixV1 = "/v1/queues" 15 | ) 16 | 17 | // HTTPEntry is the HTTP entrance of uq 18 | type HTTPEntry struct { 19 | host string 20 | port int 21 | server *http.Server 22 | stopListener *utils.StopListener 23 | messageQueue queue.MessageQueue 24 | } 25 | 26 | // NewHTTPEntry returns a new HTTPEntry server 27 | func NewHTTPEntry(host string, port int, messageQueue queue.MessageQueue) (*HTTPEntry, error) { 28 | h := new(HTTPEntry) 29 | 30 | addr := utils.Addrcat(host, port) 31 | server := new(http.Server) 32 | server.Addr = addr 33 | server.Handler = h 34 | 35 | h.host = host 36 | h.port = port 37 | h.server = server 38 | h.messageQueue = messageQueue 39 | 40 | return h, nil 41 | } 42 | 43 | // ServeHTTP implements the ServeHTTP interface of HTTPEntry 44 | func (h *HTTPEntry) ServeHTTP(w http.ResponseWriter, req *http.Request) { 45 | if !utils.AllowMethod(w, req.Method, "HEAD", "GET", "POST", "PUT", "DELETE") { 46 | return 47 | } 48 | 49 | if strings.HasPrefix(req.URL.Path, queuePrefixV1) { 50 | key := req.URL.Path[len(queuePrefixV1):] 51 | h.queueHandler(w, req, key) 52 | return 53 | } 54 | 55 | http.Error(w, "404 Not Found!", http.StatusNotFound) 56 | return 57 | } 58 | 59 | func (h *HTTPEntry) queueHandler(w http.ResponseWriter, req *http.Request, key string) { 60 | switch req.Method { 61 | case "PUT": 62 | h.addHandler(w, req, key) 63 | case "POST": 64 | h.pushHandler(w, req, key) 65 | case "GET": 66 | h.popHandler(w, req, key) 67 | case "DELETE": 68 | h.delHandler(w, req, key) 69 | default: 70 | http.Error(w, "405 Method Not Allowed!", http.StatusMethodNotAllowed) 71 | } 72 | return 73 | } 74 | 75 | func writeErrorHTTP(w http.ResponseWriter, err error) { 76 | if err == nil { 77 | return 78 | } 79 | switch e := err.(type) { 80 | case *utils.Error: 81 | e.WriteTo(w) 82 | default: 83 | // log.Printf("unexpected error: %v", err) 84 | http.Error(w, "500 Internal Error!\r\n"+err.Error(), http.StatusInternalServerError) 85 | } 86 | } 87 | 88 | func (h *HTTPEntry) addHandler(w http.ResponseWriter, req *http.Request, key string) { 89 | err := req.ParseForm() 90 | if err != nil { 91 | writeErrorHTTP(w, utils.NewError( 92 | utils.ErrInternalError, 93 | err.Error(), 94 | )) 95 | return 96 | } 97 | 98 | topicName := req.FormValue("topic") 99 | lineName := req.FormValue("line") 100 | key = topicName + "/" + lineName 101 | recycle := req.FormValue("recycle") 102 | 103 | // log.Printf("creating... %s %s", key, recycle) 104 | err = h.messageQueue.Create(key, recycle) 105 | if err != nil { 106 | writeErrorHTTP(w, err) 107 | return 108 | } 109 | w.WriteHeader(http.StatusCreated) 110 | } 111 | 112 | func (h *HTTPEntry) pushHandler(w http.ResponseWriter, req *http.Request, key string) { 113 | err := req.ParseForm() 114 | if err != nil { 115 | writeErrorHTTP(w, utils.NewError( 116 | utils.ErrInternalError, 117 | err.Error(), 118 | )) 119 | return 120 | } 121 | 122 | data := []byte(req.FormValue("value")) 123 | err = h.messageQueue.Push(key, data) 124 | if err != nil { 125 | writeErrorHTTP(w, err) 126 | return 127 | } 128 | w.WriteHeader(http.StatusNoContent) 129 | } 130 | 131 | func (h *HTTPEntry) popHandler(w http.ResponseWriter, req *http.Request, key string) { 132 | id, data, err := h.messageQueue.Pop(key) 133 | if err != nil { 134 | writeErrorHTTP(w, err) 135 | return 136 | } 137 | 138 | w.Header().Set("Content-Type", "text/plain") 139 | w.Header().Set("X-UQ-ID", id) 140 | w.WriteHeader(http.StatusOK) 141 | w.Write(data) 142 | } 143 | 144 | func (h *HTTPEntry) delHandler(w http.ResponseWriter, req *http.Request, key string) { 145 | err := h.messageQueue.Confirm(key) 146 | if err != nil { 147 | writeErrorHTTP(w, err) 148 | return 149 | } 150 | w.WriteHeader(http.StatusNoContent) 151 | } 152 | 153 | // ListenAndServe implements the ListenAndServe interface 154 | func (h *HTTPEntry) ListenAndServe() error { 155 | addr := utils.Addrcat(h.host, h.port) 156 | l, err := net.Listen("tcp", addr) 157 | if err != nil { 158 | return err 159 | } 160 | 161 | stopListener, err := utils.NewStopListener(l) 162 | if err != nil { 163 | return err 164 | } 165 | h.stopListener = stopListener 166 | 167 | log.Printf("http entrance serving at %s...", addr) 168 | return h.server.Serve(h.stopListener) 169 | } 170 | 171 | // Stop implements the Stop interface 172 | func (h *HTTPEntry) Stop() { 173 | log.Printf("http entry stoping...") 174 | h.stopListener.Stop() 175 | h.messageQueue.Close() 176 | } 177 | -------------------------------------------------------------------------------- /entry/httpEntry_test.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "bytes" 5 | "io/ioutil" 6 | "net/http" 7 | "testing" 8 | "time" 9 | 10 | "github.com/buaazp/uq/queue" 11 | "github.com/buaazp/uq/store" 12 | . "github.com/smartystreets/goconvey/convey" 13 | ) 14 | 15 | var ( 16 | storage store.Storage 17 | messageQueue queue.MessageQueue 18 | entrance Entrance 19 | client *http.Client 20 | ) 21 | 22 | func init() { 23 | client = new(http.Client) 24 | } 25 | 26 | func TestNewHTTPEntry(t *testing.T) { 27 | Convey("Test New HTTP Entry", t, func() { 28 | var err error 29 | storage, err = store.NewMemStore() 30 | So(err, ShouldBeNil) 31 | So(storage, ShouldNotBeNil) 32 | messageQueue, err = queue.NewUnitedQueue(storage, "127.0.0.1", 8801, nil, "uq") 33 | So(err, ShouldBeNil) 34 | So(messageQueue, ShouldNotBeNil) 35 | 36 | entrance, err = NewHTTPEntry("0.0.0.0", 8801, messageQueue) 37 | So(err, ShouldBeNil) 38 | So(entrance, ShouldNotBeNil) 39 | 40 | go func() { 41 | entrance.ListenAndServe() 42 | }() 43 | time.Sleep(100 * time.Millisecond) 44 | }) 45 | } 46 | 47 | func TestHttpAdd(t *testing.T) { 48 | Convey("Test Http Add Api", t, func() { 49 | bf := bytes.NewBufferString("topic=foo") 50 | body := ioutil.NopCloser(bf) 51 | req, err := http.NewRequest( 52 | "PUT", 53 | "http://127.0.0.1:8801/v1/queues", 54 | body, 55 | ) 56 | So(err, ShouldBeNil) 57 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 58 | 59 | resp, err := client.Do(req) 60 | So(err, ShouldBeNil) 61 | So(resp.StatusCode, ShouldEqual, http.StatusCreated) 62 | 63 | bf = bytes.NewBufferString("topic=foo&line=x&recycle=10s") 64 | body = ioutil.NopCloser(bf) 65 | req, err = http.NewRequest( 66 | "PUT", 67 | "http://127.0.0.1:8801/v1/queues", 68 | body, 69 | ) 70 | So(err, ShouldBeNil) 71 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 72 | 73 | resp, err = client.Do(req) 74 | So(err, ShouldBeNil) 75 | So(resp.StatusCode, ShouldEqual, http.StatusCreated) 76 | }) 77 | } 78 | 79 | func TestHttpPush(t *testing.T) { 80 | Convey("Test Http Push Api", t, func() { 81 | bf := bytes.NewBufferString("value=1") 82 | body := ioutil.NopCloser(bf) 83 | req, err := http.NewRequest( 84 | "POST", 85 | "http://127.0.0.1:8801/v1/queues/foo", 86 | body, 87 | ) 88 | So(err, ShouldBeNil) 89 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 90 | 91 | resp, err := client.Do(req) 92 | So(err, ShouldBeNil) 93 | So(resp.StatusCode, ShouldEqual, http.StatusNoContent) 94 | }) 95 | } 96 | 97 | func TestHttpPop(t *testing.T) { 98 | Convey("Test Http Pop Api", t, func() { 99 | req, err := http.NewRequest( 100 | "GET", 101 | "http://127.0.0.1:8801/v1/queues/foo/x", 102 | nil, 103 | ) 104 | So(err, ShouldBeNil) 105 | 106 | resp, err := client.Do(req) 107 | So(err, ShouldBeNil) 108 | So(resp.StatusCode, ShouldEqual, http.StatusOK) 109 | 110 | body, err := ioutil.ReadAll(resp.Body) 111 | So(err, ShouldBeNil) 112 | id := resp.Header.Get("X-UQ-ID") 113 | So(id, ShouldEqual, "foo/x/0") 114 | msg := string(body) 115 | So(msg, ShouldEqual, "1") 116 | }) 117 | } 118 | 119 | func TestHttpConfirm(t *testing.T) { 120 | Convey("Test Http Confirm Api", t, func() { 121 | req, err := http.NewRequest( 122 | "DELETE", 123 | "http://127.0.0.1:8801/v1/queues/foo/x/0", 124 | nil, 125 | ) 126 | So(err, ShouldBeNil) 127 | resp, err := client.Do(req) 128 | So(err, ShouldBeNil) 129 | So(resp.StatusCode, ShouldEqual, http.StatusNoContent) 130 | }) 131 | } 132 | 133 | func TestCloseHTTPEntry(t *testing.T) { 134 | Convey("Test Close Http Entry", t, func() { 135 | entrance.Stop() 136 | messageQueue = nil 137 | storage = nil 138 | }) 139 | } 140 | -------------------------------------------------------------------------------- /entry/mcEntry.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "bufio" 5 | "io" 6 | "log" 7 | "net" 8 | "strconv" 9 | "strings" 10 | 11 | "github.com/buaazp/uq/queue" 12 | "github.com/buaazp/uq/utils" 13 | ) 14 | 15 | // McEntry is the memcached entrance of uq 16 | type McEntry struct { 17 | host string 18 | port int 19 | stopListener *utils.StopListener 20 | messageQueue queue.MessageQueue 21 | } 22 | 23 | type item struct { 24 | flag int 25 | expTime int 26 | body []byte 27 | } 28 | 29 | type request struct { 30 | cmd string // get, set, delete, quit, etc. 31 | keys []string // key 32 | item *item 33 | noReply bool 34 | } 35 | 36 | // NewMcEntry returns a new McEntry server 37 | func NewMcEntry(host string, port int, messageQueue queue.MessageQueue) (*McEntry, error) { 38 | mc := new(McEntry) 39 | mc.host = host 40 | mc.port = port 41 | mc.messageQueue = messageQueue 42 | return mc, nil 43 | } 44 | 45 | func (m *McEntry) read(b *bufio.Reader) (*request, error) { 46 | s, err := b.ReadString('\n') 47 | if err != nil { 48 | log.Println("readstring failed: ", err) 49 | return nil, utils.NewError( 50 | utils.ErrBadRequest, 51 | err.Error(), 52 | ) 53 | } 54 | if !strings.HasSuffix(s, "\r\n") { 55 | return nil, utils.NewError( 56 | utils.ErrBadRequest, 57 | `has not suffix \r\n`, 58 | ) 59 | } 60 | parts := strings.Fields(s) 61 | if len(parts) < 1 { 62 | return nil, utils.NewError( 63 | utils.ErrBadRequest, 64 | `cmd fields error < 1`, 65 | ) 66 | } 67 | 68 | req := new(request) 69 | req.cmd = parts[0] 70 | switch req.cmd { 71 | case "get", "gets", "stats": 72 | if len(parts) < 2 { 73 | return nil, utils.NewError( 74 | utils.ErrBadRequest, 75 | `cmd parts error: < 2`, 76 | ) 77 | } 78 | req.keys = parts[1:] 79 | 80 | case "set", "add": 81 | if len(parts) < 5 || len(parts) > 7 { 82 | return nil, utils.NewError( 83 | utils.ErrBadRequest, 84 | `cmd parts error: < 5 or > 7`, 85 | ) 86 | } 87 | req.keys = parts[1:2] 88 | item := new(item) 89 | item.flag, err = strconv.Atoi(parts[2]) 90 | if err != nil { 91 | return nil, utils.NewError( 92 | utils.ErrBadRequest, 93 | `flag atoi failed: `+err.Error(), 94 | ) 95 | } 96 | item.expTime, err = strconv.Atoi(parts[3]) 97 | if err != nil { 98 | return nil, utils.NewError( 99 | utils.ErrBadRequest, 100 | `exptime atoi failed: `+err.Error(), 101 | ) 102 | } 103 | length, err := strconv.Atoi(parts[4]) 104 | if err != nil { 105 | return nil, utils.NewError( 106 | utils.ErrBadRequest, 107 | `length atoi failed: `+err.Error(), 108 | ) 109 | } 110 | if length > MaxBodyLength { 111 | return nil, utils.NewError( 112 | utils.ErrBadRequest, 113 | `bad data length`, 114 | ) 115 | } 116 | if len(parts) > 5 && parts[5] != "noreply" { 117 | return nil, utils.NewError( 118 | utils.ErrBadRequest, 119 | `cmd parts error: > 5 or part[5] != noreply`, 120 | ) 121 | } 122 | req.noReply = len(parts) > 5 && parts[5] == "noreply" 123 | 124 | item.body = make([]byte, length) 125 | _, err = io.ReadFull(b, item.body) 126 | if err != nil { 127 | return nil, utils.NewError( 128 | utils.ErrBadRequest, 129 | `readfull failed: `+err.Error(), 130 | ) 131 | } 132 | remain, _, err := b.ReadLine() 133 | if err != nil { 134 | return nil, utils.NewError( 135 | utils.ErrBadRequest, 136 | `readline failed: `+err.Error(), 137 | ) 138 | } 139 | if len(remain) != 0 { 140 | return nil, utils.NewError( 141 | utils.ErrBadRequest, 142 | `bad data chunk`, 143 | ) 144 | } 145 | req.item = item 146 | 147 | case "delete": 148 | if len(parts) < 2 || len(parts) > 4 { 149 | return nil, utils.NewError( 150 | utils.ErrBadRequest, 151 | `cmd parts error: < 2 or > 4`, 152 | ) 153 | } 154 | req.keys = parts[1:2] 155 | req.noReply = len(parts) > 2 && parts[len(parts)-1] == "noreply" 156 | 157 | case "quit", "version", "flush_all": 158 | case "replace", "cas", "append", "prepend": 159 | case "incr", "decr": 160 | case "verbosity": 161 | 162 | default: 163 | return nil, utils.NewError( 164 | utils.ErrBadRequest, 165 | `unknow command: `+req.cmd, 166 | ) 167 | } 168 | return req, nil 169 | } 170 | 171 | func writeErrorMc(resp *response, err error) { 172 | if err == nil { 173 | return 174 | } 175 | switch e := err.(type) { 176 | case *utils.Error: 177 | if e.ErrorCode >= 500 { 178 | resp.status = "SERVER_ERROR" 179 | } else { 180 | resp.status = "CLIENT_ERROR" 181 | } 182 | resp.msg = e.Error() 183 | default: 184 | // log.Printf("unexpected error: %v", err) 185 | resp.status = "SERVER_ERROR" 186 | resp.msg = e.Error() 187 | } 188 | } 189 | 190 | func (m *McEntry) process(req *request) (resp *response, quit bool) { 191 | var err error 192 | resp = new(response) 193 | quit = false 194 | resp.noreply = req.noReply 195 | 196 | switch req.cmd { 197 | case "get", "gets": 198 | for _, k := range req.keys { 199 | if len(k) > MaxKeyLength { 200 | writeErrorMc(resp, utils.NewError( 201 | utils.ErrBadKey, 202 | `key is too long`, 203 | )) 204 | return 205 | } 206 | } 207 | 208 | key := req.keys[0] 209 | resp.status = "VALUE" 210 | id, data, err := m.messageQueue.Pop(key) 211 | if err != nil { 212 | writeErrorMc(resp, err) 213 | return 214 | } 215 | 216 | itemMsg := new(item) 217 | itemMsg.body = data 218 | items := make(map[string]*item) 219 | items[key] = itemMsg 220 | 221 | if len(req.keys) > 1 { 222 | keyID := req.keys[1] 223 | itemID := new(item) 224 | itemID.body = []byte(id) 225 | items[keyID] = itemID 226 | } 227 | 228 | resp.items = items 229 | 230 | case "stats": 231 | key := req.keys[0] 232 | resp.status = "STAT" 233 | qs, err := m.messageQueue.Stat(key) 234 | if err != nil { 235 | writeErrorMc(resp, err) 236 | return 237 | } 238 | 239 | // for human reading 240 | resp.msg = qs.ToMcString() 241 | 242 | // for json format 243 | // data, err := qs.ToJSON() 244 | // if err != nil { 245 | // writeErrorMc(resp, NewError( 246 | // ErrInternalError, 247 | // err.Error(), 248 | // )) 249 | // return 250 | // } 251 | // resp.msg = string(data) 252 | 253 | case "add": 254 | key := req.keys[0] 255 | recycle := string(req.item.body) 256 | 257 | // log.Printf("creating... %s %s", key, recycle) 258 | err = m.messageQueue.Create(key, recycle) 259 | if err != nil { 260 | writeErrorMc(resp, err) 261 | return 262 | } 263 | resp.status = "STORED" 264 | 265 | case "set": 266 | key := req.keys[0] 267 | err = m.messageQueue.Push(key, req.item.body) 268 | if err != nil { 269 | writeErrorMc(resp, err) 270 | return 271 | } 272 | resp.status = "STORED" 273 | 274 | case "delete": 275 | key := req.keys[0] 276 | 277 | err = m.messageQueue.Confirm(key) 278 | if err != nil { 279 | writeErrorMc(resp, err) 280 | break 281 | } 282 | resp.status = "DELETED" 283 | 284 | case "quit": 285 | resp = nil 286 | quit = true 287 | return 288 | 289 | default: 290 | // client error 291 | writeErrorMc(resp, utils.NewError( 292 | utils.ErrBadRequest, 293 | `unknow command: `+req.cmd, 294 | )) 295 | } 296 | return 297 | } 298 | 299 | func (m *McEntry) handlerConn(conn net.Conn) { 300 | // addr := conn.RemoteAddr().String() 301 | // log.Printf("handleClient: %s", addr) 302 | 303 | rbuf := bufio.NewReader(conn) 304 | wbuf := bufio.NewWriter(conn) 305 | 306 | for { 307 | req, err := m.read(rbuf) 308 | if err != nil { 309 | if strings.Contains(err.Error(), "EOF") { 310 | break 311 | } else { 312 | resp := new(response) 313 | writeErrorMc(resp, err) 314 | resp.Write(wbuf) 315 | wbuf.Flush() 316 | continue 317 | } 318 | } 319 | 320 | resp, quit := m.process(req) 321 | if quit { 322 | break 323 | } 324 | if resp == nil { 325 | continue 326 | } 327 | if !resp.noreply { 328 | resp.Write(wbuf) 329 | wbuf.Flush() 330 | continue 331 | } 332 | } 333 | 334 | // log.Printf("conn %s closing...", addr) 335 | if err := conn.Close(); err != nil { 336 | // log.Printf("conn %s close error: %s", addr, err) 337 | } 338 | 339 | return 340 | } 341 | 342 | // ListenAndServe implements the ListenAndServe interface 343 | func (m *McEntry) ListenAndServe() error { 344 | addr := utils.Addrcat(m.host, m.port) 345 | l, err := net.Listen("tcp", addr) 346 | if err != nil { 347 | return err 348 | } 349 | 350 | stopListener, err := utils.NewStopListener(l) 351 | if err != nil { 352 | return err 353 | } 354 | m.stopListener = stopListener 355 | 356 | log.Printf("mc entrance serving at %s...", addr) 357 | for { 358 | conn, e := m.stopListener.Accept() 359 | if e != nil { 360 | // log.Printf("Accept failed: %s\n", e) 361 | return e 362 | } 363 | 364 | go m.handlerConn(conn) 365 | } 366 | } 367 | 368 | // Stop implements the Stop interface 369 | func (m *McEntry) Stop() { 370 | log.Printf("mc entry stoping...") 371 | m.stopListener.Stop() 372 | m.messageQueue.Close() 373 | log.Printf("mc entry stoped.") 374 | } 375 | -------------------------------------------------------------------------------- /entry/mcEntry_test.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "testing" 5 | "time" 6 | 7 | "github.com/bradfitz/gomemcache/memcache" 8 | "github.com/buaazp/uq/queue" 9 | "github.com/buaazp/uq/store" 10 | . "github.com/smartystreets/goconvey/convey" 11 | ) 12 | 13 | var mc *memcache.Client 14 | 15 | func init() { 16 | mc = memcache.New("localhost:8802") 17 | } 18 | 19 | func TestNewMcEntry(t *testing.T) { 20 | Convey("Test New Mc Entry", t, func() { 21 | var err error 22 | storage, err = store.NewMemStore() 23 | So(err, ShouldBeNil) 24 | So(storage, ShouldNotBeNil) 25 | messageQueue, err = queue.NewUnitedQueue(storage, "127.0.0.1", 8802, nil, "uq") 26 | So(err, ShouldBeNil) 27 | So(messageQueue, ShouldNotBeNil) 28 | 29 | entrance, err = NewMcEntry("0.0.0.0", 8802, messageQueue) 30 | So(err, ShouldBeNil) 31 | So(entrance, ShouldNotBeNil) 32 | 33 | go func() { 34 | entrance.ListenAndServe() 35 | }() 36 | time.Sleep(100 * time.Millisecond) 37 | }) 38 | } 39 | 40 | func TestMcAdd(t *testing.T) { 41 | Convey("Test Mc Add Api", t, func() { 42 | err := mc.Add(&memcache.Item{ 43 | Key: "foo", 44 | Value: []byte{}, 45 | }) 46 | So(err, ShouldBeNil) 47 | 48 | err = mc.Add(&memcache.Item{ 49 | Key: "foo/x", 50 | Value: []byte("10s"), 51 | }) 52 | So(err, ShouldBeNil) 53 | }) 54 | } 55 | 56 | func TestMcPush(t *testing.T) { 57 | Convey("Test Mc Push Api", t, func() { 58 | err := mc.Set(&memcache.Item{ 59 | Key: "foo", 60 | Value: []byte("1"), 61 | }) 62 | So(err, ShouldBeNil) 63 | }) 64 | } 65 | 66 | func TestMcPop(t *testing.T) { 67 | Convey("Test Mc Pop Api", t, func() { 68 | it, err := mc.Get("foo/x") 69 | So(err, ShouldBeNil) 70 | v := string(it.Value) 71 | So(v, ShouldEqual, "1") 72 | }) 73 | } 74 | 75 | func TestMcConfirm(t *testing.T) { 76 | Convey("Test Mc Confirm Api", t, func() { 77 | err := mc.Delete("foo/x/0") 78 | So(err, ShouldBeNil) 79 | }) 80 | } 81 | 82 | func TestCloseMcEntry(t *testing.T) { 83 | Convey("Test Close Mc Entry", t, func() { 84 | entrance.Stop() 85 | messageQueue = nil 86 | storage = nil 87 | }) 88 | } 89 | -------------------------------------------------------------------------------- /entry/mcResponse.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | ) 7 | 8 | type response struct { 9 | status string 10 | msg string 11 | noreply bool 12 | items map[string]*item 13 | } 14 | 15 | func writeFull(w io.Writer, buf []byte) error { 16 | n, e := w.Write(buf) 17 | for e != nil && n > 0 { 18 | buf = buf[n:] 19 | n, e = w.Write(buf) 20 | } 21 | return e 22 | } 23 | 24 | func (resp *response) Write(w io.Writer) error { 25 | if resp.noreply { 26 | return nil 27 | } 28 | 29 | switch resp.status { 30 | case "VALUE": 31 | if resp.items != nil { 32 | for key, item := range resp.items { 33 | fmt.Fprintf(w, "VALUE %s %d %d\r\n", key, item.flag, len(item.body)) 34 | if e := writeFull(w, item.body); e != nil { 35 | return e 36 | } 37 | writeFull(w, []byte("\r\n")) 38 | } 39 | } 40 | io.WriteString(w, "END\r\n") 41 | 42 | case "STAT": 43 | io.WriteString(w, resp.msg) 44 | io.WriteString(w, "\r\n") 45 | io.WriteString(w, "END\r\n") 46 | 47 | default: 48 | io.WriteString(w, resp.status) 49 | if resp.msg != "" { 50 | io.WriteString(w, " "+resp.msg) 51 | } 52 | io.WriteString(w, "\r\n") 53 | } 54 | return nil 55 | } 56 | -------------------------------------------------------------------------------- /entry/redisCommand.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "bytes" 5 | "errors" 6 | "fmt" 7 | "strconv" 8 | 9 | "github.com/buaazp/uq/utils" 10 | ) 11 | 12 | type command struct { 13 | args [][]byte 14 | attrs map[string]interface{} 15 | } 16 | 17 | func newCommand(args ...[]byte) (cmd *command) { 18 | cmd = &command{ 19 | args: args, 20 | attrs: make(map[string]interface{}), 21 | } 22 | return 23 | } 24 | 25 | func (cmd *command) setAttribute(name string, v interface{}) { 26 | cmd.attrs[name] = v 27 | } 28 | 29 | func (cmd *command) getAttribute(name string) interface{} { 30 | return cmd.attrs[name] 31 | } 32 | 33 | func (cmd *command) name() string { 34 | return string(bytes.ToUpper(cmd.args[0])) 35 | } 36 | 37 | func (cmd *command) stringArgs() []string { 38 | strings := make([]string, len(cmd.args)) 39 | for i, arg := range cmd.args { 40 | strings[i] = string(arg) 41 | } 42 | return strings 43 | } 44 | 45 | func (cmd *command) stringAtIndex(i int) string { 46 | if i >= cmd.length() { 47 | return "" 48 | } 49 | return string(cmd.args[i]) 50 | } 51 | 52 | func (cmd *command) argAtIndex(i int) (arg []byte, err error) { 53 | if i >= cmd.length() { 54 | err = fmt.Errorf("out of range %d/%d", i, cmd.length()) 55 | return 56 | } 57 | arg = cmd.args[i] 58 | return 59 | } 60 | 61 | func (cmd *command) intAtIndex(i int) (n int, err error) { 62 | if i >= cmd.length() { 63 | err = fmt.Errorf("out of range %d/%d", i, cmd.length()) 64 | return 65 | } 66 | n, err = strconv.Atoi(string(cmd.args[i])) 67 | return 68 | } 69 | 70 | func (cmd *command) int64AtIndex(i int) (n int64, err error) { 71 | if i >= cmd.length() { 72 | err = fmt.Errorf("out of range %d/%d", i, cmd.length()) 73 | return 74 | } 75 | n, err = strconv.ParseInt(string(cmd.args[i]), 10, 0) 76 | return 77 | } 78 | 79 | func (cmd *command) uint64AtIndex(i int) (n uint64, err error) { 80 | if i >= cmd.length() { 81 | err = fmt.Errorf("out of range %d/%d", i, cmd.length()) 82 | return 83 | } 84 | n, err = strconv.ParseUint(string(cmd.args[i]), 10, 0) 85 | return 86 | } 87 | 88 | func (cmd *command) floatAtIndex(i int) (n float64, err error) { 89 | if i >= cmd.length() { 90 | err = fmt.Errorf("out of range %d/%d", i, cmd.length()) 91 | return 92 | } 93 | n, err = strconv.ParseFloat(string(cmd.args[i]), 64) 94 | return 95 | } 96 | 97 | func (cmd *command) length() int { 98 | return len(cmd.args) 99 | } 100 | 101 | /* 102 | * CR LF 103 | $ CR LF 104 | CR LF 105 | ... 106 | $ CR LF 107 | CR LF 108 | */ 109 | func (cmd *command) bytes() []byte { 110 | buf := bytes.Buffer{} 111 | buf.WriteByte('*') 112 | argCount := cmd.length() 113 | buf.WriteString(utils.ItoaQuick(argCount)) // 114 | buf.WriteString(CRLF) 115 | for i := 0; i < argCount; i++ { 116 | buf.WriteByte('$') 117 | argSize := len(cmd.args[i]) 118 | buf.WriteString(utils.ItoaQuick(argSize)) // 119 | buf.WriteString(CRLF) 120 | buf.Write(cmd.args[i]) // 121 | buf.WriteString(CRLF) 122 | } 123 | return buf.Bytes() 124 | } 125 | 126 | func parseCommand(buf *bytes.Buffer) (*command, error) { 127 | // Read ( * CR LF ) 128 | if c, err := buf.ReadByte(); c != '*' { // io.EOF 129 | return nil, err 130 | } 131 | // number of arguments 132 | line, err := buf.ReadBytes(LF) 133 | if err != nil { 134 | return nil, err 135 | } 136 | argCount, _ := strconv.Atoi(string(line[:len(line)-2])) 137 | args := make([][]byte, argCount) 138 | for i := 0; i < argCount; i++ { 139 | // Read ( $ CR LF ) 140 | if c, err := buf.ReadByte(); c != '$' { 141 | return nil, err 142 | } 143 | 144 | line, err := buf.ReadBytes(LF) 145 | if err != nil { 146 | return nil, err 147 | } 148 | argSize, _ := strconv.Atoi(string(line[:len(line)-2])) 149 | // Read ( CR LF ) 150 | args[i] = make([]byte, argSize) 151 | n, e2 := buf.Read(args[i]) 152 | if n != argSize { 153 | return nil, errors.New("argSize too short") 154 | } 155 | if e2 != nil { 156 | return nil, e2 157 | } 158 | 159 | _, err = buf.ReadBytes(LF) 160 | if err != nil { 161 | return nil, err 162 | } 163 | } 164 | cmd := newCommand(args...) 165 | return cmd, nil 166 | } 167 | 168 | func (cmd *command) String() string { 169 | buf := bytes.Buffer{} 170 | for i, count := 0, cmd.length(); i < count; i++ { 171 | if i > 0 { 172 | buf.WriteString(" ") 173 | } 174 | buf.Write(cmd.args[i]) 175 | } 176 | return buf.String() 177 | } 178 | -------------------------------------------------------------------------------- /entry/redisEntry.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "log" 5 | "net" 6 | "time" 7 | 8 | "github.com/buaazp/uq/queue" 9 | "github.com/buaazp/uq/utils" 10 | ) 11 | 12 | const ( 13 | // CR is the \r 14 | CR = '\r' 15 | // LF is the \n 16 | LF = '\n' 17 | // CRLF is the \r\n 18 | CRLF = "\r\n" 19 | cSession = "session" 20 | cElapsed = "elapsed" 21 | ) 22 | 23 | // RedisEntry is the redis entrance of uq 24 | type RedisEntry struct { 25 | host string 26 | port int 27 | stopListener *utils.StopListener 28 | messageQueue queue.MessageQueue 29 | } 30 | 31 | // NewRedisEntry returns a new RedisEntry 32 | func NewRedisEntry(host string, port int, messageQueue queue.MessageQueue) (*RedisEntry, error) { 33 | rs := new(RedisEntry) 34 | rs.host = host 35 | rs.port = port 36 | rs.messageQueue = messageQueue 37 | return rs, nil 38 | } 39 | 40 | func (r *RedisEntry) onUndefined(ss *session, cmd *command) (rep *reply) { 41 | return errorReply(utils.NewError( 42 | utils.ErrBadRequest, 43 | "command not supported: "+cmd.String(), 44 | )) 45 | } 46 | 47 | func (r *RedisEntry) commandHandler(ss *session, cmd *command) (rep *reply) { 48 | cmdName := cmd.name() 49 | 50 | if cmdName == "ADD" || cmdName == "QADD" { 51 | rep = r.onQadd(cmd) 52 | } else if cmdName == "SET" || cmdName == "QPUSH" { 53 | rep = r.onQpush(cmd) 54 | } else if cmdName == "MSET" || cmdName == "QMPUSH" { 55 | rep = r.onQmpush(cmd) 56 | } else if cmdName == "GET" || cmdName == "QPOP" { 57 | rep = r.onQpop(cmd) 58 | } else if cmdName == "MGET" || cmdName == "QMPOP" { 59 | rep = r.onQmpop(cmd) 60 | } else if cmdName == "DEL" || cmdName == "QDEL" { 61 | rep = r.onQdel(cmd) 62 | } else if cmdName == "MDEL" || cmdName == "QMDEL" { 63 | rep = r.onQmdel(cmd) 64 | } else if cmdName == "EMPTY" || cmdName == "QEMPTY" { 65 | rep = r.onQempty(cmd) 66 | } else if cmdName == "INFO" || cmdName == "QINFO" { 67 | rep = r.onInfo(cmd) 68 | } else { 69 | rep = r.onUndefined(ss, cmd) 70 | } 71 | 72 | return 73 | } 74 | 75 | func (r *RedisEntry) process(ss *session, cmd *command) (rep *reply) { 76 | // invoke & time 77 | begin := time.Now() 78 | cmd.setAttribute(cSession, ss) 79 | 80 | // varify command 81 | if err := verifyCommand(cmd); err != nil { 82 | // log.Printf("[%s] bad command %s\n", ss.RemoteAddr(), cmd) 83 | return errorReply(utils.NewError( 84 | utils.ErrBadRequest, 85 | err.Error(), 86 | )) 87 | } 88 | 89 | // invoke 90 | rep = r.commandHandler(ss, cmd) 91 | 92 | elapsed := time.Now().Sub(begin) 93 | cmd.setAttribute(cElapsed, elapsed) 94 | 95 | return 96 | } 97 | 98 | func (r *RedisEntry) handlerConn(ss *session) { 99 | // addr := ss.RemoteAddr().String() 100 | // log.Printf("handleClient: %s", addr) 101 | 102 | for { 103 | // var cmd *command 104 | cmd, err := ss.readCommand() 105 | // 1) io.EOF 106 | // 2) read tcp 127.0.0.1:51863: connection reset by peer 107 | if err != nil { 108 | log.Printf("session read command error: %s", err) 109 | break 110 | } 111 | // log.Printf("cmd: %v", cmd) 112 | rep := r.process(ss, cmd) 113 | if rep != nil { 114 | err = ss.writeReply(rep) 115 | if err != nil { 116 | break 117 | } 118 | } 119 | } 120 | 121 | // log.Printf("session %s closing...", addr) 122 | if err := ss.Close(); err != nil { 123 | // log.Printf("session %s close error: %s", addr, err) 124 | } 125 | 126 | return 127 | } 128 | 129 | // ListenAndServe implements the ListenAndServe interface 130 | func (r *RedisEntry) ListenAndServe() error { 131 | addr := utils.Addrcat(r.host, r.port) 132 | l, err := net.Listen("tcp", addr) 133 | if err != nil { 134 | return err 135 | } 136 | 137 | stopListener, err := utils.NewStopListener(l) 138 | if err != nil { 139 | return err 140 | } 141 | r.stopListener = stopListener 142 | 143 | log.Printf("redis entrance serving at %s...", addr) 144 | for { 145 | conn, err := r.stopListener.Accept() 146 | if err != nil { 147 | // log.Printf("Accept failed: %s\n", err) 148 | return err 149 | } 150 | go r.handlerConn(newSession(conn)) 151 | } 152 | } 153 | 154 | // Stop implements the Stop interface 155 | func (r *RedisEntry) Stop() { 156 | log.Printf("redis entry stoping...") 157 | r.stopListener.Stop() 158 | r.messageQueue.Close() 159 | } 160 | -------------------------------------------------------------------------------- /entry/redisEntry_test.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "testing" 5 | "time" 6 | 7 | "github.com/buaazp/uq/queue" 8 | "github.com/buaazp/uq/store" 9 | "github.com/garyburd/redigo/redis" 10 | . "github.com/smartystreets/goconvey/convey" 11 | ) 12 | 13 | var conn redis.Conn 14 | 15 | func TestNewRedisEntry(t *testing.T) { 16 | Convey("Test New Redis Entry", t, func() { 17 | var err error 18 | storage, err = store.NewMemStore() 19 | So(err, ShouldBeNil) 20 | So(storage, ShouldNotBeNil) 21 | messageQueue, err = queue.NewUnitedQueue(storage, "127.0.0.1", 8803, nil, "uq") 22 | So(err, ShouldBeNil) 23 | So(messageQueue, ShouldNotBeNil) 24 | 25 | entrance, err = NewRedisEntry("0.0.0.0", 8803, messageQueue) 26 | So(err, ShouldBeNil) 27 | So(entrance, ShouldNotBeNil) 28 | 29 | go func() { 30 | entrance.ListenAndServe() 31 | }() 32 | time.Sleep(100 * time.Millisecond) 33 | }) 34 | } 35 | 36 | func TestRedisAdd(t *testing.T) { 37 | Convey("Test Redis Add Api", t, func() { 38 | conn, _ = redis.DialTimeout("tcp", "127.0.0.1:8803", 0, 1*time.Second, 1*time.Second) 39 | So(conn, ShouldNotBeNil) 40 | 41 | _, err := conn.Do("QADD", "foo") 42 | So(err, ShouldBeNil) 43 | 44 | _, err = conn.Do("QADD", "foo/x", "10s") 45 | So(err, ShouldBeNil) 46 | }) 47 | } 48 | 49 | func TestRedisPush(t *testing.T) { 50 | Convey("Test Redis Push Api", t, func() { 51 | _, err := conn.Do("QPUSH", "foo", "1") 52 | So(err, ShouldBeNil) 53 | }) 54 | } 55 | 56 | func TestRedisPop(t *testing.T) { 57 | Convey("Test Redis Pop Api", t, func() { 58 | rpl, err := redis.Values(conn.Do("QPOP", "foo/x")) 59 | So(err, ShouldBeNil) 60 | v, err := redis.String(rpl[0], err) 61 | So(err, ShouldBeNil) 62 | So(v, ShouldEqual, "1") 63 | id, err := redis.String(rpl[1], err) 64 | So(err, ShouldBeNil) 65 | So(id, ShouldEqual, "foo/x/0") 66 | }) 67 | } 68 | 69 | func TestRedisConfirm(t *testing.T) { 70 | Convey("Test Redis Confirm Api", t, func() { 71 | _, err := conn.Do("QDEL", "foo/x/0") 72 | So(err, ShouldBeNil) 73 | }) 74 | } 75 | 76 | func TestCloseRedisEntry(t *testing.T) { 77 | Convey("Test Close Redis Entry", t, func() { 78 | entrance.Stop() 79 | messageQueue = nil 80 | storage = nil 81 | }) 82 | } 83 | -------------------------------------------------------------------------------- /entry/redisQueue.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "github.com/buaazp/uq/utils" 5 | ) 6 | 7 | func (r *RedisEntry) onQadd(cmd *command) *reply { 8 | key := cmd.stringAtIndex(1) 9 | recycle := cmd.stringAtIndex(2) 10 | 11 | // log.Printf("creating... %s %s", key, recycle) 12 | err := r.messageQueue.Create(key, recycle) 13 | if err != nil { 14 | return errorReply(err) 15 | } 16 | return statusReply("OK") 17 | } 18 | 19 | func (r *RedisEntry) onQpush(cmd *command) *reply { 20 | key := cmd.stringAtIndex(1) 21 | val, err := cmd.argAtIndex(2) 22 | if err != nil { 23 | return errorReply(utils.NewError( 24 | utils.ErrBadRequest, 25 | err.Error(), 26 | )) 27 | } 28 | 29 | err = r.messageQueue.Push(key, val) 30 | if err != nil { 31 | return errorReply(err) 32 | } 33 | return statusReply("OK") 34 | } 35 | 36 | func (r *RedisEntry) onQmpush(cmd *command) *reply { 37 | key := cmd.stringAtIndex(1) 38 | vals := cmd.args[2:] 39 | 40 | err := r.messageQueue.MultiPush(key, vals) 41 | if err != nil { 42 | return errorReply(err) 43 | } 44 | 45 | return statusReply("OK") 46 | } 47 | 48 | func (r *RedisEntry) onQpop(cmd *command) *reply { 49 | key := cmd.stringAtIndex(1) 50 | 51 | id, value, err := r.messageQueue.Pop(key) 52 | if err != nil { 53 | return errorReply(err) 54 | } 55 | 56 | vals := make([]interface{}, 2) 57 | vals[0] = value 58 | vals[1] = id 59 | 60 | return multiBulksReply(vals) 61 | } 62 | 63 | func (r *RedisEntry) onQmpop(cmd *command) *reply { 64 | key := cmd.stringAtIndex(1) 65 | n, err := cmd.intAtIndex(2) 66 | if err != nil { 67 | return errorReply(utils.NewError( 68 | utils.ErrBadRequest, 69 | err.Error(), 70 | )) 71 | } 72 | 73 | ids, values, err := r.messageQueue.MultiPop(key, n) 74 | if err != nil { 75 | return errorReply(err) 76 | } 77 | 78 | np := len(ids) 79 | 80 | vals := make([]interface{}, np*2) 81 | for i, index := 0, 0; i < np; i++ { 82 | vals[index] = values[i] 83 | index++ 84 | 85 | vals[index] = ids[i] 86 | index++ 87 | } 88 | 89 | return multiBulksReply(vals) 90 | } 91 | 92 | func (r *RedisEntry) onQdel(cmd *command) *reply { 93 | key := cmd.stringAtIndex(1) 94 | 95 | err := r.messageQueue.Confirm(key) 96 | if err != nil { 97 | // log.Printf("confirm error: %s", err) 98 | return errorReply(err) 99 | } 100 | 101 | return statusReply("OK") 102 | } 103 | 104 | func (r *RedisEntry) onQmdel(cmd *command) *reply { 105 | keys := cmd.stringArgs()[1:] 106 | // log.Printf("keys: %v", keys) 107 | 108 | errs := r.messageQueue.MultiConfirm(keys) 109 | 110 | vals := make([]interface{}, len(errs)) 111 | for i, err := range errs { 112 | if err != nil { 113 | vals[i] = err.Error() 114 | } else { 115 | vals[i] = "OK" 116 | } 117 | } 118 | 119 | return multiBulksReply(vals) 120 | } 121 | 122 | func (r *RedisEntry) onQempty(cmd *command) *reply { 123 | key := cmd.stringAtIndex(1) 124 | 125 | err := r.messageQueue.Empty(key) 126 | if err != nil { 127 | return errorReply(err) 128 | } 129 | return statusReply("OK") 130 | } 131 | 132 | func (r *RedisEntry) onInfo(cmd *command) *reply { 133 | key := cmd.stringAtIndex(1) 134 | 135 | qs, err := r.messageQueue.Stat(key) 136 | if err != nil { 137 | return errorReply(err) 138 | } 139 | 140 | // for human reading 141 | strs := qs.ToRedisStrings() 142 | vals := make([]interface{}, len(strs)) 143 | for i, str := range strs { 144 | vals[i] = str 145 | } 146 | return multiBulksReply(vals) 147 | 148 | // for json format 149 | // data, err := qs.ToJSON() 150 | // if err != nil { 151 | // return errorReply(utils.NewError( 152 | // utils.ErrInternalError, 153 | // err.Error(), 154 | // )) 155 | // } 156 | 157 | // return statusReply(string(data)) 158 | } 159 | -------------------------------------------------------------------------------- /entry/redisReply.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | ) 7 | 8 | type reply struct { 9 | rType replyType 10 | value interface{} 11 | } 12 | 13 | type replyType int 14 | 15 | const ( 16 | replyTypeStatus replyType = iota 17 | replyTypeError 18 | replyTypeInteger 19 | replyTypeBulk 20 | replyTypeMultiBulks 21 | ) 22 | 23 | var replyTypeDesc = map[replyType]string{ 24 | replyTypeStatus: "statusReply", 25 | replyTypeError: "errorReply", 26 | replyTypeInteger: "integerReply", 27 | replyTypeBulk: "bulkReply", 28 | replyTypeMultiBulks: "multiBulksReply", 29 | } 30 | 31 | func statusReply(status string) (r *reply) { 32 | r = &reply{} 33 | r.rType = replyTypeStatus 34 | r.value = status 35 | return 36 | } 37 | 38 | func errorReply(err error) (r *reply) { 39 | r = &reply{} 40 | r.rType = replyTypeError 41 | if err != nil { 42 | r.value = err.Error() 43 | } 44 | return 45 | } 46 | 47 | func integerReply(i int) (r *reply) { 48 | r = &reply{} 49 | r.rType = replyTypeInteger 50 | r.value = i 51 | return 52 | } 53 | 54 | func bulkReply(bulk interface{}) (r *reply) { 55 | r = &reply{} 56 | r.rType = replyTypeBulk 57 | r.value = bulk 58 | return 59 | } 60 | 61 | func multiBulksReply(bulks []interface{}) (r *reply) { 62 | r = &reply{} 63 | r.rType = replyTypeMultiBulks 64 | r.value = bulks 65 | return 66 | } 67 | 68 | func (r *reply) String() string { 69 | buf := bytes.Buffer{} 70 | buf.WriteString("<") 71 | buf.WriteString(replyTypeDesc[r.rType]) 72 | buf.WriteString(":") 73 | switch r.value.(type) { 74 | case []byte: 75 | buf.WriteString(string(r.value.([]byte))) 76 | default: 77 | buf.WriteString(fmt.Sprint(r.value)) 78 | } 79 | buf.WriteString(">") 80 | return buf.String() 81 | } 82 | -------------------------------------------------------------------------------- /entry/redisSession.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "bufio" 5 | "bytes" 6 | "errors" 7 | "fmt" 8 | "io" 9 | "net" 10 | "strconv" 11 | 12 | "github.com/buaazp/uq/utils" 13 | ) 14 | 15 | type session struct { 16 | net.Conn 17 | rw *bufio.Reader 18 | attrs map[string]interface{} 19 | } 20 | 21 | func newSession(conn net.Conn) (s *session) { 22 | s = &session{ 23 | Conn: conn, 24 | attrs: make(map[string]interface{}), 25 | } 26 | s.rw = bufio.NewReader(s.Conn) 27 | return 28 | } 29 | 30 | func (s *session) setAttribute(name string, v interface{}) { 31 | s.attrs[name] = v 32 | } 33 | 34 | func (s *session) getAttribute(name string) interface{} { 35 | return s.attrs[name] 36 | } 37 | 38 | func (s *session) writeReply(rep *reply) (err error) { 39 | switch rep.rType { 40 | case replyTypeStatus: 41 | err = s.replyStatus(rep.value.(string)) 42 | case replyTypeError: 43 | err = s.replyError(rep.value.(string)) 44 | case replyTypeInteger: 45 | err = s.replyInteger(rep.value.(int)) 46 | case replyTypeBulk: 47 | err = s.replyBulk(rep.value) 48 | case replyTypeMultiBulks: 49 | err = s.replyMultiBulks(rep.value.([]interface{})) 50 | default: 51 | err = errors.New("Illegal ReplyType: " + utils.ItoaQuick(int(rep.rType))) 52 | } 53 | return 54 | } 55 | 56 | func (s *session) writeCommand(cmd *command) (err error) { 57 | _, err = s.Write(cmd.bytes()) 58 | return 59 | } 60 | 61 | // ReadReply reads the reply from the session 62 | // In a Status Reply the first byte of the reply is "+" 63 | // In an Error Reply the first byte of the reply is "-" 64 | // In an Integer Reply the first byte of the reply is ":" 65 | // In a Bulk Reply the first byte of the reply is "$" 66 | // In a Multi Bulk Reply the first byte of the reply s "*" 67 | func (s *session) readReply() (rep *reply, err error) { 68 | reader := s.rw 69 | var c byte 70 | if c, err = reader.ReadByte(); err != nil { 71 | return 72 | } 73 | 74 | rep = &reply{} 75 | switch c { 76 | case '+': 77 | rep.rType = replyTypeStatus 78 | rep.value, err = s.readString() 79 | case '-': 80 | rep.rType = replyTypeError 81 | rep.value, err = s.readString() 82 | case ':': 83 | rep.rType = replyTypeInteger 84 | rep.value, err = s.readInt() 85 | case '$': 86 | rep.rType = replyTypeBulk 87 | var bufsize int 88 | bufsize, err = s.readInt() 89 | if err != nil { 90 | break 91 | } 92 | buf := make([]byte, bufsize) 93 | _, err = io.ReadFull(s, buf) 94 | if err != nil { 95 | break 96 | } 97 | rep.value = buf 98 | s.skipBytes([]byte{CR, LF}) 99 | case '*': 100 | rep.rType = replyTypeMultiBulks 101 | var argCount int 102 | argCount, err = s.readInt() 103 | if err != nil { 104 | break 105 | } 106 | if argCount == -1 { 107 | rep.value = nil // *-1 108 | } else { 109 | args := make([]interface{}, argCount) 110 | for i := 0; i < argCount; i++ { 111 | // TODO multi bulk 的类型 $和: 112 | err = s.skipByte('$') 113 | if err != nil { 114 | break 115 | } 116 | var argSize int 117 | argSize, err = s.readInt() 118 | if err != nil { 119 | return 120 | } 121 | if argSize == -1 { 122 | args[i] = nil 123 | } else { 124 | arg := make([]byte, argSize) 125 | _, err = io.ReadFull(s, arg) 126 | if err != nil { 127 | break 128 | } 129 | args[i] = arg 130 | } 131 | s.skipBytes([]byte{CR, LF}) 132 | } 133 | rep.value = args 134 | } 135 | default: 136 | err = errors.New("Bad Reply Flag:" + string([]byte{c})) 137 | } 138 | return 139 | } 140 | 141 | /* 142 | * CR LF 143 | $ CR LF 144 | CR LF 145 | ... 146 | $ CR LF 147 | CR LF 148 | */ 149 | func (s *session) readCommand() (cmd *command, err error) { 150 | // Read ( * CR LF ) 151 | err = s.skipByte('*') 152 | if err != nil { // io.EOF 153 | return 154 | } 155 | // number of arguments 156 | var argCount int 157 | if argCount, err = s.readInt(); err != nil { 158 | return 159 | } 160 | args := make([][]byte, argCount) 161 | for i := 0; i < argCount; i++ { 162 | // Read ( $ CR LF ) 163 | err = s.skipByte('$') 164 | if err != nil { 165 | return 166 | } 167 | 168 | var argSize int 169 | argSize, err = s.readInt() 170 | if err != nil { 171 | return 172 | } 173 | 174 | // Read ( CR LF ) 175 | args[i] = make([]byte, argSize) 176 | _, err = io.ReadFull(s, args[i]) 177 | if err != nil { 178 | return 179 | } 180 | 181 | err = s.skipBytes([]byte{CR, LF}) 182 | if err != nil { 183 | return 184 | } 185 | } 186 | cmd = newCommand(args...) 187 | return 188 | } 189 | 190 | // Status reply 191 | func (s *session) replyStatus(status string) (err error) { 192 | buf := bytes.Buffer{} 193 | buf.WriteString("+") 194 | buf.WriteString(status) 195 | buf.WriteString(CRLF) 196 | _, err = buf.WriteTo(s) 197 | return 198 | } 199 | 200 | // Error reply 201 | func (s *session) replyError(errmsg string) (err error) { 202 | buf := bytes.Buffer{} 203 | buf.WriteString("-") 204 | buf.WriteString(errmsg) 205 | buf.WriteString(CRLF) 206 | _, err = buf.WriteTo(s) 207 | return 208 | } 209 | 210 | // Integer reply 211 | func (s *session) replyInteger(i int) (err error) { 212 | buf := bytes.Buffer{} 213 | buf.WriteString(":") 214 | buf.WriteString(utils.ItoaQuick(i)) 215 | buf.WriteString(CRLF) 216 | _, err = buf.WriteTo(s) 217 | return 218 | } 219 | 220 | // Bulk Reply 221 | func (s *session) replyBulk(bulk interface{}) (err error) { 222 | // NULL Bulk Reply 223 | isnil := bulk == nil 224 | if !isnil { 225 | b, ok := bulk.([]byte) 226 | isnil = ok && b == nil 227 | } 228 | if isnil { 229 | _, err = s.Write([]byte("$-1\r\n")) 230 | return 231 | } 232 | buf := bytes.Buffer{} 233 | buf.WriteString("$") 234 | switch bulk.(type) { 235 | case []byte: 236 | b := bulk.([]byte) 237 | buf.WriteString(utils.ItoaQuick(len(b))) 238 | buf.WriteString(CRLF) 239 | buf.Write(b) 240 | default: 241 | b := []byte(bulk.(string)) 242 | buf.WriteString(utils.ItoaQuick(len(b))) 243 | buf.WriteString(CRLF) 244 | buf.Write(b) 245 | } 246 | buf.WriteString(CRLF) 247 | _, err = buf.WriteTo(s) 248 | return 249 | } 250 | 251 | // Multi-bulk replies 252 | func (s *session) replyMultiBulks(bulks []interface{}) (err error) { 253 | // Null Multi Bulk Reply 254 | if bulks == nil { 255 | _, err = s.Write([]byte("*-1\r\n")) 256 | return 257 | } 258 | bulkCount := len(bulks) 259 | // Empty Multi Bulk Reply 260 | if bulkCount == 0 { 261 | _, err = s.Write([]byte("*0\r\n")) 262 | return 263 | } 264 | buf := bytes.Buffer{} 265 | buf.WriteString("*") 266 | buf.WriteString(utils.ItoaQuick(bulkCount)) 267 | buf.WriteString(CRLF) 268 | for i := 0; i < bulkCount; i++ { 269 | bulk := bulks[i] 270 | switch bulk.(type) { 271 | case string: 272 | buf.WriteString("$") 273 | b := []byte(bulk.(string)) 274 | buf.WriteString(utils.ItoaQuick(len(b))) 275 | buf.WriteString(CRLF) 276 | buf.Write(b) 277 | buf.WriteString(CRLF) 278 | case []byte: 279 | b := bulk.([]byte) 280 | if b == nil { 281 | buf.WriteString("$-1") 282 | buf.WriteString(CRLF) 283 | } else { 284 | buf.WriteString("$") 285 | buf.WriteString(utils.ItoaQuick(len(b))) 286 | buf.WriteString(CRLF) 287 | buf.Write(b) 288 | buf.WriteString(CRLF) 289 | } 290 | case int: 291 | buf.WriteString(":") 292 | buf.WriteString(utils.ItoaQuick(bulk.(int))) 293 | buf.WriteString(CRLF) 294 | case uint64: 295 | buf.WriteString(":") 296 | buf.WriteString(strconv.FormatUint(bulk.(uint64), 10)) 297 | buf.WriteString(CRLF) 298 | default: 299 | // nil element 300 | buf.WriteString("$-1") 301 | buf.WriteString(CRLF) 302 | } 303 | } 304 | // flush 305 | _, err = buf.WriteTo(s) 306 | return 307 | } 308 | 309 | // ==================================== 310 | // io 311 | // ==================================== 312 | func (s *session) skipByte(c byte) (err error) { 313 | var tmp byte 314 | tmp, err = s.rw.ReadByte() 315 | if err != nil { 316 | return 317 | } 318 | if tmp != c { 319 | err = fmt.Errorf("Illegal Byte [%d] != [%d]", tmp, c) 320 | } 321 | return 322 | } 323 | 324 | func (s *session) skipBytes(bs []byte) (err error) { 325 | for _, c := range bs { 326 | err = s.skipByte(c) 327 | if err != nil { 328 | break 329 | } 330 | } 331 | return 332 | } 333 | 334 | func (s *session) readLine() (line []byte, err error) { 335 | line, err = s.rw.ReadSlice(LF) 336 | if err == bufio.ErrBufferFull { 337 | return nil, errors.New("line too long") 338 | } 339 | if err != nil { 340 | return 341 | } 342 | i := len(line) - 2 343 | if i < 0 || line[i] != CR { 344 | err = errors.New("bad line terminator:" + string(line)) 345 | } 346 | return line[:i], nil 347 | } 348 | 349 | func (s *session) readString() (str string, err error) { 350 | var line []byte 351 | if line, err = s.readLine(); err != nil { 352 | return 353 | } 354 | str = string(line) 355 | return 356 | } 357 | 358 | func (s *session) readInt() (i int, err error) { 359 | var line string 360 | if line, err = s.readString(); err != nil { 361 | return 362 | } 363 | i, err = strconv.Atoi(line) 364 | return 365 | } 366 | 367 | func (s *session) readInt64() (i int64, err error) { 368 | var line string 369 | if line, err = s.readString(); err != nil { 370 | return 371 | } 372 | i, err = strconv.ParseInt(line, 10, 64) 373 | return 374 | } 375 | 376 | func (s *session) Read(p []byte) (n int, err error) { 377 | return s.rw.Read(p) 378 | } 379 | 380 | func (s *session) readByte() (c byte, err error) { 381 | return s.rw.ReadByte() 382 | } 383 | 384 | func (s *session) peekByte() (c byte, err error) { 385 | if b, e := s.rw.Peek(1); e == nil { 386 | c = b[0] 387 | } 388 | return 389 | } 390 | 391 | func (s *session) readRDB(w io.Writer) (err error) { 392 | // Read ( $ CR LF ) 393 | if err = s.skipByte('$'); err != nil { 394 | return 395 | } 396 | 397 | var rdbSize int64 398 | if rdbSize, err = s.readInt64(); err != nil { 399 | return 400 | } 401 | 402 | var c byte 403 | for i := int64(0); i < rdbSize; i++ { 404 | c, err = s.rw.ReadByte() 405 | if err != nil { 406 | return 407 | } 408 | w.Write([]byte{c}) 409 | } 410 | return 411 | } 412 | 413 | func (s *session) String() string { 414 | return fmt.Sprintf("", s.RemoteAddr()) 415 | } 416 | -------------------------------------------------------------------------------- /entry/redisValidate.go: -------------------------------------------------------------------------------- 1 | package entry 2 | 3 | import ( 4 | "errors" 5 | "strings" 6 | ) 7 | 8 | var ( 9 | errBadCommand = errors.New("bad command") 10 | errWrongArgumentCount = errors.New("wrong argument count") 11 | errWrongCommandKey = errors.New("wrong command key") 12 | ) 13 | 14 | const ( 15 | riMinCount = iota 16 | riMaxCount // -1 for undefined 17 | ) 18 | 19 | var cmdrules = map[string][]interface{}{ 20 | // queue 21 | "ADD": []interface{}{2, 3}, 22 | "QADD": []interface{}{2, 3}, 23 | "SET": []interface{}{3, 3}, 24 | "QPUSH": []interface{}{3, 3}, 25 | "MSET": []interface{}{3, -1}, 26 | "QMPUSH": []interface{}{3, -1}, 27 | "GET": []interface{}{2, 2}, 28 | "QPOP": []interface{}{2, 2}, 29 | "MGET": []interface{}{3, -1}, 30 | "QMPOP": []interface{}{3, -1}, 31 | "DEL": []interface{}{2, 2}, 32 | "QDEL": []interface{}{2, 2}, 33 | "MDEL": []interface{}{2, -1}, 34 | "QMDEL": []interface{}{2, -1}, 35 | "EMPTY": []interface{}{2, 2}, 36 | "QEMPTY": []interface{}{2, 2}, 37 | "INFO": []interface{}{2, 2}, 38 | "QINFO": []interface{}{2, 2}, 39 | } 40 | 41 | func verifyCommand(cmd *command) error { 42 | if cmd == nil || cmd.length() == 0 { 43 | return errBadCommand 44 | } 45 | 46 | name := cmd.name() 47 | rule, exist := cmdrules[name] 48 | if !exist { 49 | return nil 50 | } 51 | 52 | for i, count := 0, len(rule); i < count; i++ { 53 | switch i { 54 | case riMinCount: 55 | if val := rule[i].(int); val != -1 && cmd.length() < val { 56 | return errWrongArgumentCount 57 | } 58 | case riMaxCount: 59 | if val := rule[i].(int); val != -1 && cmd.length() > val { 60 | return errWrongArgumentCount 61 | } 62 | } 63 | } 64 | 65 | if cmd.length() > 1 { 66 | key := cmd.stringAtIndex(1) 67 | if strings.ContainsAny(key, "#[] ") { 68 | return errWrongCommandKey 69 | } 70 | } 71 | return nil 72 | } 73 | -------------------------------------------------------------------------------- /queue/fakeQ.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "github.com/buaazp/uq/store" 5 | ) 6 | 7 | // FakeQueue is a fake queue in uq 8 | type FakeQueue struct{} 9 | 10 | // NewFakeQueue returns a new FakeQueue 11 | func NewFakeQueue(storage store.Storage, ip string, port int, etcdServers []string, etcdKey string) (*FakeQueue, error) { 12 | fq := new(FakeQueue) 13 | return fq, nil 14 | } 15 | 16 | // queue functions 17 | 18 | // Push implements Push interface 19 | func (f *FakeQueue) Push(key string, data []byte) error { 20 | return nil 21 | } 22 | 23 | // MultiPush implements MultiPush interface 24 | func (f *FakeQueue) MultiPush(key string, datas [][]byte) error { 25 | return nil 26 | } 27 | 28 | // Pop implements Pop interface 29 | func (f *FakeQueue) Pop(key string) (string, []byte, error) { 30 | return "", nil, nil 31 | } 32 | 33 | // MultiPop implements MultiPop interface 34 | func (f *FakeQueue) MultiPop(key string, n int) ([]string, [][]byte, error) { 35 | return nil, nil, nil 36 | } 37 | 38 | // Confirm implements Confirm interface 39 | func (f *FakeQueue) Confirm(key string) error { 40 | return nil 41 | } 42 | 43 | // MultiConfirm implements MultiConfirm interface 44 | func (f *FakeQueue) MultiConfirm(keys []string) []error { 45 | return nil 46 | } 47 | 48 | // admin functions 49 | 50 | // Create implements Create interface 51 | func (f *FakeQueue) Create(key, recycle string) error { 52 | return nil 53 | } 54 | 55 | // Empty implements Empty interface 56 | func (f *FakeQueue) Empty(key string) error { 57 | return nil 58 | } 59 | 60 | // Remove implements Remove interface 61 | func (f *FakeQueue) Remove(key string) error { 62 | return nil 63 | } 64 | 65 | // Stat implements Stat interface 66 | func (f *FakeQueue) Stat(key string) (*Stat, error) { 67 | return nil, nil 68 | } 69 | 70 | // Close implements Close interface 71 | func (f *FakeQueue) Close() { 72 | return 73 | } 74 | -------------------------------------------------------------------------------- /queue/queue.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | // MessageQueue is the message queue interface of uq 4 | type MessageQueue interface { 5 | // queue functions 6 | Push(key string, data []byte) error 7 | MultiPush(key string, datas [][]byte) error 8 | Pop(key string) (string, []byte, error) 9 | MultiPop(key string, n int) ([]string, [][]byte, error) 10 | Confirm(key string) error 11 | MultiConfirm(keys []string) []error 12 | // admin functions 13 | Create(key, recycle string) error 14 | Empty(key string) error 15 | Remove(key string) error 16 | Stat(key string) (*Stat, error) 17 | Close() 18 | } 19 | -------------------------------------------------------------------------------- /queue/uEtcd.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "strings" 5 | "time" 6 | 7 | "github.com/coreos/go-etcd/etcd" 8 | ) 9 | 10 | const ( 11 | etcdUqServerListValue string = "online" 12 | etcdTTL uint64 = 60 13 | oneSecond uint64 = uint64(time.Second) 14 | etcdWatchDelay time.Duration = 3 * time.Second 15 | etcdRegisterDelay time.Duration = 3 * time.Second 16 | ) 17 | 18 | func (u *UnitedQueue) nodeCreate(node *etcd.Node) error { 19 | // key: /uq/topics/foo/z 20 | key := node.Key 21 | name := strings.TrimPrefix(key, "/"+u.etcdKey+"/topics/") 22 | recycle := node.Value 23 | 24 | return u.create(name, recycle, true) 25 | } 26 | 27 | func (u *UnitedQueue) nodeRemove(node *etcd.Node) error { 28 | // key: /uq/topics/foo/z 29 | key := node.Key 30 | name := strings.TrimPrefix(key, "/"+u.etcdKey+"/topics/") 31 | 32 | return u.remove(name, true) 33 | } 34 | 35 | func (u *UnitedQueue) pullTopics() error { 36 | // log.Printf("etcd pull topics start...") 37 | 38 | resp, err := u.etcdClient.Get(u.etcdKey+"/topics", false, true) 39 | if err != nil { 40 | return err 41 | } 42 | 43 | for _, node := range resp.Node.Nodes { 44 | if node.Dir { 45 | err := u.nodeCreate(node) 46 | if err != nil { 47 | // log.Printf("nodeCreate error: %s", err) 48 | continue 49 | } 50 | for _, nd := range node.Nodes { 51 | err := u.nodeCreate(nd) 52 | if err != nil { 53 | // log.Printf("nodeCreate error: %s", err) 54 | continue 55 | } 56 | } 57 | } 58 | } 59 | return nil 60 | } 61 | 62 | func (u *UnitedQueue) watchRun(succChan, stopChan chan bool) { 63 | // log.Printf("etcd watchRun start...") 64 | 65 | u.wg.Add(1) 66 | defer u.wg.Done() 67 | 68 | for { 69 | resp, err := u.etcdClient.Watch(u.etcdKey+"/topics", 0, true, nil, stopChan) 70 | if err != nil { 71 | if strings.Contains(err.Error(), "stop channel") { 72 | close(succChan) 73 | } else { 74 | succChan <- false 75 | } 76 | // log.Printf("etcd watch error: %v", err) 77 | break 78 | } 79 | // log.Printf("resp: %v", resp) 80 | if resp.Action == "create" || resp.Action == "set" { 81 | u.nodeCreate(resp.Node) 82 | // err := u.nodeCreate(resp.Node) 83 | // if err != nil { 84 | // log.Printf("nodeCreate error: %s", err) 85 | // } 86 | } else if resp.Action == "delete" { 87 | u.nodeRemove(resp.Node) 88 | // err := u.nodeRemove(resp.Node) 89 | // if err != nil { 90 | // log.Printf("nodeRemove error: %s", err) 91 | // } 92 | } 93 | } 94 | 95 | // log.Printf("watchRun stoped.") 96 | } 97 | 98 | func (u *UnitedQueue) scanRun() { 99 | // log.Printf("etcd scanRun start...") 100 | 101 | u.wg.Add(1) 102 | defer u.wg.Done() 103 | 104 | stopChan := make(chan bool) 105 | succChan := make(chan bool, 1) 106 | succChan <- false 107 | watchDelay := time.NewTicker(etcdWatchDelay) 108 | quit := false 109 | for !quit { 110 | select { 111 | case <-watchDelay.C: 112 | // log.Printf("watchRun ticked.") 113 | select { 114 | case succStatus := <-succChan: 115 | if succStatus == false { 116 | // log.Printf("watchRun not succ.") 117 | go u.watchRun(succChan, stopChan) 118 | } 119 | default: 120 | // log.Printf("watchRun succ. just passed.") 121 | } 122 | case <-u.etcdStop: 123 | // log.Printf("scanRun stoping...") 124 | quit = true 125 | } 126 | } 127 | 128 | close(stopChan) 129 | // log.Printf("scanRun stoped.") 130 | } 131 | 132 | func (u *UnitedQueue) register() error { 133 | // log.Printf("etcd register self...") 134 | 135 | key := u.etcdKey + "/servers/" + u.selfAddr 136 | _, err := u.etcdClient.Set(key, etcdUqServerListValue, etcdTTL) 137 | if err != nil { 138 | return err 139 | } 140 | return nil 141 | } 142 | 143 | func (u *UnitedQueue) unRegister() error { 144 | // log.Printf("etcd unregister self...") 145 | 146 | key := u.etcdKey + "/servers/" + u.selfAddr 147 | _, err := u.etcdClient.Delete(key, true) 148 | if err != nil { 149 | return err 150 | } 151 | return nil 152 | } 153 | 154 | func (u *UnitedQueue) etcdRun() { 155 | if u.etcdClient == nil { 156 | return 157 | } 158 | 159 | u.wg.Add(1) 160 | defer u.wg.Done() 161 | 162 | u.pullTopics() 163 | // err := u.pullTopics() 164 | // if err != nil { 165 | // log.Printf("pull topics error: %s", err) 166 | // } 167 | go u.scanRun() 168 | 169 | registerDelay := time.NewTicker(etcdRegisterDelay) 170 | select { 171 | case <-registerDelay.C: 172 | // log.Printf("entry succ. etcdRun first register...") 173 | u.register() 174 | case <-u.etcdStop: 175 | // log.Printf("entry failed. etcdRun stoping...") 176 | return 177 | } 178 | 179 | ticker := time.NewTicker(time.Duration(etcdTTL * oneSecond)) 180 | quit := false 181 | for !quit { 182 | select { 183 | case <-ticker.C: 184 | // log.Printf("etcdRun ticked.") 185 | u.register() 186 | case <-u.etcdStop: 187 | // log.Printf("etcdRun stoping...") 188 | quit = true 189 | break 190 | } 191 | } 192 | 193 | u.unRegister() 194 | // log.Printf("etcdRun stoped.") 195 | } 196 | 197 | func (u *UnitedQueue) registerTopic(topic string) error { 198 | if u.etcdClient == nil { 199 | return nil 200 | } 201 | // log.Printf("etcd register topic[%s]...", topic) 202 | 203 | topicKey := u.etcdKey + "/topics/" + topic 204 | _, err := u.etcdClient.CreateDir(topicKey, 0) 205 | if err != nil { 206 | return err 207 | } 208 | return nil 209 | } 210 | 211 | func (u *UnitedQueue) unRegisterTopic(topic string) error { 212 | if u.etcdClient == nil { 213 | return nil 214 | } 215 | // log.Printf("etcd unregister topic[%s]...", topic) 216 | 217 | topicKey := u.etcdKey + "/topics/" + topic 218 | _, err := u.etcdClient.Delete(topicKey, true) 219 | if err != nil { 220 | return err 221 | } 222 | return nil 223 | } 224 | 225 | func (u *UnitedQueue) registerLine(topic, line, recycle string) error { 226 | if u.etcdClient == nil { 227 | return nil 228 | } 229 | // log.Printf("etcd register topic[%s]...", topic) 230 | 231 | lineKey := u.etcdKey + "/topics/" + topic + "/" + line 232 | _, err := u.etcdClient.Set(lineKey, recycle, 0) 233 | if err != nil { 234 | return err 235 | } 236 | return nil 237 | } 238 | 239 | func (u *UnitedQueue) unRegisterLine(topic, line string) error { 240 | if u.etcdClient == nil { 241 | return nil 242 | } 243 | // log.Printf("etcd register topic[%s]...", topic) 244 | 245 | lineKey := u.etcdKey + "/topics/" + topic + "/" + line 246 | _, err := u.etcdClient.Delete(lineKey, false) 247 | if err != nil { 248 | return err 249 | } 250 | return nil 251 | } 252 | -------------------------------------------------------------------------------- /queue/uLine.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "container/list" 5 | "log" 6 | "sync" 7 | "time" 8 | 9 | "github.com/buaazp/uq/utils" 10 | ) 11 | 12 | type line struct { 13 | name string 14 | head uint64 15 | headLock sync.RWMutex 16 | recycle time.Duration 17 | recycleKey string 18 | inflight *list.List 19 | inflightLock sync.RWMutex 20 | ihead uint64 21 | imap map[uint64]bool 22 | t *topic 23 | } 24 | 25 | func (l *line) exportRecycle() error { 26 | lineRecycleData := []byte(l.recycle.String()) 27 | err := l.t.q.setData(l.recycleKey, lineRecycleData) 28 | if err != nil { 29 | return err 30 | } 31 | return nil 32 | } 33 | 34 | func (l *line) removeRecycleData() error { 35 | err := l.t.q.delData(l.recycleKey) 36 | if err != nil { 37 | return err 38 | } 39 | return nil 40 | } 41 | 42 | func (l *line) genLineStore() *UnitedLineStore { 43 | inflights := make([]*InflightMessage, l.inflight.Len()) 44 | i := 0 45 | for m := l.inflight.Front(); m != nil; m = m.Next() { 46 | msg := m.Value.(*InflightMessage) 47 | inflights[i] = msg 48 | i++ 49 | } 50 | // log.Printf("inflights: %v", inflights) 51 | 52 | ls := new(UnitedLineStore) 53 | ls.Head = l.head 54 | ls.Inflights = inflights 55 | ls.Ihead = l.ihead 56 | return ls 57 | } 58 | 59 | func (l *line) exportLine() error { 60 | // log.Printf("start export line[%s]...", l.name) 61 | ls := l.genLineStore() 62 | buf, err := ls.Marshal() 63 | if err != nil { 64 | return utils.NewError( 65 | utils.ErrInternalError, 66 | err.Error(), 67 | ) 68 | } 69 | 70 | lineStoreKey := l.t.name + "/" + l.name 71 | err = l.t.q.setData(lineStoreKey, buf) 72 | if err != nil { 73 | return err 74 | } 75 | 76 | // log.Printf("line[%s] export finisded.", l.name) 77 | return nil 78 | } 79 | 80 | func (l *line) removeLineData() error { 81 | lineStoreKey := l.t.name + "/" + l.name 82 | err := l.t.q.delData(lineStoreKey) 83 | if err != nil { 84 | return err 85 | } 86 | 87 | // log.Printf("line[%s] remove finisded.", l.name) 88 | return nil 89 | } 90 | 91 | func (l *line) updateiHead() { 92 | for l.ihead < l.head { 93 | id := l.ihead 94 | fl, ok := l.imap[id] 95 | if !ok { 96 | l.ihead++ 97 | continue 98 | } 99 | if fl { 100 | return 101 | } 102 | delete(l.imap, id) 103 | l.ihead++ 104 | } 105 | } 106 | 107 | func (l *line) pop() (uint64, []byte, error) { 108 | l.inflightLock.Lock() 109 | defer l.inflightLock.Unlock() 110 | 111 | now := time.Now() 112 | if l.recycle > 0 { 113 | 114 | m := l.inflight.Front() 115 | if m != nil { 116 | msg := m.Value.(*InflightMessage) 117 | exp := time.Unix(0, msg.Exptime) 118 | if now.After(exp) { 119 | // log.Printf("key[%s/%d] is expired.", l.name, msg.Tid) 120 | msg.Exptime = now.Add(l.recycle).UnixNano() 121 | data, err := l.t.getData(msg.Tid) 122 | if err != nil { 123 | return 0, nil, err 124 | } 125 | l.inflight.Remove(m) 126 | l.inflight.PushBack(msg) 127 | // log.Printf("key[%s/%s/%d] poped.", l.t.name, l.name, msg.Tid) 128 | return msg.Tid, data, nil 129 | } 130 | } 131 | } 132 | 133 | l.headLock.Lock() 134 | defer l.headLock.Unlock() 135 | tid := l.head 136 | 137 | topicTail := l.t.getTail() 138 | if l.head >= topicTail { 139 | // log.Printf("line[%s] is blank. head:%d - tail:%d", l.name, l.head, l.t.tail) 140 | return 0, nil, utils.NewError( 141 | utils.ErrNone, 142 | `line pop`, 143 | ) 144 | } 145 | 146 | data, err := l.t.getData(tid) 147 | if err != nil { 148 | return 0, nil, err 149 | } 150 | 151 | l.head++ 152 | 153 | if l.recycle > 0 { 154 | msg := new(InflightMessage) 155 | msg.Tid = tid 156 | msg.Exptime = now.Add(l.recycle).UnixNano() 157 | 158 | l.inflight.PushBack(msg) 159 | // log.Printf("key[%s/%s/%d] flighted.", l.t.name, l.name, l.head) 160 | l.imap[tid] = true 161 | } 162 | 163 | return tid, data, nil 164 | } 165 | 166 | func (l *line) mPop(n int) ([]uint64, [][]byte, error) { 167 | l.inflightLock.Lock() 168 | defer l.inflightLock.Unlock() 169 | 170 | fc := 0 171 | var ids []uint64 172 | var datas [][]byte 173 | now := time.Now() 174 | if l.recycle > 0 { 175 | for m := l.inflight.Front(); m != nil && fc < n; m = m.Next() { 176 | msg := m.Value.(*InflightMessage) 177 | exp := time.Unix(0, msg.Exptime) 178 | if now.After(exp) { 179 | msg := m.Value.(*InflightMessage) 180 | data, err := l.t.getData(msg.Tid) 181 | if err != nil { 182 | return nil, nil, err 183 | } 184 | ids = append(ids, msg.Tid) 185 | datas = append(datas, data) 186 | fc++ 187 | } else { 188 | break 189 | } 190 | } 191 | exptime := now.Add(l.recycle).UnixNano() 192 | for i := 0; i < fc; i++ { 193 | m := l.inflight.Front() 194 | msg := m.Value.(*InflightMessage) 195 | msg.Exptime = exptime 196 | l.inflight.Remove(m) 197 | l.inflight.PushBack(msg) 198 | } 199 | if fc >= n { 200 | return ids, datas, nil 201 | } 202 | } 203 | 204 | l.headLock.Lock() 205 | defer l.headLock.Unlock() 206 | 207 | for ; fc < n; fc++ { 208 | tid := l.head 209 | topicTail := l.t.getTail() 210 | if l.head >= topicTail { 211 | // log.Printf("line[%s] is blank. head:%d - tail:%d", l.name, l.head, l.t.tail) 212 | break 213 | } 214 | 215 | data, err := l.t.getData(tid) 216 | if err != nil { 217 | log.Printf("get data failed: %s", err) 218 | break 219 | } 220 | 221 | l.head++ 222 | ids = append(ids, tid) 223 | datas = append(datas, data) 224 | 225 | if l.recycle > 0 { 226 | msg := new(InflightMessage) 227 | msg.Tid = tid 228 | msg.Exptime = now.Add(l.recycle).UnixNano() 229 | 230 | l.inflight.PushBack(msg) 231 | // log.Printf("key[%s/%s/%d] flighted.", l.t.name, l.name, l.head) 232 | l.imap[tid] = true 233 | } 234 | } 235 | 236 | if len(ids) > 0 { 237 | return ids, datas, nil 238 | } 239 | return nil, nil, utils.NewError( 240 | utils.ErrNone, 241 | `line mPop`, 242 | ) 243 | } 244 | 245 | func (l *line) confirm(id uint64) error { 246 | if l.recycle == 0 { 247 | return utils.NewError( 248 | utils.ErrNotDelivered, 249 | `line confirm`, 250 | ) 251 | } 252 | 253 | l.headLock.RLock() 254 | defer l.headLock.RUnlock() 255 | head := l.head 256 | if id >= head { 257 | return utils.NewError( 258 | utils.ErrNotDelivered, 259 | `line confirm`, 260 | ) 261 | } 262 | 263 | l.inflightLock.Lock() 264 | defer l.inflightLock.Unlock() 265 | 266 | for m := l.inflight.Front(); m != nil; m = m.Next() { 267 | msg := m.Value.(*InflightMessage) 268 | if msg.Tid == id { 269 | l.inflight.Remove(m) 270 | // log.Printf("key[%s/%s/%d] comfirmed.", l.t.name, l.name, id) 271 | l.imap[id] = false 272 | l.updateiHead() 273 | return nil 274 | } 275 | } 276 | 277 | return utils.NewError( 278 | utils.ErrNotDelivered, 279 | `line confirm`, 280 | ) 281 | } 282 | 283 | func (l *line) stat() *Stat { 284 | l.inflightLock.RLock() 285 | defer l.inflightLock.RUnlock() 286 | l.headLock.RLock() 287 | defer l.headLock.RUnlock() 288 | 289 | qs := new(Stat) 290 | qs.Name = l.t.name + "/" + l.name 291 | qs.Type = "line" 292 | qs.Recycle = l.recycle.String() 293 | qs.IHead = l.ihead 294 | inflightLen := uint64(l.inflight.Len()) 295 | qs.Head = l.head 296 | qs.Tail = l.t.getTail() 297 | qs.Count = inflightLen + qs.Tail - qs.Head 298 | 299 | return qs 300 | } 301 | 302 | func (l *line) empty() error { 303 | l.inflightLock.Lock() 304 | defer l.inflightLock.Unlock() 305 | l.inflight.Init() 306 | l.imap = make(map[uint64]bool) 307 | l.ihead = l.t.getTail() 308 | 309 | l.headLock.Lock() 310 | defer l.headLock.Unlock() 311 | l.head = l.t.getTail() 312 | 313 | err := l.exportLine() 314 | if err != nil { 315 | return err 316 | } 317 | 318 | log.Printf("line[%s] empty succ", l.name) 319 | return nil 320 | } 321 | 322 | func (l *line) remove() error { 323 | err := l.removeLineData() 324 | if err != nil { 325 | log.Printf("line[%s] removeLineData error: %s", l.name, err) 326 | } 327 | 328 | err = l.removeRecycleData() 329 | if err != nil { 330 | log.Printf("line[%s] removeRecycleData error: %s", l.name, err) 331 | } 332 | 333 | log.Printf("line[%s] remove succ", l.name) 334 | return nil 335 | } 336 | -------------------------------------------------------------------------------- /queue/uQueue.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "encoding/binary" 5 | "errors" 6 | "log" 7 | "strconv" 8 | "strings" 9 | "sync" 10 | "time" 11 | 12 | "github.com/buaazp/uq/store" 13 | "github.com/buaazp/uq/utils" 14 | "github.com/coreos/go-etcd/etcd" 15 | ) 16 | 17 | const ( 18 | storageKeyWord string = "UnitedQueueKey" 19 | bgBackupInterval time.Duration = 10 * time.Second 20 | bgCleanInterval time.Duration = 20 * time.Second 21 | bgCleanTimeout time.Duration = 5 * time.Second 22 | keyTopicStore string = ":store" 23 | keyTopicHead string = ":head" 24 | keyTopicTail string = ":tail" 25 | keyLineStore string = ":store" 26 | keyLineHead string = ":head" 27 | keyLineRecycle string = ":recycle" 28 | keyLineInflight string = ":inflight" 29 | ) 30 | 31 | // UnitedQueue is a implemention of message queue in uq 32 | type UnitedQueue struct { 33 | topics map[string]*topic 34 | topicsLock sync.RWMutex 35 | storage store.Storage 36 | etcdLock sync.RWMutex 37 | selfAddr string 38 | etcdClient *etcd.Client 39 | etcdKey string 40 | etcdStop chan bool 41 | wg sync.WaitGroup 42 | } 43 | 44 | // NewUnitedQueue returns a new UnitedQueue 45 | func NewUnitedQueue(storage store.Storage, ip string, port int, etcdServers []string, etcdKey string) (*UnitedQueue, error) { 46 | topics := make(map[string]*topic) 47 | etcdStop := make(chan bool) 48 | uq := new(UnitedQueue) 49 | uq.topics = topics 50 | uq.storage = storage 51 | uq.etcdStop = etcdStop 52 | 53 | if len(etcdServers) > 0 { 54 | selfAddr := utils.Addrcat(ip, port) 55 | uq.selfAddr = selfAddr 56 | etcdClient := etcd.NewClient(etcdServers) 57 | uq.etcdClient = etcdClient 58 | uq.etcdKey = etcdKey 59 | } 60 | 61 | err := uq.loadQueue() 62 | if err != nil { 63 | return nil, err 64 | } 65 | 66 | go uq.etcdRun() 67 | return uq, nil 68 | } 69 | 70 | func (u *UnitedQueue) setData(key string, data []byte) error { 71 | err := u.storage.Set(key, data) 72 | if err != nil { 73 | // log.Printf("key[%s] set data error: %s", key, err) 74 | return utils.NewError( 75 | utils.ErrInternalError, 76 | err.Error(), 77 | ) 78 | } 79 | return nil 80 | } 81 | 82 | func (u *UnitedQueue) getData(key string) ([]byte, error) { 83 | data, err := u.storage.Get(key) 84 | if err != nil { 85 | // log.Printf("key[%s] get data error: %s", key, err) 86 | return nil, utils.NewError( 87 | utils.ErrInternalError, 88 | err.Error(), 89 | ) 90 | } 91 | return data, nil 92 | } 93 | 94 | func (u *UnitedQueue) delData(key string) error { 95 | err := u.storage.Del(key) 96 | if err != nil { 97 | // log.Printf("key[%s] del data error: %s", key, err) 98 | return utils.NewError( 99 | utils.ErrInternalError, 100 | err.Error(), 101 | ) 102 | } 103 | return nil 104 | } 105 | 106 | func (u *UnitedQueue) exportTopics() error { 107 | u.topicsLock.RLock() 108 | defer u.topicsLock.RUnlock() 109 | 110 | for _, t := range u.topics { 111 | err := t.exportLines() 112 | if err != nil { 113 | log.Printf("topic[%s] export lines error: %s", t.name, err) 114 | continue 115 | } 116 | t.linesLock.RLock() 117 | err = t.exportTopic() 118 | t.linesLock.RUnlock() 119 | if err != nil { 120 | log.Printf("topic[%s] export error: %s", t.name, err) 121 | continue 122 | } 123 | } 124 | 125 | // log.Printf("export all topics succ.") 126 | return nil 127 | } 128 | 129 | func (u *UnitedQueue) genQueueStore() *UnitedQueueStore { 130 | topics := make([]string, len(u.topics)) 131 | i := 0 132 | for topicName := range u.topics { 133 | topics[i] = topicName 134 | i++ 135 | } 136 | 137 | qs := new(UnitedQueueStore) 138 | qs.Topics = topics 139 | return qs 140 | } 141 | 142 | func (u *UnitedQueue) exportQueue() error { 143 | // log.Printf("start export queue...") 144 | qs := u.genQueueStore() 145 | buf, err := qs.Marshal() 146 | if err != nil { 147 | return utils.NewError( 148 | utils.ErrInternalError, 149 | err.Error(), 150 | ) 151 | } 152 | 153 | err = u.setData(storageKeyWord, buf) 154 | if err != nil { 155 | return err 156 | } 157 | 158 | // log.Printf("united queue export finisded.") 159 | return nil 160 | } 161 | 162 | func (u *UnitedQueue) loadTopic(topicName string, ts UnitedTopicStore) (*topic, error) { 163 | t := new(topic) 164 | t.name = topicName 165 | t.persist = ts.Persist 166 | t.q = u 167 | t.quit = make(chan bool) 168 | 169 | t.headKey = topicName + keyTopicHead 170 | topicHeadData, err := u.getData(t.headKey) 171 | if err != nil { 172 | return nil, err 173 | } 174 | t.head = binary.LittleEndian.Uint64(topicHeadData) 175 | t.tailKey = topicName + keyTopicTail 176 | topicTailData, err := u.getData(t.tailKey) 177 | if err != nil { 178 | return nil, err 179 | } 180 | t.tail = binary.LittleEndian.Uint64(topicTailData) 181 | 182 | lines := make(map[string]*line) 183 | for _, lineName := range ts.Lines { 184 | lineStoreKey := topicName + "/" + lineName 185 | lineStoreData, err := u.getData(lineStoreKey) 186 | if err != nil { 187 | return nil, err 188 | } 189 | if len(lineStoreData) == 0 { 190 | return nil, errors.New("line backup data missing: " + lineStoreKey) 191 | } 192 | var ls UnitedLineStore 193 | err = ls.Unmarshal(lineStoreData) 194 | if err != nil { 195 | return nil, err 196 | } 197 | l, err := t.loadLine(lineName, ls) 198 | if err != nil { 199 | continue 200 | } 201 | lines[lineName] = l 202 | // log.Printf("line[%s] load succ.", lineStoreKey) 203 | } 204 | t.lines = lines 205 | 206 | u.registerTopic(t.name) 207 | 208 | t.start() 209 | // log.Printf("topic[%s] load succ.", topicName) 210 | // log.Printf("topic: %v", t) 211 | return t, nil 212 | } 213 | 214 | func (u *UnitedQueue) loadQueue() error { 215 | unitedQueueStoreData, err := u.getData(storageKeyWord) 216 | if err != nil { 217 | // log.Printf("storage not existed: %s", err) 218 | return nil 219 | } 220 | 221 | if len(unitedQueueStoreData) > 0 { 222 | var qs UnitedQueueStore 223 | err := qs.Unmarshal(unitedQueueStoreData) 224 | if err != nil { 225 | return err 226 | } 227 | for _, topicName := range qs.Topics { 228 | topicStoreData, err := u.getData(topicName) 229 | if err != nil { 230 | return err 231 | } 232 | if len(topicStoreData) == 0 { 233 | return errors.New("topic backup data missing: " + topicName) 234 | } 235 | var ts UnitedTopicStore 236 | err = ts.Unmarshal(topicStoreData) 237 | if err != nil { 238 | return err 239 | } 240 | t, err := u.loadTopic(topicName, ts) 241 | if err != nil { 242 | return err 243 | } 244 | u.topicsLock.Lock() 245 | u.topics[topicName] = t 246 | u.topicsLock.Unlock() 247 | } 248 | } 249 | 250 | // log.Printf("united queue load finisded.") 251 | // log.Printf("u.topics: %v", u.topics) 252 | return nil 253 | } 254 | 255 | func (u *UnitedQueue) newTopic(name string, persist bool) (*topic, error) { 256 | lines := make(map[string]*line) 257 | t := new(topic) 258 | t.name = name 259 | t.persist = persist 260 | t.lines = lines 261 | t.head = 0 262 | t.headKey = name + keyTopicHead 263 | t.tail = 0 264 | t.tailKey = name + keyTopicTail 265 | t.q = u 266 | t.quit = make(chan bool) 267 | 268 | err := t.exportHead() 269 | if err != nil { 270 | return nil, err 271 | } 272 | err = t.exportTail() 273 | if err != nil { 274 | return nil, err 275 | } 276 | 277 | t.start() 278 | return t, nil 279 | } 280 | 281 | func (u *UnitedQueue) createTopic(name string, persist, fromEtcd bool) error { 282 | u.topicsLock.RLock() 283 | _, ok := u.topics[name] 284 | u.topicsLock.RUnlock() 285 | if ok { 286 | return utils.NewError( 287 | utils.ErrTopicExisted, 288 | `queue createTopic`, 289 | ) 290 | } 291 | 292 | t, err := u.newTopic(name, persist) 293 | if err != nil { 294 | return err 295 | } 296 | 297 | u.topicsLock.Lock() 298 | defer u.topicsLock.Unlock() 299 | u.topics[name] = t 300 | 301 | err = u.exportQueue() 302 | if err != nil { 303 | t.remove() 304 | delete(u.topics, name) 305 | return err 306 | } 307 | 308 | if !fromEtcd { 309 | u.registerTopic(t.name) 310 | } 311 | log.Printf("topic[%s] created.", name) 312 | return nil 313 | } 314 | 315 | func (u *UnitedQueue) create(key, arg string, fromEtcd bool) error { 316 | key = strings.TrimPrefix(key, "/") 317 | key = strings.TrimSuffix(key, "/") 318 | 319 | var topicName, lineName string 320 | var err error 321 | parts := strings.Split(key, "/") 322 | if len(parts) < 1 || len(parts) > 2 { 323 | return utils.NewError( 324 | utils.ErrBadKey, 325 | `create key parts error: `+utils.ItoaQuick(len(parts)), 326 | ) 327 | } 328 | 329 | topicName = parts[0] 330 | if topicName == "" { 331 | return utils.NewError( 332 | utils.ErrBadKey, 333 | `create topic is nil`, 334 | ) 335 | } 336 | 337 | if len(parts) == 2 { 338 | lineName = parts[1] 339 | var recycle time.Duration 340 | if arg != "" { 341 | recycle, err = time.ParseDuration(arg) 342 | if err != nil { 343 | return utils.NewError( 344 | utils.ErrBadRequest, 345 | err.Error(), 346 | ) 347 | } 348 | } 349 | 350 | u.topicsLock.RLock() 351 | t, ok := u.topics[topicName] 352 | u.topicsLock.RUnlock() 353 | if !ok { 354 | return utils.NewError( 355 | utils.ErrTopicNotExisted, 356 | `queue create`, 357 | ) 358 | } 359 | 360 | err = t.createLine(lineName, recycle, fromEtcd) 361 | if err != nil { 362 | // log.Printf("create line[%s] error: %s", lineName, err) 363 | return err 364 | } 365 | } else { 366 | var persist bool 367 | if arg == "persist" { 368 | persist = true 369 | } 370 | err = u.createTopic(topicName, persist, fromEtcd) 371 | if err != nil { 372 | // log.Printf("create topic[%s] error: %s", topicName, err) 373 | return err 374 | } 375 | } 376 | 377 | return err 378 | } 379 | 380 | // Create implements Create interface 381 | func (u *UnitedQueue) Create(key, arg string) error { 382 | return u.create(key, arg, false) 383 | } 384 | 385 | // Push implements Push interface 386 | func (u *UnitedQueue) Push(key string, data []byte) error { 387 | key = strings.TrimPrefix(key, "/") 388 | key = strings.TrimSuffix(key, "/") 389 | 390 | if len(data) <= 0 { 391 | return utils.NewError( 392 | utils.ErrBadRequest, 393 | `message has no content`, 394 | ) 395 | } 396 | 397 | u.topicsLock.RLock() 398 | t, ok := u.topics[key] 399 | u.topicsLock.RUnlock() 400 | if !ok { 401 | return utils.NewError( 402 | utils.ErrTopicNotExisted, 403 | `queue push`, 404 | ) 405 | } 406 | 407 | return t.push(data) 408 | } 409 | 410 | // MultiPush implements MultiPush interface 411 | func (u *UnitedQueue) MultiPush(key string, datas [][]byte) error { 412 | key = strings.TrimPrefix(key, "/") 413 | key = strings.TrimSuffix(key, "/") 414 | 415 | for i, data := range datas { 416 | if len(data) <= 0 { 417 | cause := "message " + strconv.Itoa(i) + " has no content" 418 | return utils.NewError( 419 | utils.ErrBadRequest, 420 | cause, 421 | ) 422 | } 423 | } 424 | 425 | u.topicsLock.RLock() 426 | t, ok := u.topics[key] 427 | u.topicsLock.RUnlock() 428 | if !ok { 429 | return utils.NewError( 430 | utils.ErrTopicNotExisted, 431 | `queue multiPush`, 432 | ) 433 | } 434 | 435 | return t.mPush(datas) 436 | } 437 | 438 | // Pop implements Pop interface 439 | func (u *UnitedQueue) Pop(key string) (string, []byte, error) { 440 | key = strings.TrimPrefix(key, "/") 441 | key = strings.TrimSuffix(key, "/") 442 | 443 | parts := strings.Split(key, "/") 444 | if len(parts) != 2 { 445 | return "", nil, utils.NewError( 446 | utils.ErrBadKey, 447 | `pop key parts error: `+utils.ItoaQuick(len(parts)), 448 | ) 449 | } 450 | 451 | tName := parts[0] 452 | lName := parts[1] 453 | 454 | u.topicsLock.RLock() 455 | t, ok := u.topics[tName] 456 | u.topicsLock.RUnlock() 457 | if !ok { 458 | // log.Printf("topic[%s] not existed.", tName) 459 | return "", nil, utils.NewError( 460 | utils.ErrTopicNotExisted, 461 | `queue pop`, 462 | ) 463 | } 464 | 465 | id, data, err := t.pop(lName) 466 | if err != nil { 467 | return "", nil, err 468 | } 469 | 470 | return utils.Acatui(key, "/", id), data, nil 471 | } 472 | 473 | // MultiPop implements MultiPop interface 474 | func (u *UnitedQueue) MultiPop(key string, n int) ([]string, [][]byte, error) { 475 | key = strings.TrimPrefix(key, "/") 476 | key = strings.TrimSuffix(key, "/") 477 | 478 | parts := strings.Split(key, "/") 479 | if len(parts) != 2 { 480 | return nil, nil, utils.NewError( 481 | utils.ErrBadKey, 482 | `mPop key parts error: `+utils.ItoaQuick(len(parts)), 483 | ) 484 | } 485 | 486 | tName := parts[0] 487 | lName := parts[1] 488 | 489 | u.topicsLock.RLock() 490 | t, ok := u.topics[tName] 491 | u.topicsLock.RUnlock() 492 | if !ok { 493 | // log.Printf("topic[%s] not existed.", tName) 494 | return nil, nil, utils.NewError( 495 | utils.ErrTopicNotExisted, 496 | `queue multiPop`, 497 | ) 498 | } 499 | 500 | ids, datas, err := t.mPop(lName, n) 501 | if err != nil { 502 | return nil, nil, err 503 | } 504 | 505 | keys := make([]string, len(ids)) 506 | for i, id := range ids { 507 | keys[i] = utils.Acatui(key, "/", id) 508 | } 509 | return keys, datas, nil 510 | } 511 | 512 | // Confirm implements Confirm interface 513 | func (u *UnitedQueue) Confirm(key string) error { 514 | key = strings.TrimPrefix(key, "/") 515 | key = strings.TrimSuffix(key, "/") 516 | 517 | parts := strings.Split(key, "/") 518 | if len(parts) != 3 { 519 | return utils.NewError( 520 | utils.ErrBadKey, 521 | `confirm key parts error: `+utils.ItoaQuick(len(parts)), 522 | ) 523 | } 524 | topicName := parts[0] 525 | lineName := parts[1] 526 | id, err := strconv.ParseUint(parts[2], 10, 0) 527 | if err != nil { 528 | return utils.NewError( 529 | utils.ErrBadKey, 530 | `confirm key parse id error: `+err.Error(), 531 | ) 532 | } 533 | 534 | u.topicsLock.RLock() 535 | t, ok := u.topics[topicName] 536 | u.topicsLock.RUnlock() 537 | if !ok { 538 | // log.Printf("topic[%s] not existed.", topicName) 539 | return utils.NewError( 540 | utils.ErrTopicNotExisted, 541 | `queue confirm`, 542 | ) 543 | } 544 | 545 | return t.confirm(lineName, id) 546 | } 547 | 548 | // MultiConfirm implements MultiConfirm interface 549 | func (u *UnitedQueue) MultiConfirm(keys []string) []error { 550 | errs := make([]error, len(keys)) 551 | for i, key := range keys { 552 | errs[i] = u.Confirm(key) 553 | } 554 | return errs 555 | } 556 | 557 | // Stat implements Stat interface 558 | func (u *UnitedQueue) Stat(key string) (*Stat, error) { 559 | key = strings.TrimPrefix(key, "/") 560 | key = strings.TrimSuffix(key, "/") 561 | 562 | var topicName, lineName string 563 | parts := strings.Split(key, "/") 564 | if len(parts) < 1 || len(parts) > 2 { 565 | return nil, utils.NewError( 566 | utils.ErrBadKey, 567 | `empty key parts error: `+utils.ItoaQuick(len(parts)), 568 | ) 569 | } 570 | 571 | topicName = parts[0] 572 | if topicName == "" { 573 | return nil, utils.NewError( 574 | utils.ErrBadKey, 575 | `stat topic is nil`, 576 | ) 577 | } 578 | 579 | u.topicsLock.RLock() 580 | t, ok := u.topics[topicName] 581 | u.topicsLock.RUnlock() 582 | if !ok { 583 | return nil, utils.NewError( 584 | utils.ErrTopicNotExisted, 585 | `queue stat`, 586 | ) 587 | } 588 | 589 | if len(parts) == 2 { 590 | lineName = parts[1] 591 | return t.statLine(lineName) 592 | } 593 | 594 | qs := t.stat() 595 | return qs, nil 596 | } 597 | 598 | // Empty implements Empty interface 599 | func (u *UnitedQueue) Empty(key string) error { 600 | key = strings.TrimPrefix(key, "/") 601 | key = strings.TrimSuffix(key, "/") 602 | 603 | var topicName, lineName string 604 | parts := strings.Split(key, "/") 605 | if len(parts) < 1 || len(parts) > 2 { 606 | return utils.NewError( 607 | utils.ErrBadKey, 608 | `empty key parts error: `+utils.ItoaQuick(len(parts)), 609 | ) 610 | } 611 | 612 | topicName = parts[0] 613 | 614 | if topicName == "" { 615 | return utils.NewError( 616 | utils.ErrBadKey, 617 | `empty topic is nil`, 618 | ) 619 | } 620 | 621 | u.topicsLock.RLock() 622 | t, ok := u.topics[topicName] 623 | u.topicsLock.RUnlock() 624 | if !ok { 625 | return utils.NewError( 626 | utils.ErrTopicNotExisted, 627 | `queue empty`, 628 | ) 629 | } 630 | 631 | if len(parts) == 2 { 632 | lineName = parts[1] 633 | return t.emptyLine(lineName) 634 | // err = t.emptyLine(lineName) 635 | // if err != nil { 636 | // log.Printf("empty line[%s] error: %s", lineName, err) 637 | // } 638 | // return err 639 | } 640 | 641 | return t.empty() 642 | } 643 | 644 | func (u *UnitedQueue) removeTopic(name string, fromEtcd bool) error { 645 | u.topicsLock.Lock() 646 | defer u.topicsLock.Unlock() 647 | 648 | t, ok := u.topics[name] 649 | if !ok { 650 | return utils.NewError( 651 | utils.ErrTopicNotExisted, 652 | `queue remove`, 653 | ) 654 | } 655 | 656 | delete(u.topics, name) 657 | err := u.exportQueue() 658 | if err != nil { 659 | u.topics[name] = t 660 | return err 661 | } 662 | 663 | if !fromEtcd { 664 | u.unRegisterTopic(name) 665 | } 666 | 667 | return t.remove() 668 | } 669 | 670 | func (u *UnitedQueue) remove(key string, fromEtcd bool) error { 671 | key = strings.TrimPrefix(key, "/") 672 | key = strings.TrimSuffix(key, "/") 673 | 674 | var topicName, lineName string 675 | parts := strings.Split(key, "/") 676 | if len(parts) < 1 || len(parts) > 2 { 677 | return utils.NewError( 678 | utils.ErrBadKey, 679 | `remove key parts error: `+utils.ItoaQuick(len(parts)), 680 | ) 681 | } 682 | 683 | topicName = parts[0] 684 | if topicName == "" { 685 | return utils.NewError( 686 | utils.ErrBadKey, 687 | `rmove topic is nil`, 688 | ) 689 | } 690 | 691 | if len(parts) == 1 { 692 | return u.removeTopic(topicName, fromEtcd) 693 | } 694 | 695 | u.topicsLock.RLock() 696 | t, ok := u.topics[topicName] 697 | u.topicsLock.RUnlock() 698 | if !ok { 699 | return utils.NewError( 700 | utils.ErrTopicNotExisted, 701 | `queue remove`, 702 | ) 703 | } 704 | 705 | lineName = parts[1] 706 | return t.removeLine(lineName, fromEtcd) 707 | } 708 | 709 | // Remove implements Remove interface 710 | func (u *UnitedQueue) Remove(key string) error { 711 | return u.remove(key, false) 712 | } 713 | 714 | // Close implements Close interface 715 | func (u *UnitedQueue) Close() { 716 | log.Printf("uq stoping...") 717 | close(u.etcdStop) 718 | u.wg.Wait() 719 | 720 | for _, t := range u.topics { 721 | t.close() 722 | } 723 | 724 | err := u.exportTopics() 725 | if err != nil { 726 | log.Printf("export queue error: %s", err) 727 | } 728 | 729 | u.storage.Close() 730 | log.Printf("uq stoped.") 731 | } 732 | -------------------------------------------------------------------------------- /queue/uQueue_test.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "os" 5 | "strconv" 6 | "testing" 7 | 8 | "github.com/buaazp/uq/store" 9 | . "github.com/smartystreets/goconvey/convey" 10 | ) 11 | 12 | const ( 13 | dbPath = "/tmp/uq.queue.test.db" 14 | ) 15 | 16 | var ( 17 | err error 18 | ldb store.Storage 19 | uq *UnitedQueue 20 | ) 21 | 22 | func TestNewUnitedQueue(t *testing.T) { 23 | Convey("Test New Uq", t, func() { 24 | ldb, err = store.NewLevelStore(dbPath) 25 | So(ldb, ShouldNotBeNil) 26 | So(err, ShouldBeNil) 27 | 28 | uq, err = NewUnitedQueue(ldb, "127.0.0.1", 9689, nil, "uq") 29 | So(err, ShouldBeNil) 30 | So(uq, ShouldNotBeNil) 31 | }) 32 | } 33 | 34 | func TestCreateTopic(t *testing.T) { 35 | Convey("Test Create a Topic", t, func() { 36 | err = uq.Create("foo", "") 37 | So(err, ShouldBeNil) 38 | 39 | topic := uq.topics["foo"] 40 | So(topic, ShouldNotBeNil) 41 | 42 | err = uq.Create("zp", "") 43 | So(err, ShouldBeNil) 44 | 45 | topiczp := uq.topics["zp"] 46 | So(topiczp, ShouldNotBeNil) 47 | }) 48 | } 49 | 50 | func TestCreateLine(t *testing.T) { 51 | Convey("Test Create a Line", t, func() { 52 | topic := uq.topics["foo"] 53 | So(topic, ShouldNotBeNil) 54 | 55 | err = uq.Create("foo/x", "") 56 | So(err, ShouldBeNil) 57 | 58 | linex := topic.lines["x"] 59 | So(linex, ShouldNotBeNil) 60 | 61 | err = uq.Create("foo/y", "3s") 62 | So(err, ShouldBeNil) 63 | 64 | liney := topic.lines["y"] 65 | So(liney, ShouldNotBeNil) 66 | 67 | topiczp := uq.topics["zp"] 68 | So(topiczp, ShouldNotBeNil) 69 | 70 | err = uq.Create("zp/z", "") 71 | So(err, ShouldBeNil) 72 | 73 | linez := topiczp.lines["z"] 74 | So(linez, ShouldNotBeNil) 75 | }) 76 | } 77 | 78 | func TestPush(t *testing.T) { 79 | Convey("Test Push a Message", t, func() { 80 | data := []byte("1") 81 | err = uq.Push("foo", data) 82 | So(err, ShouldBeNil) 83 | }) 84 | } 85 | 86 | func TestMultiPush(t *testing.T) { 87 | Convey("Test Multi Push Messages", t, func() { 88 | datas := make([][]byte, 5) 89 | for i := 0; i < 5; i++ { 90 | datas[i] = []byte(strconv.Itoa(i + 2)) 91 | } 92 | err = uq.MultiPush("foo", datas) 93 | So(err, ShouldBeNil) 94 | }) 95 | } 96 | 97 | func TestPop(t *testing.T) { 98 | Convey("Test Pop a Message", t, func() { 99 | _, msg, err := uq.Pop("foo/x") 100 | So(err, ShouldBeNil) 101 | So(string(msg), ShouldEqual, "1") 102 | }) 103 | } 104 | 105 | func TestMultiPop(t *testing.T) { 106 | Convey("Test Multi Pop Messages", t, func() { 107 | _, msgs, err := uq.MultiPop("foo/x", 5) 108 | So(err, ShouldBeNil) 109 | for i := 0; i < 5; i++ { 110 | So(string(msgs[i]), ShouldEqual, strconv.Itoa(i+2)) 111 | } 112 | }) 113 | } 114 | 115 | func TestConfirm(t *testing.T) { 116 | Convey("Test Confirm a Message", t, func() { 117 | id, msg, err := uq.Pop("foo/y") 118 | So(err, ShouldBeNil) 119 | So(string(msg), ShouldEqual, "1") 120 | 121 | err = uq.Confirm(id) 122 | So(err, ShouldBeNil) 123 | }) 124 | } 125 | 126 | func TestMultiConfirm(t *testing.T) { 127 | Convey("Test Multi Confirm Messages", t, func() { 128 | ids, msgs, err := uq.MultiPop("foo/y", 5) 129 | So(err, ShouldBeNil) 130 | for i := 0; i < 5; i++ { 131 | So(string(msgs[i]), ShouldEqual, strconv.Itoa(i+2)) 132 | } 133 | 134 | errs := uq.MultiConfirm(ids) 135 | for _, err := range errs { 136 | So(err, ShouldBeNil) 137 | } 138 | }) 139 | } 140 | 141 | func TestStat(t *testing.T) { 142 | Convey("Test Stat Line", t, func() { 143 | key := "foo/y" 144 | qs, err := uq.Stat(key) 145 | So(err, ShouldBeNil) 146 | So(qs.Name, ShouldEqual, key) 147 | }) 148 | Convey("Test Stat Topic", t, func() { 149 | key := "foo" 150 | qs, err := uq.Stat(key) 151 | So(err, ShouldBeNil) 152 | So(qs.Name, ShouldEqual, key) 153 | }) 154 | } 155 | 156 | func TestEmpty(t *testing.T) { 157 | Convey("Test Empty Line", t, func() { 158 | key := "foo/y" 159 | err := uq.Empty(key) 160 | So(err, ShouldBeNil) 161 | }) 162 | 163 | Convey("Test Empty Topic", t, func() { 164 | key := "foo" 165 | err := uq.Empty(key) 166 | So(err, ShouldBeNil) 167 | }) 168 | } 169 | 170 | func TestRemove(t *testing.T) { 171 | Convey("Test Remove Line", t, func() { 172 | topic := uq.topics["foo"] 173 | So(topic, ShouldNotBeNil) 174 | 175 | key := "foo/y" 176 | err := uq.Remove(key) 177 | So(err, ShouldBeNil) 178 | 179 | line2 := topic.lines["y"] 180 | So(line2, ShouldBeNil) 181 | }) 182 | 183 | Convey("Test Empty Topic", t, func() { 184 | key := "foo" 185 | err := uq.Remove(key) 186 | So(err, ShouldBeNil) 187 | 188 | topic := uq.topics["foo"] 189 | So(topic, ShouldBeNil) 190 | }) 191 | } 192 | 193 | func TestClose(t *testing.T) { 194 | Convey("Test Close Queue", t, func() { 195 | uq.Close() 196 | }) 197 | } 198 | 199 | func TestLoad(t *testing.T) { 200 | Convey("Test Load Queue", t, func() { 201 | ldb, err = store.NewLevelStore(dbPath) 202 | So(ldb, ShouldNotBeNil) 203 | So(err, ShouldBeNil) 204 | 205 | uq, err = NewUnitedQueue(ldb, "127.0.0.1", 9689, nil, "uq") 206 | So(err, ShouldBeNil) 207 | So(uq, ShouldNotBeNil) 208 | 209 | uq.Close() 210 | 211 | err = os.RemoveAll(dbPath) 212 | So(err, ShouldBeNil) 213 | }) 214 | } 215 | -------------------------------------------------------------------------------- /queue/uStat.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "encoding/json" 5 | "strconv" 6 | "strings" 7 | ) 8 | 9 | // Stat is the Stat of a UnitedQueue 10 | type Stat struct { 11 | Name string `json:"name"` 12 | Type string `json:"type"` 13 | Lines []*Stat `json:"lines,omitempty"` 14 | Recycle string `json:"recycle,omitempty"` 15 | Head uint64 `json:"head"` 16 | IHead uint64 `json:"ihead"` 17 | Tail uint64 `json:"tail"` 18 | Count uint64 `json:"count"` 19 | } 20 | 21 | // ToString returns the string of Stat 22 | func (q *Stat) ToString() string { 23 | replys := q.ToStrings() 24 | reply := strings.Join(replys, "\r\n") 25 | return reply 26 | } 27 | 28 | // ToMcString returns the memcached string of Stat 29 | func (q *Stat) ToMcString() string { 30 | replays := q.ToStrings() 31 | for i, replay := range replays { 32 | if replay != "" { 33 | replays[i] = "STAT " + replay 34 | } 35 | } 36 | return strings.Join(replays, "\r\n") 37 | } 38 | 39 | // ToStrings returns the strings of Stat 40 | func (q *Stat) ToStrings() []string { 41 | var replys []string 42 | replys = append(replys, "name:"+q.Name) 43 | if q.Type == "line" { 44 | replys = append(replys, "recycle:"+q.Recycle) 45 | } 46 | 47 | replys = append(replys, "head:"+strconv.FormatUint(q.Head, 10)) 48 | if q.Type == "line" { 49 | replys = append(replys, "ihead:"+strconv.FormatUint(q.IHead, 10)) 50 | } 51 | replys = append(replys, "tail:"+strconv.FormatUint(q.Tail, 10)) 52 | replys = append(replys, "count:"+strconv.FormatUint(q.Count, 10)) 53 | 54 | if q.Type == "topic" && q.Lines != nil { 55 | for _, lineStat := range q.Lines { 56 | replys = append(replys, "") 57 | ls := lineStat.ToStrings() 58 | replys = append(replys, ls...) 59 | } 60 | } 61 | 62 | return replys 63 | } 64 | 65 | // ToRedisStrings returns the redis strings of Stat 66 | func (q *Stat) ToRedisStrings() []string { 67 | replys := q.ToStrings() 68 | replys = append(replys, "") 69 | return replys 70 | } 71 | 72 | // ToJSON returns the json string of Stat 73 | func (q *Stat) ToJSON() ([]byte, error) { 74 | return json.Marshal(q) 75 | } 76 | -------------------------------------------------------------------------------- /queue/uTopic.go: -------------------------------------------------------------------------------- 1 | package queue 2 | 3 | import ( 4 | "container/list" 5 | "encoding/binary" 6 | "log" 7 | "sync" 8 | "time" 9 | 10 | "github.com/buaazp/uq/utils" 11 | ) 12 | 13 | type topic struct { 14 | name string 15 | persist bool 16 | lines map[string]*line 17 | linesLock sync.RWMutex 18 | head uint64 19 | headLock sync.RWMutex 20 | headKey string 21 | tail uint64 22 | tailLock sync.RWMutex 23 | tailKey string 24 | q *UnitedQueue 25 | 26 | quit chan bool 27 | wg sync.WaitGroup 28 | } 29 | 30 | func (t *topic) getData(id uint64) ([]byte, error) { 31 | key := utils.Acatui(t.name, ":", id) 32 | return t.q.getData(key) 33 | } 34 | 35 | func (t *topic) setData(id uint64, data []byte) error { 36 | key := utils.Acatui(t.name, ":", id) 37 | return t.q.setData(key, data) 38 | } 39 | 40 | func (t *topic) getHead() uint64 { 41 | t.headLock.RLock() 42 | defer t.headLock.RUnlock() 43 | return t.head 44 | } 45 | 46 | func (t *topic) getTail() uint64 { 47 | t.tailLock.RLock() 48 | defer t.tailLock.RUnlock() 49 | return t.tail 50 | } 51 | 52 | func (t *topic) exportHead() error { 53 | topicHeadData := make([]byte, 8) 54 | binary.LittleEndian.PutUint64(topicHeadData, t.head) 55 | err := t.q.setData(t.headKey, topicHeadData) 56 | if err != nil { 57 | return err 58 | } 59 | return nil 60 | } 61 | 62 | func (t *topic) removeHeadData() error { 63 | err := t.q.delData(t.headKey) 64 | if err != nil { 65 | return err 66 | } 67 | return nil 68 | } 69 | 70 | func (t *topic) exportTail() error { 71 | topicTailData := make([]byte, 8) 72 | binary.LittleEndian.PutUint64(topicTailData, t.tail) 73 | err := t.q.setData(t.tailKey, topicTailData) 74 | if err != nil { 75 | return err 76 | } 77 | return nil 78 | } 79 | 80 | func (t *topic) removeTailData() error { 81 | err := t.q.delData(t.tailKey) 82 | if err != nil { 83 | return err 84 | } 85 | return nil 86 | } 87 | 88 | func (t *topic) genTopicStore() *UnitedTopicStore { 89 | lines := make([]string, len(t.lines)) 90 | i := 0 91 | for _, line := range t.lines { 92 | lines[i] = line.name 93 | i++ 94 | } 95 | 96 | ts := new(UnitedTopicStore) 97 | ts.Lines = lines 98 | ts.Persist = t.persist 99 | 100 | return ts 101 | } 102 | 103 | func (t *topic) exportTopic() error { 104 | ts := t.genTopicStore() 105 | buf, err := ts.Marshal() 106 | if err != nil { 107 | return utils.NewError( 108 | utils.ErrInternalError, 109 | err.Error(), 110 | ) 111 | } 112 | 113 | err = t.q.setData(t.name, buf) 114 | if err != nil { 115 | return err 116 | } 117 | 118 | // log.Printf("topic[%s] export finisded.", t.name) 119 | return nil 120 | } 121 | 122 | func (t *topic) removeTopicData() error { 123 | err := t.q.delData(t.name) 124 | if err != nil { 125 | return err 126 | } 127 | return nil 128 | } 129 | 130 | func (t *topic) exportLines() error { 131 | t.linesLock.RLock() 132 | defer t.linesLock.RUnlock() 133 | 134 | for lineName, l := range t.lines { 135 | l.inflightLock.RLock() 136 | l.headLock.RLock() 137 | err := l.exportLine() 138 | l.inflightLock.RUnlock() 139 | l.headLock.RUnlock() 140 | if err != nil { 141 | log.Printf("topic[%s] line[%s] export error: %s", t.name, lineName, err) 142 | continue 143 | } 144 | } 145 | 146 | // log.Printf("topic[%s]'s all lines exported.", t.name) 147 | return nil 148 | } 149 | 150 | func (t *topic) loadLine(lineName string, ls UnitedLineStore) (*line, error) { 151 | // log.Printf("topic[%s] loading inflights: %v", t.name, ls.Inflights) 152 | l := new(line) 153 | l.name = lineName 154 | l.recycleKey = t.name + "/" + lineName + keyLineRecycle 155 | lineRecycleData, err := t.q.getData(l.recycleKey) 156 | if err != nil { 157 | return nil, err 158 | } 159 | lineRecycle, err := time.ParseDuration(string(lineRecycleData)) 160 | if err != nil { 161 | return nil, utils.NewError( 162 | utils.ErrInternalError, 163 | err.Error(), 164 | ) 165 | } 166 | l.recycle = lineRecycle 167 | l.head = ls.Head 168 | l.ihead = ls.Ihead 169 | imap := make(map[uint64]bool) 170 | for i := l.ihead; i < l.head; i++ { 171 | imap[i] = false 172 | } 173 | l.imap = imap 174 | inflight := list.New() 175 | for index := range ls.Inflights { 176 | msg := ls.Inflights[index] 177 | inflight.PushBack(msg) 178 | imap[msg.Tid] = true 179 | } 180 | l.inflight = inflight 181 | l.t = t 182 | 183 | t.q.registerLine(t.name, l.name, l.recycle.String()) 184 | return l, nil 185 | } 186 | 187 | func (t *topic) getEnd() uint64 { 188 | var end uint64 189 | if len(t.lines) == 0 { 190 | end = t.head 191 | } else { 192 | end = t.tail 193 | for _, l := range t.lines { 194 | if l.recycle > 0 { 195 | if l.ihead < end { 196 | end = l.ihead 197 | } 198 | } else { 199 | if l.head < end { 200 | end = l.head 201 | } 202 | } 203 | } 204 | } 205 | return end 206 | } 207 | 208 | func (t *topic) clean() (quit bool) { 209 | quit = false 210 | 211 | t.headLock.Lock() 212 | defer t.headLock.Unlock() 213 | 214 | // starting := t.head 215 | endTime := time.Now().Add(bgCleanTimeout) 216 | // log.Printf("topic[%s] begin to clean at %d", t.name, starting) 217 | 218 | // defer func() { 219 | // if t.head != starting { 220 | // log.Printf("topic[%s] garbage[%d - %d] are cleaned", t.name, starting, t.head-1) 221 | // } 222 | // }() 223 | 224 | ending := t.getEnd() 225 | for t.head < ending { 226 | select { 227 | case <-t.quit: 228 | quit = true 229 | // log.Printf("topic[%s] catched quit at %d", t.name, t.head) 230 | return 231 | default: 232 | // nothing todo 233 | } 234 | 235 | if time.Now().After(endTime) { 236 | // log.Printf("topic[%s] cleaning timeout, break at %d", t.name, t.head) 237 | return 238 | } 239 | 240 | key := utils.Acatui(t.name, ":", t.head) 241 | err := t.q.delData(key) 242 | if err != nil { 243 | log.Printf("topic[%s] del %s error; %s", t.name, key, err) 244 | return 245 | } 246 | 247 | t.head++ 248 | err = t.exportHead() 249 | if err != nil { 250 | log.Printf("topic[%s] export head error: %s", t.name, err) 251 | return 252 | } 253 | } 254 | 255 | return 256 | } 257 | 258 | func (t *topic) backgroundClean() { 259 | t.wg.Add(1) 260 | defer t.wg.Done() 261 | 262 | bgQuit := false 263 | backupTick := time.NewTicker(bgBackupInterval) 264 | cleanTick := time.NewTicker(bgCleanInterval) 265 | for !bgQuit { 266 | select { 267 | case <-backupTick.C: 268 | err := t.exportLines() 269 | if err != nil { 270 | log.Printf("topic[%s] export lines error: %s", t.name, err) 271 | } 272 | case <-cleanTick.C: 273 | if !t.persist { 274 | log.Printf("cleaning... %v", t.persist) 275 | bgQuit := t.clean() 276 | if bgQuit { 277 | // log.Printf("topic[%s] t.clean return quit: %v", t.name, bgQuit) 278 | break 279 | } 280 | } 281 | case <-t.quit: 282 | // log.Printf("topic[%s] background clean catched quit", t.name) 283 | bgQuit = true 284 | break 285 | } 286 | } 287 | // log.Printf("topic[%s] background clean exit.", t.name) 288 | } 289 | 290 | func (t *topic) start() { 291 | // log.Printf("topic[%s] is starting...", t.name) 292 | go t.backgroundClean() 293 | } 294 | 295 | func (t *topic) newLine(name string, recycle time.Duration) (*line, error) { 296 | inflight := list.New() 297 | imap := make(map[uint64]bool) 298 | l := new(line) 299 | l.name = name 300 | if !t.persist { 301 | l.head = t.head 302 | } else { 303 | l.head = 0 304 | } 305 | l.recycle = recycle 306 | l.recycleKey = t.name + "/" + name + keyLineRecycle 307 | l.inflight = inflight 308 | l.ihead = l.head 309 | l.imap = imap 310 | l.t = t 311 | 312 | err := l.exportLine() 313 | if err != nil { 314 | return nil, err 315 | } 316 | err = l.exportRecycle() 317 | if err != nil { 318 | return nil, err 319 | } 320 | 321 | return l, nil 322 | } 323 | 324 | func (t *topic) createLine(name string, recycle time.Duration, fromEtcd bool) error { 325 | t.linesLock.Lock() 326 | defer t.linesLock.Unlock() 327 | _, ok := t.lines[name] 328 | if ok { 329 | return utils.NewError( 330 | utils.ErrLineExisted, 331 | `topic createLine`, 332 | ) 333 | } 334 | 335 | l, err := t.newLine(name, recycle) 336 | if err != nil { 337 | return err 338 | } 339 | 340 | t.lines[name] = l 341 | 342 | err = t.exportTopic() 343 | if err != nil { 344 | t.linesLock.Lock() 345 | delete(t.lines, name) 346 | t.linesLock.Unlock() 347 | return err 348 | } 349 | 350 | if !fromEtcd { 351 | t.q.registerLine(t.name, l.name, l.recycle.String()) 352 | } 353 | 354 | log.Printf("topic[%s] line[%s:%v] created.", t.name, name, recycle) 355 | return nil 356 | } 357 | 358 | func (t *topic) push(data []byte) error { 359 | t.tailLock.Lock() 360 | defer t.tailLock.Unlock() 361 | 362 | key := utils.Acatui(t.name, ":", t.tail) 363 | err := t.q.setData(key, data) 364 | if err != nil { 365 | return err 366 | } 367 | // log.Printf("topic[%s] %s pushed.", t.name, string(data)) 368 | 369 | t.tail++ 370 | err = t.exportTail() 371 | if err != nil { 372 | t.tail-- 373 | return err 374 | } 375 | 376 | return nil 377 | } 378 | 379 | func (t *topic) mPush(datas [][]byte) error { 380 | t.tailLock.Lock() 381 | defer t.tailLock.Unlock() 382 | 383 | oldTail := t.tail 384 | for _, data := range datas { 385 | key := utils.Acatui(t.name, ":", t.tail) 386 | err := t.q.setData(key, data) 387 | if err != nil { 388 | t.tail = oldTail 389 | return err 390 | } 391 | // log.Printf("topic[%s] %s pushed.", t.name, string(data)) 392 | t.tail++ 393 | } 394 | 395 | err := t.exportTail() 396 | if err != nil { 397 | t.tail = oldTail 398 | return err 399 | } 400 | 401 | return nil 402 | } 403 | 404 | func (t *topic) pop(name string) (uint64, []byte, error) { 405 | t.linesLock.RLock() 406 | l, ok := t.lines[name] 407 | t.linesLock.RUnlock() 408 | if !ok { 409 | // log.Printf("topic[%s] line[%s] not existed.", t.name, name) 410 | return 0, nil, utils.NewError( 411 | utils.ErrLineNotExisted, 412 | `topic pop`, 413 | ) 414 | } 415 | 416 | return l.pop() 417 | } 418 | 419 | func (t *topic) mPop(name string, n int) ([]uint64, [][]byte, error) { 420 | t.linesLock.RLock() 421 | l, ok := t.lines[name] 422 | t.linesLock.RUnlock() 423 | if !ok { 424 | // log.Printf("topic[%s] line[%s] not existed.", t.name, name) 425 | return nil, nil, utils.NewError( 426 | utils.ErrLineNotExisted, 427 | `topic mPop`, 428 | ) 429 | } 430 | 431 | return l.mPop(n) 432 | } 433 | 434 | func (t *topic) confirm(name string, id uint64) error { 435 | t.linesLock.RLock() 436 | l, ok := t.lines[name] 437 | t.linesLock.RUnlock() 438 | if !ok { 439 | // log.Printf("topic[%s] line[%s] not existed.", t.name, name) 440 | return utils.NewError( 441 | utils.ErrLineNotExisted, 442 | `topic confirm`, 443 | ) 444 | } 445 | 446 | return l.confirm(id) 447 | } 448 | 449 | func (t *topic) statLine(name string) (*Stat, error) { 450 | t.linesLock.RLock() 451 | l, ok := t.lines[name] 452 | t.linesLock.RUnlock() 453 | if !ok { 454 | // log.Printf("topic[%s] line[%s] not existed.", t.name, name) 455 | return nil, utils.NewError( 456 | utils.ErrLineNotExisted, 457 | `topic statLine`, 458 | ) 459 | } 460 | 461 | qs := l.stat() 462 | return qs, nil 463 | } 464 | 465 | func (t *topic) stat() *Stat { 466 | qs := new(Stat) 467 | qs.Name = t.name 468 | qs.Type = "topic" 469 | 470 | t.headLock.RLock() 471 | qs.Head = t.head 472 | t.headLock.RUnlock() 473 | t.tailLock.RLock() 474 | qs.Tail = t.tail 475 | t.tailLock.RUnlock() 476 | qs.Count = qs.Tail - qs.Head 477 | 478 | t.linesLock.RLock() 479 | defer t.linesLock.RUnlock() 480 | qs.Lines = make([]*Stat, 0) 481 | for _, l := range t.lines { 482 | ls := l.stat() 483 | qs.Lines = append(qs.Lines, ls) 484 | } 485 | 486 | return qs 487 | } 488 | 489 | func (t *topic) emptyLine(name string) error { 490 | t.linesLock.RLock() 491 | l, ok := t.lines[name] 492 | t.linesLock.RUnlock() 493 | if !ok { 494 | // log.Printf("topic[%s] line[%s] not existed.", t.name, name) 495 | return utils.NewError( 496 | utils.ErrLineNotExisted, 497 | `topic emptyLine`, 498 | ) 499 | } 500 | 501 | return l.empty() 502 | } 503 | 504 | func (t *topic) empty() error { 505 | t.linesLock.RLock() 506 | defer t.linesLock.RUnlock() 507 | 508 | for _, l := range t.lines { 509 | err := l.empty() 510 | if err != nil { 511 | // log.Printf("topic[%s] line[%s] empty error: %s", t.name, name, err) 512 | return err 513 | } 514 | } 515 | 516 | t.headLock.Lock() 517 | defer t.headLock.Unlock() 518 | t.tailLock.RLock() 519 | defer t.tailLock.RUnlock() 520 | t.head = t.tail 521 | err := t.exportHead() 522 | if err != nil { 523 | return err 524 | } 525 | 526 | log.Printf("topic[%s] empty succ", t.name) 527 | return nil 528 | } 529 | 530 | func (t *topic) close() { 531 | close(t.quit) 532 | t.wg.Wait() 533 | } 534 | 535 | func (t *topic) removeLine(name string, fromEtcd bool) error { 536 | t.linesLock.RLock() 537 | defer t.linesLock.RUnlock() 538 | l, ok := t.lines[name] 539 | if !ok { 540 | // log.Printf("topic[%s] line[%s] not existed.", t.name, name) 541 | return utils.NewError( 542 | utils.ErrLineNotExisted, 543 | `topic statLine`, 544 | ) 545 | } 546 | 547 | delete(t.lines, name) 548 | err := t.exportTopic() 549 | if err != nil { 550 | t.linesLock.Lock() 551 | t.lines[name] = l 552 | t.linesLock.Unlock() 553 | return err 554 | } 555 | 556 | if !fromEtcd { 557 | t.q.unRegisterLine(t.name, name) 558 | } 559 | 560 | return l.remove() 561 | } 562 | 563 | func (t *topic) removeLines() error { 564 | for lineName, l := range t.lines { 565 | err := l.remove() 566 | if err != nil { 567 | log.Printf("topic[%s] line[%s] remove error: %s", t.name, lineName, err) 568 | continue 569 | } 570 | delete(t.lines, lineName) 571 | } 572 | 573 | // log.Printf("topic[%s]'s all lines removed.", t.name) 574 | return nil 575 | } 576 | 577 | func (t *topic) removeMsgData() error { 578 | for i := t.head; i < t.tail; i++ { 579 | key := utils.Acatui(t.name, ":", i) 580 | err := t.q.delData(key) 581 | if err != nil { 582 | log.Printf("topic[%s] del data[%s] error; %s", t.name, key, err) 583 | continue 584 | } 585 | } 586 | return nil 587 | } 588 | 589 | func (t *topic) remove() error { 590 | t.close() 591 | 592 | t.linesLock.Lock() 593 | defer t.linesLock.Unlock() 594 | 595 | err := t.removeLines() 596 | if err != nil { 597 | log.Printf("topic[%s] removeLines error: %s", t.name, err) 598 | } 599 | 600 | t.headLock.Lock() 601 | defer t.headLock.Unlock() 602 | err = t.removeHeadData() 603 | if err != nil { 604 | log.Printf("topic[%s] removeHeadData error: %s", t.name, err) 605 | } 606 | 607 | t.tailLock.Lock() 608 | defer t.tailLock.Unlock() 609 | err = t.removeTailData() 610 | if err != nil { 611 | log.Printf("topic[%s] removeTailData error: %s", t.name, err) 612 | } 613 | 614 | err = t.removeTopicData() 615 | if err != nil { 616 | log.Printf("topic[%s] removeTopicData error: %s", t.name, err) 617 | } 618 | 619 | err = t.removeMsgData() 620 | if err != nil { 621 | log.Printf("topic[%s] removeMsgData error: %s", t.name, err) 622 | } 623 | 624 | log.Printf("topic[%s] remove succ", t.name) 625 | return nil 626 | } 627 | -------------------------------------------------------------------------------- /queue/uq.pb.go: -------------------------------------------------------------------------------- 1 | // Code generated by protoc-gen-gogo. 2 | // source: uq.proto 3 | // DO NOT EDIT! 4 | 5 | /* 6 | Package queue is a generated protocol buffer package. 7 | 8 | It is generated from these files: 9 | uq.proto 10 | 11 | It has these top-level messages: 12 | UnitedQueueStore 13 | UnitedTopicStore 14 | InflightMessage 15 | UnitedLineStore 16 | */ 17 | package queue 18 | 19 | import proto "github.com/gogo/protobuf/proto" 20 | import math "math" 21 | 22 | // discarding unused import gogoproto "gogoproto" 23 | 24 | import io "io" 25 | import fmt "fmt" 26 | import github_com_gogo_protobuf_proto "github.com/gogo/protobuf/proto" 27 | 28 | // Reference imports to suppress errors if they are not otherwise used. 29 | var _ = proto.Marshal 30 | var _ = math.Inf 31 | 32 | type UnitedQueueStore struct { 33 | Topics []string `protobuf:"bytes,1,rep" json:"Topics,omitempty"` 34 | XXX_unrecognized []byte `json:"-"` 35 | } 36 | 37 | func (m *UnitedQueueStore) Reset() { *m = UnitedQueueStore{} } 38 | func (m *UnitedQueueStore) String() string { return proto.CompactTextString(m) } 39 | func (*UnitedQueueStore) ProtoMessage() {} 40 | 41 | type UnitedTopicStore struct { 42 | Lines []string `protobuf:"bytes,1,rep" json:"Lines,omitempty"` 43 | Persist bool `protobuf:"varint,2,req" json:"Persist"` 44 | XXX_unrecognized []byte `json:"-"` 45 | } 46 | 47 | func (m *UnitedTopicStore) Reset() { *m = UnitedTopicStore{} } 48 | func (m *UnitedTopicStore) String() string { return proto.CompactTextString(m) } 49 | func (*UnitedTopicStore) ProtoMessage() {} 50 | 51 | type InflightMessage struct { 52 | Tid uint64 `protobuf:"varint,1,req" json:"Tid"` 53 | Exptime int64 `protobuf:"varint,2,req" json:"Exptime"` 54 | XXX_unrecognized []byte `json:"-"` 55 | } 56 | 57 | func (m *InflightMessage) Reset() { *m = InflightMessage{} } 58 | func (m *InflightMessage) String() string { return proto.CompactTextString(m) } 59 | func (*InflightMessage) ProtoMessage() {} 60 | 61 | type UnitedLineStore struct { 62 | Head uint64 `protobuf:"varint,1,req" json:"Head"` 63 | Ihead uint64 `protobuf:"varint,2,req" json:"Ihead"` 64 | Inflights []*InflightMessage `protobuf:"bytes,3,rep" json:"Inflights,omitempty"` 65 | XXX_unrecognized []byte `json:"-"` 66 | } 67 | 68 | func (m *UnitedLineStore) Reset() { *m = UnitedLineStore{} } 69 | func (m *UnitedLineStore) String() string { return proto.CompactTextString(m) } 70 | func (*UnitedLineStore) ProtoMessage() {} 71 | 72 | func (m *UnitedQueueStore) Marshal() (data []byte, err error) { 73 | size := m.Size() 74 | data = make([]byte, size) 75 | n, err := m.MarshalTo(data) 76 | if err != nil { 77 | return nil, err 78 | } 79 | return data[:n], nil 80 | } 81 | 82 | func (m *UnitedQueueStore) MarshalTo(data []byte) (int, error) { 83 | var i int 84 | _ = i 85 | var l int 86 | _ = l 87 | if len(m.Topics) > 0 { 88 | for _, s := range m.Topics { 89 | data[i] = 0xa 90 | i++ 91 | l = len(s) 92 | for l >= 1<<7 { 93 | data[i] = uint8(uint64(l)&0x7f | 0x80) 94 | l >>= 7 95 | i++ 96 | } 97 | data[i] = uint8(l) 98 | i++ 99 | i += copy(data[i:], s) 100 | } 101 | } 102 | if m.XXX_unrecognized != nil { 103 | i += copy(data[i:], m.XXX_unrecognized) 104 | } 105 | return i, nil 106 | } 107 | 108 | func (m *UnitedTopicStore) Marshal() (data []byte, err error) { 109 | size := m.Size() 110 | data = make([]byte, size) 111 | n, err := m.MarshalTo(data) 112 | if err != nil { 113 | return nil, err 114 | } 115 | return data[:n], nil 116 | } 117 | 118 | func (m *UnitedTopicStore) MarshalTo(data []byte) (int, error) { 119 | var i int 120 | _ = i 121 | var l int 122 | _ = l 123 | if len(m.Lines) > 0 { 124 | for _, s := range m.Lines { 125 | data[i] = 0xa 126 | i++ 127 | l = len(s) 128 | for l >= 1<<7 { 129 | data[i] = uint8(uint64(l)&0x7f | 0x80) 130 | l >>= 7 131 | i++ 132 | } 133 | data[i] = uint8(l) 134 | i++ 135 | i += copy(data[i:], s) 136 | } 137 | } 138 | data[i] = 0x10 139 | i++ 140 | if m.Persist { 141 | data[i] = 1 142 | } else { 143 | data[i] = 0 144 | } 145 | i++ 146 | if m.XXX_unrecognized != nil { 147 | i += copy(data[i:], m.XXX_unrecognized) 148 | } 149 | return i, nil 150 | } 151 | 152 | func (m *InflightMessage) Marshal() (data []byte, err error) { 153 | size := m.Size() 154 | data = make([]byte, size) 155 | n, err := m.MarshalTo(data) 156 | if err != nil { 157 | return nil, err 158 | } 159 | return data[:n], nil 160 | } 161 | 162 | func (m *InflightMessage) MarshalTo(data []byte) (int, error) { 163 | var i int 164 | _ = i 165 | var l int 166 | _ = l 167 | data[i] = 0x8 168 | i++ 169 | i = encodeVarintUq(data, i, uint64(m.Tid)) 170 | data[i] = 0x10 171 | i++ 172 | i = encodeVarintUq(data, i, uint64(m.Exptime)) 173 | if m.XXX_unrecognized != nil { 174 | i += copy(data[i:], m.XXX_unrecognized) 175 | } 176 | return i, nil 177 | } 178 | 179 | func (m *UnitedLineStore) Marshal() (data []byte, err error) { 180 | size := m.Size() 181 | data = make([]byte, size) 182 | n, err := m.MarshalTo(data) 183 | if err != nil { 184 | return nil, err 185 | } 186 | return data[:n], nil 187 | } 188 | 189 | func (m *UnitedLineStore) MarshalTo(data []byte) (int, error) { 190 | var i int 191 | _ = i 192 | var l int 193 | _ = l 194 | data[i] = 0x8 195 | i++ 196 | i = encodeVarintUq(data, i, uint64(m.Head)) 197 | data[i] = 0x10 198 | i++ 199 | i = encodeVarintUq(data, i, uint64(m.Ihead)) 200 | if len(m.Inflights) > 0 { 201 | for _, msg := range m.Inflights { 202 | data[i] = 0x1a 203 | i++ 204 | i = encodeVarintUq(data, i, uint64(msg.Size())) 205 | n, err := msg.MarshalTo(data[i:]) 206 | if err != nil { 207 | return 0, err 208 | } 209 | i += n 210 | } 211 | } 212 | if m.XXX_unrecognized != nil { 213 | i += copy(data[i:], m.XXX_unrecognized) 214 | } 215 | return i, nil 216 | } 217 | 218 | func encodeFixed64Uq(data []byte, offset int, v uint64) int { 219 | data[offset] = uint8(v) 220 | data[offset+1] = uint8(v >> 8) 221 | data[offset+2] = uint8(v >> 16) 222 | data[offset+3] = uint8(v >> 24) 223 | data[offset+4] = uint8(v >> 32) 224 | data[offset+5] = uint8(v >> 40) 225 | data[offset+6] = uint8(v >> 48) 226 | data[offset+7] = uint8(v >> 56) 227 | return offset + 8 228 | } 229 | func encodeFixed32Uq(data []byte, offset int, v uint32) int { 230 | data[offset] = uint8(v) 231 | data[offset+1] = uint8(v >> 8) 232 | data[offset+2] = uint8(v >> 16) 233 | data[offset+3] = uint8(v >> 24) 234 | return offset + 4 235 | } 236 | func encodeVarintUq(data []byte, offset int, v uint64) int { 237 | for v >= 1<<7 { 238 | data[offset] = uint8(v&0x7f | 0x80) 239 | v >>= 7 240 | offset++ 241 | } 242 | data[offset] = uint8(v) 243 | return offset + 1 244 | } 245 | func (m *UnitedQueueStore) Size() (n int) { 246 | var l int 247 | _ = l 248 | if len(m.Topics) > 0 { 249 | for _, s := range m.Topics { 250 | l = len(s) 251 | n += 1 + l + sovUq(uint64(l)) 252 | } 253 | } 254 | if m.XXX_unrecognized != nil { 255 | n += len(m.XXX_unrecognized) 256 | } 257 | return n 258 | } 259 | 260 | func (m *UnitedTopicStore) Size() (n int) { 261 | var l int 262 | _ = l 263 | if len(m.Lines) > 0 { 264 | for _, s := range m.Lines { 265 | l = len(s) 266 | n += 1 + l + sovUq(uint64(l)) 267 | } 268 | } 269 | n += 2 270 | if m.XXX_unrecognized != nil { 271 | n += len(m.XXX_unrecognized) 272 | } 273 | return n 274 | } 275 | 276 | func (m *InflightMessage) Size() (n int) { 277 | var l int 278 | _ = l 279 | n += 1 + sovUq(uint64(m.Tid)) 280 | n += 1 + sovUq(uint64(m.Exptime)) 281 | if m.XXX_unrecognized != nil { 282 | n += len(m.XXX_unrecognized) 283 | } 284 | return n 285 | } 286 | 287 | func (m *UnitedLineStore) Size() (n int) { 288 | var l int 289 | _ = l 290 | n += 1 + sovUq(uint64(m.Head)) 291 | n += 1 + sovUq(uint64(m.Ihead)) 292 | if len(m.Inflights) > 0 { 293 | for _, e := range m.Inflights { 294 | l = e.Size() 295 | n += 1 + l + sovUq(uint64(l)) 296 | } 297 | } 298 | if m.XXX_unrecognized != nil { 299 | n += len(m.XXX_unrecognized) 300 | } 301 | return n 302 | } 303 | 304 | func sovUq(x uint64) (n int) { 305 | for { 306 | n++ 307 | x >>= 7 308 | if x == 0 { 309 | break 310 | } 311 | } 312 | return n 313 | } 314 | func sozUq(x uint64) (n int) { 315 | return sovUq(uint64((x << 1) ^ uint64((int64(x) >> 63)))) 316 | } 317 | func (m *UnitedQueueStore) Unmarshal(data []byte) error { 318 | l := len(data) 319 | iNdEx := 0 320 | for iNdEx < l { 321 | var wire uint64 322 | for shift := uint(0); ; shift += 7 { 323 | if iNdEx >= l { 324 | return io.ErrUnexpectedEOF 325 | } 326 | b := data[iNdEx] 327 | iNdEx++ 328 | wire |= (uint64(b) & 0x7F) << shift 329 | if b < 0x80 { 330 | break 331 | } 332 | } 333 | fieldNum := int32(wire >> 3) 334 | wireType := int(wire & 0x7) 335 | switch fieldNum { 336 | case 1: 337 | if wireType != 2 { 338 | return fmt.Errorf("proto: wrong wireType = %d for field Topics", wireType) 339 | } 340 | var stringLen uint64 341 | for shift := uint(0); ; shift += 7 { 342 | if iNdEx >= l { 343 | return io.ErrUnexpectedEOF 344 | } 345 | b := data[iNdEx] 346 | iNdEx++ 347 | stringLen |= (uint64(b) & 0x7F) << shift 348 | if b < 0x80 { 349 | break 350 | } 351 | } 352 | postIndex := iNdEx + int(stringLen) 353 | if stringLen < 0 { 354 | return ErrInvalidLengthUq 355 | } 356 | if postIndex > l { 357 | return io.ErrUnexpectedEOF 358 | } 359 | m.Topics = append(m.Topics, string(data[iNdEx:postIndex])) 360 | iNdEx = postIndex 361 | default: 362 | var sizeOfWire int 363 | for { 364 | sizeOfWire++ 365 | wire >>= 7 366 | if wire == 0 { 367 | break 368 | } 369 | } 370 | iNdEx -= sizeOfWire 371 | skippy, err := skipUq(data[iNdEx:]) 372 | if err != nil { 373 | return err 374 | } 375 | if skippy < 0 { 376 | return ErrInvalidLengthUq 377 | } 378 | if (iNdEx + skippy) > l { 379 | return io.ErrUnexpectedEOF 380 | } 381 | m.XXX_unrecognized = append(m.XXX_unrecognized, data[iNdEx:iNdEx+skippy]...) 382 | iNdEx += skippy 383 | } 384 | } 385 | 386 | return nil 387 | } 388 | func (m *UnitedTopicStore) Unmarshal(data []byte) error { 389 | var hasFields [1]uint64 390 | l := len(data) 391 | iNdEx := 0 392 | for iNdEx < l { 393 | var wire uint64 394 | for shift := uint(0); ; shift += 7 { 395 | if iNdEx >= l { 396 | return io.ErrUnexpectedEOF 397 | } 398 | b := data[iNdEx] 399 | iNdEx++ 400 | wire |= (uint64(b) & 0x7F) << shift 401 | if b < 0x80 { 402 | break 403 | } 404 | } 405 | fieldNum := int32(wire >> 3) 406 | wireType := int(wire & 0x7) 407 | switch fieldNum { 408 | case 1: 409 | if wireType != 2 { 410 | return fmt.Errorf("proto: wrong wireType = %d for field Lines", wireType) 411 | } 412 | var stringLen uint64 413 | for shift := uint(0); ; shift += 7 { 414 | if iNdEx >= l { 415 | return io.ErrUnexpectedEOF 416 | } 417 | b := data[iNdEx] 418 | iNdEx++ 419 | stringLen |= (uint64(b) & 0x7F) << shift 420 | if b < 0x80 { 421 | break 422 | } 423 | } 424 | postIndex := iNdEx + int(stringLen) 425 | if stringLen < 0 { 426 | return ErrInvalidLengthUq 427 | } 428 | if postIndex > l { 429 | return io.ErrUnexpectedEOF 430 | } 431 | m.Lines = append(m.Lines, string(data[iNdEx:postIndex])) 432 | iNdEx = postIndex 433 | case 2: 434 | if wireType != 0 { 435 | return fmt.Errorf("proto: wrong wireType = %d for field Persist", wireType) 436 | } 437 | var v int 438 | for shift := uint(0); ; shift += 7 { 439 | if iNdEx >= l { 440 | return io.ErrUnexpectedEOF 441 | } 442 | b := data[iNdEx] 443 | iNdEx++ 444 | v |= (int(b) & 0x7F) << shift 445 | if b < 0x80 { 446 | break 447 | } 448 | } 449 | m.Persist = bool(v != 0) 450 | hasFields[0] |= uint64(0x00000001) 451 | default: 452 | var sizeOfWire int 453 | for { 454 | sizeOfWire++ 455 | wire >>= 7 456 | if wire == 0 { 457 | break 458 | } 459 | } 460 | iNdEx -= sizeOfWire 461 | skippy, err := skipUq(data[iNdEx:]) 462 | if err != nil { 463 | return err 464 | } 465 | if skippy < 0 { 466 | return ErrInvalidLengthUq 467 | } 468 | if (iNdEx + skippy) > l { 469 | return io.ErrUnexpectedEOF 470 | } 471 | m.XXX_unrecognized = append(m.XXX_unrecognized, data[iNdEx:iNdEx+skippy]...) 472 | iNdEx += skippy 473 | } 474 | } 475 | if hasFields[0]&uint64(0x00000001) == 0 { 476 | return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Persist") 477 | } 478 | 479 | return nil 480 | } 481 | func (m *InflightMessage) Unmarshal(data []byte) error { 482 | var hasFields [1]uint64 483 | l := len(data) 484 | iNdEx := 0 485 | for iNdEx < l { 486 | var wire uint64 487 | for shift := uint(0); ; shift += 7 { 488 | if iNdEx >= l { 489 | return io.ErrUnexpectedEOF 490 | } 491 | b := data[iNdEx] 492 | iNdEx++ 493 | wire |= (uint64(b) & 0x7F) << shift 494 | if b < 0x80 { 495 | break 496 | } 497 | } 498 | fieldNum := int32(wire >> 3) 499 | wireType := int(wire & 0x7) 500 | switch fieldNum { 501 | case 1: 502 | if wireType != 0 { 503 | return fmt.Errorf("proto: wrong wireType = %d for field Tid", wireType) 504 | } 505 | m.Tid = 0 506 | for shift := uint(0); ; shift += 7 { 507 | if iNdEx >= l { 508 | return io.ErrUnexpectedEOF 509 | } 510 | b := data[iNdEx] 511 | iNdEx++ 512 | m.Tid |= (uint64(b) & 0x7F) << shift 513 | if b < 0x80 { 514 | break 515 | } 516 | } 517 | hasFields[0] |= uint64(0x00000001) 518 | case 2: 519 | if wireType != 0 { 520 | return fmt.Errorf("proto: wrong wireType = %d for field Exptime", wireType) 521 | } 522 | m.Exptime = 0 523 | for shift := uint(0); ; shift += 7 { 524 | if iNdEx >= l { 525 | return io.ErrUnexpectedEOF 526 | } 527 | b := data[iNdEx] 528 | iNdEx++ 529 | m.Exptime |= (int64(b) & 0x7F) << shift 530 | if b < 0x80 { 531 | break 532 | } 533 | } 534 | hasFields[0] |= uint64(0x00000002) 535 | default: 536 | var sizeOfWire int 537 | for { 538 | sizeOfWire++ 539 | wire >>= 7 540 | if wire == 0 { 541 | break 542 | } 543 | } 544 | iNdEx -= sizeOfWire 545 | skippy, err := skipUq(data[iNdEx:]) 546 | if err != nil { 547 | return err 548 | } 549 | if skippy < 0 { 550 | return ErrInvalidLengthUq 551 | } 552 | if (iNdEx + skippy) > l { 553 | return io.ErrUnexpectedEOF 554 | } 555 | m.XXX_unrecognized = append(m.XXX_unrecognized, data[iNdEx:iNdEx+skippy]...) 556 | iNdEx += skippy 557 | } 558 | } 559 | if hasFields[0]&uint64(0x00000001) == 0 { 560 | return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Tid") 561 | } 562 | if hasFields[0]&uint64(0x00000002) == 0 { 563 | return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Exptime") 564 | } 565 | 566 | return nil 567 | } 568 | func (m *UnitedLineStore) Unmarshal(data []byte) error { 569 | var hasFields [1]uint64 570 | l := len(data) 571 | iNdEx := 0 572 | for iNdEx < l { 573 | var wire uint64 574 | for shift := uint(0); ; shift += 7 { 575 | if iNdEx >= l { 576 | return io.ErrUnexpectedEOF 577 | } 578 | b := data[iNdEx] 579 | iNdEx++ 580 | wire |= (uint64(b) & 0x7F) << shift 581 | if b < 0x80 { 582 | break 583 | } 584 | } 585 | fieldNum := int32(wire >> 3) 586 | wireType := int(wire & 0x7) 587 | switch fieldNum { 588 | case 1: 589 | if wireType != 0 { 590 | return fmt.Errorf("proto: wrong wireType = %d for field Head", wireType) 591 | } 592 | m.Head = 0 593 | for shift := uint(0); ; shift += 7 { 594 | if iNdEx >= l { 595 | return io.ErrUnexpectedEOF 596 | } 597 | b := data[iNdEx] 598 | iNdEx++ 599 | m.Head |= (uint64(b) & 0x7F) << shift 600 | if b < 0x80 { 601 | break 602 | } 603 | } 604 | hasFields[0] |= uint64(0x00000001) 605 | case 2: 606 | if wireType != 0 { 607 | return fmt.Errorf("proto: wrong wireType = %d for field Ihead", wireType) 608 | } 609 | m.Ihead = 0 610 | for shift := uint(0); ; shift += 7 { 611 | if iNdEx >= l { 612 | return io.ErrUnexpectedEOF 613 | } 614 | b := data[iNdEx] 615 | iNdEx++ 616 | m.Ihead |= (uint64(b) & 0x7F) << shift 617 | if b < 0x80 { 618 | break 619 | } 620 | } 621 | hasFields[0] |= uint64(0x00000002) 622 | case 3: 623 | if wireType != 2 { 624 | return fmt.Errorf("proto: wrong wireType = %d for field Inflights", wireType) 625 | } 626 | var msglen int 627 | for shift := uint(0); ; shift += 7 { 628 | if iNdEx >= l { 629 | return io.ErrUnexpectedEOF 630 | } 631 | b := data[iNdEx] 632 | iNdEx++ 633 | msglen |= (int(b) & 0x7F) << shift 634 | if b < 0x80 { 635 | break 636 | } 637 | } 638 | postIndex := iNdEx + msglen 639 | if msglen < 0 { 640 | return ErrInvalidLengthUq 641 | } 642 | if postIndex > l { 643 | return io.ErrUnexpectedEOF 644 | } 645 | m.Inflights = append(m.Inflights, &InflightMessage{}) 646 | if err := m.Inflights[len(m.Inflights)-1].Unmarshal(data[iNdEx:postIndex]); err != nil { 647 | return err 648 | } 649 | iNdEx = postIndex 650 | default: 651 | var sizeOfWire int 652 | for { 653 | sizeOfWire++ 654 | wire >>= 7 655 | if wire == 0 { 656 | break 657 | } 658 | } 659 | iNdEx -= sizeOfWire 660 | skippy, err := skipUq(data[iNdEx:]) 661 | if err != nil { 662 | return err 663 | } 664 | if skippy < 0 { 665 | return ErrInvalidLengthUq 666 | } 667 | if (iNdEx + skippy) > l { 668 | return io.ErrUnexpectedEOF 669 | } 670 | m.XXX_unrecognized = append(m.XXX_unrecognized, data[iNdEx:iNdEx+skippy]...) 671 | iNdEx += skippy 672 | } 673 | } 674 | if hasFields[0]&uint64(0x00000001) == 0 { 675 | return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Head") 676 | } 677 | if hasFields[0]&uint64(0x00000002) == 0 { 678 | return github_com_gogo_protobuf_proto.NewRequiredNotSetError("Ihead") 679 | } 680 | 681 | return nil 682 | } 683 | func skipUq(data []byte) (n int, err error) { 684 | l := len(data) 685 | iNdEx := 0 686 | for iNdEx < l { 687 | var wire uint64 688 | for shift := uint(0); ; shift += 7 { 689 | if iNdEx >= l { 690 | return 0, io.ErrUnexpectedEOF 691 | } 692 | b := data[iNdEx] 693 | iNdEx++ 694 | wire |= (uint64(b) & 0x7F) << shift 695 | if b < 0x80 { 696 | break 697 | } 698 | } 699 | wireType := int(wire & 0x7) 700 | switch wireType { 701 | case 0: 702 | for { 703 | if iNdEx >= l { 704 | return 0, io.ErrUnexpectedEOF 705 | } 706 | iNdEx++ 707 | if data[iNdEx-1] < 0x80 { 708 | break 709 | } 710 | } 711 | return iNdEx, nil 712 | case 1: 713 | iNdEx += 8 714 | return iNdEx, nil 715 | case 2: 716 | var length int 717 | for shift := uint(0); ; shift += 7 { 718 | if iNdEx >= l { 719 | return 0, io.ErrUnexpectedEOF 720 | } 721 | b := data[iNdEx] 722 | iNdEx++ 723 | length |= (int(b) & 0x7F) << shift 724 | if b < 0x80 { 725 | break 726 | } 727 | } 728 | iNdEx += length 729 | if length < 0 { 730 | return 0, ErrInvalidLengthUq 731 | } 732 | return iNdEx, nil 733 | case 3: 734 | for { 735 | var innerWire uint64 736 | var start int = iNdEx 737 | for shift := uint(0); ; shift += 7 { 738 | if iNdEx >= l { 739 | return 0, io.ErrUnexpectedEOF 740 | } 741 | b := data[iNdEx] 742 | iNdEx++ 743 | innerWire |= (uint64(b) & 0x7F) << shift 744 | if b < 0x80 { 745 | break 746 | } 747 | } 748 | innerWireType := int(innerWire & 0x7) 749 | if innerWireType == 4 { 750 | break 751 | } 752 | next, err := skipUq(data[start:]) 753 | if err != nil { 754 | return 0, err 755 | } 756 | iNdEx = start + next 757 | } 758 | return iNdEx, nil 759 | case 4: 760 | return iNdEx, nil 761 | case 5: 762 | iNdEx += 4 763 | return iNdEx, nil 764 | default: 765 | return 0, fmt.Errorf("proto: illegal wireType %d", wireType) 766 | } 767 | } 768 | panic("unreachable") 769 | } 770 | 771 | var ( 772 | ErrInvalidLengthUq = fmt.Errorf("proto: negative length found during unmarshaling") 773 | ) 774 | -------------------------------------------------------------------------------- /queue/uq.proto: -------------------------------------------------------------------------------- 1 | // PATH_GOGOPROTOBUF=$GOPATH/src/github.com/gogo/protobuf 2 | // protoc --proto_path=$PATH_GOGOPROTOBUF:$PATH_GOGOPROTOBUF/protobuf:. --gogo_out=. *.proto 3 | 4 | package queue; 5 | import "gogoproto/gogo.proto"; 6 | 7 | option (gogoproto.marshaler_all) = true; 8 | option (gogoproto.sizer_all) = true; 9 | option (gogoproto.unmarshaler_all) = true; 10 | option (gogoproto.goproto_getters_all) = false; 11 | 12 | message UnitedQueueStore { 13 | repeated string Topics = 1 [(gogoproto.nullable) = true]; 14 | } 15 | 16 | message UnitedTopicStore { 17 | repeated string Lines = 1 [(gogoproto.nullable) = true]; 18 | required bool Persist = 2 [(gogoproto.nullable) = false]; 19 | } 20 | 21 | message InflightMessage { 22 | required uint64 Tid = 1 [(gogoproto.nullable) = false]; 23 | required int64 Exptime = 2 [(gogoproto.nullable) = false]; 24 | } 25 | 26 | message UnitedLineStore { 27 | required uint64 Head = 1 [(gogoproto.nullable) = false]; 28 | required uint64 Ihead = 2 [(gogoproto.nullable) = false]; 29 | repeated InflightMessage Inflights = 3 [(gogoproto.nullable) = true]; 30 | } 31 | -------------------------------------------------------------------------------- /store/goleveldbStore.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/syndtr/goleveldb/leveldb" 7 | "github.com/syndtr/goleveldb/leveldb/opt" 8 | ) 9 | 10 | // LevelStore is the goleveldb storage 11 | type LevelStore struct { 12 | path string 13 | db *leveldb.DB 14 | } 15 | 16 | // NewLevelStore returns a new LevelStore 17 | func NewLevelStore(path string) (*LevelStore, error) { 18 | option := &opt.Options{Compression: opt.SnappyCompression} 19 | db, err := leveldb.OpenFile(path, option) 20 | if err != nil { 21 | return nil, err 22 | } 23 | ls := new(LevelStore) 24 | ls.path = path 25 | ls.db = db 26 | 27 | return ls, nil 28 | } 29 | 30 | // Set implements the Set interface 31 | func (l *LevelStore) Set(key string, data []byte) error { 32 | return l.db.Put([]byte(key), data, nil) 33 | 34 | // err := l.db.Put(keyByte, data, nil) 35 | // if err != nil { 36 | // return err 37 | // } 38 | 39 | // return nil 40 | } 41 | 42 | // Get implements the Get interface 43 | func (l *LevelStore) Get(key string) ([]byte, error) { 44 | return l.db.Get([]byte(key), nil) 45 | 46 | // data, err := l.db.Get(keyByte, nil) 47 | // if err != nil { 48 | // return nil, err 49 | // } 50 | 51 | // return data, nil 52 | } 53 | 54 | // Del implements the Del interface 55 | func (l *LevelStore) Del(key string) error { 56 | return l.db.Delete([]byte(key), nil) 57 | 58 | // err := l.db.Delete(keyByte, nil) 59 | // if err != nil { 60 | // return err 61 | // } 62 | // return nil 63 | } 64 | 65 | // Close implements the Close interface 66 | func (l *LevelStore) Close() error { 67 | err := l.db.Close() 68 | if err != nil { 69 | log.Printf("leveldb close error: %s", err) 70 | return err 71 | } 72 | return nil 73 | } 74 | -------------------------------------------------------------------------------- /store/goleveldbStore_test.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | import ( 4 | "os" 5 | "testing" 6 | 7 | . "github.com/smartystreets/goconvey/convey" 8 | ) 9 | 10 | var ( 11 | ldb Storage 12 | dbPath string 13 | ) 14 | 15 | func init() { 16 | dbPath = os.TempDir() + "/uq.store.test.db" 17 | } 18 | 19 | func TestNewLevelStore(t *testing.T) { 20 | Convey("Test New Level Store", t, func() { 21 | ldb, err = NewLevelStore(dbPath) 22 | So(err, ShouldBeNil) 23 | So(ldb, ShouldNotBeNil) 24 | 25 | ldb2, err2 := NewLevelStore("") 26 | So(err2, ShouldNotBeNil) 27 | So(ldb2, ShouldBeNil) 28 | }) 29 | } 30 | 31 | func TestSetLevel(t *testing.T) { 32 | Convey("Test Level Store Set", t, func() { 33 | err = ldb.Set("foo", []byte("bar")) 34 | So(err, ShouldBeNil) 35 | }) 36 | } 37 | 38 | func TestGetLevel(t *testing.T) { 39 | Convey("Test Level Store Get", t, func() { 40 | data, err := ldb.Get("foo") 41 | So(err, ShouldBeNil) 42 | So(string(data), ShouldEqual, "bar") 43 | }) 44 | } 45 | 46 | func TestDelLevel(t *testing.T) { 47 | Convey("Test Level Store Del", t, func() { 48 | err = ldb.Del("foo") 49 | So(err, ShouldBeNil) 50 | }) 51 | } 52 | 53 | func TestCloseLevel(t *testing.T) { 54 | Convey("Test Level Store Close", t, func() { 55 | err = ldb.Close() 56 | So(err, ShouldBeNil) 57 | 58 | err = ldb.Close() 59 | So(err, ShouldNotBeNil) 60 | 61 | err = os.RemoveAll(dbPath) 62 | So(err, ShouldBeNil) 63 | }) 64 | } 65 | -------------------------------------------------------------------------------- /store/memStore.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | import ( 4 | "errors" 5 | "sync" 6 | ) 7 | 8 | // MemStore is the in memory storage 9 | type MemStore struct { 10 | mu sync.RWMutex 11 | db map[string][]byte 12 | } 13 | 14 | // NewMemStore returns a new MemStore 15 | func NewMemStore() (*MemStore, error) { 16 | db := make(map[string][]byte) 17 | ms := new(MemStore) 18 | ms.db = db 19 | 20 | return ms, nil 21 | } 22 | 23 | // Set implements the Set interface 24 | func (m *MemStore) Set(key string, data []byte) error { 25 | m.mu.Lock() 26 | defer m.mu.Unlock() 27 | 28 | m.db[key] = data 29 | return nil 30 | } 31 | 32 | // Get implements the Get interface 33 | func (m *MemStore) Get(key string) ([]byte, error) { 34 | m.mu.RLock() 35 | defer m.mu.RUnlock() 36 | 37 | data, ok := m.db[key] 38 | if !ok { 39 | return nil, errors.New(errNotExisted) 40 | } 41 | return data, nil 42 | } 43 | 44 | // Del implements the Del interface 45 | func (m *MemStore) Del(key string) error { 46 | m.mu.Lock() 47 | defer m.mu.Unlock() 48 | 49 | _, ok := m.db[key] 50 | if !ok { 51 | return errors.New(errNotExisted) 52 | } 53 | 54 | delete(m.db, key) 55 | return nil 56 | } 57 | 58 | // Close implements the Close interface 59 | func (m *MemStore) Close() error { 60 | m.mu.Lock() 61 | defer m.mu.Unlock() 62 | 63 | m.db = nil 64 | return nil 65 | } 66 | -------------------------------------------------------------------------------- /store/memStore_test.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | var ( 10 | mdb Storage 11 | err error 12 | ) 13 | 14 | func TestNewMemStore(t *testing.T) { 15 | Convey("Test New Mem Store", t, func() { 16 | mdb, err = NewMemStore() 17 | So(err, ShouldBeNil) 18 | So(mdb, ShouldNotBeNil) 19 | }) 20 | } 21 | 22 | func TestSetMem(t *testing.T) { 23 | Convey("Test Mem Store Set", t, func() { 24 | err = mdb.Set("foo", []byte("bar")) 25 | So(err, ShouldBeNil) 26 | }) 27 | } 28 | 29 | func TestGetMem(t *testing.T) { 30 | Convey("Test Mem Store Get", t, func() { 31 | data, err := mdb.Get("foo") 32 | So(err, ShouldBeNil) 33 | So(string(data), ShouldEqual, "bar") 34 | 35 | data2, err := mdb.Get("bar") 36 | So(err, ShouldNotBeNil) 37 | So(data2, ShouldBeNil) 38 | }) 39 | } 40 | 41 | func TestDelMem(t *testing.T) { 42 | Convey("Test Mem Store Del", t, func() { 43 | err = mdb.Del("foo") 44 | So(err, ShouldBeNil) 45 | 46 | err = mdb.Del("bar") 47 | So(err, ShouldNotBeNil) 48 | }) 49 | } 50 | 51 | func TestCloseMem(t *testing.T) { 52 | Convey("Test Mem Store Close", t, func() { 53 | err = mdb.Close() 54 | So(err, ShouldBeNil) 55 | }) 56 | } 57 | -------------------------------------------------------------------------------- /store/rocksdbStore.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | // to avoid cgo build, hide rocksdb now. 4 | /* 5 | import ( 6 | "github.com/DanielMorsing/rocksdb" 7 | ) 8 | 9 | type RockStore struct { 10 | path string 11 | db *rocksdb.DB 12 | ro *rocksdb.ReadOptions 13 | wo *rocksdb.WriteOptions 14 | } 15 | 16 | func NewRockStore(path string) (*RockStore, error) { 17 | option := rocksdb.NewOptions() 18 | // option.SetCache(rocksdb.NewLRUCache(3 << 30)) 19 | option.SetCreateIfMissing(true) 20 | 21 | db, err := rocksdb.Open(path, option) 22 | if err != nil { 23 | return nil, err 24 | } 25 | 26 | ro := rocksdb.NewReadOptions() 27 | wo := rocksdb.NewWriteOptions() 28 | 29 | rs := new(RockStore) 30 | rs.path = path 31 | rs.db = db 32 | rs.ro = ro 33 | rs.wo = wo 34 | 35 | return rs, nil 36 | } 37 | 38 | func (r *RockStore) Set(key string, data []byte) error { 39 | return r.db.Put(r.wo, []byte(key), data) 40 | } 41 | 42 | func (r *RockStore) Get(key string) ([]byte, error) { 43 | return r.db.Get(r.ro, []byte(key)) 44 | } 45 | 46 | func (r *RockStore) Del(key string) error { 47 | return r.db.Delete(r.wo, []byte(key)) 48 | } 49 | 50 | func (r *RockStore) Close() error { 51 | r.db.Close() 52 | return nil 53 | } 54 | 55 | */ 56 | -------------------------------------------------------------------------------- /store/storage.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | const ( 4 | errNotExisted string = "Data Not Existed" 5 | errModeNotMatched string = "Storage Mode Not Matched" 6 | ) 7 | 8 | // Storage is the storage of uq 9 | type Storage interface { 10 | Set(key string, data []byte) error 11 | Get(key string) ([]byte, error) 12 | Del(key string) error 13 | Close() error 14 | } 15 | -------------------------------------------------------------------------------- /test/ab-post.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ab -c 20 -n 10000 -p foo.dat "http://localhost:8808/push/foo" 3 | -------------------------------------------------------------------------------- /test/bench_reader.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "runtime" 8 | "sync" 9 | "sync/atomic" 10 | "time" 11 | 12 | "github.com/buaazp/uq/queue" 13 | "github.com/buaazp/uq/store" 14 | "github.com/buaazp/uq/utils" 15 | ) 16 | 17 | var ( 18 | runfor = flag.Duration("runfor", 10*time.Second, "duration of time to run") 19 | topic = flag.String("topic", "foo", "topic to receive messages on") 20 | size = flag.Int("size", 200, "size of messages") 21 | deadline = flag.String("deadline", "", "deadline to start the benchmark run") 22 | storePath = flag.String("path", "./data", "store path") 23 | ) 24 | 25 | var totalMsgCount int64 26 | 27 | func main() { 28 | flag.Parse() 29 | var wg sync.WaitGroup 30 | 31 | log.SetFlags(log.Lshortfile | log.LstdFlags | log.Lmicroseconds) 32 | log.SetPrefix("[uq_bench_reader] ") 33 | 34 | var err error 35 | var storage store.Storage 36 | // storage, err = store.NewMemStore() 37 | storage, err = store.NewLevelStore(*storePath) 38 | if err != nil { 39 | fmt.Printf("store init error: %s\n", err) 40 | return 41 | } 42 | 43 | var messageQueue queue.MessageQueue 44 | messageQueue, err = queue.NewUnitedQueue(storage, "", 0, nil, "uq") 45 | if err != nil { 46 | fmt.Printf("queue init error: %s\n", err) 47 | return 48 | } 49 | 50 | key := *topic + "/x" 51 | err = messageQueue.Create(key, "") 52 | if err != nil { 53 | if e := err.(*utils.Error); e.ErrorCode != utils.ErrLineExisted { 54 | fmt.Printf("line create error: %s\n", err) 55 | return 56 | } 57 | } 58 | line := *topic + "/x" 59 | log.Printf("line: %s", line) 60 | 61 | goChan := make(chan int) 62 | rdyChan := make(chan int) 63 | for j := 0; j < runtime.GOMAXPROCS(runtime.NumCPU()); j++ { 64 | log.Printf("runner: %d", j) 65 | wg.Add(1) 66 | go func() { 67 | subWorker(messageQueue, *runfor, line, rdyChan, goChan) 68 | wg.Done() 69 | }() 70 | <-rdyChan 71 | } 72 | 73 | if *deadline != "" { 74 | t, err := time.Parse("2006-01-02 15:04:05", *deadline) 75 | if err != nil { 76 | log.Fatal(err) 77 | } 78 | d := t.Sub(time.Now()) 79 | log.Printf("sleeping until %s (%s)", t, d) 80 | time.Sleep(d) 81 | } 82 | 83 | start := time.Now() 84 | close(goChan) 85 | wg.Wait() 86 | end := time.Now() 87 | duration := end.Sub(start) 88 | tmc := atomic.LoadInt64(&totalMsgCount) 89 | log.Printf("duration: %s - %.03fmb/s - %.03fops/s - %.03fus/op", 90 | duration, 91 | float64(tmc*int64(*size))/duration.Seconds()/1024/1024, 92 | float64(tmc)/duration.Seconds(), 93 | float64(duration/time.Microsecond)/float64(tmc)) 94 | 95 | messageQueue.Close() 96 | } 97 | 98 | func subWorker(mq queue.MessageQueue, td time.Duration, line string, rdyChan chan int, goChan chan int) { 99 | rdyChan <- 1 100 | <-goChan 101 | var msgCount int64 102 | endTime := time.Now().Add(td) 103 | for { 104 | _, _, err := mq.Pop(line) 105 | if err != nil { 106 | // log.Printf("mq pop error: %s\n", err) 107 | } 108 | msgCount++ 109 | if time.Now().After(endTime) { 110 | break 111 | } 112 | } 113 | atomic.AddInt64(&totalMsgCount, msgCount) 114 | } 115 | -------------------------------------------------------------------------------- /test/bench_writer.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "runtime" 8 | "sync" 9 | "sync/atomic" 10 | "time" 11 | 12 | "github.com/buaazp/uq/queue" 13 | "github.com/buaazp/uq/store" 14 | "github.com/buaazp/uq/utils" 15 | ) 16 | 17 | var ( 18 | runfor = flag.Duration("runfor", 10*time.Second, "duration of time to run") 19 | topic = flag.String("topic", "foo", "topic to receive messages on") 20 | size = flag.Int("size", 200, "size of messages") 21 | deadline = flag.String("deadline", "", "deadline to start the benchmark run") 22 | storePath = flag.String("path", "./data", "store path") 23 | ) 24 | 25 | var totalMsgCount int64 26 | 27 | func main() { 28 | flag.Parse() 29 | var wg sync.WaitGroup 30 | 31 | log.SetFlags(log.Lshortfile | log.LstdFlags | log.Lmicroseconds) 32 | log.SetPrefix("[uq_bench_writer] ") 33 | 34 | msg := make([]byte, *size) 35 | 36 | var err error 37 | var storage store.Storage 38 | // storage, err = store.NewMemStore() 39 | storage, err = store.NewLevelStore(*storePath) 40 | if err != nil { 41 | fmt.Printf("store init error: %s\n", err) 42 | return 43 | } 44 | 45 | var messageQueue queue.MessageQueue 46 | messageQueue, err = queue.NewUnitedQueue(storage, "", 0, nil, "uq") 47 | if err != nil { 48 | fmt.Printf("queue init error: %s\n", err) 49 | return 50 | } 51 | 52 | err = messageQueue.Create(*topic, "") 53 | if err != nil { 54 | if e := err.(*utils.Error); e.ErrorCode != utils.ErrTopicExisted { 55 | fmt.Printf("topic create error: %s\n", err) 56 | return 57 | } 58 | } 59 | 60 | goChan := make(chan int) 61 | rdyChan := make(chan int) 62 | for j := 0; j < runtime.GOMAXPROCS(runtime.NumCPU()); j++ { 63 | log.Printf("runner: %d", j) 64 | wg.Add(1) 65 | go func() { 66 | pubWorker(messageQueue, *runfor, msg, *topic, rdyChan, goChan) 67 | wg.Done() 68 | }() 69 | <-rdyChan 70 | } 71 | 72 | if *deadline != "" { 73 | t, err := time.Parse("2006-01-02 15:04:05", *deadline) 74 | if err != nil { 75 | log.Fatal(err) 76 | } 77 | d := t.Sub(time.Now()) 78 | log.Printf("sleeping until %s (%s)", t, d) 79 | time.Sleep(d) 80 | } 81 | 82 | start := time.Now() 83 | close(goChan) 84 | wg.Wait() 85 | end := time.Now() 86 | duration := end.Sub(start) 87 | tmc := atomic.LoadInt64(&totalMsgCount) 88 | log.Printf("duration: %s - %.03fmb/s - %.03fops/s - %.03fus/op", 89 | duration, 90 | float64(tmc*int64(*size))/duration.Seconds()/1024/1024, 91 | float64(tmc)/duration.Seconds(), 92 | float64(duration/time.Microsecond)/float64(tmc)) 93 | 94 | messageQueue.Close() 95 | } 96 | 97 | func pubWorker(mq queue.MessageQueue, td time.Duration, data []byte, topic string, rdyChan chan int, goChan chan int) { 98 | rdyChan <- 1 99 | <-goChan 100 | var msgCount int64 101 | endTime := time.Now().Add(td) 102 | for { 103 | err := mq.Push(topic, data) 104 | if err != nil { 105 | log.Printf("mq push error: %s\n", err) 106 | } 107 | msgCount++ 108 | if time.Now().After(endTime) { 109 | break 110 | } 111 | } 112 | atomic.AddInt64(&totalMsgCount, msgCount) 113 | } 114 | -------------------------------------------------------------------------------- /test/foo.dat: -------------------------------------------------------------------------------- 1 | bar -------------------------------------------------------------------------------- /test/genHttpJsonReq.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "time" 7 | 8 | "github.com/buaazp/uq/entry" 9 | ) 10 | 11 | func encode(qr *entry.HttpQueueRequest) ([]byte, error) { 12 | jsonData, err := json.Marshal(qr) 13 | if err != nil { 14 | return nil, err 15 | } 16 | fmt.Printf("encode succ: len = %d json: %s\n", len(jsonData), string(jsonData)) 17 | return jsonData, nil 18 | } 19 | 20 | func decode(data []byte) error { 21 | qr := new(entry.HttpQueueRequest) 22 | err := json.Unmarshal(data, qr) 23 | if err != nil { 24 | return err 25 | } 26 | if qr.Recycle != "" { 27 | recycle, err := time.ParseDuration(qr.Recycle) 28 | if err != nil { 29 | return err 30 | } 31 | fmt.Printf("decode succ: %s\n", recycle.String()) 32 | } else { 33 | fmt.Printf("decode succ: %v\n", qr) 34 | } 35 | 36 | return nil 37 | } 38 | 39 | func main() { 40 | qr := new(entry.HttpQueueRequest) 41 | qr.TopicName = "foo" 42 | qr.LineName = "x" 43 | qr.Recycle = "1h10m30s" 44 | 45 | jsonData, err := encode(qr) 46 | if err != nil { 47 | fmt.Printf("encode failed: %s\n", err) 48 | } 49 | 50 | err = decode(jsonData) 51 | if err != nil { 52 | fmt.Printf("decode failed: %s\n", err) 53 | } 54 | return 55 | } 56 | -------------------------------------------------------------------------------- /test/mc_bench.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "os" 8 | "runtime" 9 | "time" 10 | 11 | "github.com/bradfitz/gomemcache/memcache" 12 | "github.com/buaazp/uq/utils" 13 | ) 14 | 15 | var host, method, topicName, lineName string 16 | var port, testCount, concurrency, dataSize, bucket int 17 | 18 | func init() { 19 | flag.StringVar(&host, "h", "127.0.0.1", "hostname") 20 | flag.IntVar(&port, "p", 8808, "port") 21 | flag.IntVar(&concurrency, "c", 10, "concurrency level") 22 | flag.IntVar(&testCount, "n", 10000, "test count") 23 | flag.IntVar(&dataSize, "d", 200, "data size") 24 | flag.StringVar(&method, "m", "push", "test method") 25 | flag.IntVar(&bucket, "b", 50, "bucket size when mpush") 26 | flag.StringVar(&topicName, "t", "StressTestTool", "topic to test") 27 | flag.StringVar(&lineName, "l", "Line", "line to test") 28 | flag.Usage = func() { 29 | fmt.Fprintf(os.Stderr, "Usage: %s -h host -p port -c concurrency -n count -m push -t topic -l line\n", os.Args[0]) 30 | flag.PrintDefaults() 31 | } 32 | } 33 | 34 | func initQueue() { 35 | var mc *memcache.Client 36 | conn := utils.Addrcat(host, port) 37 | mc = memcache.New(conn) 38 | 39 | err := mc.Add(&memcache.Item{Key: topicName, Value: []byte{}}) 40 | if err != nil { 41 | log.Printf("add error: %v", err) 42 | } 43 | 44 | fullLineName := topicName + "/" + lineName 45 | err = mc.Add(&memcache.Item{Key: fullLineName, Value: []byte{}}) 46 | if err != nil { 47 | log.Printf("add error: %v", err) 48 | } 49 | } 50 | 51 | func setTestSingle(ch chan bool, cn, n int) { 52 | var err error 53 | var mc *memcache.Client 54 | conn := utils.Addrcat(host, port) 55 | v := make([]byte, dataSize) 56 | mc = memcache.New(conn) 57 | for i := 0; i < n; i++ { 58 | start := time.Now() 59 | err = mc.Set(&memcache.Item{Key: topicName, Value: v}) 60 | if err != nil { 61 | log.Printf("set error: c%d %v", cn, err) 62 | } else { 63 | end := time.Now() 64 | duration := end.Sub(start).Seconds() 65 | log.Printf("set succ: %s spend: %.3fms", topicName, duration*1000) 66 | } 67 | } 68 | ch <- true 69 | } 70 | 71 | func setTest(c, n int) { 72 | ch := make(chan bool) 73 | singleCount := n / c 74 | for i := 0; i < c; i++ { 75 | go setTestSingle(ch, i, singleCount) 76 | } 77 | for i := 0; i < c; i++ { 78 | select { 79 | case <-ch: 80 | log.Printf("set single succ: %s - c%d", topicName, i) 81 | } 82 | } 83 | } 84 | 85 | func getTestSingle(ch chan bool, cn, n int) { 86 | var mc *memcache.Client 87 | conn := utils.Addrcat(host, port) 88 | key := topicName + "/" + lineName 89 | keys := []string{key, "id"} 90 | mc = memcache.New(conn) 91 | for i := 0; i < n; i++ { 92 | start := time.Now() 93 | items, err := mc.GetMulti(keys) 94 | if err != nil { 95 | log.Printf("get error: c%d %v", cn, err) 96 | } else { 97 | end := time.Now() 98 | duration := end.Sub(start).Seconds() 99 | id := string(items["id"].Value) 100 | log.Printf("get succ: %s spend: %.3fms", id, duration*1000) 101 | } 102 | } 103 | ch <- true 104 | } 105 | 106 | func getTest(c, n int) { 107 | ch := make(chan bool) 108 | singleCount := n / c 109 | for i := 0; i < c; i++ { 110 | go getTestSingle(ch, i, singleCount) 111 | } 112 | for i := 0; i < c; i++ { 113 | select { 114 | case <-ch: 115 | log.Printf("get single succ: %s/%s - c%d", topicName, lineName, i) 116 | } 117 | } 118 | } 119 | 120 | func main() { 121 | runtime.GOMAXPROCS(runtime.NumCPU()) 122 | flag.Parse() 123 | 124 | if method != "push" && method != "pop" { 125 | fmt.Printf("test method not supported!\n") 126 | return 127 | } 128 | 129 | now := time.Now() 130 | year := now.Year() 131 | month := now.Month() 132 | day := now.Day() 133 | hour := now.Hour() 134 | minute := now.Minute() 135 | second := now.Second() 136 | logName := fmt.Sprintf("uq_mc_%s_%d-%d-%d_%d:%d:%d_c%d_n%d.log", method, year, month, day, hour, minute, second, concurrency, testCount) 137 | logfile, err := os.OpenFile(logName, os.O_RDWR|os.O_CREATE, 0666) 138 | if err != nil { 139 | fmt.Printf("%s\r\n", err.Error()) 140 | os.Exit(-1) 141 | } 142 | defer logfile.Close() 143 | 144 | log.SetFlags(log.Lshortfile | log.LstdFlags | log.Lmicroseconds) 145 | log.SetOutput(logfile) 146 | 147 | initQueue() 148 | 149 | start := time.Now() 150 | if method == "push" { 151 | setTest(concurrency, testCount) 152 | } else if method == "pop" { 153 | getTest(concurrency, testCount) 154 | } 155 | end := time.Now() 156 | 157 | duration := end.Sub(start) 158 | dSecond := duration.Seconds() 159 | 160 | fmt.Printf("StressTest Done! ") 161 | fmt.Printf("Spend: %.3fs Speed: %.3f msg/s Throughput: %.3f MB/s", dSecond, float64(testCount)/dSecond, float64(testCount*dataSize)/(1024*1024*dSecond)) 162 | 163 | log.Printf("StressTest Done!") 164 | log.Printf("Spend: %.3fs Speed: %.3f msg/s Throughput: %.3f MB/s", dSecond, float64(testCount)/dSecond, float64(testCount*dataSize)/(1024*1024*dSecond)) 165 | return 166 | } 167 | -------------------------------------------------------------------------------- /test/mc_bench_sc.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "os" 8 | "runtime" 9 | "time" 10 | 11 | "github.com/bradfitz/gomemcache/memcache" 12 | "github.com/buaazp/uq/utils" 13 | ) 14 | 15 | var host, method, topicName, lineName string 16 | var port, testCount, concurrency, dataSize, bucket int 17 | 18 | func init() { 19 | flag.StringVar(&host, "h", "127.0.0.1", "hostname") 20 | flag.IntVar(&port, "p", 8808, "port") 21 | flag.IntVar(&concurrency, "c", 10, "goroutine level") 22 | flag.IntVar(&testCount, "n", 10000, "test count") 23 | flag.IntVar(&dataSize, "d", 200, "data size") 24 | flag.StringVar(&method, "m", "push", "test method") 25 | flag.IntVar(&bucket, "b", 50, "bucket size when mpush") 26 | flag.StringVar(&topicName, "t", "StressTestTool", "topic to test") 27 | flag.StringVar(&lineName, "l", "Line", "line to test") 28 | flag.Usage = func() { 29 | fmt.Fprintf(os.Stderr, "Usage: %s -h host -p port -c concurrency -n count -m push -t topic -l line\n", os.Args[0]) 30 | flag.PrintDefaults() 31 | } 32 | } 33 | 34 | func initQueue() { 35 | var mc *memcache.Client 36 | conn := utils.Addrcat(host, port) 37 | mc = memcache.New(conn) 38 | 39 | err := mc.Add(&memcache.Item{Key: topicName, Value: []byte{}}) 40 | if err != nil { 41 | log.Printf("add error: %v", err) 42 | } 43 | 44 | fullLineName := topicName + "/" + lineName 45 | err = mc.Add(&memcache.Item{Key: fullLineName, Value: []byte{}}) 46 | if err != nil { 47 | log.Printf("add error: %v", err) 48 | } 49 | } 50 | 51 | func setTestSingle(ch chan bool, mc *memcache.Client, cn, n int) { 52 | var err error 53 | v := make([]byte, dataSize) 54 | for i := 0; i < n; i++ { 55 | start := time.Now() 56 | err = mc.Set(&memcache.Item{Key: topicName, Value: v}) 57 | if err != nil { 58 | log.Printf("set error: c%d %v", cn, err) 59 | } else { 60 | end := time.Now() 61 | duration := end.Sub(start).Seconds() 62 | log.Printf("set succ: %s spend: %.3fms", topicName, duration*1000) 63 | } 64 | } 65 | ch <- true 66 | } 67 | 68 | func setTest(c, n int) { 69 | ch := make(chan bool) 70 | conn := utils.Addrcat(host, port) 71 | mc := memcache.New(conn) 72 | singleCount := n / c 73 | for i := 0; i < c; i++ { 74 | go setTestSingle(ch, mc, i, singleCount) 75 | } 76 | for i := 0; i < c; i++ { 77 | select { 78 | case <-ch: 79 | log.Printf("set single succ: %s - c%d", topicName, i) 80 | } 81 | } 82 | } 83 | 84 | func getTestSingle(ch chan bool, cn, n int) { 85 | var mc *memcache.Client 86 | conn := utils.Addrcat(host, port) 87 | key := topicName + "/" + lineName 88 | keys := []string{key, "id"} 89 | mc = memcache.New(conn) 90 | for i := 0; i < n; i++ { 91 | start := time.Now() 92 | items, err := mc.GetMulti(keys) 93 | if err != nil { 94 | log.Printf("get error: c%d %v", cn, err) 95 | } else { 96 | end := time.Now() 97 | duration := end.Sub(start).Seconds() 98 | id := string(items["id"].Value) 99 | log.Printf("get succ: %s spend: %.3fms", id, duration*1000) 100 | } 101 | } 102 | ch <- true 103 | } 104 | 105 | func getTest(c, n int) { 106 | ch := make(chan bool) 107 | singleCount := n / c 108 | for i := 0; i < c; i++ { 109 | go getTestSingle(ch, i, singleCount) 110 | } 111 | for i := 0; i < c; i++ { 112 | select { 113 | case <-ch: 114 | log.Printf("get single succ: %s/%s - c%d", topicName, lineName, i) 115 | } 116 | } 117 | } 118 | 119 | func main() { 120 | runtime.GOMAXPROCS(runtime.NumCPU()) 121 | flag.Parse() 122 | 123 | if method != "push" && method != "pop" { 124 | fmt.Printf("test method not supported!\n") 125 | return 126 | } 127 | 128 | now := time.Now() 129 | year := now.Year() 130 | month := now.Month() 131 | day := now.Day() 132 | hour := now.Hour() 133 | minute := now.Minute() 134 | second := now.Second() 135 | logName := fmt.Sprintf("uq_mc_%s_%d-%d-%d_%d:%d:%d_c%d_n%d.log", method, year, month, day, hour, minute, second, concurrency, testCount) 136 | logfile, err := os.OpenFile(logName, os.O_RDWR|os.O_CREATE, 0666) 137 | if err != nil { 138 | fmt.Printf("%s\r\n", err.Error()) 139 | os.Exit(-1) 140 | } 141 | defer logfile.Close() 142 | 143 | log.SetFlags(log.Lshortfile | log.LstdFlags | log.Lmicroseconds) 144 | log.SetOutput(logfile) 145 | 146 | initQueue() 147 | 148 | start := time.Now() 149 | if method == "push" { 150 | setTest(concurrency, testCount) 151 | } else if method == "pop" { 152 | getTest(concurrency, testCount) 153 | } 154 | end := time.Now() 155 | 156 | duration := end.Sub(start) 157 | dSecond := duration.Seconds() 158 | 159 | fmt.Printf("StressTest Done! ") 160 | fmt.Printf("Spend: %.3fs Speed: %.3f msg/s Throughput: %.3f MB/s", dSecond, float64(testCount)/dSecond, float64(testCount*dataSize)/(1024*1024*dSecond)) 161 | 162 | log.Printf("StressTest Done!") 163 | log.Printf("Spend: %.3fs Speed: %.3f msg/s Throughput: %.3f MB/s", dSecond, float64(testCount)/dSecond, float64(testCount*dataSize)/(1024*1024*dSecond)) 164 | return 165 | } 166 | -------------------------------------------------------------------------------- /test/post.lua: -------------------------------------------------------------------------------- 1 | -- example HTTP POST script which demonstrates setting the 2 | -- HTTP method, body, and adding a header 3 | 4 | wrk.method = "POST" 5 | wrk.body = "value=barbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbar!barbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbarbar!" 6 | wrk.headers["Content-Type"] = "application/x-www-form-urlencoded" 7 | -------------------------------------------------------------------------------- /test/redis_bench.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "os" 8 | "runtime" 9 | "time" 10 | 11 | "github.com/buaazp/uq/utils" 12 | "github.com/garyburd/redigo/redis" 13 | ) 14 | 15 | var host, method, topicName, lineName string 16 | var port, testCount, concurrency, dataSize, bucket int 17 | 18 | func init() { 19 | flag.StringVar(&host, "h", "127.0.0.1", "hostname") 20 | flag.IntVar(&port, "p", 8808, "port") 21 | flag.IntVar(&concurrency, "c", 10, "concurrency level") 22 | flag.IntVar(&testCount, "n", 10000, "test count") 23 | flag.IntVar(&dataSize, "d", 200, "data size") 24 | flag.StringVar(&method, "m", "push", "test method") 25 | flag.IntVar(&bucket, "b", 50, "bucket size when mpush") 26 | flag.StringVar(&topicName, "t", "StressTestTool", "topic to test") 27 | flag.StringVar(&lineName, "l", "Line", "line to test") 28 | flag.Usage = func() { 29 | fmt.Fprintf(os.Stderr, "Usage: %s -h host -p port -c concurrency -n count -m push -t topic -l line\n", os.Args[0]) 30 | flag.PrintDefaults() 31 | } 32 | } 33 | 34 | func initQueue() error { 35 | addr := utils.Addrcat(host, port) 36 | conn, err := redis.DialTimeout("tcp", addr, 0, 1*time.Second, 1*time.Second) 37 | if err != nil { 38 | log.Printf("redis conn error: %s", err) 39 | return err 40 | } 41 | defer conn.Close() 42 | 43 | _, err = conn.Do("ADD", topicName) 44 | if err != nil { 45 | log.Printf("add error: %v", err) 46 | return err 47 | } 48 | 49 | fullLineName := topicName + "/" + lineName 50 | _, err = conn.Do("ADD", fullLineName) 51 | if err != nil { 52 | log.Printf("add error: %v", err) 53 | return err 54 | } 55 | return nil 56 | } 57 | 58 | func setTestSingle(ch chan bool, cn, n int) error { 59 | var err error 60 | addr := utils.Addrcat(host, port) 61 | conn, err := redis.DialTimeout("tcp", addr, 0, 1*time.Second, 1*time.Second) 62 | if err != nil { 63 | log.Printf("redis conn error: %s", err) 64 | return err 65 | } 66 | defer conn.Close() 67 | v := make([]byte, dataSize) 68 | for i := 0; i < n; i++ { 69 | start := time.Now() 70 | _, err = conn.Do("SET", topicName, v) 71 | if err != nil { 72 | log.Printf("set error: c%d %v", cn, err) 73 | } else { 74 | end := time.Now() 75 | duration := end.Sub(start).Seconds() 76 | log.Printf("set succ: %s spend: %.3fms", topicName, duration*1000) 77 | } 78 | } 79 | ch <- true 80 | return nil 81 | } 82 | 83 | func setTest(c, n int) { 84 | ch := make(chan bool) 85 | singleCount := n / c 86 | for i := 0; i < c; i++ { 87 | go setTestSingle(ch, i, singleCount) 88 | } 89 | for i := 0; i < c; i++ { 90 | select { 91 | case <-ch: 92 | log.Printf("set single succ: %s - c%d", topicName, i) 93 | } 94 | } 95 | } 96 | 97 | func msetTestSingle(ch chan bool, cn, n int) error { 98 | var err error 99 | addr := utils.Addrcat(host, port) 100 | conn, err := redis.DialTimeout("tcp", addr, 0, 1*time.Second, 1*time.Second) 101 | if err != nil { 102 | log.Printf("redis conn error: %s", err) 103 | return err 104 | } 105 | defer conn.Close() 106 | v := make([]byte, dataSize) 107 | b := make([]interface{}, bucket+1) 108 | b[0] = topicName 109 | for i := 1; i < bucket+1; i++ { 110 | b[i] = v 111 | } 112 | count := n / bucket 113 | for i := 0; i < count; i++ { 114 | start := time.Now() 115 | _, err = conn.Do("MSET", b...) 116 | if err != nil { 117 | log.Printf("set error: c%d %v", cn, err) 118 | } else { 119 | end := time.Now() 120 | duration := end.Sub(start).Seconds() 121 | log.Printf("set succ: %s spend: %.3fms", topicName, duration*1000) 122 | } 123 | } 124 | ch <- true 125 | return nil 126 | } 127 | 128 | func msetTest(c, n int) { 129 | ch := make(chan bool) 130 | singleCount := n / c 131 | for i := 0; i < c; i++ { 132 | go msetTestSingle(ch, i, singleCount) 133 | } 134 | for i := 0; i < c; i++ { 135 | select { 136 | case <-ch: 137 | log.Printf("set single succ: %s - c%d", topicName, i) 138 | } 139 | } 140 | } 141 | 142 | func getTestSingle(ch chan bool, cn, n int) error { 143 | addr := utils.Addrcat(host, port) 144 | conn, err := redis.DialTimeout("tcp", addr, 0, 1*time.Second, 1*time.Second) 145 | if err != nil { 146 | log.Printf("redis conn error: %s", err) 147 | return err 148 | } 149 | defer conn.Close() 150 | key := topicName + "/" + lineName 151 | for i := 0; i < n; i++ { 152 | start := time.Now() 153 | reply, err := conn.Do("GET", key) 154 | if err != nil { 155 | log.Printf("get error: c%d %v", cn, err) 156 | } else { 157 | rpl, err := redis.Strings(reply, err) 158 | if err != nil { 159 | fmt.Printf("redis.values error: c%d %v\n", cn, err) 160 | return err 161 | } 162 | // fmt.Printf("redis.strings %v\n", v) 163 | end := time.Now() 164 | duration := end.Sub(start).Seconds() 165 | id := rpl[1] 166 | log.Printf("get succ: %s spend: %.3fms", id, duration*1000) 167 | } 168 | } 169 | ch <- true 170 | return nil 171 | } 172 | 173 | func getTest(c, n int) { 174 | ch := make(chan bool) 175 | singleCount := n / c 176 | for i := 0; i < c; i++ { 177 | go getTestSingle(ch, i, singleCount) 178 | } 179 | for i := 0; i < c; i++ { 180 | select { 181 | case <-ch: 182 | log.Printf("get single succ: %s/%s - c%d", topicName, lineName, i) 183 | } 184 | } 185 | } 186 | 187 | func main() { 188 | runtime.GOMAXPROCS(runtime.NumCPU()) 189 | flag.Parse() 190 | 191 | if method != "mpush" && method != "push" && method != "pop" { 192 | fmt.Printf("test method not supported!\n") 193 | return 194 | } 195 | 196 | now := time.Now() 197 | year := now.Year() 198 | month := now.Month() 199 | day := now.Day() 200 | hour := now.Hour() 201 | minute := now.Minute() 202 | second := now.Second() 203 | logName := fmt.Sprintf("uq_redis_%s_%d-%d-%d_%d:%d:%d_c%d_n%d.log", method, year, month, day, hour, minute, second, concurrency, testCount) 204 | logfile, err := os.OpenFile(logName, os.O_RDWR|os.O_CREATE, 0666) 205 | if err != nil { 206 | fmt.Printf("%s\r\n", err.Error()) 207 | os.Exit(-1) 208 | } 209 | defer logfile.Close() 210 | 211 | log.SetFlags(log.Lshortfile | log.LstdFlags | log.Lmicroseconds) 212 | log.SetOutput(logfile) 213 | 214 | err = initQueue() 215 | if err != nil { 216 | fmt.Printf("init queue error: %s\n", err) 217 | // return 218 | } 219 | 220 | start := time.Now() 221 | if method == "push" { 222 | setTest(concurrency, testCount) 223 | } else if method == "mpush" { 224 | msetTest(concurrency, testCount) 225 | } else if method == "pop" { 226 | getTest(concurrency, testCount) 227 | } 228 | end := time.Now() 229 | 230 | duration := end.Sub(start) 231 | dSecond := duration.Seconds() 232 | 233 | fmt.Printf("StressTest Done! ") 234 | fmt.Printf("Spend: %.3fs Speed: %.3f msg/s Throughput: %.3f MB/s", dSecond, float64(testCount)/dSecond, float64(testCount*dataSize)/(1024*1024*dSecond)) 235 | 236 | log.Printf("StressTest Done!") 237 | log.Printf("Spend: %.3fs Speed: %.3f msg/s Throughput: %.3f MB/s", dSecond, float64(testCount)/dSecond, float64(testCount*dataSize)/(1024*1024*dSecond)) 238 | return 239 | } 240 | -------------------------------------------------------------------------------- /test/test.goconvey: -------------------------------------------------------------------------------- 1 | ignore 2 | -------------------------------------------------------------------------------- /test/testAcati.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "log" 6 | "strconv" 7 | "time" 8 | ) 9 | 10 | // func Acatui(str, b string, n uint64) string { 11 | // ns := strconv.FormatUint(n, 10) 12 | // log.Printf("ns: %v len: %d", ns, len([]byte(ns))) 13 | // log.Printf("ns: %v len: %d", ns, len(ns)) 14 | 15 | // bs := make([]byte, 8) 16 | // // binary.LittleEndian.PutUint64(bs, n) 17 | // binary.BigEndian.PutUint64(bs, n) 18 | // log.Printf("bs: %v", string(bs)) 19 | 20 | // key := make([]byte, 0) 21 | // key = append(key, []byte(str)...) 22 | // key = append(key, []byte(b)...) 23 | // key = append(key, bs...) 24 | 25 | // rst := str + b + ns 26 | // log.Printf("rst: %v", rst) 27 | // log.Printf("key: %v", string(key)) 28 | // return rst 29 | // } 30 | 31 | // Acatui implements Adress cat uint64 32 | func Acatui(str, b string, n uint64) string { 33 | ns := strconv.FormatUint(n, 10) 34 | return str + b + ns 35 | } 36 | 37 | // Fmt returns the fmt version of cat 38 | func Fmt(str, b string, n uint64) string { 39 | return fmt.Sprintf("%s%s%d", str, b, n) 40 | } 41 | 42 | func main() { 43 | n := 1000000 44 | start := time.Now() 45 | for i := 0; i < n; i++ { 46 | _ = Acatui("foo", "/", 999) 47 | } 48 | end := time.Now() 49 | duration := end.Sub(start) 50 | log.Printf("Acatui duration: %v", duration) 51 | 52 | start2 := time.Now() 53 | for i := 0; i < n; i++ { 54 | _ = Fmt("foo", "/", 999) 55 | } 56 | end2 := time.Now() 57 | duration2 := end2.Sub(start2) 58 | log.Printf("Fmt duration: %v", duration2) 59 | } 60 | -------------------------------------------------------------------------------- /test/wrk-post.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | wrk -t 4 -c 4 -d 10s -s post.lua "http://localhost:8808/push/foo" 3 | -------------------------------------------------------------------------------- /uq.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "os" 8 | "os/signal" 9 | "path" 10 | "runtime" 11 | "strings" 12 | "sync" 13 | "syscall" 14 | 15 | "github.com/buaazp/uq/admin" 16 | "github.com/buaazp/uq/entry" 17 | "github.com/buaazp/uq/queue" 18 | "github.com/buaazp/uq/store" 19 | ) 20 | 21 | var ( 22 | ip string 23 | host string 24 | port int 25 | adminPort int 26 | protocol string 27 | db string 28 | dir string 29 | logFile string 30 | etcd string 31 | cluster string 32 | ) 33 | 34 | func init() { 35 | flag.StringVar(&ip, "ip", "127.0.0.1", "self ip/host address") 36 | flag.StringVar(&host, "host", "0.0.0.0", "listen ip") 37 | flag.IntVar(&port, "port", 8808, "listen port") 38 | flag.IntVar(&adminPort, "admin-port", 8809, "admin listen port") 39 | flag.StringVar(&protocol, "protocol", "redis", "frontend interface type [redis/mc/http]") 40 | flag.StringVar(&db, "db", "goleveldb", "backend storage type [goleveldb/memdb]") 41 | flag.StringVar(&dir, "dir", "./data", "backend storage path") 42 | flag.StringVar(&logFile, "log", "", "uq log path") 43 | flag.StringVar(&etcd, "etcd", "", "etcd service location") 44 | flag.StringVar(&cluster, "cluster", "uq", "cluster name in etcd") 45 | } 46 | 47 | func belong(single string, team []string) bool { 48 | for _, one := range team { 49 | if single == one { 50 | return true 51 | } 52 | } 53 | return false 54 | } 55 | 56 | func checkArgs() bool { 57 | if !belong(db, []string{"goleveldb", "memdb"}) { 58 | fmt.Printf("db mode %s is not supported!\n", db) 59 | return false 60 | } 61 | if !belong(protocol, []string{"redis", "mc", "http"}) { 62 | fmt.Printf("protocol %s is not supported!\n", protocol) 63 | return false 64 | } 65 | return true 66 | } 67 | 68 | func main() { 69 | runtime.GOMAXPROCS(runtime.NumCPU()) 70 | defer func() { 71 | fmt.Printf("byebye! uq see u later! 😄\n") 72 | }() 73 | 74 | flag.Parse() 75 | 76 | if !checkArgs() { 77 | return 78 | } 79 | 80 | err := os.MkdirAll(dir, 0755) 81 | if err != nil { 82 | fmt.Printf("mkdir %s error: %s\n", dir, err) 83 | return 84 | } 85 | if logFile != "" { 86 | logFile = path.Join(dir, "uq.log") 87 | logf, err := os.OpenFile(logFile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) 88 | if err != nil { 89 | fmt.Printf("log open error: %s\n", err) 90 | return 91 | } 92 | defer logf.Close() 93 | log.SetOutput(logf) 94 | } 95 | log.SetFlags(log.Lshortfile | log.LstdFlags | log.Lmicroseconds) 96 | log.SetPrefix("[uq] ") 97 | 98 | fmt.Printf("uq started! 😄\n") 99 | 100 | var storage store.Storage 101 | // if db == "rocksdb" { 102 | // dbpath := path.Clean(path.Join(dir, "uq.db")) 103 | // log.Printf("dbpath: %s", dbpath) 104 | // storage, err = store.NewRockStore(dbpath) 105 | // } else if db == "goleveldb" { 106 | if db == "goleveldb" { 107 | dbpath := path.Clean(path.Join(dir, "uq.db")) 108 | log.Printf("dbpath: %s", dbpath) 109 | storage, err = store.NewLevelStore(dbpath) 110 | } else if db == "memdb" { 111 | storage, err = store.NewMemStore() 112 | } else { 113 | fmt.Printf("store %s is not supported!\n", db) 114 | return 115 | } 116 | if err != nil { 117 | fmt.Printf("store init error: %s\n", err) 118 | return 119 | } 120 | 121 | var etcdServers []string 122 | if etcd != "" { 123 | etcdServers = strings.Split(etcd, ",") 124 | for i, etcdServer := range etcdServers { 125 | if !strings.HasPrefix(etcdServer, "http://") { 126 | etcdServers[i] = "http://" + etcdServer 127 | } 128 | } 129 | } 130 | var messageQueue queue.MessageQueue 131 | // messageQueue, err = queue.NewFakeQueue(storage, ip, port, etcdServers, cluster) 132 | // if err != nil { 133 | // fmt.Printf("queue init error: %s\n", err) 134 | // storage.Close() 135 | // return 136 | // } 137 | messageQueue, err = queue.NewUnitedQueue(storage, ip, port, etcdServers, cluster) 138 | if err != nil { 139 | fmt.Printf("queue init error: %s\n", err) 140 | storage.Close() 141 | return 142 | } 143 | 144 | var entrance entry.Entrance 145 | if protocol == "http" { 146 | entrance, err = entry.NewHTTPEntry(host, port, messageQueue) 147 | } else if protocol == "mc" { 148 | entrance, err = entry.NewMcEntry(host, port, messageQueue) 149 | } else if protocol == "redis" { 150 | entrance, err = entry.NewRedisEntry(host, port, messageQueue) 151 | } else { 152 | fmt.Printf("protocol %s is not supported!\n", protocol) 153 | return 154 | } 155 | if err != nil { 156 | fmt.Printf("entry init error: %s\n", err) 157 | messageQueue.Close() 158 | return 159 | } 160 | 161 | stop := make(chan os.Signal) 162 | entryFailed := make(chan bool) 163 | adminFailed := make(chan bool) 164 | signal.Notify(stop, 165 | syscall.SIGHUP, 166 | syscall.SIGINT, 167 | syscall.SIGTERM, 168 | syscall.SIGQUIT) 169 | var wg sync.WaitGroup 170 | 171 | // start entrance server 172 | go func(c chan bool) { 173 | wg.Add(1) 174 | defer wg.Done() 175 | err := entrance.ListenAndServe() 176 | if err != nil { 177 | if !strings.Contains(err.Error(), "stopped") { 178 | fmt.Printf("entry listen error: %s\n", err) 179 | } 180 | close(c) 181 | } 182 | }(entryFailed) 183 | 184 | var adminServer admin.Administrator 185 | adminServer, err = admin.NewUnitedAdmin(host, adminPort, messageQueue) 186 | if err != nil { 187 | fmt.Printf("admin init error: %s\n", err) 188 | entrance.Stop() 189 | return 190 | } 191 | 192 | // start admin server 193 | go func(c chan bool) { 194 | wg.Add(1) 195 | defer wg.Done() 196 | err := adminServer.ListenAndServe() 197 | if err != nil { 198 | if !strings.Contains(err.Error(), "stopped") { 199 | fmt.Printf("entry listen error: %s\n", err) 200 | } 201 | close(c) 202 | } 203 | }(adminFailed) 204 | 205 | select { 206 | case <-stop: 207 | // log.Printf("got signal: %v", signal) 208 | adminServer.Stop() 209 | log.Printf("admin server stoped.") 210 | entrance.Stop() 211 | log.Printf("entrance stoped.") 212 | case <-entryFailed: 213 | messageQueue.Close() 214 | case <-adminFailed: 215 | entrance.Stop() 216 | } 217 | wg.Wait() 218 | } 219 | -------------------------------------------------------------------------------- /uq_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | func TestArgs(t *testing.T) { 10 | Convey("Test UQ CMD Args", t, func() { 11 | So(checkArgs(), ShouldEqual, true) 12 | db = "mysql" 13 | So(checkArgs(), ShouldEqual, false) 14 | db = "memdb" 15 | protocol = "http2" 16 | So(checkArgs(), ShouldEqual, false) 17 | }) 18 | } 19 | -------------------------------------------------------------------------------- /utils/error.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "net/http" 7 | ) 8 | 9 | const ( 10 | // ErrNone is none error 11 | ErrNone = 100 12 | // ErrTopicNotExisted is topic not existed error 13 | ErrTopicNotExisted = 101 14 | // ErrLineNotExisted is line not existed error 15 | ErrLineNotExisted = 102 16 | // ErrNotDelivered is the message not delivered error 17 | ErrNotDelivered = 103 18 | // ErrBadKey is key bad error 19 | ErrBadKey = 104 20 | // ErrTopicExisted is topic has been existed error 21 | ErrTopicExisted = 105 22 | // ErrLineExisted is line has been existed error 23 | ErrLineExisted = 106 24 | // ErrBadRequest is bad request error 25 | ErrBadRequest = 400 26 | // ErrInternalError is internal error 27 | ErrInternalError = 500 28 | ) 29 | 30 | var errorMap = map[int]string{ 31 | // 404 32 | ErrNone: "No Message", 33 | ErrTopicNotExisted: "Topic Not Existed", 34 | ErrLineNotExisted: "Line Not Existed", 35 | ErrNotDelivered: "Message Not Delivered", 36 | 37 | // 400 38 | ErrBadKey: "Bad Key Format", 39 | ErrTopicExisted: "Topic Has Existed", 40 | ErrLineExisted: "Line Has Existed", 41 | ErrBadRequest: "Bad Client Request", 42 | 43 | // 500 44 | ErrInternalError: "Internal Error", 45 | } 46 | 47 | var errorStatus = map[int]int{ 48 | ErrNone: http.StatusNotFound, 49 | ErrTopicNotExisted: http.StatusNotFound, 50 | ErrLineNotExisted: http.StatusNotFound, 51 | ErrNotDelivered: http.StatusNotFound, 52 | ErrInternalError: http.StatusInternalServerError, 53 | } 54 | 55 | // Error is the error type in uq 56 | type Error struct { 57 | ErrorCode int `json:"errorCode"` 58 | Message string `json:"message"` 59 | Cause string `json:"cause,omitempty"` 60 | } 61 | 62 | // NewError returns a uq error 63 | func NewError(errorCode int, cause string) *Error { 64 | return &Error{ 65 | ErrorCode: errorCode, 66 | Message: errorMap[errorCode], 67 | Cause: cause, 68 | } 69 | } 70 | 71 | // Only for error interface 72 | func (e Error) Error() string { 73 | return ItoaQuick(e.ErrorCode) + " " + e.Message + " (" + e.Cause + ")" 74 | } 75 | 76 | func (e Error) statusCode() int { 77 | status, ok := errorStatus[e.ErrorCode] 78 | if !ok { 79 | status = http.StatusBadRequest 80 | } 81 | return status 82 | } 83 | 84 | func (e Error) toJSONString() string { 85 | b, _ := json.Marshal(e) 86 | return string(b) 87 | } 88 | 89 | // WriteTo writes the error to a http response 90 | func (e Error) WriteTo(w http.ResponseWriter) { 91 | w.Header().Set("Content-Type", "application/json") 92 | w.WriteHeader(e.statusCode()) 93 | fmt.Fprintln(w, e.toJSONString()) 94 | } 95 | -------------------------------------------------------------------------------- /utils/error_test.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | func TestError(t *testing.T) { 10 | Convey("Test Error", t, func() { 11 | err := NewError( 12 | ErrInternalError, 13 | `This is a test error`, 14 | ) 15 | So(err.statusCode(), ShouldEqual, 500) 16 | So(err.Error(), ShouldEqual, "500 Internal Error (This is a test error)") 17 | So(err.toJSONString(), ShouldEqual, 18 | `{"errorCode":500,"message":"Internal Error","cause":"This is a test error"}`, 19 | ) 20 | }) 21 | } 22 | -------------------------------------------------------------------------------- /utils/httpUtils.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "net/http" 5 | "strings" 6 | ) 7 | 8 | // AllowMethod verifies that the given method is one of the allowed methods, 9 | // and if not, it writes an error to w. A boolean is returned indicating 10 | // whether or not the method is allowed. 11 | func AllowMethod(w http.ResponseWriter, m string, ms ...string) bool { 12 | for _, meth := range ms { 13 | if m == meth { 14 | return true 15 | } 16 | } 17 | w.Header().Set("Allow", strings.Join(ms, ",")) 18 | http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed) 19 | return false 20 | } 21 | -------------------------------------------------------------------------------- /utils/limitedBufReader.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import "io" 4 | 5 | type limitedBufferReader struct { 6 | r io.Reader 7 | n int 8 | } 9 | 10 | // NewLimitedBufferReader returns a reader that reads from the given reader 11 | // but limits the amount of data returned to at most n bytes. 12 | func NewLimitedBufferReader(r io.Reader, n int) io.Reader { 13 | return &limitedBufferReader{ 14 | r: r, 15 | n: n, 16 | } 17 | } 18 | 19 | func (r *limitedBufferReader) Read(p []byte) (n int, err error) { 20 | np := p 21 | if len(np) > r.n { 22 | np = np[:r.n] 23 | } 24 | return r.r.Read(np) 25 | } 26 | -------------------------------------------------------------------------------- /utils/limitedBufReader_test.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "bytes" 5 | "testing" 6 | 7 | . "github.com/smartystreets/goconvey/convey" 8 | ) 9 | 10 | func TestLimitedBufferReaderRead(t *testing.T) { 11 | Convey("Test LimitedBufferReader", t, func() { 12 | buf := bytes.NewBuffer(make([]byte, 10)) 13 | ln := 1 14 | lr := NewLimitedBufferReader(buf, ln) 15 | n, err := lr.Read(make([]byte, 10)) 16 | So(err, ShouldBeNil) 17 | So(n, ShouldEqual, ln) 18 | }) 19 | } 20 | -------------------------------------------------------------------------------- /utils/stopListener.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "errors" 5 | "net" 6 | "time" 7 | ) 8 | 9 | // StopListener is a stopable listener 10 | type StopListener struct { 11 | *net.TCPListener //Wrapped listener 12 | stop chan int //Channel used only to indicate listener should shutdown 13 | } 14 | 15 | var errStopped = errors.New("Listener stopped") 16 | 17 | // NewStopListener returns a new StopListener with a net listener 18 | func NewStopListener(l net.Listener) (*StopListener, error) { 19 | tcpL, ok := l.(*net.TCPListener) 20 | 21 | if !ok { 22 | return nil, errors.New("Cannot wrap listener") 23 | } 24 | 25 | retval := &StopListener{} 26 | retval.TCPListener = tcpL 27 | retval.stop = make(chan int) 28 | 29 | return retval, nil 30 | } 31 | 32 | // Accept implements the Accept interface 33 | func (sl *StopListener) Accept() (net.Conn, error) { 34 | for { 35 | //Wait up to one second for a new connection 36 | sl.SetDeadline(time.Now().Add(time.Second)) 37 | 38 | newConn, err := sl.TCPListener.Accept() 39 | 40 | //Check for the channel being closed 41 | select { 42 | case <-sl.stop: 43 | return nil, errStopped 44 | default: 45 | //If the channel is still open, continue as normal 46 | } 47 | 48 | if err != nil { 49 | netErr, ok := err.(net.Error) 50 | 51 | //If this is a timeout, then continue to wait for 52 | //new connections 53 | if ok && netErr.Timeout() && netErr.Temporary() { 54 | continue 55 | } 56 | } 57 | 58 | return newConn, err 59 | } 60 | } 61 | 62 | // Stop implements the Stop interface 63 | func (sl *StopListener) Stop() { 64 | close(sl.stop) 65 | } 66 | -------------------------------------------------------------------------------- /utils/strings.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "strconv" 5 | "strings" 6 | ) 7 | 8 | const ( 9 | itoaCount int = 1000 10 | ) 11 | 12 | var itoaNums []string 13 | 14 | func init() { 15 | itoaNums = make([]string, itoaCount) 16 | for i, count := 0, len(itoaNums); i < count; i++ { 17 | itoaNums[i] = strconv.Itoa(i) 18 | } 19 | } 20 | 21 | // ItoaQuick returns the cached itoa string 22 | func ItoaQuick(i int) string { 23 | if i > 0 && i < itoaCount { 24 | return itoaNums[i] 25 | } 26 | return strconv.Itoa(i) 27 | } 28 | 29 | // Acatui implements string cat uint64 30 | func Acatui(str, b string, n uint64) string { 31 | ns := strconv.FormatUint(n, 10) 32 | return str + b + ns 33 | } 34 | 35 | // Acati implements string cat int 36 | func Acati(str, b string, n int) string { 37 | return Acatui(str, b, uint64(n)) 38 | } 39 | 40 | // Addrcat implements net address cat 41 | func Addrcat(host string, port int) string { 42 | return Acati(host, ":", port) 43 | } 44 | 45 | // Atoi implements cached atoi 46 | func Atoi(str string) uint64 { 47 | str = strings.Trim(str, " ") 48 | if len(str) > 0 { 49 | i, err := strconv.ParseUint(str, 10, 0) 50 | if err != nil { 51 | return 0 52 | } 53 | return i 54 | } 55 | return 0 56 | } 57 | -------------------------------------------------------------------------------- /utils/strings_test.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | func TestItoaQuick(t *testing.T) { 10 | Convey("Test ItoaQuick", t, func() { 11 | i1, i2 := 100, 4869 12 | str1 := ItoaQuick(i1) 13 | So(str1, ShouldEqual, "100") 14 | str2 := ItoaQuick(i2) 15 | So(str2, ShouldEqual, "4869") 16 | }) 17 | } 18 | 19 | func TestAcatui(t *testing.T) { 20 | Convey("Test Acatui", t, func() { 21 | key := "foo/x" 22 | var i uint64 = 100 23 | str := Acatui(key, "/", i) 24 | So(str, ShouldEqual, "foo/x/100") 25 | }) 26 | } 27 | 28 | func TestAcati(t *testing.T) { 29 | Convey("Test Acati", t, func() { 30 | key := "foo/x" 31 | var i = 100 32 | str := Acati(key, "/", i) 33 | So(str, ShouldEqual, "foo/x/100") 34 | }) 35 | } 36 | 37 | func TestAddrcat(t *testing.T) { 38 | Convey("Test Addrcat", t, func() { 39 | addr := "localhost" 40 | var port = 9689 41 | str := Addrcat(addr, port) 42 | So(str, ShouldEqual, "localhost:9689") 43 | }) 44 | } 45 | 46 | func TestAtoi(t *testing.T) { 47 | Convey("Test Atoi", t, func() { 48 | str := "31415926" 49 | num := Atoi(str) 50 | So(num, ShouldEqual, 31415926) 51 | str2 := "" 52 | num2 := Atoi(str2) 53 | So(num2, ShouldEqual, 0) 54 | str3 := "abcd" 55 | num3 := Atoi(str3) 56 | So(num3, ShouldEqual, 0) 57 | }) 58 | } 59 | -------------------------------------------------------------------------------- /wercker.yml: -------------------------------------------------------------------------------- 1 | # This references the default golang container from 2 | # the Docker Hub: https://registry.hub.docker.com/u/library/golang/ 3 | # If you want Google's container you would reference google/golang 4 | # Read more about containers on our dev center 5 | # http://devcenter.wercker.com/docs/containers/index.html 6 | box: golang 7 | # This is the build pipeline. Pipelines are the core of wercker 8 | # Read more about pipelines on our dev center 9 | # http://devcenter.wercker.com/docs/pipelines/index.html 10 | 11 | # You can also use services such as databases. Read more on our dev center: 12 | # http://devcenter.wercker.com/docs/services/index.html 13 | # services: 14 | # - postgres 15 | # http://devcenter.wercker.com/docs/services/postgresql.html 16 | 17 | # - mongodb 18 | # http://devcenter.wercker.com/docs/services/mongodb.html 19 | build: 20 | # The steps that will be executed on build 21 | # Steps make up the actions in your pipeline 22 | # Read more about steps on our dev center: 23 | # http://devcenter.wercker.com/docs/steps/index.html 24 | steps: 25 | # Sets the go workspace and places you package 26 | # at the right place in the workspace tree 27 | - setup-go-workspace 28 | 29 | # Gets the dependencies 30 | - script: 31 | name: go get 32 | code: | 33 | go get -v github.com/golang/lint/golint 34 | go get -d -t -v ./... 35 | 36 | # Build the project 37 | - script: 38 | name: go build 39 | code: | 40 | go build -v 41 | 42 | # Test the project 43 | - script: 44 | name: go test 45 | code: | 46 | go test -v ./admin 47 | go test -v ./entry 48 | go test -v ./queue 49 | go test -v ./store 50 | go test -v ./utils 51 | go test -v . 52 | go vet ./... 53 | golint ./... 54 | --------------------------------------------------------------------------------