├── .gitignore ├── LICENSE ├── README.md ├── buffered_socket ├── .gitignore ├── Makefile ├── README.md ├── buffered_socket.c ├── buffered_socket.h └── demo.c ├── conftest.py ├── domain_socket ├── .gitignore ├── Makefile ├── README.md ├── demo.c ├── domain_socket.c └── domain_socket.h ├── host_pool ├── Makefile ├── host_pool.c └── host_pool.h ├── jujufly ├── Makefile ├── README ├── j_arg_d.c ├── j_arg_d.h └── jujufly.c ├── profiler_stats ├── .gitignore ├── Makefile ├── README.md ├── profiler_stats.c ├── profiler_stats.h └── test.c ├── ps_to_file ├── Makefile ├── README.md └── ps_to_file.c ├── ps_to_http ├── .gitignore ├── Makefile ├── README.md └── ps_to_http.c ├── pubsub ├── Makefile ├── README.md ├── http-internal.h └── pubsub.c ├── pubsub_filtered ├── .gitignore ├── Makefile ├── README.md ├── http-internal.h ├── md5.c ├── md5.h ├── pubsub_filtered.c ├── shared.c ├── shared.h └── stream_filter.c ├── pubsubclient ├── Makefile ├── pubsubclient.c ├── pubsubclient.h ├── pubsubclient_pycurl.py ├── stream_request.c └── stream_request.h ├── pysimplehttp ├── .gitignore ├── scripts │ ├── file_to_sq.py │ └── ps_to_sq.py └── src │ ├── BackoffTimer.py │ ├── BaseReader.py │ ├── __init__.py │ ├── file_to_simplequeue.py │ ├── formatters.py │ ├── http.py │ └── pubsub_reader.py ├── qrencode ├── Makefile └── qrencode.c ├── queuereader ├── Makefile ├── queuereader.c └── queuereader.h ├── setup.py ├── shared_tests └── test_shunt.py ├── simpleattributes ├── Makefile └── simpleattributes.c ├── simplegeo ├── Makefile ├── geo.lua ├── simplegeo.c └── test.sh ├── simplehttp ├── Makefile ├── async_simplehttp.c ├── async_simplehttp.h ├── log.c ├── options.c ├── options.h ├── queue.h ├── request.c ├── request.h ├── simplehttp.c ├── simplehttp.h ├── stat.c ├── stat.h ├── testserver.c ├── timer.c ├── utarray.h ├── uthash.h ├── util.c ├── utlist.h └── utstring.h ├── simpleleveldb ├── Makefile ├── README.md ├── csv_to_leveldb.c ├── http-internal.h ├── leveldb_to_csv.c ├── simpleleveldb.c ├── str_list_set.c ├── str_list_set.h └── test_simpleleveldb.py ├── simplememdb ├── Makefile ├── README.md └── simplememdb.c ├── simplequeue ├── Makefile ├── echo.py ├── simplequeue.c └── test_simplequeue.py ├── simpletokyo ├── Makefile ├── README.md ├── simpletokyo.c ├── test.expected └── test.sh └── sortdb ├── Makefile ├── README.md ├── sortdb.c ├── test.expected ├── test.sh ├── test.tab └── test2.tab /.gitignore: -------------------------------------------------------------------------------- 1 | *.o 2 | *.a 3 | *.dSYM 4 | *.pyc 5 | *.deps 6 | build 7 | dist 8 | sortdb/sortdb 9 | simplequeue/simplequeue 10 | simpleleveldb/simpleleveldb 11 | simpleleveldb/leveldb_to_csv 12 | simpleleveldb/csv_to_leveldb 13 | test_output 14 | .sw[op] 15 | *.egg-info/ 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Permission is hereby granted, free of charge, to any person obtaining a copy 2 | of this software and associated documentation files (the "Software"), to deal 3 | in the Software without restriction, including without limitation the rights 4 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 5 | copies of the Software, and to permit persons to whom the Software is 6 | furnished to do so, subject to the following conditions: 7 | 8 | The above copyright notice and this permission notice shall be included in 9 | all copies or substantial portions of the Software. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 13 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 14 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 15 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 16 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 17 | THE SOFTWARE. 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | simplehttp 2 | ========== 3 | 4 | `simplehttp` is a family of libraries and daemons built upon libevent that make high performance HTTP servers 5 | simple and straightforward to write. 6 | 7 | The following libraries and daemons are included: 8 | 9 | * `buffered_socket` - a simple abstraction on bufferevent for arbitrary TCP sockets 10 | * `domain_socket` - an async C library for creating, listening, and communicating over unix domain sockets 11 | * `host_pool` - a library for dealing with endpoint selection, pooling, failure, recovery, and backoff 12 | * `profiler_stats` - a library to track arbitrary profiler timings for average, 95%, 99%, 100% time 13 | * `ps_to_http` - a daemon built on top of pubsubclient to write messages from a source pubsub to destination simplequeue or pubsub server 14 | * `ps_to_file` - a daemon built on top of pubsubclient to write messages from a source pubsub to time rolled output files 15 | * `pubsub` - a daemon that receives data via HTTP POST events and writes to all subscribed long-lived HTTP connections 16 | * `pubsub_filtered` - a pubsub daemon with the ability to filter/obfuscate fields of a JSON message 17 | * `pubsubclient` - a library for writing clients that read from a pubsub 18 | * `pysimplehttp` - a python library for working with pubsub and simplequeue 19 | * `qrencode` 20 | * `queuereader` - a library for writing clients that read from a simplequeue and do work 21 | * `simpleattributes` 22 | * `simplegeo` 23 | * `simplehttp` 24 | * `simpleleveldb` - a HTTP CRUD interface to leveldb 25 | * `simplememdb` - an in-memory version of simpletokyo 26 | * `simplequeue` - an in memory queue with HTTP /put and /get endpoints to push and pop data 27 | * `simpletokyo` - a HTTP CRUD interface to front tokyo cabinet's ttserver 28 | * `sortdb` - sorted database server 29 | 30 | simplehttp Install Instructions 31 | =============================== 32 | 33 | to install any of the simplehttp components you will need to install 34 | [libevent](http://www.monkey.org/~provos/libevent/) 1.4.13+ and the 'simplehttp' module first. 35 | 36 | build the main library 37 | this provides libsimplehttp.a simplehttp/simplehttp.h and simplehttp/queue.h 38 | 39 | cd simplehttp 40 | make && make install 41 | 42 | now install whichever module you would like 43 | this will compile 'simplequeue' and place it in /usr/local/bin 44 | 45 | cd simplequeue 46 | make && make install 47 | 48 | Some modules have additional dependencies: 49 | 50 | * [json-c](http://oss.metaparadigm.com/json-c/) 51 | * [leveldb](http://code.google.com/p/leveldb/) 52 | * [tokyocabinet](http://fallabs.com/tokyocabinet/) / [tokyotyrant](http://fallabs.com/tokyotyrant/) 53 | * [qrencode](http://fukuchi.org/works/qrencode/index.en.html) 54 | * [pcre](http://www.pcre.org/) 55 | 56 | pysimplehttp Install Instructions 57 | ================================= 58 | 59 | pip install pysimplehttp 60 | 61 | provides `file_to_sq.py` and `ps_to_sq.py`. It will use ujson if available. 62 | -------------------------------------------------------------------------------- /buffered_socket/.gitignore: -------------------------------------------------------------------------------- 1 | demo 2 | -------------------------------------------------------------------------------- /buffered_socket/Makefile: -------------------------------------------------------------------------------- 1 | TARGET ?= /usr/local 2 | LIBEVENT ?= /usr/local 3 | LIBSIMPLEHTTP ?= /usr/local 4 | 5 | CFLAGS += -I. -I$(LIBSIMPLEHTTP)/include -I.. -I$(LIBEVENT)/include -g -Wall -O2 6 | LIBS = -L. -L$(LIBEVENT)/lib -L/usr/local/lib -levent -lbuffered_socket 7 | AR = ar 8 | AR_FLAGS = rc 9 | RANLIB = ranlib 10 | 11 | all: libbuffered_socket.a demo 12 | 13 | libbuffered_socket.a: buffered_socket.o 14 | /bin/rm -f $@ 15 | $(AR) $(AR_FLAGS) $@ $^ 16 | $(RANLIB) $@ 17 | 18 | demo: demo.c 19 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 20 | 21 | install: 22 | /usr/bin/install -d $(TARGET)/lib/ 23 | /usr/bin/install -d $(TARGET)/bin/ 24 | /usr/bin/install -d $(TARGET)/include/buffered_socket 25 | /usr/bin/install libbuffered_socket.a $(TARGET)/lib/ 26 | /usr/bin/install buffered_socket.h $(TARGET)/include/buffered_socket 27 | 28 | clean: 29 | /bin/rm -f *.a *.o demo 30 | -------------------------------------------------------------------------------- /buffered_socket/README.md: -------------------------------------------------------------------------------- 1 | buffered_socket 2 | =============== 3 | 4 | a simple abstraction on bufferevent for arbitrary TCP sockets 5 | 6 | see `demo.c` for an example 7 | 8 | dependencies 9 | ============ 10 | 11 | * [libevent](http://www.monkey.org/~provos/libevent/) 12 | -------------------------------------------------------------------------------- /buffered_socket/buffered_socket.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include "buffered_socket.h" 10 | 11 | #ifdef DEBUG 12 | #define _DEBUG(...) fprintf(stdout, __VA_ARGS__) 13 | #else 14 | #define _DEBUG(...) do {;} while (0) 15 | #endif 16 | 17 | static void buffered_socket_readcb(struct bufferevent *bev, void *arg); 18 | static void buffered_socket_writecb(struct bufferevent *bev, void *arg); 19 | static void buffered_socket_errorcb(struct bufferevent *bev, short what, void *arg); 20 | static void buffered_socket_connectcb(int fd, short what, void *arg); 21 | 22 | struct BufferedSocket *new_buffered_socket(const char *address, int port, 23 | void (*connect_callback)(struct BufferedSocket *buffsock, void *arg), 24 | void (*close_callback)(struct BufferedSocket *buffsock, void *arg), 25 | void (*read_callback)(struct BufferedSocket *buffsock, struct evbuffer *evb, void *arg), 26 | void (*write_callback)(struct BufferedSocket *buffsock, void *arg), 27 | void (*error_callback)(struct BufferedSocket *buffsock, void *arg), 28 | void *cbarg) 29 | { 30 | struct BufferedSocket *buffsock; 31 | 32 | buffsock = malloc(sizeof(struct BufferedSocket)); 33 | buffsock->address = strdup(address); 34 | buffsock->port = port; 35 | buffsock->bev = NULL; 36 | buffsock->fd = -1; 37 | buffsock->state = BS_INIT; 38 | buffsock->connect_callback = connect_callback; 39 | buffsock->close_callback = close_callback; 40 | buffsock->read_callback = read_callback; 41 | buffsock->write_callback = write_callback; 42 | buffsock->error_callback = error_callback; 43 | buffsock->cbarg = cbarg; 44 | 45 | return buffsock; 46 | } 47 | 48 | void free_buffered_socket(struct BufferedSocket *buffsock) 49 | { 50 | if (buffsock) { 51 | buffered_socket_close(buffsock); 52 | free(buffsock->address); 53 | free(buffsock); 54 | } 55 | } 56 | 57 | int buffered_socket_connect(struct BufferedSocket *buffsock) 58 | { 59 | struct addrinfo ai, *aitop; 60 | char strport[32]; 61 | struct sockaddr *sa; 62 | int slen; 63 | 64 | if ((buffsock->state == BS_CONNECTED) || (buffsock->state == BS_CONNECTING)) { 65 | return 0; 66 | } 67 | 68 | memset(&ai, 0, sizeof(struct addrinfo)); 69 | ai.ai_family = AF_INET; 70 | ai.ai_socktype = SOCK_STREAM; 71 | snprintf(strport, sizeof(strport), "%d", buffsock->port); 72 | if (getaddrinfo(buffsock->address, strport, &ai, &aitop) != 0) { 73 | _DEBUG("%s: getaddrinfo() failed\n", __FUNCTION__); 74 | return -1; 75 | } 76 | sa = aitop->ai_addr; 77 | slen = aitop->ai_addrlen; 78 | 79 | if ((buffsock->fd = socket(AF_INET, SOCK_STREAM, 0)) == -1) { 80 | _DEBUG("%s: socket() failed\n", __FUNCTION__); 81 | return -1; 82 | } 83 | 84 | if (evutil_make_socket_nonblocking(buffsock->fd) == -1) { 85 | close(buffsock->fd); 86 | _DEBUG("%s: evutil_make_socket_nonblocking() failed\n"); 87 | return -1; 88 | } 89 | 90 | if (connect(buffsock->fd, sa, slen) == -1) { 91 | if (errno != EINPROGRESS) { 92 | close(buffsock->fd); 93 | _DEBUG("%s: connect() failed\n"); 94 | return -1; 95 | } 96 | } 97 | 98 | freeaddrinfo(aitop); 99 | 100 | struct timeval tv = { 2, 0 }; 101 | event_set(&buffsock->conn_ev, buffsock->fd, EV_WRITE, buffered_socket_connectcb, buffsock); 102 | event_add(&buffsock->conn_ev, &tv); 103 | 104 | buffsock->state = BS_CONNECTING; 105 | 106 | return buffsock->fd; 107 | } 108 | 109 | static void buffered_socket_connectcb(int fd, short what, void *arg) 110 | { 111 | struct BufferedSocket *buffsock = (struct BufferedSocket *)arg; 112 | int error; 113 | socklen_t errsz = sizeof(error); 114 | 115 | if (what == EV_TIMEOUT) { 116 | _DEBUG("%s: connection timeout for \"%s:%d\" on %d\n", 117 | __FUNCTION__, buffsock->address, buffsock->port, buffsock->fd); 118 | buffered_socket_close(buffsock); 119 | return; 120 | } 121 | 122 | if (getsockopt(buffsock->fd, SOL_SOCKET, SO_ERROR, (void *)&error, &errsz) == -1) { 123 | _DEBUG("%s: getsockopt failed for \"%s:%d\" on %d\n", 124 | __FUNCTION__, buffsock->address, buffsock->port, buffsock->fd); 125 | buffered_socket_close(buffsock); 126 | return; 127 | } 128 | 129 | if (error) { 130 | _DEBUG("%s: \"%s\" for \"%s:%d\" on %d\n", 131 | __FUNCTION__, strerror(error), buffsock->address, buffsock->port, buffsock->fd); 132 | buffered_socket_close(buffsock); 133 | return; 134 | } 135 | 136 | _DEBUG("%s: connected to \"%s:%d\" on %d\n", 137 | __FUNCTION__, buffsock->address, buffsock->port, buffsock->fd); 138 | 139 | buffsock->state = BS_CONNECTED; 140 | buffsock->bev = bufferevent_new(buffsock->fd, 141 | buffered_socket_readcb, buffered_socket_writecb, buffered_socket_errorcb, 142 | (void *)buffsock); 143 | bufferevent_enable(buffsock->bev, EV_READ); 144 | 145 | if (buffsock->connect_callback) { 146 | (*buffsock->connect_callback)(buffsock, buffsock->cbarg); 147 | } 148 | } 149 | 150 | void buffered_socket_close(struct BufferedSocket *buffsock) 151 | { 152 | _DEBUG("%s: closing \"%s:%d\" on %d\n", 153 | __FUNCTION__, buffsock->address, buffsock->port, buffsock->fd); 154 | 155 | buffsock->state = BS_DISCONNECTED; 156 | 157 | if (event_initialized(&buffsock->conn_ev)) { 158 | event_del(&buffsock->conn_ev); 159 | } 160 | 161 | if (buffsock->fd != -1) { 162 | if (buffsock->close_callback) { 163 | (*buffsock->close_callback)(buffsock, buffsock->cbarg); 164 | } 165 | close(buffsock->fd); 166 | buffsock->fd = -1; 167 | } 168 | 169 | if (buffsock->bev) { 170 | bufferevent_free(buffsock->bev); 171 | buffsock->bev = NULL; 172 | } 173 | } 174 | 175 | size_t buffered_socket_write(struct BufferedSocket *buffsock, void *data, size_t len) 176 | { 177 | if (buffsock->state != BS_CONNECTED) { 178 | return -1; 179 | } 180 | 181 | _DEBUG("%s: writing %lu bytes starting at %p\n", __FUNCTION__, len, data); 182 | 183 | bufferevent_write(buffsock->bev, data, len); 184 | bufferevent_enable(buffsock->bev, EV_WRITE); 185 | 186 | return len; 187 | } 188 | 189 | void buffered_socket_readcb(struct bufferevent *bev, void *arg) 190 | { 191 | struct BufferedSocket *buffsock = (struct BufferedSocket *)arg; 192 | struct evbuffer *evb; 193 | 194 | _DEBUG("%s: %lu bytes read\n", __FUNCTION__, len); 195 | 196 | // client's responsibility to drain the buffer 197 | evb = EVBUFFER_INPUT(bev); 198 | if (buffsock->read_callback) { 199 | (*buffsock->read_callback)(buffsock, evb, buffsock->cbarg); 200 | } 201 | } 202 | 203 | void buffered_socket_writecb(struct bufferevent *bev, void *arg) 204 | { 205 | struct BufferedSocket *buffsock = (struct BufferedSocket *)arg; 206 | struct evbuffer *evb; 207 | 208 | evb = EVBUFFER_OUTPUT(bev); 209 | if (EVBUFFER_LENGTH(evb) == 0) { 210 | bufferevent_disable(bev, EV_WRITE); 211 | } 212 | 213 | _DEBUG("%s: left to write %lu\n", __FUNCTION__, EVBUFFER_LENGTH(evb)); 214 | 215 | if (buffsock->write_callback) { 216 | (*buffsock->write_callback)(buffsock, buffsock->cbarg); 217 | } 218 | } 219 | 220 | void buffered_socket_errorcb(struct bufferevent *bev, short what, void *arg) 221 | { 222 | struct BufferedSocket *buffsock = (struct BufferedSocket *)arg; 223 | 224 | _DEBUG("%s\n", __FUNCTION__); 225 | 226 | if (buffsock->error_callback) { 227 | (*buffsock->error_callback)(buffsock, buffsock->cbarg); 228 | } 229 | 230 | buffered_socket_close(buffsock); 231 | } 232 | -------------------------------------------------------------------------------- /buffered_socket/buffered_socket.h: -------------------------------------------------------------------------------- 1 | #ifndef __buffered_socket_h 2 | #define __buffered_socket_h 3 | 4 | #define BUFFERED_SOCKET_VERSION "0.1" 5 | 6 | enum BufferedSocketStates { 7 | BS_INIT, 8 | BS_CONNECTING, 9 | BS_CONNECTED, 10 | BS_DISCONNECTED 11 | }; 12 | 13 | struct BufferedSocket { 14 | char *address; 15 | int port; 16 | int fd; 17 | int state; 18 | struct event conn_ev; 19 | struct bufferevent *bev; 20 | void (*connect_callback)(struct BufferedSocket *buffsock, void *arg); 21 | void (*close_callback)(struct BufferedSocket *buffsock, void *arg); 22 | void (*read_callback)(struct BufferedSocket *buffsock, struct evbuffer *evb, void *arg); 23 | void (*write_callback)(struct BufferedSocket *buffsock, void *arg); 24 | void (*error_callback)(struct BufferedSocket *buffsock, void *arg); 25 | void *cbarg; 26 | }; 27 | 28 | struct BufferedSocket *new_buffered_socket(const char *address, int port, 29 | void (*connect_callback)(struct BufferedSocket *buffsock, void *arg), 30 | void (*close_callback)(struct BufferedSocket *buffsock, void *arg), 31 | void (*read_callback)(struct BufferedSocket *buffsock, struct evbuffer *evb, void *arg), 32 | void (*write_callback)(struct BufferedSocket *buffsock, void *arg), 33 | void (*error_callback)(struct BufferedSocket *buffsock, void *arg), 34 | void *cbarg); 35 | void free_buffered_socket(struct BufferedSocket *socket); 36 | int buffered_socket_connect(struct BufferedSocket *buffsock); 37 | void buffered_socket_close(struct BufferedSocket *socket); 38 | size_t buffered_socket_write(struct BufferedSocket *buffsock, void *data, size_t len); 39 | 40 | #endif 41 | -------------------------------------------------------------------------------- /buffered_socket/demo.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include "buffered_socket.h" 8 | 9 | // buffered_socket demo app 10 | // 11 | // $ nc -l 5150 (in another window) 12 | // $ make 13 | // $ ./demo 14 | 15 | static struct BufferedSocket *buffsock; 16 | static struct event reconnect_ev; 17 | static struct event pipe_ev; 18 | 19 | static void connect(int sig, short what, void *arg) 20 | { 21 | buffered_socket_connect(buffsock); 22 | fprintf(stdout, "%s: attempting to connect\n", __FUNCTION__); 23 | } 24 | 25 | static void set_reconnect_timer() 26 | { 27 | struct timeval tv = { 5, 0 }; 28 | 29 | evtimer_del(&reconnect_ev); 30 | evtimer_set(&reconnect_ev, connect, NULL); 31 | evtimer_add(&reconnect_ev, &tv); 32 | } 33 | 34 | static void termination_handler(int signum) 35 | { 36 | event_loopbreak(); 37 | } 38 | 39 | static void ignore_cb(int sig, short what, void *arg) 40 | { 41 | signal_set(&pipe_ev, SIGPIPE, ignore_cb, NULL); 42 | signal_add(&pipe_ev, NULL); 43 | } 44 | 45 | static void connect_cb(struct BufferedSocket *buffsock, void *arg) 46 | { 47 | fprintf(stdout, "%s: connected to %s:%d\n", __FUNCTION__, buffsock->address, buffsock->port); 48 | buffered_socket_write(buffsock, arg, strlen(arg)); 49 | } 50 | 51 | static void close_cb(struct BufferedSocket *buffsock, void *arg) 52 | { 53 | fprintf(stdout, "%s: connection closed to %s:%d\n", __FUNCTION__, buffsock->address, buffsock->port); 54 | set_reconnect_timer(); 55 | } 56 | 57 | static void read_cb(struct BufferedSocket *buffsock, struct evbuffer *evb, void *arg) 58 | { 59 | fprintf(stdout, "%s: read %lu bytes\n", __FUNCTION__, EVBUFFER_LENGTH(evb)); 60 | fwrite(EVBUFFER_DATA(evb), EVBUFFER_LENGTH(evb), 1, stdout); 61 | evbuffer_drain(evb, EVBUFFER_LENGTH(evb)); 62 | } 63 | 64 | static void write_cb(struct BufferedSocket *buffsock, void *arg) 65 | { 66 | // normally dont have to do anything here 67 | } 68 | 69 | static void error_cb(struct BufferedSocket *buffsock, void *arg) 70 | { 71 | // track errors, make reconnect decisions 72 | } 73 | 74 | int main(int argc, char **argv) 75 | { 76 | event_init(); 77 | 78 | buffsock = new_buffered_socket("127.0.0.1", 5150, 79 | connect_cb, close_cb, read_cb, write_cb, error_cb, "hello world\n"); 80 | 81 | signal(SIGINT, termination_handler); 82 | signal(SIGQUIT, termination_handler); 83 | signal(SIGTERM, termination_handler); 84 | 85 | signal_set(&pipe_ev, SIGPIPE, ignore_cb, NULL); 86 | signal_add(&pipe_ev, NULL); 87 | 88 | set_reconnect_timer(); 89 | 90 | event_dispatch(); 91 | 92 | free_buffered_socket(buffsock); 93 | 94 | return 0; 95 | } 96 | -------------------------------------------------------------------------------- /conftest.py: -------------------------------------------------------------------------------- 1 | # needed for py.test to accept the --valgrind option 2 | def pytest_addoption(parser): 3 | parser.addoption("--no-valgrind", action="store_true", help="disable valgrind analysis") 4 | -------------------------------------------------------------------------------- /domain_socket/.gitignore: -------------------------------------------------------------------------------- 1 | demo 2 | -------------------------------------------------------------------------------- /domain_socket/Makefile: -------------------------------------------------------------------------------- 1 | TARGET ?= /usr/local 2 | LIBEVENT ?= /usr/local 3 | 4 | CFLAGS += -I. -I$(LIBEVENT)/include -g -Wall -O2 5 | LIBS = -L. -L$(LIBEVENT)/lib -L/usr/local/lib -levent -ldomain_socket 6 | AR = ar 7 | AR_FLAGS = rc 8 | RANLIB = ranlib 9 | 10 | all: libdomain_socket.a demo 11 | 12 | libdomain_socket.a: domain_socket.o 13 | /bin/rm -f $@ 14 | $(AR) $(AR_FLAGS) $@ $^ 15 | $(RANLIB) $@ 16 | 17 | demo: demo.c 18 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 19 | 20 | install: 21 | /usr/bin/install -d $(TARGET)/lib/ 22 | /usr/bin/install -d $(TARGET)/bin/ 23 | /usr/bin/install -d $(TARGET)/include/domain_socket 24 | /usr/bin/install libdomain_socket.a $(TARGET)/lib/ 25 | /usr/bin/install domain_socket.h $(TARGET)/include/domain_socket 26 | 27 | clean: 28 | /bin/rm -f *.a *.o 29 | -------------------------------------------------------------------------------- /domain_socket/README.md: -------------------------------------------------------------------------------- 1 | domain_socket 2 | ============= 3 | 4 | an async C library for creating, listening, and communicating over unix domain sockets. 5 | 6 | see `demo.c` for an example 7 | 8 | dependencies 9 | ============ 10 | 11 | * [libevent](http://www.monkey.org/~provos/libevent/) 12 | -------------------------------------------------------------------------------- /domain_socket/demo.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include "domain_socket.h" 9 | 10 | /* 11 | domain_socket demo app 12 | 13 | $ python -c 'import socket; \ 14 | s = socket.socket(socket.AF_UNIX); \ 15 | s.connect("/tmp/domain_socket_test"); \ 16 | s.send("hi\n"); \ 17 | o = s.recv(4096); \ 18 | print o' 19 | */ 20 | 21 | static struct DomainSocket *uds; 22 | static struct event pipe_ev; 23 | 24 | static void termination_handler(int signum) 25 | { 26 | event_loopbreak(); 27 | } 28 | 29 | static void ignore_cb(int sig, short what, void *arg) 30 | { 31 | signal_set(&pipe_ev, SIGPIPE, ignore_cb, NULL); 32 | signal_add(&pipe_ev, NULL); 33 | } 34 | 35 | static void uds_on_read(struct DSClient *client) 36 | { 37 | struct bufferevent *bev = client->bev; 38 | struct evbuffer *evb; 39 | char *cmd; 40 | 41 | while ((cmd = evbuffer_readline(bev->input)) != NULL) { 42 | if (strcmp(cmd, "hi") == 0) { 43 | evb = evbuffer_new(); 44 | evbuffer_add_printf(evb, "hello world\n"); 45 | domain_socket_client_write(client, EVBUFFER_DATA(evb), EVBUFFER_LENGTH(evb)); 46 | evbuffer_free(evb); 47 | } 48 | free(cmd); 49 | } 50 | } 51 | 52 | static void uds_on_write(struct DSClient *client) 53 | {} 54 | 55 | static void uds_on_error(struct DSClient *client) 56 | {} 57 | 58 | int main(int argc, char **argv) 59 | { 60 | event_init(); 61 | 62 | if (!(uds = new_domain_socket("/tmp/domain_socket_test", 63 | S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH, 64 | uds_on_read, uds_on_write, uds_on_error, 64))) { 65 | fprintf(stdout, "ERROR: new_domain_socket() failed\n"); 66 | exit(1); 67 | } 68 | 69 | signal(SIGINT, termination_handler); 70 | signal(SIGQUIT, termination_handler); 71 | signal(SIGTERM, termination_handler); 72 | 73 | signal_set(&pipe_ev, SIGPIPE, ignore_cb, NULL); 74 | signal_add(&pipe_ev, NULL); 75 | 76 | event_dispatch(); 77 | 78 | free_domain_socket(uds); 79 | 80 | return 0; 81 | } 82 | -------------------------------------------------------------------------------- /domain_socket/domain_socket.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include "domain_socket.h" 12 | 13 | #ifdef DEBUG 14 | #define _DEBUG(...) fprintf(stdout, __VA_ARGS__) 15 | #else 16 | #define _DEBUG(...) do {;} while (0) 17 | #endif 18 | 19 | static struct DSClient *new_domain_socket_client(struct DomainSocket *uds, 20 | int client_fd, struct sockaddr *sa, socklen_t salen); 21 | static void free_domain_socket_client(struct DSClient *client); 22 | static void accept_socket(int fd, short what, void *arg); 23 | 24 | struct DomainSocket *new_domain_socket(const char *path, int access_mask, 25 | void (*read_callback)(struct DSClient *client), 26 | void (*write_callback)(struct DSClient *client), 27 | void (*error_callback)(struct DSClient *client), 28 | int listen_backlog) 29 | { 30 | struct linger ling = {0, 0}; 31 | struct sockaddr_un addr; 32 | struct stat tstat; 33 | int flags = 1; 34 | int old_umask; 35 | struct DomainSocket *uds; 36 | 37 | assert(path != NULL); 38 | 39 | uds = malloc(sizeof(struct DomainSocket)); 40 | uds->path = strdup(path); 41 | uds->fd = -1; 42 | uds->read_callback = read_callback; 43 | uds->write_callback = write_callback; 44 | uds->error_callback = error_callback; 45 | 46 | if ((uds->fd = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) { 47 | _DEBUG("%s: socket() failed\n", __FUNCTION__); 48 | free_domain_socket(uds); 49 | return NULL; 50 | } 51 | 52 | // clean up a previous socket file if we left it around 53 | if (lstat(path, &tstat) == 0) { 54 | if (S_ISSOCK(tstat.st_mode)) { 55 | unlink(path); 56 | } 57 | } 58 | 59 | // @jayridge doesn't think this does anything here 60 | setsockopt(uds->fd, SOL_SOCKET, SO_REUSEADDR, (void *)&flags, sizeof(flags)); 61 | // @jayridge doesn't think this does anything here 62 | setsockopt(uds->fd, SOL_SOCKET, SO_KEEPALIVE, (void *)&flags, sizeof(flags)); 63 | setsockopt(uds->fd, SOL_SOCKET, SO_LINGER, (void *)&ling, sizeof(ling)); 64 | 65 | // clears nonstandard fields in some impementations that otherwise mess things up 66 | memset(&addr, 0, sizeof(addr)); 67 | 68 | addr.sun_family = AF_UNIX; 69 | strncpy(addr.sun_path, path, sizeof(addr.sun_path) - 1); 70 | assert(strcmp(addr.sun_path, path) == 0); 71 | old_umask = umask(~(access_mask & 0777)); 72 | if (bind(uds->fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) { 73 | _DEBUG("%s: bind() failed\n", __FUNCTION__); 74 | free_domain_socket(uds); 75 | umask(old_umask); 76 | return NULL; 77 | } 78 | umask(old_umask); 79 | 80 | if (listen(uds->fd, listen_backlog) == -1) { 81 | _DEBUG("%s: listen() failed\n", __FUNCTION__); 82 | free_domain_socket(uds); 83 | return NULL; 84 | } 85 | 86 | if (evutil_make_socket_nonblocking(uds->fd) == -1) { 87 | _DEBUG("%s: evutil_make_socket_nonblocking() failed\n", __FUNCTION__); 88 | free_domain_socket(uds); 89 | return NULL; 90 | } 91 | 92 | event_set(&uds->ev, uds->fd, EV_READ | EV_PERSIST, accept_socket, uds); 93 | if (event_add(&uds->ev, NULL) == -1) { 94 | _DEBUG("%s: event_add() failed\n", __FUNCTION__); 95 | free_domain_socket(uds); 96 | return NULL; 97 | } 98 | 99 | return uds; 100 | } 101 | 102 | void free_domain_socket(struct DomainSocket *uds) 103 | { 104 | struct stat tstat; 105 | 106 | if (uds) { 107 | event_del(&uds->ev); 108 | 109 | if (uds->fd != -1) { 110 | close(uds->fd); 111 | } 112 | 113 | if (lstat(uds->path, &tstat) == 0) { 114 | if (S_ISSOCK(tstat.st_mode)) { 115 | unlink(uds->path); 116 | } 117 | } 118 | 119 | free(uds->path); 120 | free(uds); 121 | } 122 | } 123 | 124 | static void accept_socket(int fd, short what, void *arg) 125 | { 126 | struct DomainSocket *uds = (struct DomainSocket *)arg; 127 | struct sockaddr_storage ss; 128 | socklen_t addrlen = sizeof(ss); 129 | int nfd; 130 | 131 | _DEBUG("%s: %d\n", __FUNCTION__, fd); 132 | 133 | if ((nfd = accept(fd, (struct sockaddr *)&ss, &addrlen)) == -1) { 134 | if (errno != EAGAIN && errno != EINTR) { 135 | _DEBUG("%s: bad accept", __FUNCTION__); 136 | } 137 | return; 138 | } 139 | 140 | if (evutil_make_socket_nonblocking(nfd) < 0) { 141 | return; 142 | } 143 | 144 | new_domain_socket_client(uds, nfd, (struct sockaddr *)&ss, addrlen); 145 | } 146 | 147 | // called by libevent when there is data to read. 148 | static void buffered_on_read(struct bufferevent *bev, void *arg) 149 | { 150 | struct DSClient *client = (struct DSClient *)arg; 151 | struct DomainSocket *uds = (struct DomainSocket *)client->uds; 152 | 153 | _DEBUG("%s: %d\n", __FUNCTION__, client->fd); 154 | 155 | if (*uds->read_callback) { 156 | (*uds->read_callback)(client); 157 | } 158 | } 159 | 160 | // called by libevent when the write buffer reaches 0. 161 | static void buffered_on_write(struct bufferevent *bev, void *arg) 162 | { 163 | struct DSClient *client = (struct DSClient *)arg; 164 | struct DomainSocket *uds = (struct DomainSocket *)client->uds; 165 | struct evbuffer *evb; 166 | 167 | _DEBUG("%s: %d\n", __FUNCTION__, client->fd); 168 | 169 | evb = EVBUFFER_OUTPUT(bev); 170 | if (EVBUFFER_LENGTH(evb) == 0) { 171 | bufferevent_disable(bev, EV_WRITE); 172 | } 173 | 174 | if (*uds->write_callback) { 175 | (*uds->write_callback)(client); 176 | } 177 | } 178 | 179 | // called by libevent when there is an error on the underlying socket descriptor. 180 | static void buffered_on_error(struct bufferevent *bev, short what, void *arg) 181 | { 182 | struct DSClient *client = (struct DSClient *)arg; 183 | struct DomainSocket *uds = (struct DomainSocket *)client->uds; 184 | 185 | _DEBUG("%s: client socket error, disconnecting\n", __FUNCTION__); 186 | 187 | if (*uds->error_callback) { 188 | (*uds->error_callback)(client); 189 | } 190 | 191 | free_domain_socket_client(client); 192 | } 193 | 194 | void domain_socket_client_write(struct DSClient *client, void *data, size_t len) 195 | { 196 | bufferevent_write(client->bev, data, len); 197 | bufferevent_enable(client->bev, EV_WRITE); 198 | } 199 | 200 | static struct DSClient *new_domain_socket_client(struct DomainSocket *uds, 201 | int client_fd, struct sockaddr *sa, socklen_t salen) 202 | { 203 | struct DSClient *client; 204 | 205 | client = malloc(sizeof(struct DSClient)); 206 | client->uds = uds; 207 | client->fd = client_fd; 208 | client->bev = bufferevent_new(client_fd, buffered_on_read, buffered_on_write, buffered_on_error, client); 209 | bufferevent_enable(client->bev, EV_READ); 210 | 211 | _DEBUG("%s: %d\n", __FUNCTION__, client->fd); 212 | 213 | return client; 214 | } 215 | 216 | static void free_domain_socket_client(struct DSClient *client) 217 | { 218 | if (client) { 219 | bufferevent_free(client->bev); 220 | if (client->fd != -1) { 221 | close(client->fd); 222 | } 223 | free(client); 224 | } 225 | } 226 | -------------------------------------------------------------------------------- /domain_socket/domain_socket.h: -------------------------------------------------------------------------------- 1 | #ifndef __domain_socket_h 2 | #define __domain_socket_h 3 | 4 | struct DSClient; 5 | 6 | struct DomainSocket { 7 | int fd; 8 | char *path; 9 | void (*read_callback)(struct DSClient *client); 10 | void (*write_callback)(struct DSClient *client); 11 | void (*error_callback)(struct DSClient *client); 12 | struct event ev; 13 | }; 14 | 15 | struct DSClient { 16 | int fd; 17 | struct bufferevent *bev; 18 | struct DomainSocket *uds; 19 | }; 20 | 21 | struct DomainSocket *new_domain_socket(const char *path, int access_mask, 22 | void (*read_callback)(struct DSClient *client), 23 | void (*write_callback)(struct DSClient *client), 24 | void (*error_callback)(struct DSClient *client), 25 | int listen_backlog); 26 | void free_domain_socket(struct DomainSocket *uds); 27 | void domain_socket_client_write(struct DSClient *client, void *data, size_t len); 28 | 29 | #endif 30 | -------------------------------------------------------------------------------- /host_pool/Makefile: -------------------------------------------------------------------------------- 1 | TARGET ?= /usr/local 2 | LIBEVENT ?= /usr/local 3 | LIBSIMPLEHTTP ?= /usr/local 4 | 5 | CFLAGS = -I. -I$(LIBSIMPLEHTTP)/include -I.. -I../simplehttp -I$(LIBEVENT)/include -g -Wall -O2 6 | AR = ar 7 | AR_FLAGS = rc 8 | RANLIB = ranlib 9 | 10 | libhost_pool.a: host_pool.o host_pool.h 11 | /bin/rm -f $@ 12 | $(AR) $(AR_FLAGS) $@ $^ 13 | $(RANLIB) $@ 14 | 15 | all: libhost_pool.a 16 | 17 | install: 18 | /usr/bin/install -d $(TARGET)/lib/ 19 | /usr/bin/install -d $(TARGET)/bin/ 20 | /usr/bin/install -d $(TARGET)/include/host_pool 21 | /usr/bin/install libhost_pool.a $(TARGET)/lib/ 22 | /usr/bin/install host_pool.h $(TARGET)/include/host_pool 23 | 24 | clean: 25 | /bin/rm -f *.a *.o 26 | -------------------------------------------------------------------------------- /host_pool/host_pool.h: -------------------------------------------------------------------------------- 1 | #ifndef __host_pool_h 2 | #define __host_pool_h 3 | 4 | #include 5 | #include 6 | 7 | struct HostPoolEndpoint { 8 | int id; 9 | int alive; 10 | int retry_count; 11 | int retry_delay; 12 | time_t next_retry; 13 | char *address; 14 | int port; 15 | char *path; 16 | UT_hash_handle hh; 17 | }; 18 | 19 | struct HostPool { 20 | int count; 21 | int retry_failed_hosts; 22 | int retry_interval; 23 | time_t max_retry_interval; 24 | int reset_on_all_failed; 25 | struct HostPoolEndpoint *endpoints; 26 | struct HostPoolEndpoint *current_endpoint; 27 | int64_t checkpoint; 28 | }; 29 | 30 | enum HostPoolEndpointSelectionMode { 31 | HOST_POOL_RANDOM, 32 | HOST_POOL_ROUND_ROBIN, 33 | HOST_POOL_SINGLE 34 | }; 35 | 36 | struct HostPool *new_host_pool(int retry_failed_hosts, int retry_interval, 37 | int max_retry_interval, int reset_on_all_failed); 38 | void free_host_pool(struct HostPool *host_pool); 39 | struct HostPoolEndpoint *new_host_pool_endpoint(struct HostPool *host_pool, 40 | const char *address, int port, char *path); 41 | void free_host_pool_endpoint(struct HostPoolEndpoint *host_pool_endpoint); 42 | void host_pool_from_json(struct HostPool *host_pool, json_object *host_pool_endpoint_list); 43 | struct HostPoolEndpoint *host_pool_get_endpoint(struct HostPool *host_pool, 44 | enum HostPoolEndpointSelectionMode mode, int64_t state); 45 | struct HostPoolEndpoint *host_pool_next_endpoint(struct HostPool *host_pool, 46 | enum HostPoolEndpointSelectionMode mode, int64_t state); 47 | void host_pool_mark_success(struct HostPool *host_pool, int id); 48 | void host_pool_mark_failed(struct HostPool *host_pool, int id); 49 | void host_pool_reset(struct HostPool *host_pool); 50 | 51 | #endif 52 | -------------------------------------------------------------------------------- /jujufly/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -Wall -g -O2 8 | LIBS = -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -levent -lsimplehttp -lm -lJudy -lpcre -ltcmalloc 9 | 10 | jujufly: jujufly.c j_arg_d.c 11 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin 15 | /usr/bin/install jujufly $(TARGET)/bin 16 | 17 | clean: 18 | rm -rf *.a *.o jujufly *.dSYM test_output test.db 19 | -------------------------------------------------------------------------------- /jujufly/README: -------------------------------------------------------------------------------- 1 | Jujufly is a searchable moving window for streams 2 | oriented data. 3 | 4 | api 5 | --- 6 | 7 | Search requires at least one "equals" against an indexed 8 | field. 9 | 10 | /search?= 11 | 12 | supported comparators are: 13 | equal: '', this is default 14 | not equal: '!', this is default 15 | less than: '<', url encoded 16 | less than or equal: ',' 17 | great than: '>', url encoded 18 | greater than or equal: '.' 19 | pcre: '//' 20 | 21 | 22 | Printindex prints the values for a given field name and 23 | their counts. 24 | 25 | /printindex?field= 26 | 27 | 28 | Dbstats prints size and record statistics for each 29 | db file. 30 | 31 | /dbstats 32 | -------------------------------------------------------------------------------- /jujufly/j_arg_d.c: -------------------------------------------------------------------------------- 1 | #include "j_arg_d.h" 2 | 3 | void j_arg_d_init(j_arg_d *jad) 4 | { 5 | memset(jad, 0, sizeof(*jad)); 6 | jad->argc = 0; 7 | jad->size = J_ARG_D_STATIC_SIZE; 8 | jad->argv = (char **)jad->static_space; 9 | } 10 | 11 | int j_arg_d_append(j_arg_d *jad, char *arg) 12 | { 13 | int i; 14 | if (jad->argc == jad->size) { 15 | char **tmpv = (char **)calloc(jad->size * 2, sizeof(char *)); 16 | for (i = 0; i < jad->argc; i++) { 17 | tmpv[i] = jad->argv[i]; 18 | } 19 | if (jad->size != J_ARG_D_STATIC_SIZE) { 20 | free(jad->argv); 21 | } 22 | jad->argv = tmpv; 23 | jad->size *= 2; 24 | } 25 | jad->argv[jad->argc++] = arg; 26 | return jad->argc; 27 | } 28 | 29 | void j_arg_d_reset(j_arg_d *jad) 30 | { 31 | jad->argc = 0; 32 | } 33 | 34 | void j_arg_d_print(FILE *out, j_arg_d *jad) 35 | { 36 | int i; 37 | fprintf(out, "j_arg_d: %d items %d space\n", jad->argc, jad->size); 38 | for (i = 0; i < jad->argc; i++) { 39 | fprintf(out, "\t%d: %s\n", i, jad->argv[i]); 40 | } 41 | } 42 | 43 | void j_arg_d_free(j_arg_d *jad) 44 | { 45 | if (jad->size != J_ARG_D_STATIC_SIZE) { 46 | free(jad->argv); 47 | } 48 | } 49 | 50 | -------------------------------------------------------------------------------- /jujufly/j_arg_d.h: -------------------------------------------------------------------------------- 1 | /* 2 | * j_arg_d 3 | * 4 | * Dynamic stack oriented array routines. 5 | * 6 | * Uses a static space to store arguments on the heap. Past J_ARG_D_STATIC_SIZE 7 | * this switches to a power of two allocator using malloc. Redefine 8 | * J_ARG_D_STATIC_SIZE to change the static space for your application. 9 | * 10 | */ 11 | 12 | #ifndef __J_ARG_D_H__ 13 | #define __J_ARG_D_H__ 14 | 15 | #include 16 | #include 17 | #include 18 | 19 | #ifndef J_ARG_D_STATIC_SIZE 20 | #define J_ARG_D_STATIC_SIZE 32 // set to power of 2 bitches 21 | #endif /* J_ARG_D_STATIC_SIZE */ 22 | 23 | typedef struct j_arg_d { 24 | int argc; 25 | int size; 26 | char **argv; 27 | char *static_space[J_ARG_D_STATIC_SIZE]; 28 | } j_arg_d; 29 | 30 | void j_arg_d_init(j_arg_d *jad); 31 | int j_arg_d_append(j_arg_d *jad, char *arg); 32 | void j_arg_d_reset(j_arg_d *jad); 33 | void j_arg_d_print(FILE *out, j_arg_d *jad); 34 | void j_arg_d_free(j_arg_d *jad); 35 | 36 | #endif /* __J_ARG_D_H__ */ 37 | 38 | -------------------------------------------------------------------------------- /profiler_stats/.gitignore: -------------------------------------------------------------------------------- 1 | *.dSYM 2 | test 3 | -------------------------------------------------------------------------------- /profiler_stats/Makefile: -------------------------------------------------------------------------------- 1 | TARGET ?= /usr/local 2 | 3 | CFLAGS = -I. -I.. -g -Wall -O2 4 | AR = ar 5 | AR_FLAGS = rc 6 | RANLIB = ranlib 7 | 8 | LIBS = -L. -lm -lprofiler_stats 9 | 10 | all: libprofiler_stats.a test 11 | 12 | libprofiler_stats.a: profiler_stats.o profiler_stats.h 13 | /bin/rm -f $@ 14 | $(AR) $(AR_FLAGS) $@ $^ 15 | $(RANLIB) $@ 16 | 17 | test: test.c 18 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 19 | 20 | install: 21 | /usr/bin/install -d $(TARGET)/lib/ 22 | /usr/bin/install -d $(TARGET)/bin/ 23 | /usr/bin/install -d $(TARGET)/include/profiler_stats 24 | /usr/bin/install libprofiler_stats.a $(TARGET)/lib/ 25 | /usr/bin/install profiler_stats.h $(TARGET)/include/profiler_stats 26 | 27 | clean: 28 | /bin/rm -f *.a *.o test 29 | -------------------------------------------------------------------------------- /profiler_stats/README.md: -------------------------------------------------------------------------------- 1 | profiler_stats 2 | ============= 3 | 4 | a library to track arbitrary profiler timings for average, 95%, 99%, 100% time 5 | 6 | the library enforces a rolling window for data collection. currently it imposes 7 | a hard-coded limit of 5000 data points but allows the client to configure the 8 | window of time to consider via `profiler_stats_init();` 9 | 10 | compiling 11 | --------- 12 | 13 | there are no additional dependencies other than client applications needing to link against `libm` aka `-lm` 14 | 15 | ``` 16 | cd simplehttp/profiler_stats 17 | make 18 | make install 19 | ``` 20 | 21 | usage 22 | ----- 23 | 24 | some initialization is required and it should be noted that individual stats 25 | allocate their data structures up front. 26 | 27 | the general process is as follows (NOTE: client is responsible for `free()` of 28 | all returned structures as well as a final call to `profiler_stats_free()`): 29 | 30 | ``` 31 | #include 32 | #include 33 | 34 | void my_function() 35 | { 36 | profiler_ts start_ts; 37 | 38 | profiler_get_ts(&start_ts); 39 | 40 | // do some work 41 | 42 | profiler_stats_store("stat_name", start_ts); 43 | } 44 | 45 | void my_stats_output() 46 | { 47 | struct ProfilerStat *pstat; 48 | struct ProfilerReturn *ret; 49 | 50 | // LL_FOREACH() is convenience macro to walk a linked list 51 | // from utlist.h http://uthash.sourceforge.net/ 52 | LL_FOREACH(profiler_stats_get_all(), pstat) { 53 | ret = profiler_get_stats(pstat); 54 | fprintf(stdout, "100%%: %"PRIu64"\n", ret->hundred_percent); 55 | fprintf(stdout, "99%%: %"PRIu64"\n", ret->ninety_nine_percent); 56 | fprintf(stdout, "95%%: %"PRIu64"\n", ret->ninety_five_percent); 57 | fprintf(stdout, "avg: %"PRIu64"\n", ret->average); 58 | fprintf(stdout, "count: %"PRIu64"\n", ret->count); 59 | free(ret); 60 | } 61 | } 62 | 63 | int main(int argc, char **argv) 64 | { 65 | // window in usec 66 | profiler_stats_init(300000000); 67 | profiler_new_stat("stat_name"); 68 | 69 | my_function(); 70 | my_stats_output(); 71 | 72 | free_profiler_stats(); 73 | 74 | return 0; 75 | } 76 | ``` -------------------------------------------------------------------------------- /profiler_stats/profiler_stats.h: -------------------------------------------------------------------------------- 1 | #ifndef __profiler_stats_h 2 | #define __profiler_stats_h 3 | 4 | #include 5 | #include 6 | #include 7 | 8 | #define PROFILER_STATS_VERSION "0.3" 9 | 10 | #if _POSIX_TIMERS > 0 11 | 12 | typedef struct timespec profiler_ts; 13 | 14 | inline void profiler_ts_get(struct timespec *ts); 15 | inline unsigned int profiler_ts_diff(struct timespec start, struct timespec end); 16 | 17 | #else 18 | 19 | typedef struct timeval profiler_ts; 20 | 21 | inline void profiler_ts_get(struct timeval *ts); 22 | inline unsigned int profiler_ts_diff(struct timeval start, struct timeval end); 23 | 24 | #endif 25 | 26 | struct ProfilerReturn { 27 | uint64_t count; 28 | uint64_t average; 29 | uint64_t hundred_percent; 30 | uint64_t ninety_nine_percent; 31 | uint64_t ninety_five_percent; 32 | }; 33 | 34 | struct ProfilerData { 35 | int64_t value; 36 | profiler_ts ts; 37 | }; 38 | 39 | struct ProfilerStat { 40 | char *name; 41 | struct ProfilerData **data; 42 | uint64_t count; 43 | int index; 44 | struct ProfilerStat *next; 45 | }; 46 | 47 | void profiler_stats_init(int window_usec); 48 | struct ProfilerStat *profiler_new_stat(const char *name); 49 | void free_profiler_stats(); 50 | void profiler_stats_reset(); 51 | inline void profiler_stats_store(const char *name, profiler_ts start_ts); 52 | inline void profiler_stats_store_for_name(const char *name, uint64_t val, profiler_ts ts); 53 | inline void profiler_stats_store_value(struct ProfilerStat *pstat, uint64_t val, profiler_ts ts); 54 | struct ProfilerReturn *profiler_get_stats_for_name(const char *name); 55 | struct ProfilerReturn *profiler_get_stats(struct ProfilerStat *pstat); 56 | struct ProfilerStat *profiler_stats_get_all(); 57 | 58 | #endif 59 | -------------------------------------------------------------------------------- /profiler_stats/test.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include "profiler_stats.h" 7 | 8 | int main(int argc, char **argv) 9 | { 10 | int i; 11 | profiler_ts ts; 12 | struct ProfilerStat *pstat; 13 | struct ProfilerReturn *ret; 14 | 15 | profiler_stats_init(300000000); 16 | 17 | pstat = profiler_new_stat("test"); 18 | 19 | for (i = 0; i < 20000; i++) { 20 | profiler_ts_get(&ts); 21 | profiler_stats_store_value(pstat, i, ts); 22 | } 23 | 24 | LL_FOREACH(profiler_stats_get_all(), pstat) { 25 | ret = profiler_get_stats(pstat); 26 | assert(ret->average == 17499); 27 | assert(ret->ninety_five_percent == 19751); 28 | assert(ret->ninety_nine_percent == 19951); 29 | assert(ret->hundred_percent == 19999); 30 | assert(ret->count == 20000); 31 | free(ret); 32 | } 33 | 34 | fprintf(stdout, "ok\n"); 35 | 36 | return 0; 37 | } 38 | -------------------------------------------------------------------------------- /ps_to_file/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= /usr/local 4 | LIBPUBSUBCLIENT ?= /usr/local 5 | 6 | CFLAGS = -I. -I$(LIBSIMPLEHTTP)/include -I$(LIBPUBSUBCLIENT)/include -I.. -I$(LIBEVENT)/include -g -Wall -O2 7 | LIBS = -L. -L$(LIBSIMPLEHTTP)/lib -L$(LIBPUBSUBCLIENT)/lib -L../simplehttp -L../pubsubclient -L$(LIBEVENT)/lib -levent -lpubsubclient -lsimplehttp -lm 8 | 9 | all: ps_to_file 10 | 11 | ps_to_file: ps_to_file.c 12 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 13 | 14 | install: 15 | /usr/bin/install -d $(TARGET)/bin/ 16 | /usr/bin/install ps_to_file $(TARGET)/bin/ 17 | 18 | clean: 19 | rm -f *.o *.a ps_to_file 20 | -------------------------------------------------------------------------------- /ps_to_file/README.md: -------------------------------------------------------------------------------- 1 | ps_to_file 2 | ====== 3 | 4 | helper application to subscribe to a pubsub and write incoming messages 5 | to time rolled files. 6 | 7 | source pubsub should output non-multipart, chunked data where each 8 | message is newline terminated. 9 | 10 | OPTIONS 11 | ------- 12 | 13 | ``` 14 | --pubsub-url= source pubsub url in the form of 15 | http://domain.com:port/path 16 | --filename-format= output filename format (strftime compatible) 17 | /var/log/pubsub.%%Y-%%m-%%d_%%H.log 18 | --version 19 | ``` -------------------------------------------------------------------------------- /ps_to_file/ps_to_file.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | #ifdef DEBUG 11 | #define _DEBUG(...) fprintf(stdout, __VA_ARGS__) 12 | #else 13 | #define _DEBUG(...) do {;} while (0) 14 | #endif 15 | 16 | #define VERSION "1.3" 17 | 18 | struct output_metadata { 19 | char *filename_format; 20 | char current_filename[255]; 21 | char temp_filename[255]; 22 | FILE *output_file; 23 | }; 24 | 25 | void process_message_cb(char *message, void *cbarg) 26 | { 27 | struct output_metadata *data; 28 | time_t timer; 29 | struct tm *time_struct; 30 | 31 | _DEBUG("process_message_cb()\n"); 32 | 33 | if (message == NULL || strlen(message) < 3) { 34 | return; 35 | } 36 | 37 | data = (struct output_metadata *)cbarg; 38 | 39 | timer = time(NULL); 40 | time_struct = gmtime(&timer); 41 | _DEBUG("strftime format %s\n", data->filename_format); 42 | strftime(data->temp_filename, 255, data->filename_format, time_struct); 43 | _DEBUG("after strftime %s\n", data->temp_filename); 44 | if (strcmp(data->temp_filename, data->current_filename) != 0) { 45 | _DEBUG("rolling file\n"); 46 | // roll file or open file 47 | if (data->output_file) { 48 | _DEBUG("closing file %s\n", data->current_filename); 49 | fclose(data->output_file); 50 | } 51 | _DEBUG("opening file %s\n", data->temp_filename); 52 | strcpy(data->current_filename, data->temp_filename); 53 | data->output_file = fopen(data->current_filename, "ab"); 54 | } 55 | 56 | fprintf(data->output_file, "%s\n", message); 57 | } 58 | 59 | void error_cb(int status_code, void *cb_arg) 60 | { 61 | event_loopbreak(); 62 | } 63 | 64 | int version_cb(int value) 65 | { 66 | fprintf(stdout, "Version: %s\n", VERSION); 67 | return 0; 68 | } 69 | 70 | int main(int argc, char **argv) 71 | { 72 | char *pubsub_url; 73 | char *address; 74 | int port; 75 | char *path; 76 | char *filename_format = NULL; 77 | struct output_metadata *data; 78 | 79 | define_simplehttp_options(); 80 | option_define_bool("version", OPT_OPTIONAL, 0, NULL, version_cb, VERSION); 81 | option_define_str("pubsub_url", OPT_REQUIRED, "http://127.0.0.1:80/sub?multipart=0", &pubsub_url, NULL, "url of pubsub to read from"); 82 | option_define_str("filename_format", OPT_REQUIRED, NULL, &filename_format, NULL, "/var/log/pubsub.%%Y-%%m-%%d_%%H.log"); 83 | 84 | if (!option_parse_command_line(argc, argv)) { 85 | return 1; 86 | } 87 | 88 | data = calloc(1, sizeof(struct output_metadata)); 89 | data->filename_format = filename_format; 90 | data->current_filename[0] = '\0'; 91 | data->temp_filename[0] = '\0'; 92 | data->output_file = NULL; 93 | 94 | if (simplehttp_parse_url(pubsub_url, strlen(pubsub_url), &address, &port, &path)) { 95 | pubsubclient_main(address, port, path, process_message_cb, error_cb, data); 96 | 97 | if (data->output_file) { 98 | fclose(data->output_file); 99 | } 100 | 101 | free(address); 102 | free(path); 103 | } else { 104 | fprintf(stderr, "ERROR: failed to parse pubsub_url\n"); 105 | } 106 | 107 | free(data); 108 | free_options(); 109 | free(pubsub_url); 110 | free(filename_format); 111 | 112 | return 0; 113 | } 114 | -------------------------------------------------------------------------------- /ps_to_http/.gitignore: -------------------------------------------------------------------------------- 1 | *.sw[po] 2 | ps_to_http 3 | -------------------------------------------------------------------------------- /ps_to_http/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= /usr/local 4 | LIBPUBSUBCLIENT ?= /usr/local 5 | 6 | CFLAGS = -I. -I$(LIBSIMPLEHTTP)/include -I$(LIBPUBSUBCLIENT)/include -I.. -I$(LIBEVENT)/include -g -Wall -O2 7 | LIBS = -L. -L$(LIBSIMPLEHTTP)/lib -L$(LIBPUBSUBCLIENT)/lib -L../simplehttp -L../pubsubclient -L$(LIBEVENT)/lib -levent -lpubsubclient -lsimplehttp -lm 8 | 9 | all: ps_to_http 10 | 11 | ps_to_http: ps_to_http.c 12 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 13 | 14 | install: 15 | /usr/bin/install -d $(TARGET)/bin/ 16 | /usr/bin/install ps_to_http $(TARGET)/bin/ 17 | 18 | clean: 19 | rm -rf *.o ps_to_http *.dSYM 20 | -------------------------------------------------------------------------------- /ps_to_http/README.md: -------------------------------------------------------------------------------- 1 | ps_to_http 2 | ========== 3 | 4 | helper application to subscribe to a pubsub and write incoming messages 5 | to simplequeue(s). 6 | 7 | supports multiple destination simplequeues via round robin. 8 | 9 | source pubsub should output non-multipart, chunked data where each 10 | message is newline terminated. 11 | 12 | OPTIONS 13 | ------- 14 | ``` 15 | --destination-get-url= (multiple) url(s) to HTTP GET to 16 | This URL must contain a %s for the message data 17 | for a simplequeue use "http://127.0.0.1:8080/put?data=%s" 18 | --destination-post-url= (multiple) url(s) to HTTP POST to 19 | For a pubsub endpoint use "http://127.0.0.1:8080/pub" 20 | --help list usage 21 | --pubsub-url= (multiple) pubsub url(s) to read from 22 | default: http://127.0.0.1:80/sub?multipart=0 23 | --round-robin write round-robin to destination urls 24 | --max-silence Maximum amount of time (in seconds) between messages from 25 | the source pubsub before quitting. 26 | ``` 27 | -------------------------------------------------------------------------------- /pubsub/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I. -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -O2 -g 8 | LIBS = -L. -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -levent -lsimplehttp -lm -lcrypto 9 | 10 | pubsub: pubsub.c 11 | $(CC) $(CFLAGS) -o $@ $< $(LIBS) 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin 15 | /usr/bin/install pubsub $(TARGET)/bin/ 16 | 17 | clean: 18 | rm -rf *.o pubsub *.dSYM 19 | -------------------------------------------------------------------------------- /pubsub/README.md: -------------------------------------------------------------------------------- 1 | pubsub 2 | ====== 3 | 4 | pubsub (short for publish/subscribe) is a server that brokers new messages 5 | to all connected subscribers at the time a message is received. 6 | http://en.wikipedia.org/wiki/Publish/subscribe 7 | 8 | Commandline Options: 9 | 10 | --address= address to listen on 11 | default: 0.0.0.0 12 | --daemon daemonize process 13 | --enable-logging request logging 14 | --group= run as this group 15 | --help list usage 16 | --port= port to listen on 17 | default: 8080 18 | --root= chdir and run from this directory 19 | --user= run as this user 20 | --version 21 | 22 | API endpoints: 23 | -------------- 24 | 25 | * /pub 26 | parameter: body 27 | 28 | * /sub 29 | request parameter: multipart=(1|0). turns on/off chunked response format (on by default) 30 | long lived connection which will stream back new messages. 31 | 32 | * /stats 33 | request parameter: reset=1 (resets the counters since last reset) 34 | response: Active connections, Total connections, Messages received, Messages sent, Kicked clients. 35 | 36 | * /clients 37 | response: list of remote clients, their connect time, and their current outbound buffer size. 38 | 39 | Nginx Configuration 40 | ------------------- 41 | 42 | Because connections to /sub are long lived, there is a special nginx 43 | configuration needed to proxy connections. Most importantly, don't specify a read timeout 44 | and turn off buffering. Note: this has been tested with Nginx 0.7 series. 45 | 46 | upstream pubsub_upstream { 47 | server 127.0.0.1:8080; 48 | } 49 | 50 | location /path/sub { 51 | # allow 127.0.0.1; 52 | rewrite ^/path/sub$ /sub?multipart=0 break; 53 | proxy_buffering off; 54 | proxy_set_header Host $host; 55 | proxy_set_header X-Real-IP $remote_addr; 56 | proxy_pass http://pubsub_upstream; 57 | proxy_connect_timeout 5; 58 | proxy_send_timeout 5; 59 | proxy_next_upstream off; 60 | charset utf-8; 61 | } 62 | -------------------------------------------------------------------------------- /pubsub/http-internal.h: -------------------------------------------------------------------------------- 1 | /* 2 | NOTE: this is included copyied from libevent-1.4.13 with the addition 3 | of a definition for socklen_t so that we can give statistics on the 4 | client connection outgoing buffer size 5 | */ 6 | 7 | /* 8 | * Copyright 2001 Niels Provos 9 | * All rights reserved. 10 | * 11 | * This header file contains definitions for dealing with HTTP requests 12 | * that are internal to libevent. As user of the library, you should not 13 | * need to know about these. 14 | */ 15 | 16 | #ifndef _HTTP_H_ 17 | #define _HTTP_H_ 18 | 19 | #define HTTP_CONNECT_TIMEOUT 45 20 | #define HTTP_WRITE_TIMEOUT 50 21 | #define HTTP_READ_TIMEOUT 50 22 | 23 | #define HTTP_PREFIX "http://" 24 | #define HTTP_DEFAULTPORT 80 25 | #define socklen_t unsigned int 26 | 27 | enum message_read_status { 28 | ALL_DATA_READ = 1, 29 | MORE_DATA_EXPECTED = 0, 30 | DATA_CORRUPTED = -1, 31 | REQUEST_CANCELED = -2 32 | }; 33 | 34 | enum evhttp_connection_error { 35 | EVCON_HTTP_TIMEOUT, 36 | EVCON_HTTP_EOF, 37 | EVCON_HTTP_INVALID_HEADER 38 | }; 39 | 40 | struct evbuffer; 41 | struct addrinfo; 42 | struct evhttp_request; 43 | 44 | /* A stupid connection object - maybe make this a bufferevent later */ 45 | 46 | enum evhttp_connection_state { 47 | EVCON_DISCONNECTED, /**< not currently connected not trying either*/ 48 | EVCON_CONNECTING, /**< tries to currently connect */ 49 | EVCON_IDLE, /**< connection is established */ 50 | EVCON_READING_FIRSTLINE,/**< reading Request-Line (incoming conn) or 51 | **< Status-Line (outgoing conn) */ 52 | EVCON_READING_HEADERS, /**< reading request/response headers */ 53 | EVCON_READING_BODY, /**< reading request/response body */ 54 | EVCON_READING_TRAILER, /**< reading request/response chunked trailer */ 55 | EVCON_WRITING /**< writing request/response headers/body */ 56 | }; 57 | 58 | struct event_base; 59 | 60 | struct evhttp_connection { 61 | /* we use tailq only if they were created for an http server */ 62 | TAILQ_ENTRY(evhttp_connection) (next); 63 | 64 | int fd; 65 | struct event ev; 66 | struct event close_ev; 67 | struct evbuffer *input_buffer; 68 | struct evbuffer *output_buffer; 69 | 70 | char *bind_address; /* address to use for binding the src */ 71 | u_short bind_port; /* local port for binding the src */ 72 | 73 | char *address; /* address to connect to */ 74 | u_short port; 75 | 76 | int flags; 77 | #define EVHTTP_CON_INCOMING 0x0001 /* only one request on it ever */ 78 | #define EVHTTP_CON_OUTGOING 0x0002 /* multiple requests possible */ 79 | #define EVHTTP_CON_CLOSEDETECT 0x0004 /* detecting if persistent close */ 80 | 81 | int timeout; /* timeout in seconds for events */ 82 | int retry_cnt; /* retry count */ 83 | int retry_max; /* maximum number of retries */ 84 | 85 | enum evhttp_connection_state state; 86 | 87 | /* for server connections, the http server they are connected with */ 88 | struct evhttp *http_server; 89 | 90 | TAILQ_HEAD(evcon_requestq, evhttp_request) requests; 91 | 92 | void (*cb)(struct evhttp_connection *, void *); 93 | void *cb_arg; 94 | 95 | void (*closecb)(struct evhttp_connection *, void *); 96 | void *closecb_arg; 97 | 98 | struct event_base *base; 99 | }; 100 | 101 | struct evhttp_cb { 102 | TAILQ_ENTRY(evhttp_cb) next; 103 | 104 | char *what; 105 | 106 | void (*cb)(struct evhttp_request *req, void *); 107 | void *cbarg; 108 | }; 109 | 110 | /* both the http server as well as the rpc system need to queue connections */ 111 | TAILQ_HEAD(evconq, evhttp_connection); 112 | 113 | /* each bound socket is stored in one of these */ 114 | struct evhttp_bound_socket { 115 | TAILQ_ENTRY(evhttp_bound_socket) (next); 116 | 117 | struct event bind_ev; 118 | }; 119 | 120 | struct evhttp { 121 | TAILQ_HEAD(boundq, evhttp_bound_socket) sockets; 122 | 123 | TAILQ_HEAD(httpcbq, evhttp_cb) callbacks; 124 | struct evconq connections; 125 | 126 | int timeout; 127 | 128 | void (*gencb)(struct evhttp_request *req, void *); 129 | void *gencbarg; 130 | 131 | struct event_base *base; 132 | }; 133 | 134 | /* resets the connection; can be reused for more requests */ 135 | void evhttp_connection_reset(struct evhttp_connection *); 136 | 137 | /* connects if necessary */ 138 | int evhttp_connection_connect(struct evhttp_connection *); 139 | 140 | /* notifies the current request that it failed; resets connection */ 141 | void evhttp_connection_fail(struct evhttp_connection *, 142 | enum evhttp_connection_error error); 143 | 144 | void evhttp_get_request(struct evhttp *, int, struct sockaddr *, socklen_t); 145 | 146 | int evhttp_hostportfile(char *, char **, u_short *, char **); 147 | 148 | int evhttp_parse_firstline(struct evhttp_request *, struct evbuffer*); 149 | int evhttp_parse_headers(struct evhttp_request *, struct evbuffer*); 150 | 151 | void evhttp_start_read(struct evhttp_connection *); 152 | void evhttp_make_header(struct evhttp_connection *, struct evhttp_request *); 153 | 154 | void evhttp_write_buffer(struct evhttp_connection *, 155 | void (*)(struct evhttp_connection *, void *), void *); 156 | 157 | /* response sending HTML the data in the buffer */ 158 | void evhttp_response_code(struct evhttp_request *, int, const char *); 159 | void evhttp_send_page(struct evhttp_request *, struct evbuffer *); 160 | 161 | #endif /* _HTTP_H */ 162 | -------------------------------------------------------------------------------- /pubsub_filtered/.gitignore: -------------------------------------------------------------------------------- 1 | /pubsub_filtered 2 | /stream_filter 3 | /build/ 4 | -------------------------------------------------------------------------------- /pubsub_filtered/Makefile: -------------------------------------------------------------------------------- 1 | TARGET ?= /usr/local 2 | LIBEVENT ?= /usr/local 3 | LIBSIMPLEHTTP ?= /usr/local 4 | BLDDIR = build 5 | 6 | BINARIES = pubsub_filtered stream_filter 7 | 8 | SRCS_pubsub_filtered = pubsub_filtered.c md5.c shared.c 9 | SRCS_stream_filter = stream_filter.c md5.c shared.c 10 | 11 | all: $(BINARIES) 12 | 13 | CFLAGS = -I. -I$(LIBSIMPLEHTTP)/include -I.. -I$(LIBEVENT)/include -g 14 | LDFLAGS_pubsub_filtered = -L. -L$(LIBSIMPLEHTTP)/lib -L../simplehttp -L$(LIBEVENT)/lib -levent -lsimplehttp -ljson -lpcre -lm -lpubsubclient -lcrypto 15 | LDFLAGS_stream_filter = -L. -L$(LIBSIMPLEHTTP)/lib -L../simplehttp -lsimplehttp -ljson 16 | 17 | OBJS_pubsub_filtered := $(patsubst %.c, $(BLDDIR)/%.o, $(SRCS_pubsub_filtered)) 18 | OBJS_stream_filter := $(patsubst %.c, $(BLDDIR)/%.o, $(SRCS_stream_filter)) 19 | OBJS_all := $(OBJS_pubsub_filtered) $(OBJS_stream_filter) 20 | 21 | DEPS := $(patsubst %.o, %.o.deps, $(OBJS_all)) 22 | # sort removes duplicates 23 | -include $(sort $(DEPS)) 24 | 25 | pubsub_filtered: $(OBJS_pubsub_filtered) 26 | stream_filter: $(OBJS_stream_filter) 27 | 28 | $(BINARIES): %: 29 | @echo " LD " $@ 30 | @mkdir -p $(dir $@) 31 | @$(CC) -o $@ $(OBJS_$@) $(LDFLAGS_$@) 32 | 33 | $(BLDDIR)/%.o: %.c 34 | @echo " CC " $@ 35 | @mkdir -p $(dir $@) 36 | @$(CC) -o $@ -c $< $(CFLAGS) -MMD -MF $@.deps 37 | 38 | install: $(patsubst %, $(TARGET)/bin/%, $(BINARIES)) 39 | 40 | $(TARGET)/bin/%: % 41 | @echo " INSTALL " $@ 42 | @/usr/bin/install -d $(dir $@) 43 | @/usr/bin/install -D $< $@ 44 | 45 | clean: 46 | @echo " RM " $(BINARIES) $(OBJS_all) $(DEPS) 47 | @rm -rf $(BINARIES) $(OBJS_all) $(DEPS) 48 | 49 | .PHONY: all install clean 50 | -------------------------------------------------------------------------------- /pubsub_filtered/README.md: -------------------------------------------------------------------------------- 1 | pubsub_filtered 2 | =============== 3 | 4 | pubsub_filtered connects to a remote pubsub server and filters out or 5 | hashes entries before re-publishing as a pubsub stream 6 | 7 | will not filter heartbeat messages but will pass them through to all clients 8 | 9 | if you have a message like '{desc:"ip added", ip:"127.0.0.1"}' to encrypt the ip you would start pubsub_filtered with '-e ip' 10 | 11 | OPTIONS 12 | 13 | ``` 14 | --address= address to listen on 15 | default: 0.0.0.0 16 | --blacklist-fields= comma separated list of fields to remove 17 | --daemon daemonize process 18 | --enable-logging request logging 19 | --encrypted-fields= comma separated list of fields to encrypt 20 | --expected-key= key to expect in messages before echoing to clients 21 | --expected-value= value to expect in --expected-key field in messages before echoing to clients 22 | --expected-value-regex= regular expression matching expected value in --expected-key field before echoing to clients 23 | --group= run as this group 24 | --help list usage 25 | --port= port to listen on 26 | default: 8080 27 | --pubsub-url= url of pubsub to read from 28 | default: http://127.0.0.1/sub?multipart=0 29 | --root= chdir and run from this directory 30 | --user= run as this user 31 | --version 32 | ``` 33 | 34 | API endpoints: 35 | 36 | * /sub?filter_subject=x&filter_pattern=^a 37 | long lived connection which will stream back new messages; one per line. 38 | filter_subject (optional): the filter key, exact match 39 | filter_pattern (optional): the filter pattern, pcre 40 | 41 | * /stats 42 | response: Active connections, Total connections, Messages received, Messages sent, Kicked clients, upstream reconnect. 43 | 44 | * /clients 45 | response: list of remote clients, their connect time, and their current outbound buffer size. 46 | -------------------------------------------------------------------------------- /pubsub_filtered/http-internal.h: -------------------------------------------------------------------------------- 1 | /* 2 | NOTE: this is included copyied from libevent-1.4.13 with the addition 3 | of a definition for socklen_t so that we can give statistics on the 4 | client connection outgoing buffer size 5 | */ 6 | 7 | /* 8 | * Copyright 2001 Niels Provos 9 | * All rights reserved. 10 | * 11 | * This header file contains definitions for dealing with HTTP requests 12 | * that are internal to libevent. As user of the library, you should not 13 | * need to know about these. 14 | */ 15 | 16 | #ifndef _HTTP_H_ 17 | #define _HTTP_H_ 18 | 19 | #define HTTP_CONNECT_TIMEOUT 45 20 | #define HTTP_WRITE_TIMEOUT 50 21 | #define HTTP_READ_TIMEOUT 50 22 | 23 | #define HTTP_PREFIX "http://" 24 | #define HTTP_DEFAULTPORT 80 25 | #define socklen_t unsigned int 26 | 27 | enum message_read_status { 28 | ALL_DATA_READ = 1, 29 | MORE_DATA_EXPECTED = 0, 30 | DATA_CORRUPTED = -1, 31 | REQUEST_CANCELED = -2 32 | }; 33 | 34 | enum evhttp_connection_error { 35 | EVCON_HTTP_TIMEOUT, 36 | EVCON_HTTP_EOF, 37 | EVCON_HTTP_INVALID_HEADER 38 | }; 39 | 40 | struct evbuffer; 41 | struct addrinfo; 42 | struct evhttp_request; 43 | 44 | /* A stupid connection object - maybe make this a bufferevent later */ 45 | 46 | enum evhttp_connection_state { 47 | EVCON_DISCONNECTED, /**< not currently connected not trying either*/ 48 | EVCON_CONNECTING, /**< tries to currently connect */ 49 | EVCON_IDLE, /**< connection is established */ 50 | EVCON_READING_FIRSTLINE,/**< reading Request-Line (incoming conn) or 51 | **< Status-Line (outgoing conn) */ 52 | EVCON_READING_HEADERS, /**< reading request/response headers */ 53 | EVCON_READING_BODY, /**< reading request/response body */ 54 | EVCON_READING_TRAILER, /**< reading request/response chunked trailer */ 55 | EVCON_WRITING /**< writing request/response headers/body */ 56 | }; 57 | 58 | struct event_base; 59 | 60 | struct evhttp_connection { 61 | /* we use tailq only if they were created for an http server */ 62 | TAILQ_ENTRY(evhttp_connection) (next); 63 | 64 | int fd; 65 | struct event ev; 66 | struct event close_ev; 67 | struct evbuffer *input_buffer; 68 | struct evbuffer *output_buffer; 69 | 70 | char *bind_address; /* address to use for binding the src */ 71 | u_short bind_port; /* local port for binding the src */ 72 | 73 | char *address; /* address to connect to */ 74 | u_short port; 75 | 76 | int flags; 77 | #define EVHTTP_CON_INCOMING 0x0001 /* only one request on it ever */ 78 | #define EVHTTP_CON_OUTGOING 0x0002 /* multiple requests possible */ 79 | #define EVHTTP_CON_CLOSEDETECT 0x0004 /* detecting if persistent close */ 80 | 81 | int timeout; /* timeout in seconds for events */ 82 | int retry_cnt; /* retry count */ 83 | int retry_max; /* maximum number of retries */ 84 | 85 | enum evhttp_connection_state state; 86 | 87 | /* for server connections, the http server they are connected with */ 88 | struct evhttp *http_server; 89 | 90 | TAILQ_HEAD(evcon_requestq, evhttp_request) requests; 91 | 92 | void (*cb)(struct evhttp_connection *, void *); 93 | void *cb_arg; 94 | 95 | void (*closecb)(struct evhttp_connection *, void *); 96 | void *closecb_arg; 97 | 98 | struct event_base *base; 99 | }; 100 | 101 | struct evhttp_cb { 102 | TAILQ_ENTRY(evhttp_cb) next; 103 | 104 | char *what; 105 | 106 | void (*cb)(struct evhttp_request *req, void *); 107 | void *cbarg; 108 | }; 109 | 110 | /* both the http server as well as the rpc system need to queue connections */ 111 | TAILQ_HEAD(evconq, evhttp_connection); 112 | 113 | /* each bound socket is stored in one of these */ 114 | struct evhttp_bound_socket { 115 | TAILQ_ENTRY(evhttp_bound_socket) (next); 116 | 117 | struct event bind_ev; 118 | }; 119 | 120 | struct evhttp { 121 | TAILQ_HEAD(boundq, evhttp_bound_socket) sockets; 122 | 123 | TAILQ_HEAD(httpcbq, evhttp_cb) callbacks; 124 | struct evconq connections; 125 | 126 | int timeout; 127 | 128 | void (*gencb)(struct evhttp_request *req, void *); 129 | void *gencbarg; 130 | 131 | struct event_base *base; 132 | }; 133 | 134 | /* resets the connection; can be reused for more requests */ 135 | void evhttp_connection_reset(struct evhttp_connection *); 136 | 137 | /* connects if necessary */ 138 | int evhttp_connection_connect(struct evhttp_connection *); 139 | 140 | /* notifies the current request that it failed; resets connection */ 141 | void evhttp_connection_fail(struct evhttp_connection *, 142 | enum evhttp_connection_error error); 143 | 144 | void evhttp_get_request(struct evhttp *, int, struct sockaddr *, socklen_t); 145 | 146 | int evhttp_hostportfile(char *, char **, u_short *, char **); 147 | 148 | int evhttp_parse_firstline(struct evhttp_request *, struct evbuffer*); 149 | int evhttp_parse_headers(struct evhttp_request *, struct evbuffer*); 150 | 151 | void evhttp_start_read(struct evhttp_connection *); 152 | void evhttp_make_header(struct evhttp_connection *, struct evhttp_request *); 153 | 154 | void evhttp_write_buffer(struct evhttp_connection *, 155 | void (*)(struct evhttp_connection *, void *), void *); 156 | 157 | /* response sending HTML the data in the buffer */ 158 | void evhttp_response_code(struct evhttp_request *, int, const char *); 159 | void evhttp_send_page(struct evhttp_request *, struct evbuffer *); 160 | 161 | #endif /* _HTTP_H */ 162 | -------------------------------------------------------------------------------- /pubsub_filtered/md5.h: -------------------------------------------------------------------------------- 1 | /* See md5.c for explanation and copyright information. */ 2 | 3 | /* 4 | * $FreeBSD: src/contrib/cvs/lib/md5.h,v 1.2 1999/12/11 15:10:02 peter Exp $ 5 | */ 6 | 7 | /* Add prototype support. */ 8 | #ifndef PROTO 9 | #if defined (USE_PROTOTYPES) ? USE_PROTOTYPES : defined (__STDC__) 10 | #define PROTO(ARGS) ARGS 11 | #else 12 | #define PROTO(ARGS) () 13 | #endif 14 | #endif 15 | 16 | #ifndef MD5_H 17 | #define MD5_H 18 | 19 | /* Unlike previous versions of this code, uint32 need not be exactly 20 | 32 bits, merely 32 bits or more. Choosing a data type which is 32 21 | bits instead of 64 is not important; speed is considerably more 22 | important. ANSI guarantees that "unsigned long" will be big enough, 23 | and always using it seems to have few disadvantages. */ 24 | typedef unsigned long cvs_uint32; 25 | 26 | struct cvs_MD5Context { 27 | cvs_uint32 buf[4]; 28 | cvs_uint32 bits[2]; 29 | unsigned char in[64]; 30 | }; 31 | 32 | void cvs_MD5Init PROTO ((struct cvs_MD5Context *context)); 33 | void cvs_MD5Update PROTO ((struct cvs_MD5Context *context, 34 | unsigned char const *buf, unsigned len)); 35 | void cvs_MD5Final PROTO ((unsigned char digest[16], 36 | struct cvs_MD5Context *context)); 37 | void cvs_MD5Transform PROTO ((cvs_uint32 buf[4], const unsigned char in[64])); 38 | 39 | #endif /* !MD5_H */ 40 | -------------------------------------------------------------------------------- /pubsub_filtered/shared.c: -------------------------------------------------------------------------------- 1 | #include "shared.h" 2 | 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | #include "md5.h" 9 | 10 | /* 11 | * Parse a comma-delimited list of strings and put copies of them 12 | * in an array of (char *) with a specified max capacity. 13 | * 14 | * type and log are optional 15 | * 16 | * returns the number of fields present, even if more than could 17 | * be stored in the array, but does not overflow the array 18 | */ 19 | int parse_fields(const char *str, char **field_array, int array_len, const char *type, FILE *log) 20 | { 21 | int cnt; 22 | const char delim[] = ","; 23 | char *tok, *str_ptr, *save_ptr; 24 | 25 | if (!str) { 26 | return; 27 | } 28 | str_ptr = strdup(str); 29 | 30 | cnt = 0; 31 | for (tok = strtok_r(str_ptr, delim, &save_ptr); 32 | tok != NULL; 33 | tok = strtok_r(NULL, delim, &save_ptr) ) { 34 | if (log) { 35 | fprintf(log, "%s field: \"%s\"\n", type ? type : "parsed", tok); 36 | } 37 | if (cnt < array_len) { 38 | field_array[cnt] = strdup(tok); 39 | } 40 | cnt++; 41 | } 42 | 43 | if (cnt > array_len) { 44 | fprintf(stderr, "Exceeded maximum of %d %s fields\n", 45 | array_len, type ? type : "parsed" ); 46 | } 47 | free(str_ptr); 48 | return cnt; 49 | } 50 | 51 | void free_fields(char **field_array, int array_len) 52 | { 53 | int i; 54 | 55 | for (i = 0; i < array_len; i++) { 56 | free(field_array[i]); 57 | } 58 | } 59 | 60 | /* md5 encrypt a string */ 61 | char *md5_hash(const char *string) 62 | { 63 | char *output = calloc(1, 33 * sizeof(char)); 64 | struct cvs_MD5Context context; 65 | unsigned char checksum[16]; 66 | int i; 67 | 68 | cvs_MD5Init (&context); 69 | cvs_MD5Update (&context, string, strlen(string)); 70 | cvs_MD5Final (checksum, &context); 71 | for (i = 0; i < 16; i++) { 72 | sprintf(&output[i * 2], "%02x", (unsigned int) checksum[i]); 73 | } 74 | output[32] = '\0'; 75 | return output; 76 | } 77 | 78 | /* 79 | * Filters a JSON object based on simple key/value 80 | * return non-zero if message should be kept 81 | */ 82 | int filter_message_simple(const char *key, const char *value, struct json_object *msg) 83 | { 84 | const char *raw_string; 85 | json_object *element; 86 | 87 | element = json_object_object_get(msg, key); 88 | if (element == NULL) { 89 | return 0; 90 | } 91 | if (json_object_is_type(element, json_type_null)) { 92 | return 0; 93 | } 94 | raw_string = json_object_get_string(element); 95 | if (raw_string == NULL || !strlen(raw_string) || strcmp(raw_string, value) != 0) { 96 | return 0; 97 | } 98 | return 1; 99 | } 100 | 101 | /* 102 | * delete fields from a json message 103 | */ 104 | void delete_fields(char **fields, int len, struct json_object *msg) 105 | { 106 | const char *field_key; 107 | int i; 108 | 109 | for (i = 0; i < len; i++) { 110 | field_key = fields[i]; 111 | json_object_object_del(msg, field_key); 112 | } 113 | } 114 | 115 | /* 116 | * encrypt fields in a json message 117 | */ 118 | void encrypt_fields(char **fields, int len, struct json_object *msg) 119 | { 120 | const char *field_key, *raw_string; 121 | char *encrypted_string; 122 | json_object *element; 123 | int i; 124 | 125 | for (i = 0; i < len; i++) { 126 | field_key = fields[i]; 127 | element = json_object_object_get(msg, field_key); 128 | if (element) { 129 | raw_string = json_object_get_string(element); 130 | } else { 131 | continue; 132 | } 133 | encrypted_string = md5_hash(raw_string); 134 | json_object_object_add(msg, field_key, json_object_new_string(encrypted_string)); 135 | free(encrypted_string); 136 | } 137 | } 138 | -------------------------------------------------------------------------------- /pubsub_filtered/shared.h: -------------------------------------------------------------------------------- 1 | #ifndef PUBSUB_FILTERED_SHARED_H 2 | #define PUBSUB_FILTERED_SHARED_H 3 | 4 | #include 5 | #include 6 | 7 | int parse_fields(const char *str, char **field_array, int array_len, const char *type, FILE *log); 8 | void free_fields(char **field_array, int array_len); 9 | char *md5_hash(const char *string); 10 | int filter_message_simple(const char *key, const char *value, struct json_object *msg); 11 | void delete_fields(char **fields, int len, struct json_object *msg); 12 | void encrypt_fields(char **fields, int len, struct json_object *msg); 13 | 14 | #endif 15 | -------------------------------------------------------------------------------- /pubsub_filtered/stream_filter.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | #include "shared.h" 9 | 10 | #define VERSION "1.2" 11 | 12 | #define BUF_SZ (1024 * 16) 13 | #define MAX_FIELDS 64 14 | 15 | int parse_encrypted_fields(char *str); 16 | int parse_blacklisted_fields(char *str); 17 | 18 | static char *encrypted_fields[MAX_FIELDS]; 19 | static int num_encrypted_fields = 0; 20 | static char *blacklisted_fields[MAX_FIELDS]; 21 | static int num_blacklisted_fields = 0; 22 | static char *expected_key = NULL; 23 | static char *expected_value = NULL; 24 | static int verbose; 25 | 26 | static uint64_t msgRecv = 0; 27 | static uint64_t msgFail = 0; 28 | 29 | static char buf[BUF_SZ]; 30 | 31 | /* 32 | * populate the blacklisted_fields array 33 | */ 34 | int parse_blacklisted_fields(char *str) 35 | { 36 | num_blacklisted_fields = parse_fields(str, blacklisted_fields, MAX_FIELDS, "Blacklist", stderr); 37 | return (num_blacklisted_fields <= MAX_FIELDS); 38 | } 39 | 40 | /* 41 | * populate the encrypted_fields array with the results. 42 | */ 43 | int parse_encrypted_fields(char *str) 44 | { 45 | num_encrypted_fields = parse_fields(str, encrypted_fields, MAX_FIELDS, "Encrypted", stderr); 46 | return (num_encrypted_fields <= MAX_FIELDS); 47 | } 48 | 49 | /* 50 | * for each message. 51 | */ 52 | void process_message(const char *source) 53 | { 54 | struct json_object *json_in, *element; 55 | const char *field_key, *raw_string, *json_out; 56 | char *encrypted_string; 57 | int i; 58 | 59 | json_in = json_tokener_parse(source); 60 | 61 | if (json_in == NULL) { 62 | msgFail++; 63 | if (verbose) { 64 | fprintf(stderr, "ERR: unable to parse json '%s'\n", source); 65 | } 66 | return; 67 | } 68 | 69 | // filter messages 70 | if (expected_value && !filter_message_simple(expected_key, expected_value, json_in)) { 71 | json_object_put(json_in); 72 | return; 73 | } 74 | 75 | // remove the blacklisted fields 76 | delete_fields(blacklisted_fields, num_blacklisted_fields, json_in); 77 | 78 | // fields we need to encrypt 79 | encrypt_fields(encrypted_fields, num_encrypted_fields, json_in); 80 | 81 | json_out = json_object_to_json_string(json_in); 82 | 83 | /* new line terminated */ 84 | printf("%s\n", json_out); 85 | 86 | json_object_put(json_in); 87 | } 88 | 89 | int version_cb(int value) 90 | { 91 | fprintf(stderr, "Version: %s\n", VERSION); 92 | exit(0); 93 | } 94 | 95 | int main(int argc, char **argv) 96 | { 97 | option_define_bool("version", OPT_OPTIONAL, 0, NULL, version_cb, VERSION); 98 | option_define_bool("verbose", OPT_OPTIONAL, 0, &verbose, NULL, "verbose output (to stderr)"); 99 | option_define_str("blacklist_fields", OPT_OPTIONAL, NULL, NULL, parse_blacklisted_fields, "comma separated list of fields to remove"); 100 | option_define_str("encrypted_fields", OPT_OPTIONAL, NULL, NULL, parse_encrypted_fields, "comma separated list of fields to encrypt"); 101 | option_define_str("expected_key", OPT_OPTIONAL, NULL, &expected_key, NULL, "key to expect in messages before echoing to clients"); 102 | option_define_str("expected_value", OPT_OPTIONAL, NULL, &expected_value, NULL, "value to expect in --expected-key field in messages before echoing to clients"); 103 | 104 | if (!option_parse_command_line(argc, argv)) { 105 | return 1; 106 | } 107 | 108 | if ( !!expected_key ^ !!expected_value ) { 109 | fprintf(stderr, "--expected-key and --expected-value must be used together\n"); 110 | exit(1); 111 | } 112 | 113 | while (fgets(buf, BUF_SZ, stdin)) { 114 | msgRecv++; 115 | process_message(buf); 116 | } 117 | 118 | fprintf(stderr, "processed %lu lines, failed to parse %lu of them\n", msgRecv, msgFail); 119 | free_options(); 120 | free_fields(blacklisted_fields, num_blacklisted_fields); 121 | free_fields(encrypted_fields, num_encrypted_fields); 122 | 123 | return 0; 124 | } 125 | -------------------------------------------------------------------------------- /pubsubclient/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | 4 | CFLAGS = -I. -I$(LIBEVENT)/include -g 5 | AR = ar 6 | AR_FLAGS = rc 7 | RANLIB = ranlib 8 | 9 | libpubsubclient.a: pubsubclient.o stream_request.o 10 | /bin/rm -f $@ 11 | $(AR) $(AR_FLAGS) $@ $^ 12 | $(RANLIB) $@ 13 | 14 | all: libpubsubclient.a 15 | 16 | install: 17 | /usr/bin/install -d $(TARGET)/lib/ 18 | /usr/bin/install -d $(TARGET)/bin/ 19 | /usr/bin/install -d $(TARGET)/include/pubsubclient 20 | /usr/bin/install libpubsubclient.a $(TARGET)/lib/ 21 | /usr/bin/install pubsubclient.h $(TARGET)/include/pubsubclient 22 | 23 | clean: 24 | /bin/rm -f *.a *.o 25 | -------------------------------------------------------------------------------- /pubsubclient/pubsubclient.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include "pubsubclient.h" 8 | #include "stream_request.h" 9 | 10 | #ifdef DEBUG 11 | #define _DEBUG(...) fprintf(stdout, __VA_ARGS__) 12 | #else 13 | #define _DEBUG(...) do {;} while (0) 14 | #endif 15 | 16 | static int chunked = 0; 17 | static struct GlobalData *data = NULL; 18 | static struct StreamRequest *stream_request = NULL; 19 | static struct evhttp_connection *evhttp_source_connection = NULL; 20 | static struct evhttp_request *evhttp_source_request = NULL; 21 | static struct StreamRequest *autodetect_sr = NULL; 22 | extern struct event_base *current_base; 23 | 24 | struct GlobalData { 25 | void (*message_cb)(char *data, void *cbarg); 26 | void (*error_cb)(int status_code, void *cbarg); 27 | void *cbarg; 28 | const char *source_address; 29 | int source_port; 30 | const char *path; 31 | }; 32 | 33 | void pubsubclient_termination_handler(int signum) 34 | { 35 | event_loopbreak(); 36 | } 37 | 38 | int pubsubclient_parse_one_message(struct evbuffer *evb, void *arg) 39 | { 40 | char *line; 41 | size_t line_len; 42 | struct GlobalData *client_data; 43 | char *data; 44 | int has_more = 0; 45 | 46 | _DEBUG("pubsubclient_parse_one_message()\n"); 47 | 48 | client_data = (struct GlobalData *)arg; 49 | line = evbuffer_readline(evb); 50 | if (line && (line_len = strlen(line))) { 51 | _DEBUG("line (%p): %s (%d)\n", line, line, line_len); 52 | (*client_data->message_cb)(line, client_data->cbarg); 53 | if (EVBUFFER_LENGTH(evb)) { 54 | has_more = 1; 55 | } 56 | } 57 | free(line); 58 | return has_more; 59 | } 60 | 61 | void pubsubclient_source_readcb(struct bufferevent *bev, void *arg) 62 | { 63 | while (pubsubclient_parse_one_message(EVBUFFER_INPUT(bev), arg)) {} 64 | } 65 | 66 | void pubsubclient_errorcb(struct bufferevent *bev, void *arg) 67 | { 68 | struct GlobalData *client_data = (struct GlobalData *)arg; 69 | 70 | fprintf(stderr, "ERROR: request failed\n"); 71 | 72 | if (client_data->error_cb) { 73 | (*client_data->error_cb)(-1, client_data->cbarg); 74 | } 75 | } 76 | 77 | void pubsubclient_source_request_done(struct evhttp_request *req, void *arg) 78 | { 79 | struct GlobalData *client_data = (struct GlobalData *)arg; 80 | int status_code = -1; 81 | 82 | _DEBUG("pubsubclient_source_request_done()\n"); 83 | 84 | fprintf(stderr, "ERROR: request failed\n"); 85 | 86 | if (req) { 87 | status_code = req->response_code; 88 | } 89 | 90 | if (client_data->error_cb) { 91 | (*client_data->error_cb)(status_code, client_data->cbarg); 92 | } 93 | } 94 | 95 | void pubsubclient_source_callback(struct evhttp_request *req, void *arg) 96 | { 97 | // keep parsing out messages until there aren't any left 98 | while (pubsubclient_parse_one_message(req->input_buffer, arg)) {} 99 | } 100 | 101 | int pubsubclient_connect() 102 | { 103 | fprintf(stdout, "CONNECTING TO SOURCE http://%s:%d%s\n", data->source_address, data->source_port, data->path); 104 | if (chunked) { 105 | if (evhttp_source_connection) { 106 | evhttp_connection_free(evhttp_source_connection); 107 | } 108 | 109 | // use libevent's built in evhttp methods to parse chunked responses 110 | evhttp_source_connection = evhttp_connection_new(data->source_address, data->source_port); 111 | if (evhttp_source_connection == NULL) { 112 | fprintf(stderr, "ERROR: evhttp_connection_new() failed for %s:%d\n", data->source_address, data->source_port); 113 | return 0; 114 | } 115 | 116 | evhttp_source_request = evhttp_request_new(pubsubclient_source_request_done, data); 117 | evhttp_add_header(evhttp_source_request->output_headers, "Host", data->source_address); 118 | evhttp_request_set_chunked_cb(evhttp_source_request, pubsubclient_source_callback); 119 | 120 | if (evhttp_make_request(evhttp_source_connection, evhttp_source_request, EVHTTP_REQ_GET, data->path) == -1) { 121 | fprintf(stderr, "ERROR: evhttp_make_request() failed for %s\n", data->path); 122 | evhttp_connection_free(evhttp_source_connection); 123 | return 0; 124 | } 125 | } else { 126 | if (stream_request) { 127 | free_stream_request(stream_request); 128 | } 129 | 130 | // use our stream_request library to handle non-chunked 131 | stream_request = new_stream_request("GET", data->source_address, data->source_port, data->path, 132 | NULL, pubsubclient_source_readcb, pubsubclient_errorcb, data); 133 | if (!stream_request) { 134 | fprintf(stderr, "ERROR: new_stream_request() failed for %s:%d%s\n", data->source_address, data->source_port, data->path); 135 | return 0; 136 | } 137 | } 138 | 139 | return 1; 140 | } 141 | 142 | void pubsubclient_autodetect_headercb(struct bufferevent *bev, struct evkeyvalq *headers, void *arg) 143 | { 144 | const char *encoding_header = NULL; 145 | 146 | _DEBUG("pubsubclient_autodetect_headercb() headers: %p\n", headers); 147 | 148 | if ((encoding_header = evhttp_find_header(headers, "Transfer-Encoding")) != NULL) { 149 | if (strncmp(encoding_header, "chunked", 7) == 0) { 150 | chunked = 1; 151 | } 152 | } 153 | 154 | // turn off the events for this buffer 155 | // its free'd later 156 | bufferevent_disable(bev, EV_READ); 157 | 158 | _DEBUG("chunked = %d\n", chunked); 159 | 160 | if (!pubsubclient_connect(arg)) { 161 | event_loopbreak(); 162 | } 163 | } 164 | 165 | void pubsubclient_init(const char *source_address, int source_port, const char *path, 166 | void (*message_cb)(char *data, void *arg), 167 | void (*error_cb)(int status_code, void *arg), 168 | void *cbarg) 169 | { 170 | 171 | signal(SIGINT, pubsubclient_termination_handler); 172 | signal(SIGQUIT, pubsubclient_termination_handler); 173 | signal(SIGTERM, pubsubclient_termination_handler); 174 | signal(SIGHUP, pubsubclient_termination_handler); 175 | 176 | if (!current_base) { 177 | event_init(); 178 | } 179 | 180 | data = calloc(1, sizeof(struct GlobalData)); 181 | data->message_cb = message_cb; 182 | data->error_cb = error_cb; 183 | data->cbarg = cbarg; 184 | data->source_address = source_address; 185 | data->source_port = source_port; 186 | data->path = path; 187 | 188 | // perform a request for headers so we can autodetect whether or not we're 189 | // getting a chunked response 190 | fprintf(stdout, "AUTODETECTING ENCODING FOR http://%s:%d%s\n", source_address, source_port, path); 191 | autodetect_sr = new_stream_request("HEAD", source_address, source_port, path, 192 | pubsubclient_autodetect_headercb, NULL, pubsubclient_errorcb, data); 193 | if (!autodetect_sr) { 194 | fprintf(stderr, "ERROR: new_stream_request() failed for %s:%d%s\n", source_address, source_port, path); 195 | exit(1); 196 | } 197 | } 198 | 199 | int pubsubclient_main(const char *source_address, int source_port, const char *path, 200 | void (*message_cb)(char *data, void *arg), 201 | void (*error_cb)(int status_code, void *arg), 202 | void *cbarg) 203 | { 204 | pubsubclient_init(source_address, source_port, path, message_cb, error_cb, cbarg); 205 | event_dispatch(); 206 | pubsubclient_free(); 207 | return 0; 208 | } 209 | 210 | void pubsubclient_run() 211 | { 212 | event_dispatch(); 213 | } 214 | 215 | void pubsubclient_free() 216 | { 217 | free_stream_request(autodetect_sr); 218 | 219 | if (chunked) { 220 | evhttp_connection_free(evhttp_source_connection); 221 | } else { 222 | free_stream_request(stream_request); 223 | } 224 | 225 | free(data); 226 | } 227 | -------------------------------------------------------------------------------- /pubsubclient/pubsubclient.h: -------------------------------------------------------------------------------- 1 | #ifndef __pubsubclient_h 2 | #define __pubsubclient_h 3 | 4 | #include 5 | 6 | struct StreamRequest; 7 | 8 | int pubsubclient_main(const char *source_address, int source_port, const char *path, 9 | void (*message_cb)(char *data, void *arg), 10 | void (*error_cb)(int status_code, void *arg), 11 | void *cbarg); 12 | void pubsubclient_init(const char *source_address, int source_port, const char *path, 13 | void (*message_cb)(char *data, void *arg), 14 | void (*error_cb)(int status_code, void *arg), 15 | void *cbarg); 16 | int pubsubclient_connect(); 17 | void pubsubclient_run(); 18 | void pubsubclient_free(); 19 | 20 | struct StreamRequest *new_stream_request(const char *method, const char *source_address, int source_port, const char *path, 21 | void (*header_cb)(struct bufferevent *bev, struct evkeyvalq *headers, void *arg), 22 | void (*read_cb)(struct bufferevent *bev, void *arg), 23 | void (*error_cb)(struct bufferevent *bev, void *arg), 24 | void *arg); 25 | void free_stream_request(struct StreamRequest *sr); 26 | 27 | #endif 28 | -------------------------------------------------------------------------------- /pubsubclient/pubsubclient_pycurl.py: -------------------------------------------------------------------------------- 1 | import pycurl 2 | 3 | def read_cb(data): 4 | # note: this may not be called with per-message blocks of data 5 | print "data: **%s**" % (data) 6 | 7 | def dbg_cb(debug_type, debug_msg): 8 | print "debug(%d): %s" % (debug_type, debug_msg) 9 | 10 | def geturl(url): 11 | c = pycurl.Curl() 12 | c.setopt(pycurl.URL, url) 13 | #c.setopt(pycurl.VERBOSE, 1) 14 | c.setopt(pycurl.WRITEFUNCTION, read_cb) 15 | c.setopt(pycurl.FOLLOWLOCATION, 1) 16 | c.setopt(pycurl.NOSIGNAL, 1) 17 | c.setopt(pycurl.DEBUGFUNCTION, dbg_cb) 18 | c.perform() 19 | c.close() 20 | 21 | geturl("http://localhost:8080/sub") 22 | -------------------------------------------------------------------------------- /pubsubclient/stream_request.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include "stream_request.h" 12 | 13 | #ifdef DEBUG 14 | #define _DEBUG(...) fprintf(stdout, __VA_ARGS__) 15 | #else 16 | #define _DEBUG(...) do {;} while (0) 17 | #endif 18 | 19 | enum read_states { read_firstline, read_headers, read_body }; 20 | 21 | struct StreamRequest *new_stream_request(const char *method, const char *source_address, int source_port, const char *path, 22 | void (*header_cb)(struct bufferevent *bev, struct evkeyvalq *headers, void *arg), 23 | void (*read_cb)(struct bufferevent *bev, void *arg), 24 | void (*error_cb)(struct bufferevent *bev, void *arg), 25 | void *arg) 26 | { 27 | struct StreamRequest *sr; 28 | int fd; 29 | struct evbuffer *http_request; 30 | 31 | fd = stream_request_connect(source_address, source_port); 32 | if (fd == -1) { 33 | return NULL; 34 | } 35 | 36 | sr = malloc(sizeof(struct StreamRequest)); 37 | sr->fd = fd; 38 | sr->state = read_firstline; 39 | sr->header_cb = header_cb; 40 | sr->read_cb = read_cb; 41 | sr->error_cb = error_cb; 42 | sr->arg = arg; 43 | sr->bev = bufferevent_new(sr->fd, stream_request_readcb, stream_request_writecb, stream_request_errorcb, sr); 44 | http_request = evbuffer_new(); 45 | evbuffer_add_printf(http_request, "%s %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n", method, path, source_address); 46 | bufferevent_write(sr->bev, (char *)EVBUFFER_DATA(http_request), EVBUFFER_LENGTH(http_request)); 47 | evbuffer_free(http_request); 48 | 49 | return sr; 50 | } 51 | 52 | void free_stream_request(struct StreamRequest *sr) 53 | { 54 | if (sr) { 55 | stream_request_disconnect(sr->fd); 56 | bufferevent_free(sr->bev); 57 | free(sr); 58 | } 59 | } 60 | 61 | int stream_request_connect(const char *address, int port) 62 | { 63 | struct addrinfo ai, *aitop; 64 | char strport[NI_MAXSERV]; 65 | struct sockaddr *sa; 66 | int slen; 67 | int fd; 68 | 69 | memset(&ai, 0, sizeof(struct addrinfo)); 70 | ai.ai_family = AF_INET; 71 | ai.ai_socktype = SOCK_STREAM; 72 | snprintf(strport, sizeof(strport), "%d", port); 73 | if (getaddrinfo(address, strport, &ai, &aitop) != 0) { 74 | fprintf(stderr, "ERROR: getaddrinfo() failed\n"); 75 | return -1; 76 | } 77 | sa = aitop->ai_addr; 78 | slen = aitop->ai_addrlen; 79 | 80 | fd = socket(AF_INET, SOCK_STREAM, 0); 81 | if (fd == -1) { 82 | fprintf(stderr, "ERROR: socket() failed\n"); 83 | return -1; 84 | } 85 | 86 | if (fcntl(fd, F_SETFL, O_NONBLOCK) == -1) { 87 | fprintf(stderr, "ERROR: fcntl(O_NONBLOCK) failed\n"); 88 | return -1; 89 | } 90 | 91 | if (connect(fd, sa, slen) == -1) { 92 | if (errno != EINPROGRESS) { 93 | fprintf(stderr, "ERROR: connect() failed\n"); 94 | return -1; 95 | } 96 | } 97 | 98 | freeaddrinfo(aitop); 99 | 100 | return fd; 101 | } 102 | 103 | int stream_request_disconnect(int fd) 104 | { 105 | EVUTIL_CLOSESOCKET(fd); 106 | } 107 | 108 | void stream_request_readcb(struct bufferevent *bev, void *arg) 109 | { 110 | struct evhttp_request *req; 111 | struct StreamRequest *sr = (struct StreamRequest *)arg; 112 | 113 | _DEBUG("stream_request_readcb()\n"); 114 | 115 | switch (sr->state) { 116 | case read_firstline: 117 | req = evhttp_request_new(NULL, NULL); 118 | req->kind = EVHTTP_RESPONSE; 119 | // 1 is the constant ALL_DATA_READ in http-internal.h 120 | if (evhttp_parse_firstline(req, EVBUFFER_INPUT(bev)) == 1) { 121 | sr->state = read_headers; 122 | } 123 | evhttp_request_free(req); 124 | // dont break, try to parse the headers too 125 | case read_headers: 126 | req = evhttp_request_new(NULL, NULL); 127 | req->kind = EVHTTP_RESPONSE; 128 | // 1 is the constant ALL_DATA_READ in http-internal.h 129 | if (evhttp_parse_headers(req, EVBUFFER_INPUT(bev)) == 1) { 130 | if (sr->header_cb) { 131 | sr->header_cb(sr->bev, req->input_headers, sr->arg); 132 | } 133 | sr->state = read_body; 134 | } 135 | evhttp_request_free(req); 136 | break; 137 | case read_body: 138 | if (sr->read_cb) { 139 | sr->read_cb(sr->bev, sr->arg); 140 | } 141 | break; 142 | } 143 | } 144 | 145 | void stream_request_writecb(struct bufferevent *bev, void *arg) 146 | { 147 | _DEBUG("stream_request_writecb()\n"); 148 | 149 | if (EVBUFFER_LENGTH(EVBUFFER_OUTPUT(bev)) == 0) { 150 | bufferevent_enable(bev, EV_READ); 151 | } 152 | } 153 | 154 | void stream_request_errorcb(struct bufferevent *bev, short what, void *arg) 155 | { 156 | struct StreamRequest *sr = (struct StreamRequest *)arg; 157 | 158 | _DEBUG("stream_request_errorcb()\n"); 159 | 160 | if (sr->error_cb) { 161 | sr->error_cb(sr->bev, sr->arg); 162 | } 163 | } 164 | -------------------------------------------------------------------------------- /pubsubclient/stream_request.h: -------------------------------------------------------------------------------- 1 | #ifndef __stream_request_h 2 | #define __stream_request_h 3 | 4 | struct StreamRequest { 5 | int fd; 6 | struct bufferevent *bev; 7 | void *arg; 8 | int state; 9 | void (*header_cb)(struct bufferevent *bev, struct evkeyvalq *headers, void *arg); 10 | void (*read_cb)(struct bufferevent *bev, void *arg); 11 | void (*error_cb)(struct bufferevent *bev, void *arg); 12 | }; 13 | 14 | int stream_request_connect(const char *address, int port); 15 | int stream_request_disconnect(int fd); 16 | void stream_request_readcb(struct bufferevent *bev, void *arg); 17 | void stream_request_writecb(struct bufferevent *bev, void *arg); 18 | void stream_request_errorcb(struct bufferevent *bev, short what, void *arg); 19 | 20 | #endif 21 | -------------------------------------------------------------------------------- /pysimplehttp/.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | dist/ 3 | *.egg-info/ 4 | -------------------------------------------------------------------------------- /pysimplehttp/scripts/file_to_sq.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import tornado.options 3 | import os 4 | import sys 5 | import pysimplehttp.file_to_simplequeue 6 | 7 | if __name__ == "__main__": 8 | tornado.options.define("input_file", type=str, default=None, help="File to load") 9 | tornado.options.define("max_queue_depth", type=int, default=2500, help="only fill the queue to DEPTH entries") 10 | tornado.options.define("simplequeue_url", type=str, multiple=True, help="(multiple) url(s) for simplequeue to write to") 11 | tornado.options.define("stats_interval", type=int, default=60, help="seconds between displaying stats") 12 | tornado.options.define("concurrent_requests", type=int, default=20, help="number of simultaneous requests") 13 | tornado.options.define("check_simplequeue_interval", type=int, default=1, help="seconds between checking simplequeue depth") 14 | tornado.options.define('filter_require', type=str, multiple=True, help="filter json message to require for key=value") 15 | tornado.options.define('filter_exclude', type=str, multiple=True, help="filter json message to exclude for key=value") 16 | tornado.options.parse_command_line() 17 | 18 | options = tornado.options.options 19 | if not options.input_file or not os.path.exists(options.input_file): 20 | sys.stderr.write("ERROR: --input_file=%r does not exist\n" % options.input_file) 21 | sys.exit(1) 22 | if not options.simplequeue_url: 23 | sys.stderr.write('ERROR: --simplequeue_url required\n' ) 24 | sys.exit(1) 25 | 26 | file_to_sq = pysimplehttp.file_to_simplequeue.FileToSimplequeue(options.input_file, 27 | options.concurrent_requests, 28 | options.max_queue_depth, 29 | options.simplequeue_url, 30 | options.check_simplequeue_interval, 31 | options.stats_interval, 32 | filter_require=options.filter_require, 33 | filter_exclude=options.filter_exclude, 34 | ) 35 | file_to_sq.start() 36 | 37 | 38 | -------------------------------------------------------------------------------- /pysimplehttp/scripts/ps_to_sq.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | generic pubsub to simplequeue daemon that takes command line arguments: 4 | --pubsub-url= 5 | (multiple) --simplequeue-url= 6 | 7 | when multiple destination simplequeue arguments are specified, the daemon will 8 | randomly choose one endpoint to write a message to 9 | """ 10 | import logging 11 | import tornado.httpclient 12 | import tornado.options 13 | import sys 14 | import urllib 15 | import random 16 | try: 17 | import ujson as json 18 | except ImportError: 19 | import json 20 | 21 | from pysimplehttp.pubsub_reader import PubsubReader 22 | 23 | 24 | class PubsubToSimplequeue(PubsubReader): 25 | def __init__(self, simplequeue_urls, filter_require, filter_exclude, **kwargs): 26 | assert isinstance(simplequeue_urls, (list, tuple)) 27 | self.simplequeue_urls = simplequeue_urls 28 | self.filter_require = dict([data.split('=', 1) for data in filter_require]) 29 | for key, value in self.filter_require.items(): 30 | logging.info("requiring json key=%s value=%s" % (key, value) ) 31 | self.filter_exclude = dict([data.split('=', 1) for data in filter_exclude]) 32 | for key, value in self.filter_exclude.items(): 33 | logging.info("excluding json key=%s value=%s" % (key, value) ) 34 | self.http = tornado.httpclient.AsyncHTTPClient() 35 | super(PubsubToSimplequeue, self).__init__(**kwargs) 36 | 37 | def http_fetch(self, url, params, callback, headers={}): 38 | url += '?' + urllib.urlencode(params) 39 | req = tornado.httpclient.HTTPRequest(url=url, 40 | method='GET', 41 | follow_redirects=False, 42 | headers=headers, 43 | user_agent='ps_to_sq') 44 | self.http.fetch(req, callback=callback) 45 | 46 | def _finish(self, response): 47 | if response.code != 200: 48 | logging.info(response) 49 | 50 | def callback(self, data): 51 | """ 52 | handle a single pubsub message 53 | """ 54 | if not data or len(data) == 1: 55 | return 56 | assert isinstance(data, str) 57 | if self.filter_require or self.filter_exclude: 58 | try: 59 | msg = json.loads(data) 60 | except Exception: 61 | logging.error('failed json.loads(%r)' % data) 62 | return 63 | for key, value in self.filter_require.items(): 64 | if msg.get(key) != value: 65 | return 66 | for key, value in self.filter_exclude.items(): 67 | if msg.get(key) == value: 68 | return 69 | endpoint = random.choice(self.simplequeue_urls) + '/put' 70 | self.http_fetch(endpoint, dict(data=data), callback=self._finish) 71 | 72 | 73 | if __name__ == "__main__": 74 | tornado.options.define('pubsub_url', type=str, default="http://127.0.0.1:8080/sub?multipart=0", help="url for pubsub to read from") 75 | tornado.options.define('simplequeue_url', type=str, multiple=True, help="(multiple) url(s) for simplequeue to write to") 76 | tornado.options.define('filter_require', type=str, multiple=True, help="filter json message to require for key=value") 77 | tornado.options.define('filter_exclude', type=str, multiple=True, help="filter json message to exclude for key=value") 78 | tornado.options.parse_command_line() 79 | if not tornado.options.options.pubsub_url: 80 | sys.stderr.write('--pubsub-url requrired\n') 81 | sys.exit(1) 82 | 83 | if not tornado.options.options.simplequeue_url: 84 | sys.stderr.write('--simplequeue-url requrired\n') 85 | sys.exit(1) 86 | 87 | reader = PubsubToSimplequeue( 88 | simplequeue_urls=tornado.options.options.simplequeue_url, 89 | filter_require=tornado.options.options.filter_require, 90 | filter_exclude=tornado.options.options.filter_exclude, 91 | pubsub_url=tornado.options.options.pubsub_url 92 | ) 93 | reader.start() 94 | -------------------------------------------------------------------------------- /pysimplehttp/src/BackoffTimer.py: -------------------------------------------------------------------------------- 1 | from decimal import Decimal 2 | 3 | 4 | def _Decimal(v): 5 | if not isinstance(v, Decimal): 6 | return Decimal(str(v)) 7 | return v 8 | 9 | 10 | class BackoffTimer(object): 11 | """ 12 | This is a timer that is smart about backing off exponentially when there are problems 13 | """ 14 | def __init__(self, min_interval, max_interval, ratio=.25, short_length=10, long_length=250): 15 | assert isinstance(min_interval, (int, float, Decimal)) 16 | assert isinstance(max_interval, (int, float, Decimal)) 17 | 18 | self.min_interval = _Decimal(min_interval) 19 | self.max_interval = _Decimal(max_interval) 20 | 21 | self.max_short_timer = (self.max_interval - self.min_interval) * _Decimal(ratio) 22 | self.max_long_timer = (self.max_interval - self.min_interval) * (1 - _Decimal(ratio)) 23 | self.short_unit = self.max_short_timer / _Decimal(short_length) 24 | self.long_unit = self.max_long_timer / _Decimal(long_length) 25 | # print self.short_unit, self.long_unit 26 | # logging.info('short unit %s' % self.short_unit) 27 | # logging.info('long unit %s' % self.long_unit) 28 | 29 | self.short_interval = Decimal(0) 30 | self.long_interval = Decimal(0) 31 | 32 | def success(self): 33 | """Update the timer to reflect a successfull call""" 34 | self.short_interval -= self.short_unit 35 | self.long_interval -= self.long_unit 36 | self.short_interval = max(self.short_interval, Decimal(0)) 37 | self.long_interval = max(self.long_interval, Decimal(0)) 38 | 39 | def failure(self): 40 | """Update the timer to reflect a failed call""" 41 | self.short_interval += self.short_unit 42 | self.long_interval += self.long_unit 43 | self.short_interval = min(self.short_interval, self.max_short_timer) 44 | self.long_interval = min(self.long_interval, self.max_long_timer) 45 | 46 | def get_interval(self): 47 | return float(self.min_interval + self.short_interval + self.long_interval) 48 | 49 | 50 | def test_timer(): 51 | timer = BackoffTimer(.1, 120, long_length=1000) 52 | assert timer.get_interval() == .1 53 | timer.success() 54 | assert timer.get_interval() == .1 55 | timer.failure() 56 | interval = '%0.2f' % timer.get_interval() 57 | assert interval == '3.19' 58 | assert timer.min_interval == Decimal('.1') 59 | assert timer.short_interval == Decimal('2.9975') 60 | assert timer.long_interval == Decimal('0.089925') 61 | 62 | timer.failure() 63 | interval = '%0.2f' % timer.get_interval() 64 | assert interval == '6.27' 65 | timer.success() 66 | interval = '%0.2f' % timer.get_interval() 67 | assert interval == '3.19' 68 | for i in range(25): 69 | timer.failure() 70 | interval = '%0.2f' % timer.get_interval() 71 | assert interval == '32.41' 72 | 73 | -------------------------------------------------------------------------------- /pysimplehttp/src/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = "0.2.1" 2 | 3 | import file_to_simplequeue 4 | import pubsub_reader 5 | import BackoffTimer 6 | import BaseReader 7 | import http 8 | import formatters 9 | -------------------------------------------------------------------------------- /pysimplehttp/src/file_to_simplequeue.py: -------------------------------------------------------------------------------- 1 | import tornado.ioloop 2 | import tornado.httpclient 3 | import os 4 | import functools 5 | import gzip 6 | import logging 7 | import urllib 8 | try: 9 | import ujson as json 10 | except ImportError: 11 | import json 12 | 13 | class FileToSimplequeue(object): 14 | http = tornado.httpclient.AsyncHTTPClient() 15 | def __init__(self, input_file, max_concurrent, max_queue_depth, simplequeue_urls, 16 | check_simplequeue_interval, stats_interval, filter_require=None, filter_exclude=None, io_loop=None): 17 | assert isinstance(simplequeue_urls, (list, tuple)) 18 | assert isinstance(max_queue_depth, int) 19 | assert isinstance(max_concurrent, int) 20 | assert isinstance(check_simplequeue_interval, int) 21 | assert isinstance(stats_interval, int) 22 | assert isinstance(filter_require, (None.__class__, list, tuple)) 23 | assert isinstance(filter_exclude, (None.__class__, list, tuple)) 24 | for entry in simplequeue_urls: 25 | assert entry.startswith("http://") or entry.startswith("https://"), "simplequeue url %s is not valid" % entry 26 | self.simplequeue_urls = simplequeue_urls 27 | self.input = self.open_file(input_file) 28 | self.concurrent = 0 29 | self.finished = False 30 | self.fill_check = False 31 | self.max_queue_depth = max_queue_depth 32 | self.max_concurrent = max_concurrent 33 | self.check_simplequeue_interval = check_simplequeue_interval 34 | self.pending = dict([[simplequeue, 0] for simplequeue in simplequeue_urls]) 35 | self.stats_interval = stats_interval 36 | self.filter_require = dict([data.split('=', 1) for data in (filter_require or [])]) 37 | for key, value in self.filter_require.items(): 38 | logging.info("requiring json key=%s value=%s" % (key, value) ) 39 | self.filter_exclude = dict([data.split('=', 1) for data in (filter_exclude or [])]) 40 | for key, value in self.filter_exclude.items(): 41 | logging.info("excluding json key=%s value=%s" % (key, value) ) 42 | self.stats_reset() 43 | self.io_loop = io_loop or tornado.ioloop.IOLoop.instance() 44 | 45 | def stats_incr(self, successful=True, filtered=False): 46 | if filtered: 47 | self.filtered += 1 48 | return 49 | if successful: 50 | self.success += 1 51 | else: 52 | self.failed += 1 53 | 54 | def stats_reset(self): 55 | self.success = 0 56 | self.failed = 0 57 | self.filtered = 0 58 | 59 | def print_and_reset_stats(self): 60 | logging.warning('success: %5d failed: %5d filtered: %5d concurrent: %2d' % (self.success, self.failed, self.filtered, self.concurrent)) 61 | self.stats_reset() 62 | 63 | def start(self): 64 | self.stats_timer = tornado.ioloop.PeriodicCallback(self.print_and_reset_stats, self.stats_interval * 1000) 65 | self.stats_timer.start() 66 | self.check_timer = tornado.ioloop.PeriodicCallback(self.check_simplequeue_depth, self.check_simplequeue_interval * 1000) 67 | self.check_timer.start() 68 | self.check_simplequeue_depth() # seed the loop 69 | self.io_loop.start() 70 | 71 | def open_file(self, filename): 72 | assert os.path.exists(filename), "%r is not accessible" % filename 73 | if filename.endswith('.gz'): 74 | return gzip.open(filename, 'rb') 75 | else: 76 | return open(filename, 'rb') 77 | 78 | def check_simplequeue_depth(self): 79 | """query the simplequeue and fill it based on where it's dept should be""" 80 | if self.finished: 81 | return self.finish() 82 | for simplequeue in self.simplequeue_urls: 83 | self.http.fetch(simplequeue + '/stats?format=json', 84 | callback=functools.partial(self.finish_check_simplequeue_depth, simplequeue=simplequeue)) 85 | 86 | def finish_check_simplequeue_depth(self, response, simplequeue): 87 | if response.code != 200: 88 | logging.error('failed checking simplequeue depth %s/stats?format=json' % simplequeue) 89 | self.continue_fill() 90 | return 91 | stats = json.loads(response.body) 92 | entries_needed = self.max_queue_depth - stats['depth'] 93 | entries_needed = max(0, entries_needed) 94 | logging.info('%s needs %d entries' % (simplequeue, entries_needed)) 95 | self.pending[simplequeue] = entries_needed 96 | self.continue_fill() 97 | 98 | def continue_fill(self): 99 | if not self.fill_check: 100 | self.fill_check = True 101 | self.io_loop.add_callback(self.fill_as_needed) 102 | 103 | def fill_as_needed(self): 104 | """ 105 | as needed based on how many more should go in a simplequeue, and the current concurrency 106 | """ 107 | self.fill_check = False 108 | if self.finished: 109 | return self.finish() 110 | available_concurrency = self.max_concurrent - self.concurrent 111 | for simplequeue in self.pending.keys(): 112 | while available_concurrency and self.pending[simplequeue] > 0: 113 | if self.fill_one(simplequeue): 114 | available_concurrency -= 1 115 | self.pending[simplequeue] -= 1 116 | 117 | def fill_one(self, endpoint): 118 | """read one line from `self.input` and send it to a simplequeue""" 119 | data = self.input.readline() 120 | if not data: 121 | if not self.finished: 122 | logging.info('at end of input stream') 123 | self.finish() 124 | return True 125 | 126 | if self.filter_require or self.filter_exclude: 127 | try: 128 | msg = json.loads(data) 129 | except Exception: 130 | logging.error('failed json.loads(%r)' % data) 131 | self.stats_incr(successful=False) 132 | return False 133 | for key, value in self.filter_require.items(): 134 | if msg.get(key) != value: 135 | self.stats_incr(filtered=True) 136 | return False 137 | for key, value in self.filter_exclude.items(): 138 | if msg.get(key) == value: 139 | self.stats_incr(filtered=True) 140 | return False 141 | 142 | self.concurrent += 1 143 | url = endpoint + '/put?' + urllib.urlencode(dict(data=data)) 144 | self.http.fetch(url, self.finish_fill_one) 145 | return True 146 | 147 | def finish_fill_one(self, response): 148 | self.concurrent -= 1 149 | if response.code != 200: 150 | logging.info(response) 151 | self.failed += 1 152 | else: 153 | self.success += 1 154 | 155 | # continue loop 156 | if self.max_concurrent > self.concurrent: 157 | self.continue_fill() 158 | 159 | def finish(self): 160 | self.finished = True 161 | if self.concurrent == 0: 162 | logging.info('stopping ioloop') 163 | self.io_loop.stop() 164 | -------------------------------------------------------------------------------- /pysimplehttp/src/formatters.py: -------------------------------------------------------------------------------- 1 | import re 2 | import binascii 3 | import calendar 4 | 5 | 6 | def _crc(key): 7 | """crc32 hash a string""" 8 | return binascii.crc32(_utf8(key)) & 0xffffffff 9 | 10 | 11 | def _b32(number): 12 | """convert positive integer to a base32 string""" 13 | assert isinstance(number, (int, long)) 14 | alphabet = '0123456789abcdefghijklmnopqrstuv' 15 | alphabet_len = 32 16 | 17 | if number == 0: 18 | return alphabet[0] 19 | 20 | base32 = '' 21 | 22 | sign = '' 23 | if number < 0: 24 | sign = '-' 25 | number = -number 26 | 27 | while number != 0: 28 | number, i = divmod(number, alphabet_len) 29 | base32 = alphabet[i] + base32 30 | 31 | return sign + base32 32 | 33 | 34 | def _idn(domain): 35 | """idn encode a domain name""" 36 | if not domain: 37 | return domain 38 | if 'xn--' in domain: 39 | return domain.decode('idna') 40 | return _unicode(domain) 41 | 42 | 43 | def _punycode(domain): 44 | """idna encode (punycode) a domain name""" 45 | if not domain: 46 | return domain 47 | domain = _unicode(domain) 48 | if re.findall(r'[^-_a-zA-Z0-9\.]', domain): 49 | return domain.encode('idna') 50 | return _utf8(domain) 51 | 52 | 53 | def _unicode(value): 54 | """decode a utf-8 string as unicode""" 55 | if isinstance(value, str): 56 | return value.decode("utf-8") 57 | assert isinstance(value, unicode) 58 | return value 59 | 60 | 61 | def _utf8(s): 62 | """encode a unicode string as utf-8""" 63 | if isinstance(s, unicode): 64 | return s.encode("utf-8") 65 | assert isinstance(s, str) 66 | return s 67 | 68 | 69 | def _utc_ts(dt): 70 | """convert a datetime object into a UNIX epoch timestamp""" 71 | return calendar.timegm(dt.utctimetuple()) 72 | 73 | 74 | def _utf8_params(params): 75 | """encode a dictionary of URL parameters (including iterables) as utf-8""" 76 | isinstance(params, dict) 77 | encoded_params = [] 78 | for k, v in params.items(): 79 | if isinstance(v, (list, tuple)): 80 | v = [_utf8(x) for x in v] 81 | else: 82 | v = _utf8(v) 83 | encoded_params.append((k, v)) 84 | return dict(encoded_params) 85 | 86 | 87 | class _O(dict): 88 | """Makes a dictionary behave like an object.""" 89 | def __getattr__(self, name): 90 | try: 91 | return self[name] 92 | except KeyError: 93 | # raise AttributeError(name) 94 | return None 95 | 96 | def __setattr__(self, name, value): 97 | self[name] = value 98 | -------------------------------------------------------------------------------- /pysimplehttp/src/http.py: -------------------------------------------------------------------------------- 1 | import tornado.httpclient 2 | import urllib 3 | 4 | from formatters import _utf8, _utf8_params 5 | 6 | try: 7 | import ujson as json 8 | except ImportError: 9 | import json 10 | 11 | _HTTPCLIENT = None 12 | def get_http_client(): 13 | global _HTTPCLIENT 14 | if _HTTPCLIENT is None: 15 | _HTTPCLIENT = tornado.httpclient.HTTPClient() 16 | return _HTTPCLIENT 17 | 18 | def http_fetch(url, params={}, headers=None, method='GET', body=None, timeout=5.0, client_options=None, connect_timeout=None, request_timeout=None): 19 | headers = headers or {} 20 | client_options = client_options or {} 21 | 22 | for key, value in params.items(): 23 | if isinstance(value, (int, long, float)): 24 | params[key] = str(value) 25 | if value is None: 26 | del params[key] 27 | 28 | params = _utf8_params(params) 29 | if method in ['GET', 'HEAD'] and params: 30 | url += '?' + urllib.urlencode(params, doseq=1) 31 | body = None 32 | elif params: 33 | body = urllib.urlencode(params, doseq=1) 34 | headers['Content-type'] = 'application/x-www-form-urlencoded' 35 | if body: 36 | body = _utf8(body) 37 | if 'follow_redirects' not in client_options: 38 | client_options['follow_redirects'] = False 39 | req = tornado.httpclient.HTTPRequest(url=_utf8(url), \ 40 | method=method, 41 | body=body, 42 | headers=headers, 43 | connect_timeout=connect_timeout or timeout, 44 | request_timeout=request_timeout or timeout, 45 | validate_cert=False, 46 | **client_options) 47 | 48 | # sync client raises errors on non-200 responses 49 | http = get_http_client() 50 | response = http.fetch(req) 51 | if method == 'HEAD': 52 | return True 53 | return response.body 54 | 55 | 56 | def pubsub_write(endpoint, data): 57 | if isinstance(data, dict): 58 | data = json.dumps(data) 59 | return http_fetch(endpoint + '/pub', body=data, method='POST', timeout=1.5) 60 | 61 | def simplequeue_write(endpoint, data): 62 | assert isinstance(data, dict) 63 | data = json.dumps(data) 64 | result = http_fetch(endpoint + '/put', dict(data=data)) 65 | # simplequeue success is a 200 response w/ an empty response body 66 | return result == '' 67 | -------------------------------------------------------------------------------- /pysimplehttp/src/pubsub_reader.py: -------------------------------------------------------------------------------- 1 | import tornado.iostream 2 | import tornado.ioloop 3 | import socket 4 | import logging 5 | import urlparse 6 | import functools 7 | import base64 8 | 9 | 10 | class HTTPError(Exception): 11 | def __init__(self, code, msg=None): 12 | self.code = code 13 | self.msg = msg 14 | super(HTTPError, self).__init__('%s %s' % (code , msg)) 15 | 16 | class PubsubReader(object): 17 | def __init__(self, pubsub_url, io_loop=None): 18 | self.io_loop = io_loop or tornado.ioloop.IOLoop.instance() 19 | self.socket = None 20 | 21 | urlinfo = urlparse.urlparse(pubsub_url) 22 | assert urlinfo.scheme == 'http' 23 | netloc = urlinfo.netloc 24 | self.basic_auth = None 25 | port = 80 26 | if '@' in netloc: 27 | self.basic_auth, netloc = netloc.split('@', 1) 28 | if ':' in netloc: 29 | netloc, port = netloc.rsplit(':', 1) 30 | port = int(port) 31 | self.host = netloc 32 | self.port = port 33 | self.get_line = urlparse.urlunparse(('', '', urlinfo.path, urlinfo.params, urlinfo.query, urlinfo.fragment)) 34 | 35 | def start(self): 36 | self.open(self.host, self.port) 37 | self.io_loop.start() 38 | 39 | def _callback(self, data): 40 | try: 41 | self.callback(data[:-1]) 42 | except Exception: 43 | logging.exception('failed in callback') 44 | # NOTE: to work around a maximum recursion error (later fix by https://github.com/facebook/tornado/commit/f8f3a9bf08f1cab1d2ab232074a14e7a94eaa4b1) 45 | # we schedule the read_until for later call by the io_loop 46 | # this can go away w/ tornado 2.0 47 | callback = functools.partial(self.stream.read_until, '\n', self._callback) 48 | self.io_loop.add_callback(callback) 49 | 50 | def callback(self, data): 51 | raise Exception("Not Implemented") 52 | 53 | def close(self): 54 | logging.info('closed') 55 | self.io_loop.stop() 56 | 57 | def http_get_line(self): 58 | line = "GET %s HTTP/1.0\r\n" % self.get_line 59 | if self.basic_auth: 60 | line += "Authorization: Basic %s\r\n" % base64.b64encode(self.basic_auth) 61 | return line + "\r\n" 62 | 63 | def open(self, host, port): 64 | assert isinstance(port, int) 65 | self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) 66 | logging.info('opening socket to %s:%s' % (host, port)) 67 | self.socket.connect((host, port)) 68 | self.stream = tornado.iostream.IOStream(self.socket) 69 | self.stream.set_close_callback(self.close) 70 | get_line = self.http_get_line() 71 | logging.info(get_line) 72 | self.stream.write(get_line) 73 | self.stream.read_until("\r\n\r\n", self.on_headers) 74 | 75 | def on_headers(self, data): 76 | headers = {} 77 | lines = data.split("\r\n") 78 | status_line = lines[0] 79 | if status_line.count(' ') < 2: 80 | raise HTTPError(599, 'connect error') 81 | status_code = status_line.split(' ', 2)[1] 82 | if status_code != "200": 83 | raise HTTPError(status_code) 84 | for line in lines[1:]: 85 | parts = line.split(":") 86 | if len(parts) == 2: 87 | headers[parts[0].strip()] = parts[1].strip() 88 | self.stream.read_until('\n', self._callback) 89 | 90 | -------------------------------------------------------------------------------- /qrencode/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I. -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -g 8 | LIBS = -L. -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -L/usr/local/lib -levent -lqrencode -lpng -lz -lbz2 -lresolv -ldl -lpthread -lm -lc 9 | 10 | qrencode: qrencode.c 11 | $(CC) $(CFLAGS) -o $@ $< $(LIBS) -lsimplehttp 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin 15 | /usr/bin/install qrencode $(TARGET)/bin 16 | 17 | clean: 18 | rm -rf *.o qrencode *.dSYM 19 | -------------------------------------------------------------------------------- /qrencode/qrencode.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include "png.h" 9 | #include "qrencode.h" 10 | 11 | /* 12 | * libqrencode 13 | * http://megaui.net/fukuchi/works/qrencode/index.en.html 14 | * 15 | * os x installation 16 | * http://blog.loudhush.ro/2009/12/creating-qr-codes-on-mac-os-x-snow.html 17 | * 18 | * libpng 19 | * http://www.libpng.org/pub/png/libpng.html 20 | * 21 | */ 22 | 23 | static FILE *fp; // avoid clobbering by setjmp. 24 | static int casesensitive = 1; 25 | static int eightbit = 0; 26 | static int version = 0; 27 | static int size = 3; 28 | static int margin = 4; 29 | static int structured = 0; 30 | static QRecLevel level = QR_ECLEVEL_L; 31 | static QRencodeMode hint = QR_MODE_8; 32 | 33 | static int writePNG(QRcode *qrcode) 34 | { 35 | png_structp png_ptr; 36 | png_infop info_ptr; 37 | unsigned char *row, *p, *q; 38 | int x, y, xx, yy, bit; 39 | int realwidth; 40 | 41 | realwidth = (qrcode->width + margin * 2) * size; 42 | row = (unsigned char *)malloc((realwidth + 7) / 8); 43 | if (row == NULL) { 44 | fprintf(stderr, "Failed to allocate memory.\n"); 45 | exit(EXIT_FAILURE); 46 | } 47 | 48 | png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL); 49 | if (png_ptr == NULL) { 50 | fprintf(stderr, "Failed to initialize PNG writer.\n"); 51 | exit(EXIT_FAILURE); 52 | } 53 | 54 | info_ptr = png_create_info_struct(png_ptr); 55 | if (info_ptr == NULL) { 56 | fprintf(stderr, "Failed to initialize PNG write.\n"); 57 | exit(EXIT_FAILURE); 58 | } 59 | 60 | if (setjmp(png_jmpbuf(png_ptr))) { 61 | png_destroy_write_struct(&png_ptr, &info_ptr); 62 | fprintf(stderr, "Failed to write PNG image.\n"); 63 | exit(EXIT_FAILURE); 64 | } 65 | 66 | fseek(fp, 0, SEEK_SET); 67 | ftruncate(fileno(fp), 0); 68 | png_init_io(png_ptr, fp); 69 | png_set_IHDR(png_ptr, info_ptr, 70 | realwidth, realwidth, 71 | 1, 72 | PNG_COLOR_TYPE_GRAY, 73 | PNG_INTERLACE_NONE, 74 | PNG_COMPRESSION_TYPE_DEFAULT, 75 | PNG_FILTER_TYPE_DEFAULT); 76 | png_write_info(png_ptr, info_ptr); 77 | 78 | /* top margin */ 79 | memset(row, 0xff, (realwidth + 7) / 8); 80 | for (y = 0; y < margin * size; y++) { 81 | png_write_row(png_ptr, row); 82 | } 83 | 84 | /* data */ 85 | p = qrcode->data; 86 | for (y = 0; y < qrcode->width; y++) { 87 | bit = 7; 88 | memset(row, 0xff, (realwidth + 7) / 8); 89 | q = row; 90 | q += margin * size / 8; 91 | bit = 7 - (margin * size % 8); 92 | for (x = 0; x < qrcode->width; x++) { 93 | for (xx = 0; xx < size; xx++) { 94 | *q ^= (*p & 1) << bit; 95 | bit--; 96 | if (bit < 0) { 97 | q++; 98 | bit = 7; 99 | } 100 | } 101 | p++; 102 | } 103 | for (yy = 0; yy < size; yy++) { 104 | png_write_row(png_ptr, row); 105 | } 106 | } 107 | /* bottom margin */ 108 | memset(row, 0xff, (realwidth + 7) / 8); 109 | for (y = 0; y < margin * size; y++) { 110 | png_write_row(png_ptr, row); 111 | } 112 | 113 | png_write_end(png_ptr, info_ptr); 114 | png_destroy_write_struct(&png_ptr, &info_ptr); 115 | 116 | free(row); 117 | fflush(fp); 118 | 119 | return 0; 120 | } 121 | 122 | static QRcode *encode(const char *intext) 123 | { 124 | QRcode *code; 125 | 126 | if (eightbit) { 127 | code = QRcode_encodeString8bit(intext, version, level); 128 | } else { 129 | code = QRcode_encodeString(intext, version, level, hint, casesensitive); 130 | } 131 | 132 | return code; 133 | } 134 | 135 | static int qrencode(const char *intext) 136 | { 137 | QRcode *qrcode; 138 | int ret; 139 | 140 | qrcode = encode(intext); 141 | if (qrcode == NULL) { 142 | perror("Failed to encode the input data:"); 143 | exit(EXIT_FAILURE); 144 | } 145 | ret = writePNG(qrcode); 146 | QRcode_free(qrcode); 147 | return ret; 148 | } 149 | 150 | void cb(struct evhttp_request *req, struct evbuffer *evb, void *ctx) 151 | { 152 | int ret, fd; 153 | struct stat st; 154 | void *pngdata; 155 | const char *txt; 156 | struct evkeyvalq args; 157 | 158 | evhttp_parse_query(req->uri, &args); 159 | txt = evhttp_find_header(&args, "txt"); 160 | 161 | ret = qrencode(txt); 162 | if (ret == 0) { 163 | fd = fileno(fp); 164 | fstat(fd, &st); 165 | pngdata = mmap(0, st.st_size, PROT_READ, MAP_SHARED, fd, 0); 166 | evhttp_add_header(req->output_headers, "content-type", "image/png"); 167 | evbuffer_add(evb, pngdata, st.st_size); 168 | evhttp_send_reply(req, HTTP_OK, "OK", evb); 169 | munmap(pngdata, st.st_size); 170 | } else { 171 | evhttp_send_reply(req, HTTP_SERVUNAVAIL, "ERROR", evb); 172 | } 173 | } 174 | 175 | int main(int argc, char **argv) 176 | { 177 | 178 | char *outfile = "/tmp/qrencode.png"; 179 | 180 | define_simplehttp_options(); 181 | option_define_str("temp_file", OPT_OPTIONAL, "/tmp/qrencode.png", &outfile, NULL, NULL); 182 | if (!option_parse_command_line(argc, argv)) { 183 | return 1; 184 | } 185 | 186 | fp = fopen(outfile, "a+"); 187 | if (fp == NULL) { 188 | fprintf(stderr, "Failed to create file: %s\n", outfile); 189 | perror(NULL); 190 | exit(EXIT_FAILURE); 191 | } 192 | 193 | simplehttp_init(); 194 | simplehttp_set_cb("/qr*", cb, NULL); 195 | simplehttp_main(); 196 | free_options(); 197 | 198 | fclose(fp); 199 | return 0; 200 | } 201 | -------------------------------------------------------------------------------- /queuereader/Makefile: -------------------------------------------------------------------------------- 1 | TARGET ?= /usr/local 2 | LIBEVENT ?= /usr/local 3 | LIBSIMPLEHTTP ?= /usr/local 4 | 5 | CFLAGS = -I. -I$(LIBSIMPLEHTTP)/include -I.. -I$(LIBEVENT)/include -g -Wall -O2 6 | AR = ar 7 | AR_FLAGS = rc 8 | RANLIB = ranlib 9 | 10 | libqueuereader.a: queuereader.o queuereader.h 11 | /bin/rm -f $@ 12 | $(AR) $(AR_FLAGS) $@ $^ 13 | $(RANLIB) $@ 14 | 15 | all: libqueuereader.a 16 | 17 | install: 18 | /usr/bin/install -d $(TARGET)/lib/ 19 | /usr/bin/install -d $(TARGET)/bin/ 20 | /usr/bin/install -d $(TARGET)/include/queuereader 21 | /usr/bin/install libqueuereader.a $(TARGET)/lib/ 22 | /usr/bin/install queuereader.h $(TARGET)/include/queuereader 23 | 24 | clean: 25 | /bin/rm -f *.a *.o 26 | -------------------------------------------------------------------------------- /queuereader/queuereader.h: -------------------------------------------------------------------------------- 1 | #ifndef __queuereader_h 2 | #define __queuereader_h 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | 14 | #define QR_CONT 0 15 | #define QR_EMPTY 1 16 | #define QR_FAILURE 2 17 | #define QR_SUCCESS 3 18 | #define QR_CONT_SOURCE_REQUEST 4 19 | #define QR_REQUEUE_WITHOUT_BACKOFF 5 20 | #define QR_REQUEUE_WITHOUT_DELAY 6 21 | 22 | 23 | int queuereader_main(struct json_object *tasks, 24 | const char *source_address, int source_port, const char *path, 25 | int (*message_cb)(struct json_object *json_msg, void *arg), 26 | void (*error_cb)(int status_code, void *arg), 27 | void *cbarg); 28 | void queuereader_run(); 29 | void queuereader_free(); 30 | void queuereader_init(struct json_object *tasks, 31 | const char *source_address, int source_port, const char *path, 32 | int (*message_cb)(struct json_object *json_msg, void *arg), 33 | void (*error_cb)(int status_code, void *arg), 34 | void *cbarg); 35 | void queuereader_finish_message(int return_code); 36 | void queuereader_set_sleeptime_queue_empty_ms(int milliseconds); 37 | struct json_object *queuereader_copy_tasks(struct json_object *input_array); 38 | void queuereader_finish_task_by_name(const char *finished_task); 39 | 40 | #endif 41 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | # release steps 4 | # --------------- 5 | # update version variable (below) 6 | # update pysimplehttp/src/__init__.__version__ 7 | # run python setup.py sdist 8 | # upload .tar.gz to github 9 | # run python setup.py register to update pypi 10 | 11 | __version__ = "0.2.1" 12 | scripts = ['pysimplehttp/scripts/ps_to_sq.py', 13 | 'pysimplehttp/scripts/file_to_sq.py'] 14 | 15 | setup( 16 | name='pysimplehttp', 17 | version=__version__, 18 | author='Jehiah Czebotar', 19 | author_email='jehiah@gmail.com', 20 | description='Python libraries for simplehttp', 21 | url='https://github.com/bitly/simplehttp', 22 | classifiers=[ 23 | 'Intended Audience :: Developers', 24 | 'Programming Language :: Python', 25 | ], 26 | download_url="http://github.com/downloads/bitly/simplehttp/pysimplehttp-%s.tar.gz" %__version__, 27 | 28 | packages=['pysimplehttp'], 29 | package_dir = {'pysimplehttp' : 'pysimplehttp/src'}, 30 | 31 | scripts = scripts, 32 | install_requires = [ 33 | 'tornado', 34 | ], 35 | requires = [ 36 | 'ujson', 37 | 'host_pool', 38 | ], 39 | ) 40 | -------------------------------------------------------------------------------- /shared_tests/test_shunt.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import sys 3 | import os 4 | import unittest 5 | import subprocess 6 | import signal 7 | import time 8 | import simplejson as json 9 | import urllib 10 | import tornado.httpclient 11 | 12 | logging.basicConfig(stream=sys.stdout, level=logging.INFO, 13 | format='%(asctime)s %(process)d %(filename)s %(lineno)d %(levelname)s #| %(message)s', 14 | datefmt='%H:%M:%S') 15 | 16 | def http_fetch_json(endpoint, params=None, status_code=200, status_txt="OK", body=None): 17 | body = http_fetch(endpoint, params, 200, body) 18 | data = json.loads(body) 19 | assert data['status_code'] == status_code 20 | assert data['status_txt'] == status_txt 21 | return data['data'] 22 | 23 | def http_fetch(endpoint, params=None, response_code=200, body=None): 24 | http_client = tornado.httpclient.HTTPClient() 25 | url = 'http://127.0.0.1:8080' + endpoint 26 | if params: 27 | url += '?' + urllib.urlencode(params, doseq=1) 28 | method = "POST" if body else "GET" 29 | try: 30 | res = http_client.fetch(url, method=method, body=body) 31 | except tornado.httpclient.HTTPError, e: 32 | logging.info(e) 33 | res = e.response 34 | assert res.code == response_code 35 | return res.body 36 | 37 | def valgrind_cmd(test_output_dir, *options): 38 | assert isinstance(options, (list, tuple)) 39 | cmdlist = list(options) 40 | if '--no-valgrind' not in sys.argv: 41 | cmdlist = [ 42 | 'valgrind', 43 | '-v', 44 | '--tool=memcheck', 45 | '--trace-children=yes', 46 | '--log-file=%s/vg.out' % test_output_dir, 47 | '--leak-check=full', 48 | #'--show-reachable=yes', 49 | '--run-libc-freeres=yes', 50 | ] + cmdlist 51 | return cmdlist 52 | 53 | def check_valgrind_output(filename): 54 | if '--no-valgrind' in sys.argv: 55 | return 56 | 57 | assert os.path.exists(filename) 58 | time.sleep(.15) 59 | vg_output = open(filename, 'r').readlines() 60 | logging.info('checking valgrind output %d lines' % len(vg_output)) 61 | 62 | # strip the process prefix '==pid==' 63 | prefix_len = vg_output[0].find(' ') 64 | vg_output = [line[prefix_len+1:] for line in vg_output] 65 | 66 | error_summary = [line.strip() for line in vg_output if line.startswith("ERROR SUMMARY:")] 67 | assert error_summary, "vg.out not finished" 68 | assert error_summary[0].startswith("ERROR SUMMARY: 0 errors") 69 | 70 | lost = [line.strip() for line in vg_output if line.strip().startswith("definitely lost:")] 71 | assert lost 72 | assert lost[0] == "definitely lost: 0 bytes in 0 blocks" 73 | 74 | lost = [line.strip() for line in vg_output if line.strip().startswith("possibly lost:")] 75 | assert lost 76 | assert lost[0] == "possibly lost: 0 bytes in 0 blocks" 77 | 78 | class SubprocessTest(unittest.TestCase): 79 | process_options = [] 80 | binary_name = "" 81 | working_dir = None 82 | test_output_dir = None 83 | 84 | @classmethod 85 | def setUpClass(self): 86 | """setup method that starts up target instances using `self.process_options`""" 87 | self.temp_dirs = [] 88 | self.processes = [] 89 | assert self.binary_name, "you must override self.binary_name" 90 | assert self.working_dir, "set workign dir to os.path.dirname(__file__)" 91 | 92 | # make should update the executable if needed 93 | logging.info('running make') 94 | pipe = subprocess.Popen(['make', '-C', self.working_dir]) 95 | assert pipe.wait() == 0, "compile failed" 96 | 97 | test_output_dir = self.test_output_dir 98 | if os.path.exists(test_output_dir): 99 | logging.info('removing %s' % test_output_dir) 100 | pipe = subprocess.Popen(['rm', '-rf', test_output_dir]) 101 | pipe.wait() 102 | 103 | if not os.path.exists(test_output_dir): 104 | os.makedirs(test_output_dir) 105 | 106 | for options in self.process_options: 107 | logging.info(' '.join(options)) 108 | pipe = subprocess.Popen(options) 109 | self.processes.append(pipe) 110 | logging.debug('started process %s' % pipe.pid) 111 | 112 | self.wait_for('http://127.0.0.1:8080/', max_time=9) 113 | 114 | @classmethod 115 | def wait_for(self, url, max_time): 116 | # check up to 15 times till the endpoint specified is available waiting max_time 117 | step = max_time / float(15) 118 | http = tornado.httpclient.HTTPClient() 119 | for x in range(15): 120 | try: 121 | logging.info('try number %d for %s' % (x, url)) 122 | http.fetch(url, connect_timeout=.5, request_timeout=.5) 123 | return 124 | except: 125 | pass 126 | time.sleep(step) 127 | 128 | @classmethod 129 | def graceful_shutdown(self): 130 | try: 131 | http_fetch('/exit', dict()) 132 | except: 133 | # we never get a reply if this works correctly 134 | time.sleep(1) 135 | 136 | @classmethod 137 | def tearDownClass(self): 138 | """teardown method that cleans up child target instances, and removes their temporary data files""" 139 | logging.debug('teardown') 140 | try: 141 | self.graceful_shutdown() 142 | except: 143 | logging.exception('failed graceful shutdown') 144 | 145 | for process in self.processes: 146 | logging.info('%s' % process.poll()) 147 | if process.poll() is None: 148 | logging.debug('killing process %s' % process.pid) 149 | os.kill(process.pid, signal.SIGKILL) 150 | process.wait() 151 | for dirname in self.temp_dirs: 152 | logging.debug('cleaning up %s' % dirname) 153 | pipe = subprocess.Popen(['rm', '-rf', dirname]) 154 | pipe.wait() 155 | check_valgrind_output(os.path.join(self.test_output_dir, 'vg.out')) 156 | -------------------------------------------------------------------------------- /simpleattributes/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I. -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -g 8 | LIBS = -L. -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -L/usr/local/lib -lsimplehttp -levent -ljson -ltokyotyrant -ltokyocabinet -lz -lbz2 -lresolv -ldl -lpthread -lm -lc 9 | 10 | 11 | simpleattributes: simpleattributes.c 12 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 13 | 14 | install: 15 | /usr/bin/install -d $(TARGET)/bin 16 | /usr/bin/install simpleattributes $(TARGET)/bin 17 | 18 | clean: 19 | rm -rf *.o simpleattributes *.dSYM 20 | -------------------------------------------------------------------------------- /simplegeo/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I. -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -g 8 | LIBS = -L. -L$(LIBSIMPLEHTTP_INC) -L$(LIBEVENT)/lib -L/usr/local/lib -lsimplehttp -levent -ljson -ltokyotyrant -ltokyocabinet -lz -lbz2 -lresolv -ldl -lpthread -lm -lc 9 | 10 | simplegeo: simplegeo.c 11 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin 15 | /usr/bin/install simplegeo $(TARGET)/bin 16 | 17 | clean: 18 | rm -rf *.o simplegeo *.dSYM 19 | -------------------------------------------------------------------------------- /simplegeo/geo.lua: -------------------------------------------------------------------------------- 1 | -- geo.lua 2 | -- Geographic bounding box based search routines. 3 | -- Nate Folkman and Jay Ridgeway, 2009 4 | 5 | --[[ 6 | 7 | ttserver -ext geo.lua "casket.tct#idx=x:dec#idx=y:dec" 8 | 9 | tcrmgr ext localhost put 1 "lat,40.709092,lng,-73.921412" 10 | tcrmgr ext localhost put 2 "lat,40.738007,lng,-73.883625" 11 | tcrmgr ext localhost put 3 "lat,40.680177,lng,-73.959199" 12 | tcrmgr ext localhost put 4 "lat,40.739843,lng,-74.003047" 13 | tcrmgr ext localhost put 5 "lat,40.700616,lng,-73.917979" 14 | tcrmgr list -pv -sep ", " localhost 15 | 16 | tcrmgr ext localhost distance "40.738007,-73.883625" "40.700616,-73.917979" 17 | 18 | tcrmgr ext localhost geosearch "40.700616,-73.917979" 50 19 | 20 | --]] 21 | 22 | 23 | -- namespace for geo routines 24 | geo = {} 25 | 26 | 27 | -- 28 | -- constants 29 | -- 30 | 31 | 32 | -- kilometers in a mile (not used) 33 | geo.kminamile = 1.609344 34 | 35 | -- ~radius of the earth in miles 36 | geo.radius = 3958.75587 37 | 38 | -- miles per degree of lattitude 39 | geo.latDistance = 69.169144 40 | 41 | -- table of miles between successive longitudinal degrees 42 | geo.longDistance = {69.1691, 69.1586, 69.1270, 69.0743, 69.0006, 68.9059, 68.7902, 68.6536, 68.4960, 68.3175, 68.1183, 67.8983, 67.6576, 67.3963, 67.1145, 66.8122, 66.4896, 66.1467, 65.7837, 65.4006, 64.9976, 64.5749, 64.1324, 63.6704, 63.1890, 62.6884, 62.1687, 61.6300, 61.0726, 60.4965, 59.9020, 59.2893, 58.6586, 58.0099, 57.3436, 56.6598, 55.9588, 55.2407, 54.5058, 53.7543, 52.9864, 52.2023, 51.4024, 50.5868, 49.7558, 48.9097, 48.0486, 47.1729, 46.2829, 45.3787, 44.4607, 43.5292, 42.5844, 41.6267, 40.6563, 39.6735, 38.6786, 37.6719, 36.6537, 35.6244, 34.5842, 33.5335, 32.4726, 31.4018, 30.3214, 29.2318, 28.1333, 27.0262, 25.9109, 24.7877, 23.6570, 22.5190, 21.3742, 20.2229, 19.0654, 17.9021, 16.7333, 15.5595, 14.3809, 13.1979, 12.0109, 10.8203, 9.6264, 8.4295, 7.2301, 6.0284, 4.8249, 3.6200, 2.4139, 1.2072, 0.0000, 1.2072, 2.4139, 3.6200, 4.8249, 6.0284, 7.2301, 8.4295, 9.6264, 10.8203, 12.0109, 13.1979, 14.3809, 15.5595, 16.7333, 17.9021, 19.0654, 20.2229, 21.3742, 22.5190, 23.6570, 24.7877, 25.9109, 27.0262, 28.1333, 29.2318, 30.3214, 31.4018, 32.4726, 33.5335, 34.5842, 35.6244, 36.6537, 37.6719, 38.6786, 39.6735, 40.6563, 41.6267, 42.5844, 43.5292, 44.4607, 45.3787, 46.2829, 47.1729, 48.0486, 48.9097, 49.7558, 50.5868, 51.4024, 52.2023, 52.9864, 53.7543, 54.5058, 55.2407, 55.9588, 56.6598, 57.3436, 58.0099, 58.6586, 59.2893, 59.9020, 60.4965, 61.0726, 61.6300, 62.1687, 62.6884, 63.1890, 63.6704, 64.1324, 64.5749, 64.9976, 65.4006, 65.7837, 66.1467, 66.4896, 66.8122, 67.1145, 67.3963, 67.6576, 67.8983, 68.1183, 68.3175, 68.4960, 68.6536, 68.7902, 68.9059, 69.0006, 69.0743, 69.1270, 69.1586, 69.1691} 43 | 44 | 45 | -- 46 | -- functions 47 | -- 48 | 49 | 50 | -- convert from float to natural number 51 | function geo.tonatural(pt) 52 | return math.floor(pt * 10000 + 1800000) 53 | end 54 | 55 | 56 | -- miles between point a and point b 57 | function geo.distance(lat1, lng1, lat2, lng2) 58 | local rlat = lat1*math.pi/180; 59 | local rlng = lng1*math.pi/180; 60 | local rlat2 = lat2*math.pi/180; 61 | local rlng2 = lng2*math.pi/180; 62 | 63 | if (rlat == rlat2 and rlng == rlng2) then 64 | return 0 65 | else 66 | -- Spherical Law of Cosines 67 | return geo.radius*math.acos(math.sin(rlat)*math.sin(rlat2) 68 | +math.cos(rlng-rlng2)*math.cos(rlat)*math.cos(rlat2)) 69 | end 70 | end 71 | 72 | 73 | -- returns bounding box vertices 74 | function geo.box(lat, lng, miles) 75 | local lngD = miles/geo.longDistance[math.abs(string.format("%.0f", lat))+1] 76 | local latD = miles/geo.latDistance 77 | 78 | local llat = (lat+latD > 180 and 180 - lat+latD or lat+latD) 79 | local llng = (lng+lngD > 180 and 180 - lng+lngD or lng+lngD) 80 | local ulat = (lat-latD < -180 and 180 + lat-latD or lat-latD) 81 | local ulng = (lng-lngD < -180 and 180 + lng-lngD or lng-lngD) 82 | return ulat, ulng, llat, llng 83 | end 84 | 85 | 86 | -- 87 | -- tcrmgr routines 88 | -- 89 | 90 | 91 | - wrapper for geo.distance 92 | function distance(a, b) 93 | a = _split(a, ',') 94 | b = _split(b, ',') 95 | return geo.distance(a[1], a[2], b[1], b[2]) 96 | end 97 | 98 | 99 | -- puts a new entry into the db 100 | function put(id, data) 101 | local t = {} 102 | local cols = _split(data, ',') 103 | 104 | for i = 1, #cols, 2 do 105 | t[cols[i]] = cols[i+1] 106 | end 107 | if t["lat"] == nil or t["lng"] == nil then 108 | return "lat/lng required" 109 | end 110 | if t["x"] ~= nil or t["y"] ~= nil or t["distance"] ~= nil then 111 | return "x/y/distance are reserved" 112 | end 113 | 114 | local x = geo.tonatural(t["lat"]) 115 | local y = geo.tonatural(t["lng"]) 116 | table.insert(cols, "x") 117 | table.insert(cols, x) 118 | table.insert(cols, "y") 119 | table.insert(cols, y) 120 | 121 | return _put(id, table.concat(cols, '\0')) 122 | end 123 | 124 | 125 | - tcrmgr geo search 126 | function geosearch(pt, miles) 127 | pt = _split(pt, ",") 128 | local X = pt[1] 129 | local Y = pt[2] 130 | local lx,ly,ux,uy = geo.box(X, Y, miles) 131 | 132 | local args = {} 133 | table.insert(args, "addcond\0x\0NUMGT\0" .. geo.tonatural(lx)) 134 | table.insert(args, "addcond\0x\0NUMLT\0" .. geo.tonatural(ux)) 135 | table.insert(args, "addcond\0y\0NUMGT\0" .. geo.tonatural(ly)) 136 | table.insert(args, "addcond\0y\0NUMLT\0" .. geo.tonatural(uy)) 137 | table.insert(args, "get") 138 | 139 | local res = _misc("search", args) 140 | if res == nil then 141 | return "" 142 | end 143 | 144 | table.sort(res, function (a,b) 145 | local sa = _split(a) 146 | local sb = _split(b) 147 | local ta = {} 148 | local tb = {} 149 | for i=1, #sa, 2 do ta[sa[i]] = sa[i+1] end 150 | for i=1, #sb, 2 do tb[sb[i]] = sb[i+1] end 151 | 152 | local da = geo.distance(X, Y, ta["lat"], ta["lng"]) 153 | local db = geo.distance(X, Y, tb["lat"], tb["lng"]) 154 | return (da < db) 155 | end) 156 | 157 | local val = {} 158 | for i=1, #res do 159 | local ary = {} 160 | local row = _split(res[i]) 161 | table.remove(row, 1) 162 | table.insert(row, 1, "id") 163 | for i=1, #row, 2 do 164 | if row[i] == nil then row[i] = "id" end 165 | ary[row[i]] = row[i+1] 166 | end 167 | table.insert(row, "distance") 168 | table.insert(row, geo.distance(X, Y, ary["lat"], ary["lng"])) 169 | table.insert(val, table.concat(row,",")) 170 | end 171 | 172 | return table.concat(val, "\n") 173 | end 174 | -------------------------------------------------------------------------------- /simplegeo/test.sh: -------------------------------------------------------------------------------- 1 | curl '127.0.0.1:8080/distance?lat1=40.709092&lng1=-73.921412&lat2=40.739843698929995&lng2=-74.00304794311523' 2 | 3 | curl '127.0.0.1:8080/box?lat=40.709092&lng=-73.921412&miles=2' 4 | 5 | curl '127.0.0.1:8080/put?lat=40.709092&lng=-73.921412&id=1' 6 | curl '127.0.0.1:8080/put?lat=40.738007&lng=-73.883625&id=2' 7 | curl '127.0.0.1:8080/put?lat=40.680177&lng=-73.959199&id=3' 8 | curl '127.0.0.1:8080/put?lat=40.739843&lng=-74.003047&id=4' 9 | curl '127.0.0.1:8080/put?lat=40.700616&lng=-73.917979&id=5' 10 | 11 | curl '127.0.0.1:8080/search?lat=40.700616&lng=-73.917979&miles=5' 12 | -------------------------------------------------------------------------------- /simplehttp/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | 4 | CFLAGS = -I. -I$(LIBEVENT)/include -Wall -g 5 | LIBS = -L. -L$(LIBEVENT)/lib -levent -lm 6 | 7 | AR = ar 8 | AR_FLAGS = rc 9 | RANLIB = ranlib 10 | 11 | libsimplehttp.a: simplehttp.o async_simplehttp.o timer.o log.o util.o stat.o request.o options.o 12 | /bin/rm -f $@ 13 | $(AR) $(AR_FLAGS) $@ $^ 14 | $(RANLIB) $@ 15 | 16 | testserver: testserver.c 17 | $(CC) $(CFLAGS) -o $@ $< $(LIBS) -lsimplehttp 18 | 19 | all: libsimplehttp.a testserver 20 | 21 | install: 22 | /usr/bin/install -d $(TARGET)/lib/ 23 | /usr/bin/install -d $(TARGET)/include/simplehttp/ 24 | /usr/bin/install libsimplehttp.a $(TARGET)/lib/ 25 | /usr/bin/install simplehttp.h $(TARGET)/include/simplehttp/ 26 | /usr/bin/install simplehttp.h $(TARGET)/include/simplehttp/ 27 | /usr/bin/install queue.h $(TARGET)/include/simplehttp/ 28 | /usr/bin/install uthash.h $(TARGET)/include/simplehttp/ 29 | /usr/bin/install utlist.h $(TARGET)/include/simplehttp/ 30 | /usr/bin/install utstring.h $(TARGET)/include/simplehttp/ 31 | /usr/bin/install utarray.h $(TARGET)/include/simplehttp/ 32 | /usr/bin/install options.h $(TARGET)/include/simplehttp/ 33 | 34 | clean: 35 | rm -rf *.a *.o testserver *.dSYM 36 | -------------------------------------------------------------------------------- /simplehttp/async_simplehttp.h: -------------------------------------------------------------------------------- 1 | #ifndef _ASYNC_SIMPLEHTTP_H_ 2 | #define _ASYNC_SIMPLEHTTP_H_ 3 | 4 | #include 5 | #include 6 | 7 | #ifdef ASYNC_DEBUG 8 | #define AS_DEBUG(...) fprintf(stdout, __VA_ARGS__) 9 | #else 10 | #define AS_DEBUG(...) do {;} while (0) 11 | #endif 12 | 13 | #define ASYNC_PER_HOST_CONNECTION_LIMIT 10 14 | 15 | struct AsyncCallbackGroup; 16 | struct AsyncCallback; 17 | struct Connection; 18 | 19 | /* handling for callbacks */ 20 | struct AsyncCallback { 21 | simplehttp_ts start_ts; 22 | struct Connection *conn; 23 | struct evhttp_connection *evcon; 24 | struct evhttp_request *request; 25 | uint64_t id; 26 | void (*cb)(struct evhttp_request *req, void *); 27 | void *cb_arg; 28 | struct AsyncCallbackGroup *callback_group; 29 | TAILQ_ENTRY(AsyncCallback) entries; 30 | }; 31 | 32 | struct AsyncCallbackGroup { 33 | struct evhttp_request *original_request; 34 | struct evbuffer *evb; 35 | uint64_t id; 36 | unsigned int count; 37 | time_t connect_time; 38 | void (*finished_cb)(struct evhttp_request *, void *); 39 | void *finished_cb_arg; 40 | TAILQ_HEAD(, AsyncCallback) callback_list; 41 | }; 42 | 43 | struct Connection { 44 | struct evhttp_connection *evcon[ASYNC_PER_HOST_CONNECTION_LIMIT]; 45 | uint64_t next_evcon; 46 | char *address; 47 | int port; 48 | TAILQ_ENTRY(Connection) next; 49 | }; 50 | 51 | void finish_async_request(struct evhttp_request *req, void *cb_arg); 52 | struct evhttp_connection *get_connection(const char *address, int port, struct Connection **store_conn); 53 | void free_async_callback(struct AsyncCallback *callback); 54 | 55 | #endif 56 | -------------------------------------------------------------------------------- /simplehttp/log.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include "simplehttp.h" 7 | 8 | const char *simplehttp_method(struct evhttp_request *req) 9 | { 10 | const char *method; 11 | 12 | switch (req->type) { 13 | case EVHTTP_REQ_GET: 14 | method = "GET"; 15 | break; 16 | case EVHTTP_REQ_POST: 17 | method = "POST"; 18 | break; 19 | case EVHTTP_REQ_HEAD: 20 | method = "HEAD"; 21 | break; 22 | default: 23 | method = NULL; 24 | break; 25 | } 26 | 27 | return method; 28 | } 29 | 30 | void simplehttp_log(const char *host, struct evhttp_request *req, uint64_t req_time, const char *id, int display_post) 31 | { 32 | // NOTE: this is localtime not gmtime 33 | time_t now; 34 | struct tm *tm_now; 35 | char datetime_buf[64]; 36 | char code; 37 | const char *method; 38 | char *uri; 39 | int response_code; 40 | int type; 41 | 42 | time(&now); 43 | tm_now = localtime(&now); 44 | strftime(datetime_buf, 64, "%y%m%d %H:%M:%S", tm_now); 45 | 46 | if (req) { 47 | if (req->response_code >= 500 && req->response_code < 600) { 48 | code = 'E'; 49 | } else if (req->response_code >= 400 && req->response_code < 500) { 50 | code = 'W'; 51 | } else { 52 | code = 'I'; 53 | } 54 | response_code = req->response_code; 55 | method = simplehttp_method(req); 56 | uri = req->uri; 57 | type = req->type; 58 | } else { 59 | code = 'E'; 60 | response_code = 0; 61 | method = "NA"; 62 | uri = ""; 63 | type = -1; 64 | } 65 | 66 | fprintf(stdout, "[%c %s %s] %d %s %s%s", code, datetime_buf, id, response_code, method, host, uri); 67 | 68 | if (display_post && (type == EVHTTP_REQ_POST)) { 69 | if (req->input_buffer == NULL || EVBUFFER_DATA(req->input_buffer) == NULL) { 70 | fprintf(stdout, "input_buffer=%p, EVBUFFER_DATA=%p>", req->input_buffer, /* must be */ NULL); 71 | } else { 72 | fprintf(stdout, "?"); 73 | fwrite(EVBUFFER_DATA(req->input_buffer), EVBUFFER_LENGTH(req->input_buffer), 1, stdout); 74 | } 75 | } 76 | 77 | fprintf(stdout, " %.3fms\n", req_time / 1000.0); 78 | } 79 | -------------------------------------------------------------------------------- /simplehttp/options.h: -------------------------------------------------------------------------------- 1 | #ifndef _SIMPLEHTTP_OPTIONS_H 2 | #define _SIMPLEHTTP_OPTIONS_H 3 | 4 | enum required_option { 5 | OPT_REQUIRED = 1, 6 | OPT_OPTIONAL = 0 7 | }; 8 | 9 | int option_parse_command_line(int argc, char **argv); 10 | int option_define_int(const char *option_name, int required, int default_val, int *dest, int(*cb)(int value), const char *help); 11 | int option_define_str(const char *option_name, int required, const char *default_val, char **dest, int(*cb)(char *value), const char *help); 12 | int option_define_bool(const char *option_name, int required, int default_val, int *dest, int(*cb)(int value), const char *help); 13 | int option_define_char(const char *option_name, int required, char default_val, char *dest, int(*cb)(char value), const char *help); 14 | 15 | void option_help(); 16 | int option_get_int(const char *option_name); 17 | char *option_get_str(const char *option_name); 18 | char option_get_char(const char *option_name); 19 | 20 | void free_options(); 21 | 22 | #endif 23 | -------------------------------------------------------------------------------- /simplehttp/request.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "queue.h" 6 | #include "simplehttp.h" 7 | #include "async_simplehttp.h" 8 | #include "request.h" 9 | #include "stat.h" 10 | 11 | extern int simplehttp_logging; 12 | 13 | struct simplehttp_request *simplehttp_request_new(struct evhttp_request *req, uint64_t id) 14 | { 15 | struct simplehttp_request *s_req; 16 | simplehttp_ts start_ts; 17 | 18 | simplehttp_ts_get(&start_ts); 19 | s_req = malloc(sizeof(struct simplehttp_request)); 20 | s_req->req = req; 21 | s_req->start_ts = start_ts; 22 | s_req->id = id; 23 | s_req->async = 0; 24 | s_req->index = -1; 25 | TAILQ_INSERT_TAIL(&simplehttp_reqs, s_req, entries); 26 | 27 | AS_DEBUG("simplehttp_request_new (%p)\n", s_req); 28 | 29 | return s_req; 30 | } 31 | 32 | struct simplehttp_request *simplehttp_request_get(struct evhttp_request *req) 33 | { 34 | struct simplehttp_request *entry; 35 | 36 | TAILQ_FOREACH(entry, &simplehttp_reqs, entries) { 37 | if (req == entry->req) { 38 | return entry; 39 | } 40 | } 41 | 42 | return NULL; 43 | } 44 | 45 | uint64_t simplehttp_request_id(struct evhttp_request *req) 46 | { 47 | struct simplehttp_request *entry; 48 | 49 | entry = simplehttp_request_get(req); 50 | 51 | return entry ? entry->id : 0; 52 | } 53 | 54 | struct simplehttp_request *simplehttp_async_check(struct evhttp_request *req) 55 | { 56 | struct simplehttp_request *entry; 57 | 58 | entry = simplehttp_request_get(req); 59 | if (entry && entry->async) { 60 | return entry; 61 | } 62 | 63 | return NULL; 64 | } 65 | 66 | void simplehttp_async_enable(struct evhttp_request *req) 67 | { 68 | struct simplehttp_request *entry; 69 | 70 | if ((entry = simplehttp_request_get(req)) != NULL) { 71 | AS_DEBUG("simplehttp_async_enable (%p)\n", req); 72 | entry->async = 1; 73 | } 74 | } 75 | 76 | void simplehttp_request_finish(struct evhttp_request *req, struct simplehttp_request *s_req) 77 | { 78 | simplehttp_ts end_ts; 79 | uint64_t req_time; 80 | char id_buf[64]; 81 | 82 | AS_DEBUG("simplehttp_request_finish (%p, %p)\n", req, s_req); 83 | 84 | simplehttp_ts_get(&end_ts); 85 | req_time = simplehttp_ts_diff(s_req->start_ts, end_ts); 86 | 87 | if (s_req->index != -1) { 88 | simplehttp_stats_store(s_req->index, req_time); 89 | } 90 | 91 | if (simplehttp_logging) { 92 | sprintf(id_buf, "%"PRIu64, s_req->id); 93 | simplehttp_log("", req, req_time, id_buf, 1); 94 | } 95 | 96 | AS_DEBUG("\n"); 97 | 98 | TAILQ_REMOVE(&simplehttp_reqs, s_req, entries); 99 | free(s_req); 100 | } 101 | 102 | void simplehttp_async_finish(struct evhttp_request *req) 103 | { 104 | struct simplehttp_request *entry; 105 | 106 | AS_DEBUG("simplehttp_async_finish (%p)\n", req); 107 | if ((entry = simplehttp_async_check(req))) { 108 | AS_DEBUG("simplehttp_async_check found (%p)\n", entry); 109 | simplehttp_request_finish(req, entry); 110 | } 111 | } 112 | 113 | 114 | int get_argument_format(struct evkeyvalq *args) 115 | { 116 | int format_code = json_format; 117 | const char *format = evhttp_find_header(args, "format"); 118 | if (format && !strncmp(format, "txt", 3)) { 119 | format_code = txt_format; 120 | } 121 | return format_code; 122 | } 123 | 124 | int get_int_argument(struct evkeyvalq *args, const char *key, int default_value) 125 | { 126 | const char *tmp; 127 | if (!key) { 128 | return default_value; 129 | } 130 | tmp = evhttp_find_header(args, key); 131 | if (tmp) { 132 | return atoi(tmp); 133 | } 134 | return default_value; 135 | } 136 | 137 | double get_double_argument(struct evkeyvalq *args, const char *key, double default_value) 138 | { 139 | const char *tmp; 140 | if (!key) { 141 | return default_value; 142 | } 143 | tmp = evhttp_find_header(args, key); 144 | if (tmp) { 145 | return atof(tmp); 146 | } 147 | return default_value; 148 | } 149 | 150 | -------------------------------------------------------------------------------- /simplehttp/request.h: -------------------------------------------------------------------------------- 1 | #ifndef _REQUEST_H 2 | #define _REQUEST_H 3 | 4 | struct simplehttp_request { 5 | struct evhttp_request *req; 6 | simplehttp_ts start_ts; 7 | uint64_t id; 8 | int index; 9 | int async; 10 | TAILQ_ENTRY(simplehttp_request) entries; 11 | }; 12 | TAILQ_HEAD(, simplehttp_request) simplehttp_reqs; 13 | 14 | struct simplehttp_request *simplehttp_request_new(struct evhttp_request *req, uint64_t id); 15 | struct simplehttp_request *simplehttp_request_get(struct evhttp_request *req); 16 | struct simplehttp_request *simplehttp_async_check(struct evhttp_request *req); 17 | void simplehttp_request_finish(struct evhttp_request *req, struct simplehttp_request *s_req);; 18 | 19 | #endif 20 | -------------------------------------------------------------------------------- /simplehttp/simplehttp.h: -------------------------------------------------------------------------------- 1 | #ifndef _SIMPLEHTTP_H 2 | #define _SIMPLEHTTP_H 3 | 4 | #include "queue.h" 5 | #include "options.h" 6 | #include 7 | #include 8 | 9 | #define SIMPLEHTTP_VERSION "0.1.3" 10 | 11 | #if _POSIX_TIMERS > 0 12 | 13 | typedef struct timespec simplehttp_ts; 14 | 15 | void simplehttp_ts_get(struct timespec *ts); 16 | unsigned int simplehttp_ts_diff(struct timespec start, struct timespec end); 17 | 18 | #else 19 | 20 | typedef struct timeval simplehttp_ts; 21 | 22 | void simplehttp_ts_get(struct timeval *ts); 23 | unsigned int simplehttp_ts_diff(struct timeval start, struct timeval end); 24 | 25 | #endif 26 | 27 | struct simplehttp_stats { 28 | uint64_t requests; 29 | uint64_t *stats_counts; 30 | uint64_t *average_requests; 31 | uint64_t *ninety_five_percents; 32 | char **stats_labels; 33 | int callback_count; 34 | }; 35 | 36 | void simplehttp_init(); 37 | int simplehttp_main(); 38 | int simplehttp_listen(); 39 | void simplehttp_run(); 40 | void simplehttp_free(); 41 | void simplehttp_set_cb(const char *path, void (*cb)(struct evhttp_request *, struct evbuffer *, void *), void *ctx); 42 | 43 | uint64_t simplehttp_request_id(struct evhttp_request *req); 44 | void simplehttp_async_enable(struct evhttp_request *req); 45 | void simplehttp_async_finish(struct evhttp_request *req); 46 | 47 | void simplehttp_log(const char *host, struct evhttp_request *req, uint64_t req_time, const char *id, int display_post); 48 | 49 | char *simplehttp_strnstr(const char *s, const char *find, size_t slen); 50 | uint64_t ninety_five_percent(int64_t *int_array, int length); 51 | struct simplehttp_stats *simplehttp_stats_new(); 52 | void simplehttp_stats_get(struct simplehttp_stats *st); 53 | void simplehttp_stats_free(struct simplehttp_stats *st); 54 | uint64_t ninety_five_percent(int64_t *int_array, int length); 55 | char **simplehttp_callback_names(); 56 | 57 | struct AsyncCallbackGroup; 58 | struct AsyncCallback; 59 | struct RequestHeader { 60 | char *name; 61 | char *value; 62 | struct RequestHeader *next; 63 | }; 64 | 65 | /* start a new callback_group. memory will be freed after a call to 66 | release_callback_group or when all the callbacks have been run */ 67 | struct AsyncCallbackGroup *new_async_callback_group(struct evhttp_request *req, 68 | void (*finished_cb)(struct evhttp_request *, void *), void *finished_cb_arg); 69 | /* create a new AsyncCallback. delegation of memory for this callback 70 | will be passed to callback_group */ 71 | int new_async_callback(struct AsyncCallbackGroup *callback_group, const char *address, int port, 72 | const char *path, void (*cb)(struct evhttp_request *, void *), void *cb_arg); 73 | struct AsyncCallback *new_async_request(const char *address, int port, const char *path, 74 | void (*cb)(struct evhttp_request *, void *), void *cb_arg); 75 | struct AsyncCallback *new_async_request_with_body(int request_method, const char *address, int port, const char *path, 76 | struct RequestHeader *header_list, const char *body, void (*cb)(struct evhttp_request *, void *), void *cb_arg); 77 | void free_async_callback_group(struct AsyncCallbackGroup *callback_group); 78 | void init_async_connection_pool(int enable_request_logging); 79 | void free_async_connection_pool(); 80 | 81 | enum response_formats {json_format, txt_format}; 82 | int get_argument_format(struct evkeyvalq *args); 83 | int get_int_argument(struct evkeyvalq *args, const char *key, int default_value); 84 | double get_double_argument(struct evkeyvalq *args, const char *key, double default_value); 85 | 86 | void define_simplehttp_options(); 87 | 88 | int simplehttp_parse_url(const char *endpoint, size_t endpoint_len, char **address, int *port, char **path); 89 | char *simplehttp_encode_uri(const char *uri); 90 | 91 | #endif 92 | -------------------------------------------------------------------------------- /simplehttp/stat.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include "stat.h" 5 | #include "simplehttp.h" 6 | 7 | static int64_t *stats = NULL; 8 | static int *stats_idx = NULL; 9 | static uint64_t *stats_counts = NULL; 10 | 11 | extern int callback_count; 12 | extern uint64_t request_count; 13 | 14 | void simplehttp_stats_store(int index, uint64_t val) 15 | { 16 | stats[(index * STAT_WINDOW) + stats_idx[index]] = val; 17 | stats_idx[index]++; 18 | stats_counts[index]++; 19 | 20 | if (stats_idx[index] >= STAT_WINDOW) { 21 | stats_idx[index] = 0; 22 | } 23 | } 24 | 25 | void simplehttp_stats_init() 26 | { 27 | int i; 28 | stats = malloc(STAT_WINDOW * callback_count * sizeof(int64_t)); 29 | for (i = 0; i < (STAT_WINDOW * callback_count); i++) { 30 | stats[i] = -1; 31 | } 32 | stats_idx = calloc(1, callback_count * sizeof(int)); 33 | stats_counts = calloc(1, callback_count * sizeof(uint64_t)); 34 | } 35 | 36 | void simplehttp_stats_destruct() 37 | { 38 | free(stats); 39 | free(stats_idx); 40 | free(stats_counts); 41 | } 42 | 43 | struct simplehttp_stats *simplehttp_stats_new() 44 | { 45 | struct simplehttp_stats *st; 46 | 47 | st = malloc(sizeof(struct simplehttp_stats)); 48 | 49 | return st; 50 | } 51 | 52 | void simplehttp_stats_free(struct simplehttp_stats *st) 53 | { 54 | int i; 55 | 56 | if (st) { 57 | if (st->stats_counts) { 58 | free(st->stats_counts); 59 | } 60 | 61 | if (st->average_requests) { 62 | free(st->average_requests); 63 | } 64 | 65 | if (st->ninety_five_percents) { 66 | free(st->ninety_five_percents); 67 | } 68 | 69 | if (st->stats_labels) { 70 | for (i = 0; i < st->callback_count; i++) { 71 | free(st->stats_labels[i]); 72 | } 73 | free(st->stats_labels); 74 | } 75 | 76 | free(st); 77 | } 78 | } 79 | 80 | void simplehttp_stats_get(struct simplehttp_stats *st) 81 | { 82 | uint64_t request_total; 83 | int i, j, c, request_array_end; 84 | 85 | st->requests = request_count; 86 | st->callback_count = callback_count; 87 | st->stats_counts = malloc(callback_count * sizeof(uint64_t)); 88 | memcpy(st->stats_counts, stats_counts, callback_count * sizeof(uint64_t)); 89 | st->average_requests = calloc(1, callback_count * sizeof(uint64_t)); 90 | st->ninety_five_percents = calloc(1, callback_count * sizeof(uint64_t)); 91 | st->stats_labels = simplehttp_callback_names(); 92 | 93 | for (i = 0; i < callback_count; i++) { 94 | request_total = 0; 95 | for (j = (i * STAT_WINDOW), request_array_end = j + STAT_WINDOW, c = 0; 96 | (j < request_array_end) && (stats[j] != -1); j++, c++) { 97 | request_total += stats[j]; 98 | } 99 | if (c) { 100 | st->average_requests[i] = request_total / c; 101 | st->ninety_five_percents[i] = ninety_five_percent(stats + (i * STAT_WINDOW), c); 102 | } 103 | } 104 | } 105 | -------------------------------------------------------------------------------- /simplehttp/stat.h: -------------------------------------------------------------------------------- 1 | #ifndef _STAT_H 2 | #define _STAT_H 3 | 4 | #define STAT_WINDOW 1000 5 | 6 | void simplehttp_stats_store(int index, uint64_t val); 7 | void simplehttp_stats_init(); 8 | void simplehttp_stats_destruct(); 9 | 10 | #endif 11 | -------------------------------------------------------------------------------- /simplehttp/testserver.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | #define VERSION "0.1" 5 | 6 | void cb(struct evhttp_request *req, struct evbuffer *evb, void *ctx) 7 | { 8 | evbuffer_add_printf(evb, "Hello World!\n%s\n", req->uri); 9 | evhttp_send_reply(req, HTTP_OK, "OK", evb); 10 | } 11 | 12 | int version_cb(int value) 13 | { 14 | fprintf(stdout, "Version: %s\n", VERSION); 15 | return 0; 16 | } 17 | 18 | int main(int argc, char **argv) 19 | { 20 | define_simplehttp_options(); 21 | option_define_bool("version", OPT_OPTIONAL, 0, NULL, version_cb, VERSION); 22 | 23 | if (!option_parse_command_line(argc, argv)) { 24 | return 1; 25 | } 26 | 27 | simplehttp_init(); 28 | simplehttp_set_cb("/ass*", cb, NULL); 29 | simplehttp_set_cb("/foo*", cb, NULL); 30 | simplehttp_set_cb("/bar*", cb, NULL); 31 | simplehttp_main(); 32 | free_options(); 33 | return 0; 34 | } 35 | -------------------------------------------------------------------------------- /simplehttp/timer.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | #if _POSIX_TIMERS > 0 5 | 6 | void simplehttp_ts_get(struct timespec *ts) 7 | { 8 | clock_gettime(CLOCK_REALTIME, ts); 9 | } 10 | 11 | unsigned int simplehttp_ts_diff(struct timespec start, struct timespec end) 12 | { 13 | struct timespec temp; 14 | 15 | if ((end.tv_nsec - start.tv_nsec) < 0) { 16 | temp.tv_sec = end.tv_sec - start.tv_sec - 1; 17 | temp.tv_nsec = 1000000000 + end.tv_nsec - start.tv_nsec; 18 | } else { 19 | temp.tv_sec = end.tv_sec - start.tv_sec; 20 | temp.tv_nsec = end.tv_nsec - start.tv_nsec; 21 | } 22 | 23 | // return usec as int 24 | return (temp.tv_sec * 1000000) + (temp.tv_nsec / 1000); 25 | } 26 | 27 | #else 28 | 29 | void simplehttp_ts_get(struct timeval *ts) 30 | { 31 | gettimeofday(ts, NULL); 32 | } 33 | 34 | unsigned int simplehttp_ts_diff(struct timeval start, struct timeval end) 35 | { 36 | struct timeval temp; 37 | 38 | if ((end.tv_usec - start.tv_usec) < 0) { 39 | temp.tv_sec = end.tv_sec - start.tv_sec - 1; 40 | temp.tv_usec = 1000000 + end.tv_usec - start.tv_usec; 41 | } else { 42 | temp.tv_sec = end.tv_sec - start.tv_sec; 43 | temp.tv_usec = end.tv_usec - start.tv_usec; 44 | } 45 | 46 | // return usec as int 47 | return (temp.tv_sec * 1000000) + temp.tv_usec; 48 | } 49 | 50 | #endif 51 | -------------------------------------------------------------------------------- /simplehttp/util.c: -------------------------------------------------------------------------------- 1 | #define _GNU_SOURCE // for strndup() 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include "simplehttp.h" 9 | 10 | static const char uri_chars[256] = { 11 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13 | 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 14 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 15 | /* 64 */ 16 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 17 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 18 | 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 19 | 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 20 | /* 128 */ 21 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 23 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 24 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 25 | /* 192 */ 26 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 27 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 28 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 29 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 30 | }; 31 | 32 | int int_cmp(const void *a, const void *b) 33 | { 34 | const uint64_t *ia = (const uint64_t *)a; 35 | const uint64_t *ib = (const uint64_t *)b; 36 | 37 | return *ia - *ib; 38 | } 39 | 40 | uint64_t ninety_five_percent(int64_t *int_array, int length) 41 | { 42 | uint64_t value; 43 | int64_t *sorted_requests; 44 | int index_of_95; 45 | 46 | sorted_requests = calloc(1, length * sizeof(int64_t)); 47 | memcpy(sorted_requests, int_array, length * sizeof(int64_t)); 48 | qsort(sorted_requests, length, sizeof(int64_t), int_cmp); 49 | index_of_95 = (int)ceil(((95.0 / 100.0) * length) + 0.5); 50 | if (index_of_95 >= length) { 51 | index_of_95 = length - 1; 52 | } 53 | value = sorted_requests[index_of_95]; 54 | free(sorted_requests); 55 | 56 | return value; 57 | } 58 | 59 | int simplehttp_parse_url(const char *endpoint, size_t endpoint_len, char **address, int *port, char **path) 60 | { 61 | // parse out address, port, path 62 | const char *address_p, *path_p, *tmp_pointer, *tmp_port; 63 | size_t address_len; 64 | size_t path_len; 65 | 66 | // http://0/ 67 | if (endpoint_len < 9) { 68 | return 0; 69 | } 70 | 71 | // find the first / 72 | address_p = strchr(endpoint, '/'); // http:/ 73 | if (!address_p) { 74 | return 0; 75 | } 76 | 77 | // move past the two slashes 78 | address_p += 2; 79 | 80 | // check for the colon specifying a port 81 | tmp_pointer = strchr(address_p, ':'); 82 | path_p = strchr(address_p, '/'); 83 | 84 | if (!path_p) { 85 | return 0; 86 | } 87 | 88 | if (tmp_pointer) { 89 | address_len = tmp_pointer - address_p; 90 | tmp_port = address_p + address_len + 1; 91 | // atoi() will stop at the first non-digit which will be '/' 92 | *port = atoi(tmp_port); 93 | } else { 94 | address_len = path_p - address_p; 95 | *port = 80; 96 | } 97 | 98 | path_len = (endpoint + endpoint_len) - path_p; 99 | *address = strndup(address_p, address_len); 100 | *path = strndup(path_p, path_len); 101 | 102 | return 1; 103 | } 104 | 105 | // libevent's built in encoder does not encode spaces, this does 106 | char *simplehttp_encode_uri(const char *uri) 107 | { 108 | char *buf, *cur; 109 | char *p; 110 | 111 | // allocate 3x the memory 112 | buf = malloc(strlen(uri) * 3 + 1); 113 | cur = buf; 114 | for (p = (char *)uri; *p != '\0'; p++) { 115 | if (uri_chars[(unsigned char)(*p)]) { 116 | *cur++ = *p; 117 | } else { 118 | sprintf(cur, "%%%02X", (unsigned char)(*p)); 119 | cur += 3; 120 | } 121 | } 122 | *cur = '\0'; 123 | 124 | return buf; 125 | } 126 | 127 | /** 128 | * Find the first occurrence of find in s, where the search is limited to the 129 | * first slen characters of s. 130 | */ 131 | char *simplehttp_strnstr(const char *s, const char *find, size_t slen) 132 | { 133 | char c, sc; 134 | size_t len; 135 | 136 | // exit if the end of the strung 137 | if ((c = *find++) != '\0') { 138 | 139 | // get the length of the string to find, shortens as we loop 140 | len = strlen(find); 141 | 142 | // compare the passed and find string at the current position. we would 143 | // have iterated up to and including the first char of the find string in 144 | // the search string, now compare from there, if no match loop again 145 | do { 146 | 147 | // go until we get the starting char of the find string or until 148 | // we run out of chars to search either by count or end of string 149 | do { 150 | if (slen-- < 1 || (sc = *s++) == '\0') { 151 | return NULL; 152 | } 153 | } while (sc != c); 154 | 155 | // no more chars to search then exit 156 | if (len > slen) { 157 | return NULL; 158 | } 159 | 160 | } while (strncmp(s, find, len) != 0); 161 | 162 | // when find string matches go one position back to be at start index 163 | s--; 164 | } 165 | 166 | // return pointer to start pos of found string 167 | return (char *)s; 168 | } 169 | -------------------------------------------------------------------------------- /simplehttp/utstring.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2008-2010, Troy D. Hanson http://uthash.sourceforge.net 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | 8 | * Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 11 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 12 | IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 13 | TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 14 | PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER 15 | OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 16 | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 17 | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 18 | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 19 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 20 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 21 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | */ 23 | 24 | /* a dynamic string implementation using macros 25 | * see http://uthash.sourceforge.net/utstring 26 | */ 27 | #ifndef UTSTRING_H 28 | #define UTSTRING_H 29 | 30 | #define UTSTRING_VERSION 1.9.1 31 | 32 | #ifdef __GNUC__ 33 | #define _UNUSED_ __attribute__ ((__unused__)) 34 | #else 35 | #define _UNUSED_ 36 | #endif 37 | 38 | #include 39 | #include 40 | #include 41 | #define oom() exit(-1) 42 | 43 | typedef struct { 44 | char *d; 45 | size_t n; /* allocd size */ 46 | size_t i; /* index of first unused byte */ 47 | } UT_string; 48 | 49 | #define utstring_reserve(s,amt) \ 50 | do { \ 51 | if (((s)->n - (s)->i) < (size_t)(amt)) { \ 52 | (s)->d = (char*)realloc((s)->d, (s)->n + amt); \ 53 | if ((s)->d == NULL) oom(); \ 54 | (s)->n += amt; \ 55 | } \ 56 | } while(0) 57 | 58 | #define utstring_init(s) \ 59 | do { \ 60 | (s)->n = 0; (s)->i = 0; (s)->d = NULL; \ 61 | utstring_reserve(s,100); \ 62 | } while(0) 63 | 64 | #define utstring_done(s) \ 65 | do { \ 66 | if ((s)->d != NULL) free((s)->d); \ 67 | (s)->n = 0; \ 68 | } while(0) 69 | 70 | #define utstring_free(s) \ 71 | do { \ 72 | utstring_done(s); \ 73 | free(s); \ 74 | } while(0) 75 | 76 | #define utstring_new(s) \ 77 | do { \ 78 | s = (UT_string*)calloc(sizeof(UT_string),1); \ 79 | if (!s) oom(); \ 80 | utstring_init(s); \ 81 | } while(0) 82 | 83 | #define utstring_clear(s) \ 84 | do { \ 85 | (s)->i = 0; \ 86 | } while(0) 87 | 88 | #define utstring_bincpy(s,b,l) \ 89 | do { \ 90 | utstring_reserve(s,(l)+1); \ 91 | if (l) memcpy(&(s)->d[(s)->i], b, l); \ 92 | s->i += l; \ 93 | s->d[s->i]='\0'; \ 94 | } while(0) 95 | 96 | #define utstring_concat(dst,src) \ 97 | do { \ 98 | utstring_reserve(dst,(src->i)+1); \ 99 | if (src->i) memcpy(&(dst)->d[(dst)->i], src->d, src->i); \ 100 | dst->i += src->i; \ 101 | dst->d[dst->i]='\0'; \ 102 | } while(0) 103 | 104 | #define utstring_len(s) ((unsigned)((s)->i)) 105 | 106 | #define utstring_body(s) ((s)->d) 107 | 108 | _UNUSED_ static void utstring_printf_va(UT_string *s, const char *fmt, va_list ap) { 109 | int n; 110 | va_list cp; 111 | while (1) { 112 | #ifdef _WIN32 113 | cp = ap; 114 | #else 115 | va_copy(cp, ap); 116 | #endif 117 | n = vsnprintf (&s->d[s->i], s->n-s->i, fmt, cp); 118 | va_end(cp); 119 | 120 | if ((n > -1) && (n < (int)(s->n-s->i))) { 121 | s->i += n; 122 | return; 123 | } 124 | 125 | /* Else try again with more space. */ 126 | if (n > -1) utstring_reserve(s,n+1); /* exact */ 127 | else utstring_reserve(s,(s->n)*2); /* 2x */ 128 | } 129 | } 130 | _UNUSED_ static void utstring_printf(UT_string *s, const char *fmt, ...) { 131 | va_list ap; 132 | va_start(ap,fmt); 133 | utstring_printf_va(s,fmt,ap); 134 | va_end(ap); 135 | } 136 | 137 | #endif /* UTSTRING_H */ 138 | -------------------------------------------------------------------------------- /simpleleveldb/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | LIBLEVELDB ?= /usr/local 7 | 8 | CFLAGS = -I. -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -I$(LIBLEVELDB)/include -Wall -g -O2 9 | LIBS = -L. -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -L/usr/local/lib -L$(LIBLEVELDB)/lib -levent -ljson -lsimplehttp -lleveldb -lm -lstdc++ -lsnappy -lpthread 10 | AR = ar 11 | AR_FLAGS = rc 12 | RANLIB = ranlib 13 | 14 | TARGETS = simpleleveldb leveldb_to_csv csv_to_leveldb 15 | 16 | SOURCES_simpleleveldb = simpleleveldb.c str_list_set.c 17 | SOURCES_leveldb_to_csv = leveldb_to_csv.c 18 | SOURCES_csv_to_leveldb = csv_to_leveldb.c 19 | 20 | all: $(TARGETS) 21 | 22 | -include $(TARGETS:%=%.deps) 23 | 24 | $(TARGETS): %: %.c 25 | $(CC) $(CFLAGS) -MM -MT $@ -MF $@.deps $(SOURCES_$@) 26 | $(CC) $(CFLAGS) -o $@ $(SOURCES_$@) $(LIBS) 27 | 28 | install: 29 | /usr/bin/install -d $(TARGET)/bin/ 30 | /usr/bin/install simpleleveldb $(TARGET)/bin/ 31 | /usr/bin/install leveldb_to_csv $(TARGET)/bin/ 32 | /usr/bin/install csv_to_leveldb $(TARGET)/bin/ 33 | 34 | clean: 35 | rm -rf *.a *.o *.deps *.dSYM $(TARGETS) 36 | -------------------------------------------------------------------------------- /simpleleveldb/README.md: -------------------------------------------------------------------------------- 1 | simpleleveldb 2 | ============= 3 | 4 | HTTP based leveldb server. 5 | 6 | NOTE: there is a known issue with long-running leveldb processes resulting 7 | in unbounded memory growth of the process. This is due to the fact that 8 | the MANIFEST file is never compacted. We've provided a `/hup` handler that 9 | will close and re-open the database (twice) in order to trigger compaction 10 | and removal of the MANIFEST. 11 | (see [issue #70](https://github.com/bitly/simplehttp/issues/70)) 12 | 13 | Building 14 | -------- 15 | 16 | you need a copy of leveldb 17 | 18 | cd /tmp/ 19 | git clone https://code.google.com/p/leveldb/ 20 | cd leveldb 21 | make 22 | cp libleveldb.a /usr/local/lib/ 23 | cp -r include/leveldb /usr/local/include/ 24 | 25 | then in simpleleveldb 26 | 27 | env LIBLEVELDB=/usr/local make 28 | make install 29 | 30 | OPTIONS 31 | ------- 32 | 33 | ``` 34 | --address= address to listen on 35 | default: 0.0.0.0 36 | --block-size= block size 37 | default: 4096 38 | --compression=True|False snappy compression 39 | --create-db-if-missing=True|False Create leveldb file if missing 40 | --daemon daemonize process 41 | --db-file= path to leveldb file 42 | --enable-logging request logging 43 | --error-if-db-exists Error out if leveldb file exists 44 | --group= run as this group 45 | --help list usage 46 | --leveldb-max-open-files= leveldb max open files 47 | default: 4096 48 | --paranoid-checks=True|False leveldb paranoid checks 49 | --port= port to listen on 50 | default: 8080 51 | --root= chdir and run from this directory 52 | --user= run as this user 53 | --version 0.1 54 | --write-buffer-size= write buffer size 55 | default: 4194304 56 | ``` 57 | 58 | API endpoints: 59 | 60 | * /get 61 | 62 | parameters: `key` 63 | 64 | * /mget 65 | 66 | parameters: `key` (multiple) 67 | 68 | * /fwmatch 69 | 70 | parameters: `key`, `limit` (default 500) 71 | 72 | * /range_match 73 | 74 | parameters: `start`, `end`, `limit` (default 500) 75 | 76 | * /put 77 | 78 | parameters: `key`, `value` 79 | 80 | Note: `value` can also be specified as the raw POST body content 81 | 82 | * /mput 83 | 84 | Note: takes separator-separated key/value pairs in separate lines in the POST body 85 | 86 | * /list_append 87 | 88 | parameters: `key`, `value` (multiple) 89 | 90 | * /list_prepend 91 | 92 | parameters: `key`, `value` (multiple) 93 | 94 | * /list_remove 95 | 96 | parameters: `key`, `value` (multiple) 97 | 98 | * /list_pop 99 | 100 | parameters: `key`, `position` (default 0), `count` (default 1) 101 | 102 | Note: a negative position does a reverse count from the end of the list 103 | 104 | * /set_add 105 | 106 | parameters: `key`, `value` (multiple) 107 | 108 | * /set_remove 109 | 110 | parameters: `key`, `value` (multiple) 111 | 112 | * /set_pop 113 | 114 | parameters: `key`, `count` (default 1) 115 | 116 | * /dump_csv 117 | 118 | parameters: `key` (optional) 119 | 120 | Note: dumps the entire database starting at `key` or else at the beginning, in txt format csv 121 | 122 | * /del 123 | 124 | parameters: `key` 125 | 126 | * /stats 127 | 128 | * /exit 129 | 130 | Note: causes the process to exit 131 | 132 | All endpoints take a `format` parameter which affects whether error conditions 133 | are represented by the HTTP response code (format=txt) or by the "status_code" 134 | member of the json result (format=json) (in which case the HTTP response code 135 | is always 200 if the server isn't broken). `format` also affects the output 136 | data for all endpoints except /put, /mput, /exit, /del, and /dump_csv. 137 | 138 | Output data in json format is under the "data" member of the root json object, 139 | sometimes as a string (/get), sometimes as an array (/mget), sometimes as an 140 | object with some metadata (/list_remove). 141 | 142 | Most endpoints take a `separator` parameter which defaults to "," (but can be 143 | set to any single character), which affects txt format output data. It also 144 | affects the deserialization and serialization of lists and sets stored in the 145 | db, and the input parsing of /mput. 146 | 147 | All list and set endpoints take a `return_data` parameter; set it to 1 to additionally 148 | return the new value of the list or set. However, this doesn't work for list_pop 149 | or set_pop endpoints in txt format. 150 | 151 | Utilities 152 | --------- 153 | 154 | * `leveldb_to_csv` is a utility to dump a leveldb database into csv format. It takes the same parameters as simpleleveldb plus an optional `--output-file` and `--output_deliminator` (or run `--help` for more info) 155 | * `csv_to_leveldb` loads from a csv into a leveldb database. It takes the same parameters as simpleleveldb plus an optional `--input-file` and `--input_deliminator` (or run `--help` for more info) 156 | -------------------------------------------------------------------------------- /simpleleveldb/http-internal.h: -------------------------------------------------------------------------------- 1 | /* 2 | NOTE: this is included copyied from libevent-1.4.13 with the addition 3 | of a definition for socklen_t so that we can give statistics on the 4 | client connection outgoing buffer size 5 | */ 6 | 7 | /* 8 | * Copyright 2001 Niels Provos 9 | * All rights reserved. 10 | * 11 | * This header file contains definitions for dealing with HTTP requests 12 | * that are internal to libevent. As user of the library, you should not 13 | * need to know about these. 14 | */ 15 | 16 | #ifndef _HTTP_H_ 17 | #define _HTTP_H_ 18 | 19 | #define HTTP_CONNECT_TIMEOUT 45 20 | #define HTTP_WRITE_TIMEOUT 50 21 | #define HTTP_READ_TIMEOUT 50 22 | 23 | #define HTTP_PREFIX "http://" 24 | #define HTTP_DEFAULTPORT 80 25 | #define socklen_t unsigned int 26 | 27 | enum message_read_status { 28 | ALL_DATA_READ = 1, 29 | MORE_DATA_EXPECTED = 0, 30 | DATA_CORRUPTED = -1, 31 | REQUEST_CANCELED = -2 32 | }; 33 | 34 | enum evhttp_connection_error { 35 | EVCON_HTTP_TIMEOUT, 36 | EVCON_HTTP_EOF, 37 | EVCON_HTTP_INVALID_HEADER 38 | }; 39 | 40 | struct evbuffer; 41 | struct addrinfo; 42 | struct evhttp_request; 43 | 44 | /* A stupid connection object - maybe make this a bufferevent later */ 45 | 46 | enum evhttp_connection_state { 47 | EVCON_DISCONNECTED, /**< not currently connected not trying either*/ 48 | EVCON_CONNECTING, /**< tries to currently connect */ 49 | EVCON_IDLE, /**< connection is established */ 50 | EVCON_READING_FIRSTLINE,/**< reading Request-Line (incoming conn) or 51 | **< Status-Line (outgoing conn) */ 52 | EVCON_READING_HEADERS, /**< reading request/response headers */ 53 | EVCON_READING_BODY, /**< reading request/response body */ 54 | EVCON_READING_TRAILER, /**< reading request/response chunked trailer */ 55 | EVCON_WRITING /**< writing request/response headers/body */ 56 | }; 57 | 58 | struct event_base; 59 | 60 | struct evhttp_connection { 61 | /* we use tailq only if they were created for an http server */ 62 | TAILQ_ENTRY(evhttp_connection) (next); 63 | 64 | int fd; 65 | struct event ev; 66 | struct event close_ev; 67 | struct evbuffer *input_buffer; 68 | struct evbuffer *output_buffer; 69 | 70 | char *bind_address; /* address to use for binding the src */ 71 | u_short bind_port; /* local port for binding the src */ 72 | 73 | char *address; /* address to connect to */ 74 | u_short port; 75 | 76 | int flags; 77 | #define EVHTTP_CON_INCOMING 0x0001 /* only one request on it ever */ 78 | #define EVHTTP_CON_OUTGOING 0x0002 /* multiple requests possible */ 79 | #define EVHTTP_CON_CLOSEDETECT 0x0004 /* detecting if persistent close */ 80 | 81 | int timeout; /* timeout in seconds for events */ 82 | int retry_cnt; /* retry count */ 83 | int retry_max; /* maximum number of retries */ 84 | 85 | enum evhttp_connection_state state; 86 | 87 | /* for server connections, the http server they are connected with */ 88 | struct evhttp *http_server; 89 | 90 | TAILQ_HEAD(evcon_requestq, evhttp_request) requests; 91 | 92 | void (*cb)(struct evhttp_connection *, void *); 93 | void *cb_arg; 94 | 95 | void (*closecb)(struct evhttp_connection *, void *); 96 | void *closecb_arg; 97 | 98 | struct event_base *base; 99 | }; 100 | 101 | struct evhttp_cb { 102 | TAILQ_ENTRY(evhttp_cb) next; 103 | 104 | char *what; 105 | 106 | void (*cb)(struct evhttp_request *req, void *); 107 | void *cbarg; 108 | }; 109 | 110 | /* both the http server as well as the rpc system need to queue connections */ 111 | TAILQ_HEAD(evconq, evhttp_connection); 112 | 113 | /* each bound socket is stored in one of these */ 114 | struct evhttp_bound_socket { 115 | TAILQ_ENTRY(evhttp_bound_socket) (next); 116 | 117 | struct event bind_ev; 118 | }; 119 | 120 | struct evhttp { 121 | TAILQ_HEAD(boundq, evhttp_bound_socket) sockets; 122 | 123 | TAILQ_HEAD(httpcbq, evhttp_cb) callbacks; 124 | struct evconq connections; 125 | 126 | int timeout; 127 | 128 | void (*gencb)(struct evhttp_request *req, void *); 129 | void *gencbarg; 130 | 131 | struct event_base *base; 132 | }; 133 | 134 | /* resets the connection; can be reused for more requests */ 135 | void evhttp_connection_reset(struct evhttp_connection *); 136 | 137 | /* connects if necessary */ 138 | int evhttp_connection_connect(struct evhttp_connection *); 139 | 140 | /* notifies the current request that it failed; resets connection */ 141 | void evhttp_connection_fail(struct evhttp_connection *, 142 | enum evhttp_connection_error error); 143 | 144 | void evhttp_get_request(struct evhttp *, int, struct sockaddr *, socklen_t); 145 | 146 | int evhttp_hostportfile(char *, char **, u_short *, char **); 147 | 148 | int evhttp_parse_firstline(struct evhttp_request *, struct evbuffer*); 149 | int evhttp_parse_headers(struct evhttp_request *, struct evbuffer*); 150 | 151 | void evhttp_start_read(struct evhttp_connection *); 152 | void evhttp_make_header(struct evhttp_connection *, struct evhttp_request *); 153 | 154 | void evhttp_write_buffer(struct evhttp_connection *, 155 | void (*)(struct evhttp_connection *, void *), void *); 156 | 157 | /* response sending HTML the data in the buffer */ 158 | void evhttp_response_code(struct evhttp_request *, int, const char *); 159 | void evhttp_send_page(struct evhttp_request *, struct evbuffer *); 160 | 161 | #endif /* _HTTP_H */ 162 | -------------------------------------------------------------------------------- /simpleleveldb/leveldb_to_csv.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | #define NAME "leveldb_to_csv" 12 | #define VERSION "0.1" 13 | 14 | int db_open(); 15 | void db_close(); 16 | 17 | leveldb_t *ldb; 18 | leveldb_options_t *ldb_options; 19 | leveldb_cache_t *ldb_cache; 20 | 21 | void db_close() 22 | { 23 | leveldb_close(ldb); 24 | leveldb_options_destroy(ldb_options); 25 | leveldb_cache_destroy(ldb_cache); 26 | } 27 | 28 | int db_open() 29 | { 30 | char *error = NULL; 31 | char *filename = option_get_str("db_file"); 32 | 33 | ldb_options = leveldb_options_create(); 34 | ldb_cache = leveldb_cache_create_lru(option_get_int("cache_size")); 35 | 36 | leveldb_options_set_create_if_missing(ldb_options, option_get_int("create_db_if_missing")); 37 | leveldb_options_set_error_if_exists(ldb_options, option_get_int("error_if_db_exists")); 38 | leveldb_options_set_paranoid_checks(ldb_options, option_get_int("paranoid_checks")); 39 | leveldb_options_set_write_buffer_size(ldb_options, option_get_int("write_buffer_size")); 40 | leveldb_options_set_block_size(ldb_options, option_get_int("block_size")); 41 | leveldb_options_set_cache(ldb_options, ldb_cache); 42 | leveldb_options_set_max_open_files(ldb_options, option_get_int("leveldb_max_open_files")); 43 | leveldb_options_set_block_restart_interval(ldb_options, 8); 44 | leveldb_options_set_compression(ldb_options, option_get_int("compression")); 45 | 46 | // leveldb_options_set_env(options, self->_env); 47 | leveldb_options_set_info_log(ldb_options, NULL); 48 | 49 | ldb = leveldb_open(ldb_options, filename, &error); 50 | if (error) { 51 | fprintf(stderr, "ERROR opening db:%s\n", error); 52 | return 0; 53 | } 54 | return 1; 55 | } 56 | 57 | 58 | int dump_to_csv() 59 | { 60 | leveldb_iterator_t *iter; 61 | leveldb_readoptions_t *read_options; 62 | char *output_filename = option_get_str("output_file"); 63 | const char *key, *value; 64 | size_t key_len, value_len; 65 | char *err = NULL; 66 | char *output_deliminator = option_get_str("output_deliminator"); 67 | FILE *out_file = stdout; 68 | 69 | if (output_filename) { 70 | out_file = fopen(output_filename, "w"); 71 | } 72 | 73 | read_options = leveldb_readoptions_create(); 74 | leveldb_readoptions_set_verify_checksums(read_options, option_get_int("verify_checksums")); 75 | iter = leveldb_create_iterator(ldb, read_options); 76 | leveldb_iter_seek_to_first(iter); 77 | while (leveldb_iter_valid(iter)) { 78 | key = leveldb_iter_key(iter, &key_len); 79 | value = leveldb_iter_value(iter, &value_len); 80 | fwrite(key, 1, key_len, out_file); 81 | fwrite(output_deliminator, 1, strlen(output_deliminator), out_file); 82 | fwrite(value, 1, value_len, out_file); 83 | fwrite("\n", 1, strlen("\n"), out_file); 84 | leveldb_iter_next(iter); 85 | } 86 | leveldb_iter_get_error(iter, &err); 87 | leveldb_readoptions_destroy(read_options); 88 | leveldb_iter_destroy(iter); 89 | if (err) { 90 | fprintf(stderr, "Error: %s\n", err); 91 | return 0; 92 | } 93 | 94 | return 1; 95 | } 96 | 97 | int version_cb(int value) 98 | { 99 | fprintf(stdout, "Version: %s\n", VERSION); 100 | return 0; 101 | } 102 | 103 | int main(int argc, char **argv) 104 | { 105 | option_define_bool("version", OPT_OPTIONAL, 0, NULL, version_cb, VERSION); 106 | option_define_str("db_file", OPT_REQUIRED, NULL, NULL, NULL, "path to leveldb file"); 107 | option_define_bool("create_db_if_missing", OPT_OPTIONAL, 1, NULL, NULL, "Create leveldb file if missing"); 108 | option_define_bool("error_if_db_exists", OPT_OPTIONAL, 0, NULL, NULL, "Error out if leveldb file exists"); 109 | option_define_bool("paranoid_checks", OPT_OPTIONAL, 1, NULL, NULL, "leveldb paranoid checks"); 110 | option_define_int("write_buffer_size", OPT_OPTIONAL, 4 << 20, NULL, NULL, "write buffer size"); 111 | option_define_int("cache_size", OPT_OPTIONAL, 4 << 20, NULL, NULL, "cache size (frequently used blocks)"); 112 | option_define_int("block_size", OPT_OPTIONAL, 4096, NULL, NULL, "block size"); 113 | option_define_bool("compression", OPT_OPTIONAL, 1, NULL, NULL, "snappy compression"); 114 | option_define_bool("verify_checksums", OPT_OPTIONAL, 1, NULL, NULL, "verify checksums at read time"); 115 | option_define_int("leveldb_max_open_files", OPT_OPTIONAL, 4096, NULL, NULL, "leveldb max open files"); 116 | 117 | option_define_str("output_file", OPT_OPTIONAL, NULL, NULL, NULL, "path to output file (default:stdout)"); 118 | option_define_str("output_deliminator", OPT_OPTIONAL, ",", NULL, NULL, "output deliminator"); 119 | 120 | if (!option_parse_command_line(argc, argv)) { 121 | return 1; 122 | } 123 | 124 | if (!db_open()) { 125 | return 1; 126 | } 127 | 128 | if (!dump_to_csv()) { 129 | return 1; 130 | } 131 | 132 | db_close(); 133 | free_options(); 134 | 135 | return 0; 136 | } 137 | -------------------------------------------------------------------------------- /simpleleveldb/str_list_set.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include "str_list_set.h" 4 | 5 | void prepare_token_list(struct ListInfo *list_info, char **db_data, size_t db_data_len, char sep) 6 | { 7 | // null terminate 8 | *db_data = realloc(*db_data, db_data_len + 1); 9 | *(*db_data + db_data_len) = '\0'; 10 | 11 | list_info->sep = sep; 12 | list_info->buf = db_data; 13 | list_info->buflen = db_data_len; 14 | 15 | // for strtok_r() 16 | list_info->sepstr[0] = sep; 17 | list_info->sepstr[1] = '\0'; 18 | 19 | // for reverse_tokenize() 20 | list_info->saveptr = *db_data + db_data_len; 21 | } 22 | 23 | // somewhat like strtok_r() but depends on prepare_token_list() for setup 24 | char *reverse_tokenize(struct ListInfo *list_info) 25 | { 26 | while (list_info->saveptr > *list_info->buf) { 27 | list_info->saveptr--; 28 | if (list_info->saveptr == *list_info->buf) { 29 | if (*list_info->saveptr != list_info->sep) { 30 | return list_info->saveptr; 31 | } else { 32 | return NULL; 33 | } 34 | } 35 | if (*list_info->saveptr == list_info->sep) { 36 | *list_info->saveptr = '\0'; 37 | if (*(list_info->saveptr + 1) != '\0') { 38 | return list_info->saveptr + 1; 39 | } 40 | } 41 | } 42 | return NULL; 43 | } 44 | 45 | void reserialize_list(struct evbuffer *output, struct json_object *array, char **db_data, size_t db_data_len, char sep) 46 | { 47 | struct ListInfo list_info; 48 | char *item; 49 | 50 | prepare_token_list(&list_info, db_data, db_data_len, sep); 51 | 52 | TOKEN_LIST_FOREACH(item, &list_info) { 53 | serialize_list_item(output, item, sep); 54 | 55 | if (array) { 56 | json_object_array_add(array, json_object_new_string(item)); 57 | } 58 | } 59 | } 60 | 61 | void deserialize_alloc_set(struct SetItem **set, char **db_data, size_t db_data_len, char sep) 62 | { 63 | char *token; 64 | struct SetItem *set_item; 65 | struct ListInfo list_info; 66 | 67 | prepare_token_list(&list_info, db_data, db_data_len, sep); 68 | 69 | TOKEN_LIST_FOREACH(token, &list_info) { 70 | HASH_FIND_STR(*set, token, set_item); 71 | if (!set_item) { 72 | add_new_set_item(set, token); 73 | } 74 | } 75 | } 76 | 77 | void serialize_free_set(struct evbuffer *output, struct json_object *array, struct SetItem **set, char sep) 78 | { 79 | struct SetItem *set_item, *set_tmp; 80 | 81 | HASH_ITER(hh, *set, set_item, set_tmp) { 82 | if (array) { 83 | json_object_array_add(array, json_object_new_string(set_item->value)); 84 | } 85 | serialize_list_item(output, set_item->value, sep); 86 | HASH_DEL(*set, set_item); 87 | free(set_item); 88 | } 89 | } 90 | 91 | -------------------------------------------------------------------------------- /simpleleveldb/str_list_set.h: -------------------------------------------------------------------------------- 1 | #ifndef _STR_LIST_SET_H 2 | #define _STR_LIST_SET_H 3 | 4 | #include 5 | #include 6 | 7 | struct SetItem { 8 | const char *value; 9 | UT_hash_handle hh; 10 | }; 11 | 12 | struct ListInfo { 13 | char sep; 14 | char sepstr[2]; 15 | char *saveptr; 16 | char **buf; 17 | size_t buflen; 18 | }; 19 | 20 | void prepare_token_list(struct ListInfo *list_info, char **db_data, size_t db_data_len, char sep); 21 | char *reverse_tokenize(struct ListInfo *list_info); 22 | void reserialize_list(struct evbuffer *output, struct json_object *array, char **db_data, size_t db_data_len, char sep); 23 | void deserialize_alloc_set(struct SetItem **set, char **db_data, size_t db_data_len, char sep); 24 | void serialize_free_set(struct evbuffer *output, struct json_object *array, struct SetItem **set, char sep); 25 | 26 | static inline void serialize_list_item(struct evbuffer *output, const char *item, char sep) 27 | { 28 | if (EVBUFFER_LENGTH(output) > 0) { 29 | evbuffer_add_printf(output, "%c", sep); 30 | } 31 | evbuffer_add_printf(output, "%s", item); 32 | } 33 | 34 | static inline void add_new_set_item(struct SetItem **set, const char *value_ptr) 35 | { 36 | struct SetItem *set_item; 37 | set_item = calloc(1, sizeof(struct SetItem)); 38 | set_item->value = value_ptr; 39 | HASH_ADD_KEYPTR(hh, *set, value_ptr, strlen(value_ptr), set_item); 40 | } 41 | 42 | #define TOKEN_LIST_FOREACH(item, list_info) \ 43 | for (item = strtok_r(*(list_info)->buf, \ 44 | (list_info)->sepstr, \ 45 | &(list_info)->saveptr); \ 46 | item != NULL; \ 47 | item = strtok_r(NULL, \ 48 | (list_info)->sepstr, \ 49 | &(list_info)->saveptr) )\ 50 | 51 | #define TOKEN_LIST_FOREACH_REVERSE(item, list_info) \ 52 | while ((item = reverse_tokenize(list_info)) != NULL) \ 53 | 54 | #endif 55 | -------------------------------------------------------------------------------- /simplememdb/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -Wall -g -O2 8 | LIBS = -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -levent -lsimplehttp -ltokyocabinet -ljson -lpcre 9 | 10 | simplememdb: simplememdb.c 11 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin 15 | /usr/bin/install simplememdb $(TARGET)/bin 16 | 17 | clean: 18 | rm -rf *.a *.o simplememdb *.dSYM 19 | -------------------------------------------------------------------------------- /simplememdb/README.md: -------------------------------------------------------------------------------- 1 | simplememdb 2 | =========== 3 | 4 | in-memory http accessible tokyo cabinet database 5 | 6 | Cmdline usage: 7 | 8 | -a 0.0.0.0 (address to bind to) 9 | -p 8080 (port to bind to) 10 | -D daemonize (default off) 11 | 12 | API endpoints: 13 | 14 | * /get 15 | parameter: key 16 | 17 | * /put 18 | parameter: key 19 | parameter: value 20 | 21 | * /del 22 | parameter: key 23 | 24 | * /dump (non-blocking asynchronous dump of key,value pairs) 25 | parameter:regex (optional regex to dump specific keys) 26 | 27 | * /fwmatch 28 | parameter:key 29 | parameter:max (optional) 30 | parameter:length (optional) 31 | parameter:offset (optional) 32 | 33 | * /fwmatch_int (returns values added with /incr) 34 | parameter:key 35 | parameter:max (optional) 36 | parameter:length (optional) 37 | parameter:offset (optional) 38 | 39 | * /fwmatch_int_merged (returns values added with /incr via prefix merge to single lines) 40 | parameter:key 41 | parameter:max (optional) 42 | parameter:length (optional) 43 | parameter:offset (optional) 44 | parameter:format=json|txt [default json] 45 | 46 | * /incr 47 | parameter:key 48 | parameter:value 49 | 50 | * /get_int (to return values added with /incr) 51 | parameter:key (accepts multiple &key= parameters) 52 | 53 | * /vanish (empty the database) 54 | 55 | * /stats (example output) 56 | parameter: format=json|txt 57 | 58 | * /exit 59 | 60 | Dependencies 61 | 62 | * [libevent](http://monkey.org/~provos/libevent/) 1.4.13+ 63 | * [json-c](http://oss.metaparadigm.com/json-c/) 64 | * [tokyocabinet](http://fallabs.com/tokyocabinet/) 65 | * [pcre](http://www.pcre.org/) 66 | -------------------------------------------------------------------------------- /simplequeue/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -Wall -g 8 | LIBS = -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -levent -lsimplehttp -lm 9 | 10 | simplequeue: simplequeue.c 11 | $(CC) $(CFLAGS) -o $@ $< $(LIBS) 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin 15 | /usr/bin/install simplequeue $(TARGET)/bin 16 | 17 | clean: 18 | rm -rf *.a *.o simplequeue *.dSYM 19 | -------------------------------------------------------------------------------- /simplequeue/echo.py: -------------------------------------------------------------------------------- 1 | from SimpleQueue import SimpleQueue 2 | from time import sleep 3 | 4 | queue = SimpleQueue() 5 | i = 0 6 | 7 | while 1: 8 | data = queue.get() 9 | 10 | if data == '': 11 | sleep(.5) 12 | continue 13 | 14 | i = i + 1 15 | print "%d:%s" % (i, data) -------------------------------------------------------------------------------- /simplequeue/test_simplequeue.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | sys.path.append(os.path.join(os.path.dirname(__file__), "../shared_tests")) 4 | 5 | import simplejson as json 6 | from test_shunt import valgrind_cmd, SubprocessTest, http_fetch, http_fetch_json 7 | 8 | class SimplequeueTest(SubprocessTest): 9 | binary_name = "simplequeue" 10 | working_dir = os.path.dirname(__file__) 11 | test_output_dir = os.path.join(working_dir, "test_output") 12 | process_options = [valgrind_cmd(test_output_dir, os.path.join(working_dir, binary_name), '--enable-logging')] 13 | 14 | def test_basic(self): 15 | # put/get in order 16 | http_fetch('/put', dict(data='12345')) 17 | http_fetch('/put', dict(data='23456')) 18 | data = json.loads(http_fetch('/stats', dict(format="json"))) 19 | assert data['depth'] == 2 20 | assert data['bytes'] == 10 21 | data = http_fetch('/get') 22 | assert data == "12345" 23 | data = http_fetch('/get') 24 | assert data == "23456" 25 | data = http_fetch('/get') 26 | assert data == '' 27 | 28 | data = json.loads(http_fetch('/stats', dict(format="json"))) 29 | assert data['depth'] == 0 30 | assert data['bytes'] == 0 31 | 32 | # mget 33 | http_fetch('/put', dict(data='12345')) 34 | http_fetch('/put', dict(data='23456')) 35 | http_fetch('/put', dict(data='34567')) 36 | http_fetch('/put', dict(data='45678')) 37 | data = http_fetch('/mget', dict(items=5)) 38 | assert data == '12345\n23456\n34567\n45678\n' 39 | 40 | # mput GET req 41 | http_fetch('/mput', dict(data='test1\ntest2\ntest3')) 42 | data = http_fetch('/get') 43 | assert data == 'test1' 44 | data = http_fetch('/get') 45 | assert data == 'test2' 46 | data = http_fetch('/get') 47 | assert data == 'test3' 48 | 49 | # mput POST req 50 | http_fetch('/mput', body='test1\ntest2\ntest3') 51 | data = http_fetch('/get') 52 | assert data == 'test1' 53 | data = http_fetch('/get') 54 | assert data == 'test2' 55 | data = http_fetch('/get') 56 | assert data == 'test3' 57 | 58 | 59 | if __name__ == "__main__": 60 | print "usage: py.test" 61 | -------------------------------------------------------------------------------- /simpletokyo/Makefile: -------------------------------------------------------------------------------- 1 | LIBEVENT ?= /usr/local 2 | TARGET ?= /usr/local 3 | LIBSIMPLEHTTP ?= ../simplehttp 4 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 5 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 6 | 7 | CFLAGS = -I. -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -Wall -g -O2 8 | LIBS = -L. -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -L/usr/local/lib -levent -ljson -ltokyotyrant -ltokyocabinet -lsimplehttp 9 | 10 | simpletokyo: simpletokyo.c 11 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 12 | 13 | install: 14 | /usr/bin/install -d $(TARGET)/bin/ 15 | /usr/bin/install simpletokyo $(TARGET)/bin/ 16 | 17 | clean: 18 | rm -rf *.o simpletokyo *.dSYM test_output 19 | -------------------------------------------------------------------------------- /simpletokyo/README.md: -------------------------------------------------------------------------------- 1 | simpletokyo 2 | =========== 3 | 4 | SimpleTokyo provides a light http interface to [Tokyo Tyrant](http://fallabs.com/tokyotyrant/). It supports 5 | the following commands. 6 | 7 | Command Line Options: 8 | 9 | --address= address to listen on 10 | default: 0.0.0.0 11 | --daemon daemonize process 12 | --enable-logging request logging 13 | --group= run as this group 14 | --help list usage 15 | --port= port to listen on 16 | default: 8080 17 | --root= chdir and run from this directory 18 | --ttserver-host= 19 | default: 127.0.0.1 20 | --ttserver-port= 21 | default: 1978 22 | --user= run as this user 23 | --version 24 | 25 | API endpoints: 26 | 27 | * /get 28 | parameter: key 29 | 30 | * /put 31 | parameter: key 32 | parameter: value 33 | 34 | * /del 35 | parameter: key 36 | 37 | * /fwmatch 38 | parameter:key 39 | parameter:max (optional) 40 | parameter:length (optional) 41 | parameter:offset (optional) 42 | parameter:format=json|txt [default json] 43 | 44 | * /fwmatch_int (returns values added with /incr) 45 | parameter:key 46 | parameter:max (optional) 47 | parameter:length (optional) 48 | parameter:offset (optional) 49 | parameter:format=json|txt [default json] 50 | 51 | * /fwmatch_int_merged (returns values added with /incr via prefix merge to single lines) 52 | parameter:key 53 | parameter:max (optional) 54 | parameter:length (optional) 55 | parameter:offset (optional) 56 | parameter:format=json|txt [default json] 57 | 58 | * /incr 59 | parameter:key 60 | parameter:value 61 | 62 | * /get_int (to return values added with /incr) 63 | parameter:key (accepts multiple &key= parameters) 64 | 65 | * /mget_int (to return values added with /incr) 66 | parameter:key (accepts multiple &key= parameters) 67 | 68 | * /mget 69 | parameter:key (accepts multiple &key= parameters) 70 | parameter:format=json|txt 71 | 72 | * /vanish (empty the database) 73 | 74 | * /stats (example output) 75 | parameter: format=json (optional) 76 | Total requests: 439 77 | /get requests: 0 78 | /get_int requests: 0 79 | /put requests: 303 80 | /del requests: 136 81 | /fwmatch requests: 0 82 | /fwmatch_int_ requests: 0 83 | /incr requests: 0 84 | /vanish requests: 0 85 | db opens: 1 86 | 87 | * /exit 88 | 89 | Dependencies 90 | 91 | * [libevent](http://monkey.org/~provos/libevent/) 1.4.13+ 92 | * [json-c](http://oss.metaparadigm.com/json-c/) 93 | * [tokyocabinet](http://fallabs.com/tokyocabinet/) / [tokyotyrant](http://fallabs.com/tokyotyrant/) 94 | -------------------------------------------------------------------------------- /simpletokyo/test.expected: -------------------------------------------------------------------------------- 1 | key should not exist 2 | { "status": "error", "code": 7, "message": "no record found" } 3 | key should exist 4 | { "status": "ok", "value": "1001" } 5 | should return 1005, 1007, 1009 6 | { "results": [ { "a1005": "1005" }, { "a1007": "1007" }, { "a1009": "1009" } ], "status": "ok" } 7 | set of odd data 8 | { "status": "ok", "value": "<>&this=that{} +*" } 9 | should == '<>&this=that{} +*' 10 | { "status": "ok", "value": "<>&this=that{} +*" } 11 | { "status": "ok" } 12 | test mget 13 | { "status": "ok", "value": "mget_value_A" } 14 | { "status": "ok", "value": "mget_value_B" } 15 | test mget format=json 16 | { "mget_key_A": "mget_value_A" } 17 | test mget format=txt 18 | mget_key_A,mget_value_A 19 | 20 | test mget two keys 21 | mget_key_A,mget_value_A 22 | mget_key_B,mget_value_B 23 | test mget two keys (one does not exist) 24 | mget_key_A,mget_value_A 25 | 26 | should == ' ' not '+++++' 27 | { "status": "ok", "value": " " } 28 | { "status": "ok", "value": " " } 29 | { "status": "ok" } 30 | key should not exist 31 | { "status": "error", "code": 7, "message": "no record found" } 32 | { "status": "ok" } 33 | { "status": "ok" } 34 | value should be 6 35 | { "status": "ok", "value": 6 } 36 | incr on two different keys at the same time. both values should be == 1 37 | { "status": "ok" } 38 | { "status": "ok", "value": 1 } 39 | { "status": "ok", "value": 1 } 40 | set key, get key, vanish db, and get key 41 | { "status": "ok", "value": "asdf" } 42 | { "status": "ok", "value": "asdf" } 43 | { "status": "ok", "value": 1 } 44 | now no key 45 | { "status": "error", "code": 7, "message": "no record found" } 46 | -------------------------------------------------------------------------------- /simpletokyo/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | make clean 4 | make 5 | 6 | if [ -f /tmp/simpletokyo_test.tcb ]; then 7 | rm -rf /tmp/simpletokyo_test.tcb ; 8 | fi 9 | 10 | SCRIPT=$(readlink -f "$0") 11 | SCRIPTPATH=`dirname "$SCRIPT"` 12 | 13 | testsubdir=test_output 14 | rm -rf "$testsubdir" > /dev/null 2>&1 15 | mkdir -p "$testsubdir" 16 | CMP="${CMP-cmp}" 17 | 18 | run_vg (){ 19 | TEST_COMMAND="$1" 20 | TEST_OPTIONS="$2" 21 | REDIR_OUTPUT="2>/dev/null 1>${testsubdir}/run.out" 22 | # REDIR_OUTPUT="" 23 | eval valgrind --tool=memcheck \ 24 | --trace-children=yes \ 25 | --demangle=yes \ 26 | --track-origins=yes \ 27 | --log-file="${testsubdir}/vg.out" \ 28 | --leak-check=full \ 29 | --show-reachable=yes \ 30 | --run-libc-freeres=yes \ 31 | "\"${SCRIPTPATH}/${TEST_COMMAND}\"" $TEST_OPTIONS ${REDIR_OUTPUT} & 32 | } 33 | 34 | 35 | OUT=$testsubdir/test.out 36 | 37 | ttserver -host 127.0.0.1 -port 8079 -thnum 1 /tmp/simpletokyo_test.tcb 2>/dev/null 1>/dev/null & 38 | run_vg simpletokyo "-A 127.0.0.1 -P 8079 -a 127.0.0.1 -p 8080 -V" 39 | 40 | sleep 2; 41 | 42 | for k in a b c 43 | do 44 | # put 100 entries per prefix 45 | for (( i = 1000; i <= 1100; i++ )) 46 | do 47 | curl --silent "http://localhost:8080/put?key=$k$i&value=$i" >/dev/null 48 | done 49 | 50 | # delete even entries 51 | for (( i = 1000; i <= 1100; i+=2 )) 52 | do 53 | curl --silent "http://localhost:8080/del?key=$k$i" >/dev/null 54 | done 55 | done 56 | 57 | # get a few specific keys 58 | echo "key should not exist" >> ${OUT} 59 | curl --silent "http://localhost:8080/get?key=a1000" >> ${OUT} 60 | echo "key should exist" >> ${OUT} 61 | curl --silent "http://localhost:8080/get?key=a1001" >> ${OUT} 62 | echo "should return 1005, 1007, 1009" >> ${OUT} 63 | curl --silent "http://localhost:8080/fwmatch?key=a&length=3&offset=2" >> ${OUT} 64 | echo "set of odd data" >> ${OUT} 65 | curl --silent "http://localhost:8080/put?key=odd&value=%3C%3E%26this%3Dthat%7B%7D%20%2B*" >> ${OUT} 66 | echo "should == '<>&this=that{} +*'" >> ${OUT} 67 | curl --silent "http://localhost:8080/get?key=odd" >> ${OUT} 68 | curl --silent "http://localhost:8080/del?key=odd" >> ${OUT} 69 | 70 | echo "test mget" >> ${OUT} 71 | curl --silent "http://localhost:8080/put?key=mget_key_A&value=mget_value_A" >> ${OUT} 72 | curl --silent "http://localhost:8080/put?key=mget_key_B&value=mget_value_B" >> ${OUT} 73 | echo "test mget format=json" >> ${OUT} 74 | curl --silent "http://localhost:8080/mget?key=mget_key_A" >> ${OUT} 75 | echo "test mget format=txt" >> ${OUT} 76 | curl --silent "http://localhost:8080/mget?key=mget_key_A&format=txt" >> ${OUT} 77 | echo "" >> ${OUT} 78 | echo "test mget two keys" >> ${OUT} 79 | curl --silent "http://localhost:8080/mget?key=mget_key_A&key=mget_key_B&format=txt" >> ${OUT} 80 | echo "test mget two keys (one does not exist)" >> ${OUT} 81 | curl --silent "http://localhost:8080/mget?key=mget_key_A&key=mget_key_C&format=txt" >> ${OUT} 82 | 83 | echo "" >> ${OUT} 84 | echo "should == ' ' not '+++++'" >> ${OUT} 85 | curl --silent "http://localhost:8080/put?key=plus&value=++++++" >> ${OUT} 86 | curl --silent "http://localhost:8080/get?key=plus" >> ${OUT} 87 | curl --silent "http://localhost:8080/del?key=plus" >> ${OUT} 88 | 89 | echo "key should not exist" >> ${OUT} 90 | curl --silent "http://localhost:8080/get?key=incr_test" >> ${OUT} 91 | curl --silent "http://localhost:8080/incr?key=incr_test" >> ${OUT} 92 | curl --silent "http://localhost:8080/incr?key=incr_test&value=5" >> ${OUT} 93 | echo "value should be 6" >> ${OUT} 94 | curl --silent "http://localhost:8080/get_int?key=incr_test" >> ${OUT} 95 | 96 | echo "incr on two different keys at the same time. both values should be == 1" >> ${OUT} 97 | curl --silent "http://localhost:8080/incr?key=double_key1&key=double_key2" >> ${OUT} 98 | curl --silent "http://localhost:8080/get_int?key=double_key1" >> ${OUT} 99 | curl --silent "http://localhost:8080/get_int?key=double_key2" >> ${OUT} 100 | 101 | echo "set key, get key, vanish db, and get key" >> ${OUT} 102 | curl --silent "http://localhost:8080/put?key=vanishtest&value=asdf" >> ${OUT} 103 | curl --silent "http://localhost:8080/get?key=vanishtest" >> ${OUT} 104 | curl --silent "http://localhost:8080/vanish" >> ${OUT} 105 | echo "now no key" >> ${OUT} 106 | curl --silent "http://localhost:8080/get?key=vanishtest" >> ${OUT} 107 | 108 | 109 | kill %1 # ttserver 110 | curl --silent "http://localhost:8080/exit" >> ${OUT} 111 | 112 | err=0; 113 | vg=0 114 | if ! "$CMP" -s "test.expected" "${testsubdir}/test.out" ; then 115 | echo "ERROR: test failed:" 1>&2 116 | diff -U 3 "test.expected" "${testsubdir}/test.out" 1>&2 117 | err=1 118 | else 119 | echo "FUNCTIONAL TEST PASSED" 120 | fi 121 | 122 | if ! grep -q "ERROR SUMMARY: 0 errors" "${testsubdir}/vg.out" ; then 123 | echo "ERROR: valgrind found errors during execution" 1>&2 124 | vg=1 125 | fi 126 | if ! grep -q "definitely lost: 0 bytes in 0 blocks" "${testsubdir}/vg.out" ; then 127 | echo "ERROR: valgrind found leaks during execution" 1>&2 128 | vg=1 129 | fi 130 | if ! grep -q "possibly lost: 0 bytes in 0 blocks" "${testsubdir}/vg.out" ; then 131 | echo "ERROR: valgrind found leaks during execution" 1>&2 132 | vg=1 133 | fi 134 | 135 | if [ $vg == "1" ]; then 136 | echo "see ${testsubdir}/vg.out for more details" 137 | err=1; 138 | fi 139 | 140 | exit $err; 141 | -------------------------------------------------------------------------------- /sortdb/Makefile: -------------------------------------------------------------------------------- 1 | # requires libevent-1.4 right now, will not work with libevent-2.X 2 | LIBEVENT ?= /usr/local 3 | TARGET ?= /usr/local 4 | LIBSIMPLEHTTP ?= ../simplehttp 5 | LIBSIMPLEHTTP_INC ?= $(LIBSIMPLEHTTP)/.. 6 | LIBSIMPLEHTTP_LIB ?= $(LIBSIMPLEHTTP) 7 | 8 | CFLAGS = -I$(LIBSIMPLEHTTP_INC) -I$(LIBEVENT)/include -Wall -g -O2 9 | LIBS = -L$(LIBSIMPLEHTTP_LIB) -L$(LIBEVENT)/lib -levent -lsimplehttp -lm 10 | 11 | sortdb: sortdb.c 12 | $(CC) $(CFLAGS) -o $@ $^ $(LIBS) 13 | 14 | install: 15 | /usr/bin/install -d $(TARGET)/bin 16 | /usr/bin/install sortdb $(TARGET)/bin 17 | 18 | clean: 19 | rm -rf *.a *.o sortdb *.dSYM test_output test.db 20 | -------------------------------------------------------------------------------- /sortdb/README.md: -------------------------------------------------------------------------------- 1 | sortdb 2 | ====== 3 | 4 | Sorted database server. Makes a tab (or comma) delimitated sorted file accessible via HTTP 5 | 6 | OPTIONS 7 | 8 | --address= address to listen on 9 | default: 0.0.0.0 10 | --daemon daemonize process 11 | --db-file= 12 | --memory-lock lock data file pages into memory 13 | --enable-logging request logging 14 | --field-separator= field separator (eg: comma, tab, pipe). default: TAB 15 | --group= run as this group 16 | --help list usage 17 | --port= port to listen on 18 | default: 8080 19 | --root= chdir and run from this directory 20 | --user= run as this user 21 | 22 | API endpoints: 23 | 24 | * /get?key=... 25 | 26 | * /mget?k=&k=... 27 | 28 | * /stats 29 | 30 | * /reload (reload/remap the db file) 31 | 32 | * /exit (cause the current process to exit) 33 | 34 | a HUP signal will also cause sortdb to reload/remap the db file 35 | -------------------------------------------------------------------------------- /sortdb/test.expected: -------------------------------------------------------------------------------- 1 | /get?key=a 2 | first record 3 | /get?key=b 4 | third 5 | /get?key=c 6 | d 7 | /get?key=m 8 | n 9 | /get?key=o 10 | p 11 | /get?key=zzzzzzzzzzzzzzzzzzzzzzzz 12 | almost-sleepy 13 | /get?key=zzzzzzzzzzzzzzzzzzzzzzzzz 14 | very-sleepy 15 | /get?key=zzzzzzzzzzzzzzzzzzzzzzzzzz 16 | already-asleep 17 | /mget?k=a&k=c&k=o 18 | a first record 19 | c d 20 | o p 21 | /fwmatch?key=prefix. 22 | prefix.1 how 23 | prefix.2 are 24 | prefix.3 you 25 | db reloaded 26 | /get?key=a (should be a new key 'new db') 27 | new db 28 | /get?key=b not found 29 | -------------------------------------------------------------------------------- /sortdb/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | SCRIPT=$(readlink -f "$0") 4 | SCRIPTPATH=`dirname "$SCRIPT"` 5 | testsubdir=test_output 6 | rm -rf "$testsubdir" > /dev/null 2>&1 7 | mkdir -p "$testsubdir" 8 | 9 | CMP="${CMP-cmp}" 10 | 11 | run_vg (){ 12 | TEST_COMMAND="$1" 13 | TEST_OPTIONS="$2" 14 | REDIR_OUTPUT="2>/dev/null 1>/dev/null" 15 | # REDIR_OUTPUT="" 16 | eval valgrind --tool=memcheck \ 17 | --trace-children=yes \ 18 | --demangle=yes \ 19 | --track-origins=yes \ 20 | --log-file="${testsubdir}/vg.out" \ 21 | --leak-check=full \ 22 | --show-reachable=yes \ 23 | --run-libc-freeres=yes \ 24 | "\"${SCRIPTPATH}/${TEST_COMMAND}\"" $TEST_OPTIONS ${REDIR_OUTPUT} & 25 | } 26 | err=$? 27 | 28 | ln -s -f test.tab test.db 29 | run_vg sortdb "--db-file=test.db --address=127.0.0.1 --port=8080" 30 | sleep 1 31 | for key in a b c m o zzzzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzzzzzz; do 32 | echo "/get?key=$key" >> $testsubdir/test.out 33 | curl --silent "localhost:8080/get/?key=$key" >> $testsubdir/test.out 34 | done 35 | echo "/mget?k=a&k=c&k=o" >> $testsubdir/test.out 36 | curl --silent "localhost:8080/mget?k=a&k=c&k=o" >> $testsubdir/test.out 37 | echo "/fwmatch?key=prefix." >> $testsubdir/test.out 38 | curl --silent "localhost:8080/fwmatch?key=prefix." >> $testsubdir/test.out 39 | 40 | # now swap the db and check keys again 41 | ln -s -f test2.tab test.db 42 | curl --silent "localhost:8080/reload" >> $testsubdir/test.out 43 | 44 | echo "/get?key=a (should be a new key 'new db')" >> $testsubdir/test.out 45 | curl --silent "localhost:8080/get/?key=a" >> $testsubdir/test.out 46 | echo "/get?key=b not found" >> $testsubdir/test.out 47 | curl --silent "localhost:8080/get/?key=b" >> $testsubdir/test.out 48 | 49 | curl --silent "localhost:8080/exit" 50 | sleep .25; 51 | 52 | err=0; 53 | vg=0 54 | if ! "$CMP" -s "test.expected" "${testsubdir}/test.out" ; then 55 | echo "ERROR: test failed:" 1>&2 56 | diff -U 3 "test.expected" "${testsubdir}/test.out" 1>&2 57 | err=1 58 | else 59 | echo "FUNCTIONAL TEST PASSED" 60 | fi 61 | 62 | if ! grep -q "ERROR SUMMARY: 0 errors" "${testsubdir}/vg.out" ; then 63 | echo "ERROR: valgrind found errors during execution" 1>&2 64 | vg=1 65 | fi 66 | if ! grep -q "definitely lost: 0 bytes in 0 blocks" "${testsubdir}/vg.out" ; then 67 | echo "ERROR: valgrind found leaks during execution" 1>&2 68 | vg=1 69 | fi 70 | if ! grep -q "possibly lost: 0 bytes in 0 blocks" "${testsubdir}/vg.out" ; then 71 | echo "ERROR: valgrind found leaks during execution" 1>&2 72 | vg=1 73 | fi 74 | 75 | if [ $vg == "1" ]; then 76 | echo "see ${testsubdir}/vg.out for more details" 77 | err=1; 78 | fi 79 | 80 | exit $err; 81 | -------------------------------------------------------------------------------- /sortdb/test.tab: -------------------------------------------------------------------------------- 1 | a first record 2 | aa another first 3 | b third 4 | c d 5 | e f 6 | g h 7 | i j 8 | k l 9 | m n 10 | o p 11 | prefix.1 how 12 | prefix.2 are 13 | prefix.3 you 14 | q r 15 | s t 16 | u v 17 | w x 18 | y z 19 | zzzzzzzzzzzzzzzzzzzzzzzz almost-sleepy 20 | zzzzzzzzzzzzzzzzzzzzzzzzz very-sleepy 21 | zzzzzzzzzzzzzzzzzzzzzzzzzz already-asleep 22 | -------------------------------------------------------------------------------- /sortdb/test2.tab: -------------------------------------------------------------------------------- 1 | a new db 2 | --------------------------------------------------------------------------------