├── .gitignore ├── LICENSE ├── README.md └── src ├── Makefile ├── barrier.h ├── eventmanager.cc ├── eventmanager.h ├── eventmanager_test.cc ├── future.h ├── future_test.cc ├── iobuffer.h ├── iobuffer_test.cc ├── notification.h ├── tcp.cc ├── tcp.h └── tcp_test.cc /.gitignore: -------------------------------------------------------------------------------- 1 | *.o 2 | *.a 3 | *_test 4 | src/lib 5 | src/include 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2011 Aaron Drew 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions 6 | are met: 7 | 1. Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | 2. Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | 3. Neither the name of the copyright holders nor the names of its 13 | contributors may be used to endorse or promote products derived from 14 | this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 19 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 20 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 21 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 22 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 23 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 24 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 25 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 26 | THE POSSIBILITY OF SUCH DAMAGE. 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | EventManager 2 | ============ 3 | 4 | This is a simple epoll() based thread-pool implementation for linux designed 5 | to be fast and very easy to use. 6 | 7 | How do I use it? 8 | ---------------- 9 | 10 | Basic event queing is simple: 11 | 12 | using std::tr1::bind; 13 | EventManager em; 14 | 15 | // Start 10 worker threads 16 | em.start(10); 17 | 18 | // Execute a function ASAP 19 | em.enqueue(bind(&MyFunction)); 20 | 21 | // Execute a function in 10 seconds time. 22 | em.enqueue(bind(&MyFunction), EventManager::currentTime() + 10.0); 23 | 24 | // Stop the event manager's threads. This call is optional. It will be 25 | // done if required when the EventManager is destroyed. 26 | em.stop(); 27 | 28 | The library also supports file descriptors: 29 | 30 | int socketFd = ...; 31 | 32 | using std::tr1::bind; 33 | EventManager em; 34 | em.start(10); 35 | 36 | em.watchFd(socketFd, EPOLLIN, bind(&onReadEvent)); 37 | em.watchFd(socketFd, EPOLLOUT, bind(&onWriteEvent)); 38 | em.watchFd(socketFd, EPOLLHUP, bind(&onDisconnectEvent)); 39 | 40 | // ... sleep or do something else ... 41 | 42 | em.removeFd(socketFd, EPOLLIN); 43 | em.removeFd(socketFd, EPOLLOUT); 44 | em.removeFd(socketFd, EPOLLHUP); 45 | 46 | em.stop(); 47 | 48 | All event handlers will be run on the first available EventManager worker 49 | thread. Blocking functions are strongly discouraged due to the risk of deadlock 50 | related issues. Its perfectly OK for code running on a worker thread to update 51 | the EventManager it is running on with the exception of the stop() function. 52 | The stop() function can only be called from a non-worker thread. Ideally it 53 | should be called from the thread that created the EventManager to begin with. 54 | 55 | Can you make it easier? 56 | ----------------------- 57 | 58 | If you insist. The following addititional classes are provided to allow higher 59 | level coding: 60 | 61 | - TcpSocket - A receive and send buffering TcpSocket proxy. Provides receive and disconnect callbacks. 62 | - TcpListenSocket - A listening socket that creates TcpSocket instances when someone connects. Provides an "onAccept" callback. 63 | - IOBuffer - A buffering I/O class that defers memory copying as long as possible to avoid redundant copy operations. This is designed to make dealing with received data significantly easier. 64 | - Notification - A simple pthread-based notification class for blocking until triggered. This should *not* be used on worker threads if possible but can be used to safely clean up from the main application thread. 65 | - Future - A proxy object representing a return value that may be set at some stage in the future. This allows for conventional (synchronous) and callback-driven (asynchronous) use cases with minimal fuss. 66 | - Barrier - Provides a synchronisation point. This object provides a callback that should be triggered exactly N times. On the Nth time, it calls a function and deletes itself. This functionality *may* be rolled into the Future class at some stage (by chaining Futures). 67 | 68 | Examples of how to use these classes will eventually arrive here. Until then, there are a bunch of unit tests sitting in the src/ directory that can be used as examples. 69 | 70 | Tests 71 | ----- 72 | 73 | I haven't bothered with any complex build systems. GNU make and valgrind are all that's required to build and test the library. For the paranoid (like me), you can run repeated tests with valgrind memory testing via: 74 | 75 | make long_test 76 | 77 | I'm not aware of any bugs but I would certainly be happier with more thorough unit tests. These will also probably appear with time but if you find this library useful, I'd certainly appreciate payback in the form of additional unit test patches. :) 78 | 79 | 80 | Dependencies 81 | ------------ 82 | 83 | The library depends on the following packages: 84 | 85 | - A recent version of g++ (with std::tr1 support for function, bind, shared_from_this, ...). 86 | - pthread 87 | - [glog](http://code.google.com/p/google-glog/) - used for debug logging 88 | - [googletest](http://code.google.com/p/googletest/) - for unit tests 89 | 90 | -------------------------------------------------------------------------------- /src/Makefile: -------------------------------------------------------------------------------- 1 | CPPFLAGS = -O3 2 | LDFLAGS = -lpthread -lstdc++ -lglog -lgtest -lgtest_main 3 | 4 | all: iobuffer_test eventmanager_test future_test tcp_test lib include 5 | 6 | clean: 7 | rm -f *.o eventmanager_test future_test iobuffer_test tcp_test 8 | rm -rf lib include 9 | 10 | eventmanager_test: eventmanager.o eventmanager_test.o 11 | g++ -o $@ $^ $(LDFLAGS) 12 | 13 | future_test: future_test.o eventmanager.o 14 | g++ -o $@ $^ $(LDFLAGS) 15 | 16 | iobuffer_test: iobuffer_test.o 17 | g++ -o $@ $^ $(LDFLAGS) 18 | 19 | tcp_test: eventmanager.o tcp.o tcp_test.o 20 | g++ -o $@ $^ $(LDFLAGS) 21 | 22 | .PHONY: lib 23 | lib: eventmanager.o tcp.o 24 | mkdir -p lib 25 | ar cr lib/libepoll_threadpool.a $^ 26 | 27 | .PHONY: include 28 | include: barrier.h eventmanager.h future.h iobuffer.h notification.h tcp.h 29 | mkdir -p include/epoll_threadpool 30 | cp $^ include/epoll_threadpool 31 | 32 | test: all 33 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./eventmanager_test --gtest_break_on_failure 34 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./future_test --gtest_break_on_failure 35 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./iobuffer_test --gtest_break_on_failure 36 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./tcp_test --gtest_break_on_failure 37 | 38 | long_test: all 39 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./eventmanager_test --gtest_repeat=1000 --gtest_break_on_failure 40 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./future_test --gtest_repeat=1000 --gtest_break_on_failure 41 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./iobuffer_test --gtest_repeat=1000 --gtest_break_on_failure 42 | valgrind -v --track-origins=yes --leak-check=full --show-reachable=yes ./tcp_test --gtest_repeat=1000 --gtest_break_on_failure 43 | 44 | -------------------------------------------------------------------------------- /src/barrier.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #ifndef _EPOLL_THREADPOOL_BARRIER_H_ 30 | #define _EPOLL_THREADPOOL_BARRIER_H_ 31 | 32 | #include "eventmanager.h" 33 | 34 | #include 35 | #include 36 | 37 | #include 38 | #include 39 | #include 40 | 41 | namespace epoll_threadpool { 42 | 43 | using std::tr1::bind; 44 | using std::tr1::function; 45 | 46 | /** 47 | * A simple class used to provide a synchronisation point. It can called as 48 | * a std::tr1::function a fixed number of times before triggering 49 | * a callback and destroying itself. 50 | */ 51 | class Barrier { 52 | private: 53 | pthread_mutex_t _mutex; 54 | int _n; 55 | function _callback; 56 | 57 | public: 58 | Barrier(int n, function callback) 59 | : _n(n), _callback(callback) { 60 | pthread_mutex_init(&_mutex, 0); 61 | } 62 | virtual ~Barrier() { 63 | pthread_mutex_destroy(&_mutex); 64 | } 65 | 66 | function callback() { 67 | return bind(&Barrier::signal, this); 68 | } 69 | /*operator function() { 70 | return bind(&Barrier::signal, this); 71 | }*/ 72 | 73 | private: 74 | 75 | void signal() { 76 | pthread_mutex_lock(&_mutex); 77 | --_n; 78 | if (_n > 0) { 79 | pthread_mutex_unlock(&_mutex); 80 | #ifdef NDEBUG 81 | } else if(_n < 0) { 82 | DLOG(ERROR) << "Barrier " << this 83 | << " called too many times (" << _n << ")."; 84 | #endif 85 | } else { 86 | pthread_mutex_unlock(&_mutex); 87 | _callback(); 88 | delete this; 89 | } 90 | } 91 | 92 | private: 93 | 94 | Barrier(const Barrier& b); // No copying 95 | Barrier& operator=(const Barrier& b); // No copying 96 | }; 97 | 98 | } 99 | 100 | #endif 101 | -------------------------------------------------------------------------------- /src/eventmanager.cc: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #include "eventmanager.h" 30 | 31 | #include 32 | 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | 40 | namespace epoll_threadpool { 41 | 42 | using std::tr1::function; 43 | 44 | EventManager::EventManager() : _is_running(false) { 45 | pthread_mutex_init(&_mutex, 0); 46 | _epoll_fd = epoll_create(64); 47 | 48 | // Set up and add eventfd to the epoll descriptor 49 | _event_fd = eventfd(0, 0); 50 | fcntl(_event_fd, F_SETFL, O_NONBLOCK); 51 | struct epoll_event ev; 52 | ev.data.u64 = 0; // stop valgrind whinging 53 | ev.events = EPOLLIN; 54 | ev.data.fd = _event_fd; 55 | epoll_ctl(_epoll_fd, EPOLL_CTL_ADD, _event_fd, &ev); 56 | } 57 | 58 | EventManager::~EventManager() { 59 | stop(); 60 | 61 | struct epoll_event ev; 62 | ev.data.u64 = 0; // stop valgrind whinging 63 | ev.events = EPOLLIN; 64 | ev.data.fd = _event_fd; 65 | epoll_ctl(_epoll_fd, EPOLL_CTL_DEL, _event_fd, &ev); 66 | close(_event_fd); 67 | 68 | close(_epoll_fd); 69 | pthread_mutex_destroy(&_mutex); 70 | } 71 | 72 | bool EventManager::start(int num_threads) { 73 | pthread_mutex_lock(&_mutex); 74 | 75 | // Tried to call start() from one of our worker threads? There lies madness. 76 | if (_thread_set.find(pthread_self()) != _thread_set.end()) { 77 | pthread_mutex_unlock(&_mutex); 78 | return false; 79 | } 80 | 81 | _is_running = true; 82 | for (int i = 0; i < num_threads; ++i) { 83 | pthread_t thread; 84 | pthread_create(&thread, NULL, trampoline, this); 85 | _thread_set.insert(thread); 86 | } 87 | pthread_mutex_unlock(&_mutex); 88 | return true; 89 | } 90 | 91 | bool EventManager::stop() { 92 | pthread_mutex_lock(&_mutex); 93 | 94 | // We don't allow a stop() call from one of our worker threads. 95 | if (_thread_set.find(pthread_self()) != _thread_set.end()) { 96 | pthread_mutex_unlock(&_mutex); 97 | return false; 98 | } 99 | 100 | _is_running = false; 101 | 102 | eventfd_write(_event_fd, 1); 103 | for (set::const_iterator i = _thread_set.begin(); 104 | i != _thread_set.end(); ++i) { 105 | pthread_mutex_unlock(&_mutex); 106 | pthread_join(*i, NULL); 107 | pthread_mutex_lock(&_mutex); 108 | } 109 | _thread_set.clear(); 110 | 111 | // Cancel any unprocessed tasks or fds. 112 | DLOG_IF(WARNING, !_fds.empty()) << "Stopping event manager with attached " 113 | << "file descriptors. You should consider calling removeFd first."; 114 | DLOG_IF(WARNING, !_tasks.empty()) << "Stopping event manager with pending " 115 | << "tasks."; 116 | _fds.clear(); 117 | _tasks.clear(); 118 | 119 | pthread_mutex_unlock(&_mutex); 120 | return true; 121 | } 122 | 123 | EventManager::WallTime EventManager::currentTime() { 124 | struct timeval tv; 125 | gettimeofday(&tv, NULL); 126 | return tv.tv_sec + tv.tv_usec/1000000.0; 127 | } 128 | 129 | void EventManager::enqueue(Function f, WallTime when) { 130 | pthread_mutex_lock(&_mutex); 131 | Task t = { when, f }; 132 | double oldwhen = _tasks.empty() ? -1 : _tasks.front().when; 133 | _tasks.push_back(t); 134 | push_heap(_tasks.begin(), _tasks.end(), &EventManager::compareTasks); 135 | // Do we need to wake up a worker to get this done on time? 136 | if (oldwhen != _tasks.front().when) { 137 | eventfd_write(_event_fd, 1); 138 | } 139 | pthread_mutex_unlock(&_mutex); 140 | } 141 | 142 | bool EventManager::watchFd(int fd, EventType type, function f) { 143 | pthread_mutex_lock(&_mutex); 144 | 145 | if (_fds.find(fd) == _fds.end()) { 146 | _fds[fd][type] = f; 147 | epollUpdate(fd, EPOLL_CTL_ADD); 148 | } else { 149 | _fds[fd][type] = f; 150 | epollUpdate(fd, EPOLL_CTL_MOD); 151 | } 152 | 153 | eventfd_write(_event_fd, 1); 154 | pthread_mutex_unlock(&_mutex); 155 | return true; 156 | } 157 | 158 | bool EventManager::removeFd(int fd, EventType type) { 159 | pthread_mutex_lock(&_mutex); 160 | 161 | if (_fds.find(fd) == _fds.end()) { 162 | pthread_mutex_unlock(&_mutex); 163 | return false; 164 | } 165 | 166 | if (_fds[fd].size() == 1 && 167 | _fds[fd].find(type) != _fds[fd].end()) { 168 | epollUpdate(fd, EPOLL_CTL_DEL); 169 | _fds.erase(fd); 170 | } else { 171 | _fds[fd].erase(type); 172 | epollUpdate(fd, EPOLL_CTL_MOD); 173 | } 174 | 175 | pthread_mutex_unlock(&_mutex); 176 | return true; 177 | } 178 | 179 | void EventManager::epollUpdate(int fd, int epoll_op) { 180 | struct epoll_event ev; 181 | ev.data.u64 = 0; // stop valgrind whinging 182 | ev.events = 0; 183 | for (std::map >::iterator i = _fds[fd].begin(); 185 | i != _fds[fd].end(); ++i) { 186 | switch (i->first) { 187 | case EventManager::EM_READ: 188 | ev.events |= EPOLLIN; 189 | break; 190 | case EventManager::EM_WRITE: 191 | ev.events |= EPOLLOUT; 192 | break; 193 | case EventManager::EM_ERROR: 194 | ev.events |= EPOLLRDHUP | EPOLLHUP; 195 | break; 196 | default: 197 | LOG(ERROR) << "Unknown event type " << i->first; 198 | }; 199 | } 200 | ev.data.fd = fd; 201 | 202 | int r = epoll_ctl(_epoll_fd, epoll_op, fd, &ev); 203 | DLOG_IF(WARNING, r != 0) 204 | << "epoll_ctl(" << _epoll_fd << ", " << epoll_op << ", " << fd 205 | << ", &ev) returned error " << errno; 206 | } 207 | 208 | void* EventManager::trampoline(void *arg) { 209 | EventManager *em = static_cast(arg); 210 | em->thread_main(); 211 | return NULL; 212 | } 213 | 214 | void EventManager::thread_main() { 215 | const int kMaxEvents = 32; 216 | const int kEpollDefaultWait = 10000; 217 | 218 | struct epoll_event events[kMaxEvents]; 219 | pthread_mutex_lock(&_mutex); 220 | while (_is_running) { 221 | int timeout; 222 | if (!_tasks.empty()) { 223 | timeout = static_cast((_tasks.front().when - currentTime())*1000); 224 | if (timeout < 0) { 225 | timeout = 0; 226 | } 227 | } else { 228 | timeout = kEpollDefaultWait; 229 | } 230 | pthread_mutex_unlock(&_mutex); 231 | int ret = epoll_wait(_epoll_fd, events, kMaxEvents, timeout); 232 | pthread_mutex_lock(&_mutex); 233 | 234 | if (ret < 0) { 235 | if (errno != EINTR) { 236 | LOG(ERROR) << "Epoll error: " << errno << " fd is " << _epoll_fd; 237 | } 238 | continue; 239 | } 240 | 241 | // Execute triggered fd handlers 242 | for (int i = 0; i < ret; i++) { 243 | int fd = events[i].data.fd; 244 | if (fd == _event_fd) { 245 | uint64_t val; 246 | eventfd_read(_event_fd, &val); 247 | } else { 248 | int flags = events[i].events; 249 | if ((flags | EPOLLIN) && 250 | _fds.find(fd) != _fds.end() && 251 | _fds[fd].find(EM_READ) != _fds[fd].end()) { 252 | function f = _fds[fd][EM_READ]; 253 | pthread_mutex_unlock(&_mutex); 254 | f(); 255 | pthread_mutex_lock(&_mutex); 256 | } 257 | if ((flags | EPOLLOUT) && 258 | _fds.find(fd) != _fds.end() && 259 | _fds[fd].find(EM_WRITE) != _fds[fd].end()) { 260 | function f = _fds[fd][EM_WRITE]; 261 | pthread_mutex_unlock(&_mutex); 262 | f(); 263 | pthread_mutex_lock(&_mutex); 264 | } 265 | if ((flags | EPOLLHUP | EPOLLRDHUP) && 266 | _fds.find(fd) != _fds.end() && 267 | _fds[fd].find(EM_ERROR) != _fds[fd].end()) { 268 | function f = _fds[fd][EM_ERROR]; 269 | pthread_mutex_unlock(&_mutex); 270 | f(); 271 | pthread_mutex_lock(&_mutex); 272 | } 273 | } 274 | } 275 | 276 | // Execute queued events that are due to be run. 277 | while (!_tasks.empty() && _tasks.front().when <= currentTime()) { 278 | Task t = _tasks.front(); 279 | pop_heap(_tasks.begin(), _tasks.end(), &EventManager::compareTasks); 280 | _tasks.pop_back(); 281 | pthread_mutex_unlock(&_mutex); 282 | t.f(); 283 | pthread_mutex_lock(&_mutex); 284 | } 285 | } 286 | // wake up another thread - its likely we want to shut down. 287 | eventfd_write(_event_fd, 1); 288 | pthread_mutex_unlock(&_mutex); 289 | } 290 | } 291 | -------------------------------------------------------------------------------- /src/eventmanager.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #ifndef _EPOLL_THREADPOOL_EVENTMANAGER_H_ 30 | #define _EPOLL_THREADPOOL_EVENTMANAGER_H_ 31 | 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | 40 | #include 41 | #include 42 | 43 | namespace epoll_threadpool { 44 | 45 | using namespace std; 46 | using std::tr1::function; 47 | using std::tr1::shared_ptr; 48 | 49 | /** 50 | * Tiny epoll based event manager with a configurable pool of threads. 51 | * Can watch on sockets and execute functions on event threads. 52 | */ 53 | class EventManager { 54 | public: 55 | typedef double WallTime; 56 | 57 | /** 58 | * These represent the type of events a user can watch for on a file 59 | * descriptor. 60 | * @see watchFd removeFd 61 | */ 62 | enum EventType { 63 | EM_READ, 64 | EM_WRITE, 65 | EM_ERROR 66 | }; 67 | 68 | /** 69 | * A simple std::tr1::function wrapper that adds the ability to cancel 70 | * a call if it hasn't already started. Internally reference counted 71 | * to make passing the function object around easy to do. 72 | */ 73 | class Function { 74 | public: 75 | template 76 | Function(T f) : _internal(new Internal(f)) { } 77 | void cancel() { 78 | _internal->cancel(); 79 | } 80 | void operator()() { 81 | _internal->run(); 82 | } 83 | private: 84 | class Internal { 85 | public: 86 | Internal(function f) : _f(f) { 87 | pthread_mutex_init(&_mutex, 0); 88 | } 89 | ~Internal() { 90 | pthread_mutex_destroy(&_mutex); 91 | } 92 | void cancel() { 93 | pthread_mutex_lock(&_mutex); 94 | _f = NULL; 95 | pthread_mutex_unlock(&_mutex); 96 | } 97 | void run() { 98 | pthread_mutex_lock(&_mutex); 99 | function f(_f); 100 | pthread_mutex_unlock(&_mutex); 101 | if (f != NULL) { 102 | f(); 103 | } 104 | } 105 | private: 106 | pthread_mutex_t _mutex; 107 | function _f; 108 | }; 109 | shared_ptr _internal; 110 | }; 111 | 112 | EventManager(); 113 | virtual ~EventManager(); 114 | 115 | /** 116 | * Starts a number of worker threads for the EventManager. 117 | * This can be called repeatedly to start more threads as necessary. 118 | * This function will fail if called from an EventManager thread. 119 | * Returns true on success, false on failure. 120 | */ 121 | bool start(int num_threads); 122 | 123 | /** 124 | * Gracefully shut down running worker threads. 125 | * This function will fail if called from an EventManager thread. 126 | * Returns true on success, false on failure. 127 | */ 128 | bool stop(); 129 | 130 | /** 131 | * Returns the current wall time in fractional seconds since epoch. 132 | */ 133 | static WallTime currentTime(); 134 | 135 | /** 136 | * Enqueues a function to be run on one of the EventManager's worker threads. 137 | * The function will be run on the first available thread. 138 | * It is safe to call this function from a worker thread itself. 139 | */ 140 | void enqueue(Function f) { 141 | enqueue(f, currentTime()); 142 | } 143 | 144 | /** 145 | * Enqueues a function to be run at a specified time on a worker thread. 146 | * The function will be run on the first available thread at or after the 147 | * requested time. 148 | * It is safe to call this function from a worker thread itself. 149 | */ 150 | void enqueue(Function f, WallTime when); 151 | 152 | /** 153 | * Watches for activity on a given file descriptor and triggers a callback 154 | * when an event occurs. (fd, type) can be considered a tuple. To watch 155 | * a file descriptor for multiple events, you must call this function 156 | * multiple times. 157 | * It is safe to call this function from a worker thread itself. 158 | */ 159 | bool watchFd(int fd, EventType type, function f); 160 | 161 | /** 162 | * Stops triggering callbacks when a given event type occurs for a given FD. 163 | * The type argument should match that passed in to watchFd. 164 | * It is safe to call this function from a worker thread itself. 165 | */ 166 | bool removeFd(int fd, EventType type); 167 | 168 | private: 169 | // Stores a scheduled task callback. 170 | struct Task { 171 | WallTime when; 172 | Function f; 173 | }; 174 | // Used to sort heap with earliest time at the top 175 | static bool compareTasks(const Task&a, const Task& b) { 176 | return a.when > b.when; 177 | } 178 | 179 | /** 180 | * Helper that wraps and hides epoll_ctl calls, intended to make updating 181 | * the epoll fd simpler. 182 | */ 183 | void epollUpdate(int fd, int op); 184 | 185 | pthread_mutex_t _mutex; 186 | 187 | int _epoll_fd; 188 | int _event_fd; 189 | volatile bool _is_running; 190 | 191 | std::set _thread_set; 192 | 193 | std::vector _tasks; 194 | std::map > > _fds; 195 | 196 | static void* trampoline(void *arg); 197 | void thread_main(); 198 | }; 199 | } 200 | 201 | #endif 202 | -------------------------------------------------------------------------------- /src/eventmanager_test.cc: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #include 30 | #include 31 | #include 32 | #include 33 | 34 | #include 35 | 36 | #include "eventmanager.h" 37 | #include "notification.h" 38 | 39 | using epoll_threadpool::EventManager; 40 | using epoll_threadpool::Notification; 41 | 42 | TEST(EventManagerTest, StartStop) { 43 | EventManager em; 44 | 45 | ASSERT_TRUE(em.start(2)); 46 | em.stop(); 47 | ASSERT_TRUE(em.start(2)); 48 | em.stop(); 49 | } 50 | 51 | TEST(EventManagerTest, NotificationDelayedRaise) { 52 | EventManager em; 53 | Notification n; 54 | EventManager::WallTime t = EventManager::currentTime(); 55 | 56 | ASSERT_TRUE(em.start(2)); 57 | em.enqueue(std::tr1::bind(&Notification::signal, &n), t+0.001); 58 | ASSERT_TRUE(n.tryWait(t+0.500)); 59 | em.stop(); 60 | } 61 | 62 | TEST(EventManagerTest, NotificationPreDelayRaise) { 63 | EventManager em; 64 | Notification n; 65 | EventManager::WallTime t = EventManager::currentTime(); 66 | 67 | ASSERT_TRUE(em.start(2)); 68 | em.enqueue(std::tr1::bind(&Notification::signal, &n)); 69 | usleep(10); 70 | ASSERT_TRUE(n.tryWait(t+0.500)); 71 | em.stop(); 72 | } 73 | 74 | TEST(EventManagerTest, StartEnqueueStop) { 75 | EventManager em; 76 | Notification n; 77 | EventManager::WallTime t = EventManager::currentTime(); 78 | 79 | em.start(2); 80 | em.enqueue(std::tr1::bind(&Notification::signal, &n)); 81 | ASSERT_TRUE(n.tryWait(t+0.500)); 82 | em.stop(); 83 | } 84 | 85 | TEST(EventManagerTest, StartEnqueueStop2) { 86 | EventManager em; 87 | Notification n; 88 | EventManager::WallTime t = EventManager::currentTime(); 89 | 90 | ASSERT_TRUE(em.start(2)); 91 | em.enqueue(std::tr1::bind(&Notification::signal, &n), t+0.001); 92 | n.wait(); 93 | em.stop(); 94 | } 95 | 96 | TEST(EventManagerTest, StartEnqueueStop3) { 97 | EventManager em; 98 | Notification n; 99 | EventManager::WallTime t = EventManager::currentTime(); 100 | 101 | ASSERT_TRUE(em.start(2)); 102 | em.enqueue(std::tr1::bind(&Notification::signal, &n), t+0.005); 103 | ASSERT_TRUE(n.tryWait(t+0.500)); 104 | em.stop(); 105 | } 106 | 107 | void EnqueuedCountCheck(pthread_mutex_t *mutex, int *cnt, Notification *n, int expected) { 108 | pthread_mutex_lock(mutex); 109 | EXPECT_EQ(expected, *cnt); 110 | (*cnt)++; 111 | if (n) { 112 | n->signal(); 113 | } 114 | pthread_mutex_unlock(mutex); 115 | } 116 | 117 | TEST(EventManagerTest, StartEnqueueStop4) { 118 | EventManager em; 119 | Notification n; 120 | EventManager::WallTime t = EventManager::currentTime(); 121 | 122 | pthread_mutex_t mutex; 123 | pthread_mutex_init(&mutex, 0); 124 | int cnt = 1; 125 | 126 | // Check double precision - should be ok but just to be safe... 127 | ASSERT_LT(t+0.001, t+0.002); 128 | ASSERT_LT(t+0.002, t+0.003); 129 | ASSERT_LT(t+0.003, t+0.004); 130 | ASSERT_LT(t+0.004, t+0.005); 131 | 132 | em.enqueue(std::tr1::bind(&EnqueuedCountCheck, &mutex, &cnt, &n, 5), t+0.005); 133 | em.enqueue(std::tr1::bind(&EnqueuedCountCheck, &mutex, &cnt, (Notification *)NULL, 4), t+0.004); 134 | em.enqueue(std::tr1::bind(&EnqueuedCountCheck, &mutex, &cnt, (Notification *)NULL, 2), t+0.002); 135 | em.enqueue(std::tr1::bind(&EnqueuedCountCheck, &mutex, &cnt, (Notification *)NULL, 1), t+0.001); 136 | em.enqueue(std::tr1::bind(&EnqueuedCountCheck, &mutex, &cnt, (Notification *)NULL, 3), t+0.003); 137 | 138 | // We start this AFTER adding the tasks to ensure we don't start one before 139 | // we've added them all. Note that we only start one thread because its 140 | // otherwise possible we start two tasks at close to the same time and 141 | // the second one runs first, leading to flaky tests. 142 | ASSERT_TRUE(em.start(1)); 143 | ASSERT_TRUE(n.tryWait(t+0.500)); 144 | em.stop(); 145 | 146 | pthread_mutex_destroy(&mutex); 147 | } 148 | 149 | void EnqueuedAfterCheck(volatile bool *flag) { 150 | bool flagcpy = *flag; 151 | usleep(1000); 152 | EXPECT_EQ(*flag, flagcpy); 153 | } 154 | 155 | void EnqueuedAfterCheck2(volatile bool *flag, Notification *n) { 156 | *flag = true; 157 | if (n) { 158 | n->signal(); 159 | } 160 | } 161 | 162 | void WorkerStartStop(EventManager *em) { 163 | ASSERT_FALSE(em->start(2)); 164 | ASSERT_FALSE(em->stop()); 165 | } 166 | 167 | TEST(EventManagerTest, CallMethodsFromWorkerThread) { 168 | EventManager em; 169 | Notification n; 170 | EventManager::WallTime t = EventManager::currentTime(); 171 | 172 | em.enqueue(std::tr1::bind(&WorkerStartStop, &em)); 173 | em.enqueue(std::tr1::bind(&Notification::signal, &n), t + 0.001); 174 | ASSERT_TRUE(em.start(2)); 175 | ASSERT_TRUE(n.tryWait(t+0.500)); 176 | em.stop(); 177 | } 178 | 179 | void WatchFdRead(Notification *n, int fd) { 180 | char buf[9]; 181 | ASSERT_EQ(9, ::read(fd, buf, 9)); 182 | EXPECT_STREQ("testdata", buf); 183 | n->signal(); 184 | } 185 | 186 | void WatchFdWrite(Notification *n, int fd, EventManager *em) { 187 | ASSERT_EQ(9, ::write(fd, "testdata", 9)); 188 | // Tests removeFd() from worker thread 189 | ASSERT_TRUE(em->removeFd(fd, EventManager::EM_WRITE)); 190 | n->signal(); 191 | } 192 | 193 | TEST(EventManagerTest, WatchFdAndRemoveFdFromWorker) { 194 | EventManager em; 195 | Notification n, n2; 196 | EventManager::WallTime t = EventManager::currentTime(); 197 | 198 | int fds[2]; 199 | ASSERT_EQ(0, pipe2(fds, O_NONBLOCK)); 200 | 201 | em.watchFd(fds[0], EventManager::EM_READ, 202 | std::tr1::bind(&WatchFdRead, &n, fds[0])); 203 | em.watchFd(fds[1], EventManager::EM_WRITE, 204 | std::tr1::bind(&WatchFdWrite, &n2, fds[1], &em)); 205 | ASSERT_TRUE(em.start(1)); 206 | ASSERT_TRUE(n.tryWait(t + 0.500)); 207 | ASSERT_TRUE(n2.tryWait(t + 0.500)); 208 | 209 | ASSERT_TRUE(em.removeFd(fds[0], EventManager::EM_READ)); 210 | em.stop(); 211 | close(fds[0]); 212 | close(fds[1]); 213 | } 214 | 215 | TEST(EventManagerTest, WatchFdConcurrentReadWrite) { 216 | EventManager em; 217 | Notification n, n2; 218 | EventManager::WallTime t = EventManager::currentTime(); 219 | 220 | int fds[2]; 221 | ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, fds)); 222 | 223 | // Write test data to fds[1] so fds[0] has something to read. 224 | ASSERT_EQ(9, ::write(fds[1], "testdata", 9)); 225 | 226 | em.watchFd(fds[0], EventManager::EM_READ, 227 | std::tr1::bind(&WatchFdRead, &n, fds[0])); 228 | em.watchFd(fds[0], EventManager::EM_WRITE, 229 | std::tr1::bind(&WatchFdWrite, &n2, fds[0], &em)); 230 | ASSERT_TRUE(em.start(1)); 231 | ASSERT_TRUE(n.tryWait(t + 0.500)); 232 | ASSERT_TRUE(n2.tryWait(t + 0.500)); 233 | 234 | ASSERT_TRUE(em.removeFd(fds[0], EventManager::EM_READ)); 235 | em.stop(); 236 | close(fds[0]); 237 | close(fds[1]); 238 | } 239 | 240 | void TestCancelA() { 241 | // We expect this to get called. 242 | } 243 | void TestCancelB() { 244 | // We expec this to get cancelled and never called. 245 | ASSERT_TRUE(false); 246 | } 247 | 248 | TEST(EventManagerTest, TestCancel) { 249 | EventManager em; 250 | Notification n; 251 | EventManager::WallTime t = EventManager::currentTime(); 252 | 253 | em.enqueue(&TestCancelA); 254 | em.enqueue(&TestCancelA); 255 | 256 | EventManager::Function f(&TestCancelB); 257 | em.enqueue(f); 258 | f.cancel(); 259 | 260 | em.enqueue(std::tr1::bind(&Notification::signal, &n)); 261 | 262 | ASSERT_TRUE(em.start(1)); 263 | ASSERT_TRUE(n.tryWait(t + 0.500)); 264 | em.stop(); 265 | } 266 | 267 | 268 | // TODO: Test watchFd() with two events firing one at a time. 269 | // TODO: Test watchFd() with two events firing at the same time. 270 | 271 | -------------------------------------------------------------------------------- /src/future.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #ifndef _EPOLL_THREADPOOL_FUTURE_H_ 30 | #define _EPOLL_THREADPOOL_FUTURE_H_ 31 | 32 | #include "eventmanager.h" 33 | 34 | #include 35 | 36 | #include 37 | 38 | #include 39 | #include 40 | #include 41 | #include 42 | 43 | namespace epoll_threadpool { 44 | 45 | using std::tr1::bind; 46 | using std::tr1::function; 47 | using std::tr1::shared_ptr; 48 | 49 | /** 50 | * A Future is a placeholder for an arbitrary return value that might not be 51 | * known until a later point in time. 52 | * 53 | * The user should be aware that there are several caveats to using this class. 54 | * 1. Access to the return value *may* incur a mutex lock overhead. 55 | * 2. Only types with copy constructors can be used because the class takes 56 | * a copy of the returned value to handle cases where addCallback() or 57 | * get() are called once the return value is out of scope. 58 | * 3. The return value is always const. Its possible to have multiple 59 | * callbacks registered and non-const would be dangerous. 60 | * 61 | * Aside from these conditions, the user is free to use a cast Future in place 62 | * of its representive type. In such cases, the object behaves synchronously 63 | * like a regualr return value. To use it asynchronously, simply call 64 | * addCallback() on the return value instead and let it fall out of scope. 65 | * (Futures maintain a reference counted internal state and thus will clean 66 | * themselves up.) 67 | * 68 | * @see Notification 69 | */ 70 | template 71 | class Future { 72 | public: 73 | Future() : _internal(new Internal()) { } 74 | Future(T value) : _internal(new Internal(new T(value))) { } 75 | Future(const Future &other) { 76 | _internal = other._internal; 77 | } 78 | Future &operator=(Future &other) { 79 | if (&other == this) { 80 | return *this; 81 | } 82 | if (_internal->hasValue()) { 83 | LOG(ERROR) << "Future assigned another future's value but value " 84 | << "has already been set."; 85 | return *this; 86 | } 87 | if (_internal->hasCallbacks()) { 88 | LOG(ERROR) << "Future assigned another future but we have " 89 | << "pending callbacks for this one."; 90 | } 91 | _internal = other._internal; 92 | return *this; 93 | } 94 | virtual ~Future() { } 95 | 96 | operator const T&() { return get(); } 97 | 98 | /** 99 | * Sets the return value. 100 | */ 101 | bool set(T value) { return _internal->set(new T(value)); } 102 | 103 | /** 104 | * Inherit the return value from another Future instance. 105 | */ 106 | bool set(Future &other) { *this = other; } 107 | 108 | /** 109 | * Waits until we either have a value to return or 'when' is reached. 110 | * @returns true if value available, false otherwise. 111 | */ 112 | bool tryWait(EventManager::WallTime when) { 113 | return _internal->tryWait(when); 114 | } 115 | 116 | /** 117 | * Returns the value, blocking if necessary until it becomes available. 118 | */ 119 | const T& get() { 120 | return _internal->get(); 121 | } 122 | 123 | /** 124 | * Registers a callback to get run when the Future's value is set. 125 | * If a callback is added after the value has been set, it will be 126 | * executed immediately. 127 | * @note Callbacks registered here will run on either the calling 128 | * thread or the thread that calls set(). 129 | */ 130 | void addCallback(function callback) { 131 | _internal->addCallback(callback); 132 | } 133 | 134 | /** 135 | * Deregisters a callback registered with addCallback() to ensure 136 | * it is never called after this function returns. 137 | */ 138 | void removeCallback(function callback) { 139 | _internal->removeCallback(callback); 140 | } 141 | 142 | /** 143 | * Adds an "errback", a callback that is triggered if the Future 144 | * is destroyed without ever setting a value. 145 | */ 146 | void addErrback(function errback) { 147 | _internal->addErrback(errback); 148 | } 149 | 150 | /** 151 | * Deregisters an errback. 152 | * @see addErrback 153 | */ 154 | void removeErrback(function errback) { 155 | _internal->removeErrback(errback); 156 | } 157 | 158 | private: 159 | class Internal { 160 | public: 161 | Internal() : _value(NULL) { 162 | pthread_mutex_init(&_mutex, 0); 163 | pthread_cond_init(&_cond, 0); 164 | } 165 | Internal(T* value) : _value(value) { 166 | pthread_mutex_init(&_mutex, 0); 167 | pthread_cond_init(&_cond, 0); 168 | } 169 | virtual ~Internal() { 170 | if (_value == NULL) { 171 | for (std::list< function >::iterator i = 172 | _errbacks.begin(); i != _errbacks.end(); ++i) { 173 | (*i)(); 174 | } 175 | } 176 | pthread_mutex_destroy(&_mutex); 177 | pthread_cond_destroy(&_cond); 178 | delete _value; 179 | } 180 | 181 | bool hasValue() { 182 | return (_value != NULL); 183 | } 184 | 185 | bool hasCallbacks() { 186 | return (_callbacks.size() != 0); 187 | } 188 | 189 | bool set(T* value) { 190 | if (_value != NULL) { 191 | delete value; 192 | return false; 193 | } 194 | pthread_mutex_lock(&_mutex); 195 | if (_value != NULL) { 196 | pthread_mutex_unlock(&_mutex); 197 | delete value; 198 | return false; 199 | } else { 200 | _value = value; 201 | pthread_cond_broadcast(&_cond); 202 | pthread_mutex_unlock(&_mutex); 203 | for (class std::list< function >::iterator i = 204 | _callbacks.begin(); i != _callbacks.end(); ++i) { 205 | (*i)(*_value); 206 | } 207 | // Must delete callbacks last as they may be holding references to us. 208 | _callbacks.clear(); 209 | _errbacks.clear(); 210 | return true; 211 | } 212 | } 213 | 214 | bool tryWait(EventManager::WallTime when) { 215 | if (_value != NULL) { 216 | return true; 217 | } 218 | pthread_mutex_lock(&_mutex); 219 | if (_value != NULL) { 220 | pthread_mutex_unlock(&_mutex); 221 | return true; 222 | } 223 | struct timespec ts = { (int64_t)when, 224 | (when - (int64_t)when) * 1000000000 }; 225 | int r = pthread_cond_timedwait(&_cond, &_mutex, &ts); 226 | pthread_mutex_unlock(&_mutex); 227 | return (r == 0); 228 | } 229 | 230 | const T& get() { 231 | if (_value != NULL) { 232 | return *_value; 233 | } 234 | pthread_mutex_lock(&_mutex); 235 | if (_value != NULL) { 236 | pthread_mutex_unlock(&_mutex); 237 | return *_value; 238 | } 239 | int ret = pthread_cond_wait(&_cond, &_mutex); 240 | pthread_mutex_unlock(&_mutex); 241 | return *_value; 242 | } 243 | 244 | void addCallback(function callback) { 245 | if (_value != NULL) { 246 | callback(*_value); 247 | return; 248 | } 249 | pthread_mutex_lock(&_mutex); 250 | if (_value != NULL) { 251 | pthread_mutex_unlock(&_mutex); 252 | callback(*_value); 253 | } else { 254 | _callbacks.push_back(callback); 255 | pthread_mutex_unlock(&_mutex); 256 | } 257 | } 258 | 259 | void removeCallback(function callback) { 260 | pthread_mutex_lock(&_mutex); 261 | _callbacks.erase(callback); 262 | pthread_mutex_unlock(&_mutex); 263 | } 264 | 265 | void addErrback(function errback) { 266 | pthread_mutex_lock(&_mutex); 267 | _errbacks.push_back(errback); 268 | pthread_mutex_unlock(&_mutex); 269 | } 270 | 271 | void removeErrback(function errback) { 272 | pthread_mutex_lock(&_mutex); 273 | _errbacks.erase(errback); 274 | pthread_mutex_unlock(&_mutex); 275 | } 276 | 277 | private: 278 | T* _value; 279 | pthread_mutex_t _mutex; 280 | pthread_cond_t _cond; 281 | std::list< function > _callbacks; 282 | std::list< function > _errbacks; 283 | 284 | Internal(const Internal& other); 285 | Internal& operator=(const Internal& other); 286 | }; 287 | shared_ptr _internal; 288 | }; 289 | 290 | /** 291 | * This is a convenience object that combines a counter and a set of Future 292 | * objects to create a synchronisation barrier. 293 | * 294 | * To do this cleanly, we use a special FutureSet class and virtualisation 295 | * to allow us to store different Future types in the same set. As a result 296 | * of this, we are not able to retrieve the results of any of these Future 297 | * objects so its the users responsibility to keep a reference to 298 | * the Future's around if their results are needed. 299 | * 300 | * @note This class expects to eventually be notified by all Futures. 301 | * If it is deleted before all its Future's have returned a result, 302 | * your application *will* crash. 303 | */ 304 | class FutureBarrier { 305 | public: 306 | /** 307 | * Stores a set of arbitrary Future instances. 308 | * These instances may be of different types so once added to a FutureSet 309 | * they can only ever be waited on. If the result is also required, a 310 | * copy of the original Future should be kept elsewhere. 311 | */ 312 | class FutureSet { 313 | public: 314 | FutureSet() { } 315 | virtual ~FutureSet() { } 316 | template 317 | void push_back(Future &item) { 318 | _items.insert(shared_ptr(new ConcreteItem(item))); 319 | } 320 | size_t size() const { 321 | return _items.size(); 322 | } 323 | private: 324 | friend class FutureBarrier; 325 | /** 326 | * We use virtualisation to strip the type specialisation from 327 | * Future instances for storage in a FutureSet. 328 | */ 329 | class Item { 330 | public: 331 | virtual ~Item() {} 332 | virtual void addCallback(function callback) = 0; 333 | virtual bool tryWait(EventManager::WallTime when) = 0; 334 | }; 335 | template 336 | class ConcreteItem : public Item { 337 | public: 338 | ConcreteItem(Future &val) : _val(val) { } 339 | virtual ~ConcreteItem() { } 340 | virtual void addCallback(function callback) { 341 | // Note that we register both a callback and an errback here 342 | // so the provided callback is triggered exactly once regardless 343 | // of whether or not the Future is ever set. 344 | _val.addCallback(bind(&addCallbackHelper, callback, 345 | std::tr1::placeholders::_1)); 346 | _val.addErrback(callback); 347 | } 348 | virtual bool tryWait(EventManager::WallTime when) { 349 | return _val.tryWait(when); 350 | } 351 | private: 352 | // Helper that discards argument on addCallback() callback. 353 | static void addCallbackHelper(function callback, const T&) { 354 | callback(); 355 | } 356 | Future _val; 357 | }; 358 | set< shared_ptr > _items; 359 | }; 360 | 361 | FutureBarrier(FutureSet &future_set) { 362 | pthread_mutex_init(&_mutex, 0); 363 | pthread_cond_init(&_cond, 0); 364 | _counter = future_set.size(); 365 | for (set< shared_ptr >::iterator i = 366 | future_set._items.begin(); 367 | i != future_set._items.end(); ++i) { 368 | (*i)->addCallback(bind(&FutureBarrier::signal, this)); 369 | } 370 | } 371 | 372 | ~FutureBarrier() { 373 | pthread_mutex_destroy(&_mutex); 374 | pthread_cond_destroy(&_cond); 375 | } 376 | 377 | void addCallback(function callback) { 378 | _callbacks.push_back(callback); 379 | if (!_counter) { 380 | callback(); 381 | return; 382 | } 383 | pthread_mutex_lock(&_mutex); 384 | if (!_counter) { 385 | pthread_mutex_unlock(&_mutex); 386 | callback(); 387 | } else { 388 | _callbacks.push_back(callback); 389 | pthread_mutex_unlock(&_mutex); 390 | } 391 | } 392 | 393 | bool tryWait(EventManager::WallTime when) { 394 | if (!_counter) { 395 | return true; 396 | } 397 | pthread_mutex_lock(&_mutex); 398 | if (!_counter) { 399 | pthread_mutex_unlock(&_mutex); 400 | return true; 401 | } 402 | struct timespec ts = { (int64_t)when, 403 | (when - (int64_t)when) * 1000000000 }; 404 | int r = pthread_cond_timedwait(&_cond, &_mutex, &ts); 405 | pthread_mutex_unlock(&_mutex); 406 | return (r == 0); 407 | } 408 | private: 409 | void signal() { 410 | pthread_mutex_lock(&_mutex); 411 | if (!--_counter) { 412 | pthread_cond_broadcast(&_cond); 413 | pthread_mutex_unlock(&_mutex); 414 | for (std::list< function >::iterator i = 415 | _callbacks.begin(); i != _callbacks.end(); ++i) { 416 | (*i)(); 417 | } 418 | return; 419 | } else { 420 | pthread_mutex_unlock(&_mutex); 421 | } 422 | } 423 | 424 | std::list< function > _callbacks; 425 | int _counter; 426 | pthread_mutex_t _mutex; 427 | pthread_cond_t _cond; 428 | 429 | // Illegal operators 430 | FutureBarrier(FutureBarrier &other); 431 | FutureBarrier& operator=(FutureBarrier &other); 432 | }; 433 | 434 | } 435 | 436 | #endif 437 | -------------------------------------------------------------------------------- /src/future_test.cc: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #include "future.h" 30 | 31 | #include "eventmanager.h" 32 | #include "notification.h" 33 | 34 | #include 35 | #include 36 | 37 | using epoll_threadpool::EventManager; 38 | using epoll_threadpool::Future; 39 | using epoll_threadpool::FutureBarrier; 40 | using epoll_threadpool::Notification; 41 | using std::string; 42 | 43 | TEST(FutureTest, BasicSynchronous) { 44 | Future f("apple"); 45 | ASSERT_EQ(string("apple"), f.get()); 46 | } 47 | 48 | TEST(FutureTest, TryGet) { 49 | Future f; 50 | EventManager::WallTime t = EventManager::currentTime(); 51 | ASSERT_FALSE(f.tryWait(t + 0.002)); 52 | f.set(string("apple")); 53 | ASSERT_TRUE(f.tryWait(t + 0.002)); 54 | ASSERT_EQ(string("apple"), (string)f); 55 | } 56 | 57 | void callbackHelper(Notification *n, const string &s) { 58 | ASSERT_EQ(string("apple"), s); 59 | n->signal(); 60 | } 61 | 62 | TEST(FutureTest, AddCallback) { 63 | Future f; 64 | Notification n1, n2, n3, n4; 65 | EventManager::WallTime t = EventManager::currentTime(); 66 | f.addCallback(bind(&callbackHelper, &n1, std::tr1::placeholders::_1)); 67 | f.addCallback(bind(&callbackHelper, &n2, std::tr1::placeholders::_1)); 68 | f.set("apple"); 69 | f.addCallback(bind(&callbackHelper, &n3, std::tr1::placeholders::_1)); 70 | ASSERT_EQ(string("apple"), f.get()); 71 | ASSERT_TRUE(n1.tryWait(t + 1)); 72 | ASSERT_TRUE(n2.tryWait(t + 1)); 73 | ASSERT_TRUE(n3.tryWait(t + 1)); 74 | f.addCallback(bind(&callbackHelper, &n4, std::tr1::placeholders::_1)); 75 | ASSERT_TRUE(n4.tryWait(t + 1)); 76 | } 77 | 78 | TEST(FutureBarrierTest, TryGet) { 79 | Future f1, f2, f3; 80 | 81 | FutureBarrier::FutureSet future_set; 82 | future_set.push_back(f1); 83 | future_set.push_back(f2); 84 | future_set.push_back(f3); 85 | FutureBarrier barrier(future_set); 86 | 87 | EventManager::WallTime t = EventManager::currentTime(); 88 | 89 | ASSERT_FALSE(barrier.tryWait(t + 0.001)); 90 | f1.set(string("apple")); 91 | ASSERT_FALSE(barrier.tryWait(t + 0.001)); 92 | f2.set(string("banana")); 93 | ASSERT_FALSE(barrier.tryWait(t + 0.001)); 94 | f3.set(string("carrot")); 95 | ASSERT_TRUE(barrier.tryWait(t + 0.004)); 96 | } 97 | 98 | TEST(FutureBarrierTest, AddCallback) { 99 | Future f1, f2, f3, f4; 100 | Notification n1, n2; 101 | 102 | FutureBarrier::FutureSet future_set; 103 | future_set.push_back(f1); 104 | future_set.push_back(f2); 105 | future_set.push_back(f3); 106 | FutureBarrier barrier(future_set); 107 | 108 | FutureBarrier::FutureSet future_set2; 109 | future_set2.push_back(f1); 110 | future_set2.push_back(f2); 111 | future_set2.push_back(f3); 112 | future_set2.push_back(f4); 113 | FutureBarrier barrier2(future_set2); 114 | 115 | EventManager::WallTime t = EventManager::currentTime(); 116 | 117 | f1.set(string("apple")); 118 | f2.set(string("banana")); 119 | f3.set(string("carrot")); 120 | 121 | // Add Callback after all preconditions have been met. 122 | barrier.addCallback(bind(&Notification::signal, &n1)); 123 | ASSERT_TRUE(n1.tryWait(t + 0.001)); 124 | 125 | // Add Callback before all preconditions have been met. 126 | barrier2.addCallback(bind(&Notification::signal, &n2)); 127 | f4.set(string("donut")); 128 | ASSERT_TRUE(n2.tryWait(t + 0.002)); 129 | } 130 | 131 | TEST(FutureBarrierTest, TryErrBack) { 132 | Notification n1, n2; 133 | EventManager::WallTime t = EventManager::currentTime(); 134 | 135 | FutureBarrier::FutureSet future_set; 136 | FutureBarrier barrier(future_set); 137 | { 138 | Future f1; 139 | future_set.push_back(f1); 140 | barrier.addCallback(bind(&Notification::signal, &n1)); 141 | f1.addCallback(bind(&Notification::signal, &n2)); 142 | } 143 | 144 | 145 | ASSERT_TRUE(n1.tryWait(t + 0.002)); 146 | ASSERT_TRUE(n1.tryWait(t + 0.002)); 147 | } 148 | 149 | -------------------------------------------------------------------------------- /src/iobuffer.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #ifndef _EPOLL_THREADPOOL_IOBUFFER_H_ 30 | #define _EPOLL_THREADPOOL_IOBUFFER_H_ 31 | 32 | #include 33 | #include 34 | #include 35 | #include 36 | 37 | namespace epoll_threadpool { 38 | 39 | using std::deque; 40 | using std::vector; 41 | 42 | /** 43 | * Stores blocks of data in potentially discontinuous blocks of memory. 44 | * The user can request these discontinuous blocks are pulled down into 45 | * continuous RAM for reading off. 46 | */ 47 | class IOBuffer { 48 | public: 49 | IOBuffer() : _size(0) {} 50 | 51 | /** 52 | * Convenience constructor for creating an IOBuffer out of a character 53 | * array. This *will* copy the data so avoid it if possible. 54 | */ 55 | IOBuffer(const char *data, size_t len) : _size(0) { 56 | append(data, len); 57 | } 58 | 59 | virtual ~IOBuffer() { 60 | for (int i = 0; i < _blocks.size(); i++) { 61 | delete _blocks[i]; 62 | } 63 | } 64 | 65 | /** 66 | * Appends another IOBuffer instance to this IOBuffer. All data from the 67 | * first buffer will be appended to this buffer without memcpying the 68 | * data itself. This class will take ownership of the IOBuffer instance. 69 | */ 70 | void append(IOBuffer *data) { 71 | for (int i = 0; i < data->_blocks.size(); i++) { 72 | _blocks.push_back(data->_blocks[i]); 73 | } 74 | _size += data->_size; 75 | data->_blocks.clear(); 76 | delete data; 77 | } 78 | 79 | /** 80 | * Appends a given vector of data to the IOBuffer. Ownership of the vector 81 | * is taken by the class, avoiding any data copies. 82 | */ 83 | void append(vector *data) { 84 | _size += data->size(); 85 | _blocks.push_back(data); 86 | } 87 | 88 | /** 89 | * Convenience method that allows appending of arbitrary data types. 90 | * @param data a pointer to an array of one or more elements of type T. 91 | * @param len the size of the array of data types T. 92 | */ 93 | template 94 | void append(T *data, size_t len) { 95 | vector *d = new vector(); 96 | d->resize(sizeof(T)*len); 97 | memcpy(&((*d)[0]), data, sizeof(T)*len); 98 | append(d); 99 | } 100 | 101 | /** 102 | * Equivalent to the previous append() method but required to use this class 103 | * as a destination buffer for msgpack::pack(). 104 | */ 105 | void write(const char *data, int len) { 106 | append(data, (size_t)len); 107 | } 108 | 109 | /** 110 | * Returns the current length in bytes of all blocks combined. 111 | */ 112 | size_t size() const { return _size; } 113 | 114 | /** 115 | * Ensures that the first n bytes are stored in contiguous memory. 116 | * Returns a pointer to the start of the memory on success, NULL on error. 117 | */ 118 | const char *pulldown(size_t bytes) { 119 | if (bytes > _size || _size == 0) { 120 | return NULL; 121 | } 122 | if (bytes > _blocks[0]->size()) { 123 | vector *pulldown_block = _blocks[0]; 124 | _blocks.pop_front(); 125 | while (pulldown_block->size() < bytes) { 126 | pulldown_block->insert( 127 | pulldown_block->end(), _blocks[0]->begin(), _blocks[0]->end()); 128 | delete _blocks[0]; 129 | _blocks.pop_front(); 130 | } 131 | _blocks.push_front(pulldown_block); 132 | } 133 | return &_blocks[0]->at(0); 134 | } 135 | 136 | /** 137 | * Consumes the first n bytes of the buffer. 138 | * Takes care of freeing memory, etc. 139 | * Returns false if asked to consume more data than is available. 140 | */ 141 | bool consume(size_t bytes) { 142 | if (bytes > _size || _size == 0) { 143 | return false; 144 | } 145 | while (!_blocks.empty() && bytes >= _blocks[0]->size()) { 146 | _size -= _blocks[0]->size(); 147 | bytes -= _blocks[0]->size(); 148 | delete _blocks[0]; 149 | _blocks.pop_front(); 150 | } 151 | if (_blocks.empty()) { 152 | return true; 153 | } 154 | if (bytes) { 155 | memmove(&_blocks[0]->at(0), &_blocks[0]->at(bytes), _blocks[0]->size() - bytes); 156 | _blocks[0]->resize(_blocks[0]->size() - bytes); 157 | _size -= bytes; 158 | } 159 | return true; 160 | } 161 | 162 | private: 163 | size_t _size; 164 | deque*> _blocks; 165 | 166 | private: 167 | IOBuffer(const IOBuffer&); 168 | }; 169 | } 170 | 171 | #endif 172 | -------------------------------------------------------------------------------- /src/iobuffer_test.cc: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #include "iobuffer.h" 30 | #include 31 | #include 32 | 33 | using epoll_threadpool::IOBuffer; 34 | using std::string; 35 | 36 | TEST(IOBufferTest, AppendBytes) { 37 | IOBuffer buf; 38 | const char *a = "abc", *b = "def"; 39 | 40 | buf.append(a, 3); 41 | buf.append(b, 3); 42 | ASSERT_EQ(6, buf.size()); 43 | ASSERT_EQ(NULL, buf.pulldown(7)); 44 | ASSERT_EQ(string("abcdef"), string(buf.pulldown(6), 6)); 45 | ASSERT_EQ(true, buf.consume(1)); 46 | ASSERT_EQ(5, buf.size()); 47 | ASSERT_EQ(NULL, buf.pulldown(6)); 48 | ASSERT_EQ(string("bcdef"), string(buf.pulldown(5), 5)); 49 | ASSERT_EQ(true, buf.consume(3)); 50 | ASSERT_EQ(2, buf.size()); 51 | ASSERT_EQ(string("ef"), string(buf.pulldown(2), 2)); 52 | ASSERT_EQ(NULL, buf.pulldown(3)); 53 | } 54 | 55 | TEST(IOBufferTest, ConsumeBeforePullDown) { 56 | IOBuffer buf; 57 | const char *a = "abc", *b = "def"; 58 | 59 | buf.append(a, 3); 60 | buf.append(b, 3); 61 | ASSERT_EQ(6, buf.size()); 62 | ASSERT_EQ(true, buf.consume(1)); 63 | ASSERT_EQ(5, buf.size()); 64 | ASSERT_EQ(NULL, buf.pulldown(6)); 65 | ASSERT_EQ(string("bcdef"), string(buf.pulldown(5), 5)); 66 | ASSERT_EQ(true, buf.consume(5)); 67 | ASSERT_EQ(0, buf.size()); 68 | ASSERT_EQ(NULL, buf.pulldown(1)); 69 | ASSERT_EQ(NULL, buf.pulldown(0)); 70 | } 71 | -------------------------------------------------------------------------------- /src/notification.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #ifndef _EPOLL_THREADPOOL_NOTIFICATION_H_ 30 | #define _EPOLL_THREADPOOL_NOTIFICATION_H_ 31 | 32 | #include "eventmanager.h" 33 | #include 34 | 35 | namespace epoll_threadpool { 36 | 37 | /** 38 | * Simple pthread based notification object. 39 | * Similar to pthread_cond_t but it can only be fired once and it will 40 | * stay triggered once fired. 41 | */ 42 | class Notification { 43 | public: 44 | Notification() { 45 | pthread_mutex_init(&_mutex, 0); 46 | pthread_cond_init(&_cond, 0); 47 | _signaled = false; 48 | } 49 | virtual ~Notification() { 50 | pthread_mutex_destroy(&_mutex); 51 | pthread_cond_destroy(&_cond); 52 | } 53 | 54 | void signal() { 55 | pthread_mutex_lock(&_mutex); 56 | _signaled = true; 57 | pthread_cond_broadcast(&_cond); 58 | pthread_mutex_unlock(&_mutex); 59 | } 60 | 61 | bool tryWait(EventManager::WallTime when) { 62 | pthread_mutex_lock(&_mutex); 63 | if (_signaled) { 64 | pthread_mutex_unlock(&_mutex); 65 | return true; 66 | } 67 | struct timespec ts = { (int64_t)when, (when - (int64_t)when) * 1000000000 }; 68 | int ret = pthread_cond_timedwait(&_cond, &_mutex, &ts); 69 | pthread_mutex_unlock(&_mutex); 70 | return ret == 0; 71 | } 72 | void wait() { 73 | pthread_mutex_lock(&_mutex); 74 | if (_signaled) { 75 | pthread_mutex_unlock(&_mutex); 76 | return; 77 | } 78 | int ret = pthread_cond_wait(&_cond, &_mutex); 79 | pthread_mutex_unlock(&_mutex); 80 | } 81 | protected: 82 | volatile bool _signaled; 83 | pthread_mutex_t _mutex; 84 | pthread_cond_t _cond; 85 | }; 86 | 87 | /** 88 | * Similar to Notification but required a set number of calls to signal() 89 | * before becoming 'signalled'. 90 | * TODO(aarond10): Fix terminology. Overloaded use of the word "signal". 91 | */ 92 | class CountingNotification : public Notification { 93 | public: 94 | /** 95 | * Creates a notification that will only be signalled after num calls to 96 | * signal(). 97 | */ 98 | CountingNotification(int num) : _num(num) { } 99 | virtual ~CountingNotification() { } 100 | 101 | void signal() { 102 | pthread_mutex_lock(&_mutex); 103 | if (--_num <= 0) { 104 | _signaled = true; 105 | pthread_cond_broadcast(&_cond); 106 | } 107 | pthread_mutex_unlock(&_mutex); 108 | } 109 | private: 110 | int _num; 111 | }; 112 | } 113 | #endif 114 | -------------------------------------------------------------------------------- /src/tcp.cc: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #include "tcp.h" 30 | #include "notification.h" 31 | 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include 37 | 38 | #include 39 | 40 | namespace epoll_threadpool { 41 | 42 | using std::tr1::function; 43 | using std::tr1::bind; 44 | 45 | TcpSocket::TcpSocket(EventManager* em, int fd) 46 | : _internal(new Internal(em, fd)) { 47 | } 48 | 49 | TcpSocket::~TcpSocket() { 50 | _internal->_disconnectCallback = NULL; 51 | _internal->disconnect(); 52 | } 53 | 54 | TcpSocket::Internal::Internal(EventManager* em, int fd) 55 | : _em(em), _fd(fd), _isStarted(false) { 56 | pthread_mutex_init(&_mutex, 0); 57 | fcntl(_fd, F_SETFL, O_NONBLOCK); 58 | int err; 59 | socklen_t size = sizeof(_maxSendSize); 60 | if ((err = getsockopt(fd, SOL_SOCKET, SO_SNDBUF, 61 | (char*) &_maxSendSize, &size)) != 0) { 62 | LOG(WARNING) << "Unable to determine maximum send size. Assuming 4k."; 63 | _maxSendSize = 4096; 64 | } 65 | size = sizeof(_maxReceiveSize); 66 | if ((err = getsockopt(fd, SOL_SOCKET, SO_RCVBUF, 67 | (char*) &_maxReceiveSize, &size)) != 0) { 68 | LOG(WARNING) << "Unable to determine maximum receive size. Assuming 4k."; 69 | _maxReceiveSize = 4096; 70 | } 71 | } 72 | 73 | TcpSocket::Internal::~Internal() { 74 | disconnect(); 75 | pthread_mutex_destroy(&_mutex); 76 | } 77 | 78 | void TcpSocket::start() { 79 | _internal->start(); 80 | } 81 | 82 | void TcpSocket::Internal::start() { 83 | pthread_mutex_lock(&_mutex); 84 | if (!_isStarted) { 85 | _isStarted = true; 86 | if (_fd > 0) { 87 | _em->watchFd(_fd, EventManager::EM_READ, 88 | bind(&TcpSocket::Internal::onReceive, shared_from_this())); 89 | _em->watchFd(_fd, EventManager::EM_WRITE, 90 | bind(&TcpSocket::Internal::onCanSend, shared_from_this())); 91 | 92 | // We trigger the receive handler to handle the case where we got 93 | // disconnected before start() completed. 94 | _em->enqueue( 95 | bind(&TcpSocket::Internal::onReceive, shared_from_this())); 96 | _em->enqueue( 97 | bind(&TcpSocket::Internal::onCanSend, shared_from_this())); 98 | } else { 99 | DLOG(INFO) << "disconnected in start()"; 100 | if (_disconnectCallback) { 101 | _em->enqueue(_disconnectCallback); 102 | _disconnectCallback = NULL; 103 | } 104 | } 105 | } 106 | pthread_mutex_unlock(&_mutex); 107 | } 108 | 109 | shared_ptr TcpSocket::connect( 110 | EventManager* em, string host, uint16_t port) { 111 | struct sockaddr_in sa; 112 | int fd = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); 113 | 114 | if (fd == -1) { 115 | LOG(ERROR) << "can not create socket"; 116 | return shared_ptr(); 117 | } 118 | 119 | memset(&sa, 0, sizeof(sa)); 120 | sa.sin_family = AF_INET; 121 | sa.sin_port = htons(port); 122 | if (inet_pton(AF_INET, host.c_str(), &sa.sin_addr) <= 0) { 123 | LOG(ERROR) << "Failed to resolve address: " << host; 124 | close(fd); 125 | return shared_ptr(); 126 | } 127 | 128 | if (::connect(fd, (struct sockaddr*)&sa, sizeof(sa)) == -1) { 129 | close(fd); 130 | return shared_ptr(); 131 | } 132 | 133 | return shared_ptr(new TcpSocket(em, fd)); 134 | } 135 | 136 | void TcpSocket::write(IOBuffer* data) { 137 | _internal->write(data); 138 | } 139 | 140 | void TcpSocket::disconnect() { 141 | _internal->disconnect(); 142 | } 143 | 144 | bool TcpSocket::isDisconnected() const { 145 | return _internal->_fd == -1; 146 | } 147 | 148 | void TcpSocket::setReceiveCallback(function callback) { 149 | pthread_mutex_lock(&_internal->_mutex); 150 | _internal->_recvCallback = callback; 151 | pthread_mutex_unlock(&_internal->_mutex); 152 | } 153 | 154 | void TcpSocket::setDisconnectCallback(function callback) { 155 | pthread_mutex_lock(&_internal->_mutex); 156 | _internal->_disconnectCallback = callback; 157 | pthread_mutex_unlock(&_internal->_mutex); 158 | } 159 | 160 | void TcpSocket::Internal::write(IOBuffer* data) { 161 | pthread_mutex_lock(&_mutex); 162 | bool wasBufferEmpty = (_sendBuffer.size() == 0); 163 | _sendBuffer.append(data); 164 | if (_fd >= 0 && _isStarted && wasBufferEmpty) { 165 | _em->watchFd(_fd, EventManager::EM_WRITE, 166 | bind(&TcpSocket::Internal::onCanSend, shared_from_this())); 167 | } 168 | pthread_mutex_unlock(&_mutex); 169 | } 170 | 171 | void TcpSocket::Internal::disconnect() { 172 | pthread_mutex_lock(&_mutex); 173 | if (_fd > 0) { 174 | _em->removeFd(_fd, EventManager::EM_READ); 175 | _em->removeFd(_fd, EventManager::EM_WRITE); 176 | _em->removeFd(_fd, EventManager::EM_ERROR); 177 | ::shutdown(_fd, SHUT_RDWR); 178 | ::close(_fd); 179 | _fd = -1; 180 | if (_disconnectCallback) { 181 | _em->enqueue(_disconnectCallback); 182 | _disconnectCallback = NULL; 183 | } 184 | } 185 | pthread_mutex_unlock(&_mutex); 186 | } 187 | 188 | void TcpSocket::Internal::onReceive() { 189 | pthread_mutex_lock(&_mutex); 190 | while (_fd > 0) { 191 | vector* buf = new vector(); 192 | buf->resize(_maxReceiveSize); 193 | 194 | int r = ::recv(_fd, &(*buf)[0], buf->size(), 0); 195 | if (r > 0) { 196 | buf->resize(r); 197 | _recvBuffer.append(buf); 198 | if (_recvCallback) { 199 | _recvCallback(&_recvBuffer); 200 | } 201 | } else { 202 | if (r < 0) { 203 | delete buf; 204 | if (errno != EAGAIN) { 205 | LOG(INFO) << "Read error. (" << errno << "). Disconnecting fd " << _fd; 206 | _em->enqueue(bind( 207 | &TcpSocket::Internal::disconnect, shared_from_this())); 208 | } 209 | } else { 210 | delete buf; 211 | _em->enqueue(bind( 212 | &TcpSocket::Internal::disconnect, shared_from_this())); 213 | } 214 | break; 215 | } 216 | } 217 | pthread_mutex_unlock(&_mutex); 218 | } 219 | 220 | void TcpSocket::Internal::onCanSend() { 221 | pthread_mutex_lock(&_mutex); 222 | if (_fd > 0) { 223 | while (_sendBuffer.size()) { 224 | int sz = (_sendBuffer.size() > _maxSendSize) ? 225 | _maxSendSize : _sendBuffer.size(); 226 | const char* buf = _sendBuffer.pulldown(sz); 227 | if (buf) { 228 | int r = ::send(_fd, buf, sz, MSG_NOSIGNAL); 229 | if (r > 0) { 230 | _sendBuffer.consume(r); 231 | } else if (r < 0) { 232 | if (errno != EAGAIN) { 233 | LOG(INFO) << "Write error. (" << errno << "). Disconnecting fd " << _fd; 234 | _em->enqueue(bind( 235 | &TcpSocket::Internal::disconnect, shared_from_this())); 236 | } 237 | pthread_mutex_unlock(&_mutex); 238 | return; 239 | } else { 240 | _em->enqueue(bind( 241 | &TcpSocket::Internal::disconnect, shared_from_this())); 242 | } 243 | } 244 | } 245 | _em->removeFd(_fd, EventManager::EM_WRITE); 246 | } 247 | pthread_mutex_unlock(&_mutex); 248 | } 249 | 250 | TcpListenSocket::TcpListenSocket(EventManager* em, int fd) 251 | : _internal(new Internal(em, fd)) { 252 | em->watchFd(fd, EventManager::EM_READ, bind( 253 | &TcpListenSocket::Internal::onAccept, _internal)); 254 | } 255 | 256 | TcpListenSocket::~TcpListenSocket() { 257 | _internal->shutdown(); 258 | } 259 | 260 | TcpListenSocket::Internal::Internal(EventManager* em, int fd) 261 | : _em(em), _fd(fd) { 262 | pthread_mutex_init(&_mutex, 0); 263 | fcntl(_fd, F_SETFL, O_NONBLOCK); 264 | } 265 | 266 | TcpListenSocket::Internal::~Internal() { 267 | pthread_mutex_destroy(&_mutex); 268 | } 269 | 270 | void TcpListenSocket::Internal::shutdown() { 271 | pthread_mutex_lock(&_mutex); 272 | _em->removeFd(_fd, EventManager::EM_READ); 273 | ::shutdown(_fd, SHUT_RDWR); 274 | ::close(_fd); 275 | _fd = -1; 276 | pthread_mutex_unlock(&_mutex); 277 | } 278 | 279 | shared_ptr TcpListenSocket::create( 280 | EventManager* em, uint16_t port) { 281 | struct sockaddr_in sa; 282 | int fd = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); 283 | 284 | if (fd == -1) { 285 | LOG(ERROR) << "can not create socket"; 286 | return shared_ptr(); 287 | } 288 | 289 | memset(&sa, 0, sizeof(sa)); 290 | sa.sin_family = AF_INET; 291 | sa.sin_port = htons(port); 292 | sa.sin_addr.s_addr = INADDR_ANY; 293 | 294 | if (::bind(fd, (struct sockaddr *)&sa, sizeof(sa)) == -1) { 295 | LOG(ERROR) << "error bind failed"; 296 | close(fd); 297 | return shared_ptr(); 298 | } 299 | 300 | if (::listen(fd, 5) == -1) { 301 | LOG(ERROR) << "error listen failed"; 302 | close(fd); 303 | return shared_ptr(); 304 | } 305 | 306 | return shared_ptr(new TcpListenSocket(em, fd)); 307 | } 308 | 309 | void TcpListenSocket::setAcceptCallback( 310 | function)> callback) { 311 | pthread_mutex_lock(&_internal->_mutex); 312 | _internal->_callback = callback; 313 | pthread_mutex_unlock(&_internal->_mutex); 314 | } 315 | 316 | void TcpListenSocket::Internal::onAccept() { 317 | pthread_mutex_lock(&_mutex); 318 | if (_fd > 0) { 319 | int fd = ::accept(_fd, NULL, NULL); 320 | pthread_mutex_unlock(&_mutex); 321 | if (fd > 0) { 322 | shared_ptr s(new TcpSocket(_em, fd)); 323 | if (_callback) { 324 | _callback(s); 325 | // We separate registration to give AcceptCallback a chance to set up 326 | // receive / disconnect callbacks for the socket. The callback must 327 | // call start() on the socket to begin receiving traffic. 328 | } 329 | } 330 | } else { 331 | pthread_mutex_unlock(&_mutex); 332 | } 333 | } 334 | } 335 | -------------------------------------------------------------------------------- /src/tcp.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #ifndef _EPOLL_THREADPOOL_TCP_H_ 30 | #define _EPOLL_THREADPOOL_TCP_H_ 31 | 32 | #include "eventmanager.h" 33 | #include "iobuffer.h" 34 | 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include 40 | 41 | #include 42 | #include 43 | #include 44 | 45 | namespace epoll_threadpool { 46 | 47 | using std::string; 48 | using std::tr1::bind; 49 | using std::tr1::function; 50 | using std::tr1::shared_ptr; 51 | 52 | class TcpListenSocket; 53 | 54 | /** 55 | * Represents a TCP data socket connected to some remote endpoint. 56 | * 57 | * An instance of this class can only be created internally by TcpListenSocket 58 | * in the case of incoming connections or by TcpSocket::connect for outgoing 59 | * connections. In both cases, a TcpSocket is in a connected state when 60 | * first passed to the user. 61 | * 62 | * setReceiveCallback and setDisconnectCallback methods can (and should) both 63 | * be set prior to using the socket. To avoid a potential race condition, a 64 | * TcpSocket will not start processing data until start() is called, giving the 65 | * user a chance to set up callbacks safely without missing potential events. 66 | * 67 | * After a sockets disconnect callback has been triggered, we guarantee that no 68 | * subsequent callbacks will be triggered and thus that the object can be 69 | * safely deleted (by reset()ing or otherwise letting shared_ptr's to the 70 | * object fall out of scope). 71 | * 72 | * TODO(aarond10): Confirm and add test cases for the following: 73 | * Both disconnect and receive callbacks are guaranteed to run on one of socket 74 | * EventManager's threads. As a general rule of thumb, these functions should 75 | * not block as the EventManager's thread pool is of limited size. 76 | * 77 | * The disconnect callback will NOT be run if a TcpSocket is disconnected at 78 | * the time of deletion. (i.e. dereferencing to zero) 79 | */ 80 | class TcpSocket { 81 | public: 82 | virtual ~TcpSocket(); 83 | 84 | /** 85 | * Attempts to connect to the given host:port and return a TcpSocket for the 86 | * connection. If a connection cannot be made, returns NULL. 87 | */ 88 | static shared_ptr connect( 89 | EventManager* em, string host, uint16_t port); 90 | 91 | /** 92 | * Called to begin the event handling. 93 | * This should be called exactly once. It is done explicitly to give 94 | * sockets a chance to set up event callbacks. 95 | */ 96 | void start(); 97 | 98 | /** 99 | * Writes data to the TCP stream. 100 | * This class takes ownership of the provided IOBuffer instance. 101 | */ 102 | void write(IOBuffer* data); 103 | 104 | /** 105 | * Disconnects the socket endpoint. No further events will be triggered. 106 | */ 107 | void disconnect(); 108 | 109 | /** 110 | * Returns true if the socket is closed. 111 | */ 112 | bool isDisconnected() const; 113 | 114 | /** 115 | * Registers a callback to be triggered on received data. 116 | * The callback is passed a pointer to an IOBuffer. This IOBuffer remains 117 | * the property of this class. The callback is free to consume data at will. 118 | */ 119 | void setReceiveCallback(function callback); 120 | 121 | /** 122 | * Registes a callback to be triggered on disconnect. 123 | */ 124 | void setDisconnectCallback(function callback); 125 | 126 | /** 127 | * Returns the eventmanager for this class. 128 | */ 129 | EventManager* getEventManager() const { return _internal->_em; } 130 | 131 | /** 132 | * Returns the file descriptor for the socket. 133 | * @note This shouldn't be used directly. Its exposed for debugging. 134 | */ 135 | int fd() const { return _internal->_fd; } 136 | 137 | private: 138 | // Because we need to preserve data structures until we can be sure they 139 | // aren't in use on worker threads, we store them separately. 140 | class Internal : public std::tr1::enable_shared_from_this { 141 | public: 142 | Internal(EventManager* em, int fd); 143 | ~Internal(); 144 | pthread_mutex_t _mutex; 145 | EventManager* _em; 146 | int _fd; 147 | int _maxReceiveSize; 148 | int _maxSendSize; 149 | volatile bool _isStarted; 150 | IOBuffer _recvBuffer; 151 | IOBuffer _sendBuffer; 152 | function _recvCallback; 153 | function _disconnectCallback; 154 | 155 | void start(); 156 | void write(IOBuffer* data); 157 | void disconnect(); 158 | void onReceive(); 159 | void onCanSend(); 160 | 161 | static void cleanup(shared_ptr ptr) { 162 | ptr.reset(); 163 | } 164 | }; 165 | shared_ptr _internal; 166 | 167 | friend class TcpListenSocket; 168 | TcpSocket(EventManager* em, int fd); 169 | 170 | private: 171 | // Bad constructors not implemented. 172 | TcpSocket(const TcpSocket&); 173 | }; 174 | 175 | /** 176 | * Represents a TCP listening socket on a specific TCP port. 177 | * 178 | * When a connection is made to the socket, it will be accepted and a 179 | * shared_ptr passed to the registered accept callback. 180 | * If an accept callback has not been set, the TcpSocket will end up being 181 | * dereferenced to zero and deleted. 182 | * 183 | * It is the callback's responsibility to set up any event handlers on the 184 | * TcpSocket and then call its start() method to begin receiving events. 185 | */ 186 | class TcpListenSocket { 187 | public: 188 | virtual ~TcpListenSocket(); 189 | 190 | /** 191 | * Attempts to create a new TcpListenSocket on a provided TCP port. 192 | * Uses the provided EventManager to handle the incoming connections. 193 | * Returns a pointer to a new TcpListenSocket object on success, NULL on 194 | * failure. 195 | */ 196 | static shared_ptr create(EventManager* em, uint16_t port); 197 | 198 | /** 199 | * Registers a callback to be triggered when a new client connects. 200 | * The callback is passed a pointer to a new TcpSocket instance for the 201 | * connection. The callback inherits ownership of this socket. 202 | */ 203 | void setAcceptCallback(function)> callback); 204 | 205 | /** 206 | * Returns the eventmanager for this class. 207 | */ 208 | EventManager* getEventManager() const { return _internal->_em; } 209 | 210 | private: 211 | // Because we need to preserve data structures until we can be sure they 212 | // aren't in use on worker threads, we store them separately. 213 | class Internal : public std::tr1::enable_shared_from_this { 214 | public: 215 | Internal(EventManager* em, int fd); 216 | ~Internal(); 217 | 218 | pthread_mutex_t _mutex; 219 | EventManager *_em; 220 | int _fd; 221 | function)> _callback; 222 | 223 | void shutdown(); 224 | void onAccept(); 225 | 226 | static void cleanup(shared_ptr ptr) { 227 | ptr.reset(); 228 | } 229 | }; 230 | shared_ptr _internal; 231 | 232 | TcpListenSocket(EventManager* em, int fd); 233 | 234 | private: 235 | // Bad constructors not implemented. 236 | TcpListenSocket(const TcpListenSocket&); 237 | }; 238 | } 239 | #endif 240 | -------------------------------------------------------------------------------- /src/tcp_test.cc: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2011 Aaron Drew 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions 7 | are met: 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 2. Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in the 12 | documentation and/or other materials provided with the distribution. 13 | 3. Neither the name of the copyright holders nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 21 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF 27 | THE POSSIBILITY OF SUCH DAMAGE. 28 | */ 29 | #include "gtest/gtest.h" 30 | 31 | #include 32 | #include 33 | 34 | #include 35 | 36 | #include "eventmanager.h" 37 | #include "iobuffer.h" 38 | #include "notification.h" 39 | #include "tcp.h" 40 | 41 | using epoll_threadpool::EventManager; 42 | using epoll_threadpool::IOBuffer; 43 | using epoll_threadpool::Notification; 44 | using epoll_threadpool::TcpSocket; 45 | using epoll_threadpool::TcpListenSocket; 46 | 47 | using namespace std; 48 | using namespace std::tr1; 49 | using namespace std::tr1::placeholders; 50 | 51 | shared_ptr createListenSocket( 52 | EventManager *em, int &port) { 53 | shared_ptr s; 54 | while(!s) { 55 | port = (rand()%40000) + 1024; 56 | s = TcpListenSocket::create(em, port); 57 | } 58 | return s; 59 | } 60 | 61 | // Code coverage - test basic creation and listening functionality. 62 | TEST(TCPTest, ListenTest) { 63 | EventManager em; 64 | em.start(4); 65 | int port = 0; 66 | shared_ptr s = createListenSocket(&em, port); 67 | ASSERT_TRUE(s != NULL); 68 | } 69 | 70 | // Code coverage - test connect and failed connect code paths. 71 | TEST(TCPTest, ListenConnectTest) { 72 | EventManager em; 73 | em.start(4); 74 | 75 | int port = 0; 76 | shared_ptr s = createListenSocket(&em, port); 77 | // TODO(aarond10): Add TcpListenSocket::start() to avoid a race issue? 78 | shared_ptr c = TcpSocket::connect(&em, "127.0.0.1", port); 79 | ASSERT_TRUE(s); 80 | ASSERT_TRUE(c); 81 | 82 | // Expect this to fail and return NULL. On occasions we will hit a port in 83 | // use so we try a few times before failing 84 | bool found = false; 85 | for (int i = 1; i < 4 && !found; i++) { 86 | shared_ptr c2 = TcpSocket::connect(&em, "127.0.0.1", port+i); 87 | if (!c2) { 88 | found = true; 89 | } 90 | } 91 | ASSERT_TRUE(found); 92 | } 93 | 94 | void receiveChecker(Notification *n, string expected, IOBuffer *data) { 95 | uint64_t *psize = (uint64_t *)(data->pulldown(sizeof(uint64_t))); 96 | if (psize != NULL && *psize > 0 && data->size() >= *psize) { 97 | uint64_t size = *psize; 98 | ASSERT_TRUE(data->consume(sizeof(uint64_t))); 99 | const char *str = static_cast(data->pulldown(size)); 100 | ASSERT_TRUE(str != NULL); 101 | ASSERT_EQ(expected, string(str, size)); 102 | data->consume(size); 103 | n->signal(); 104 | } 105 | } 106 | 107 | void acceptHandler(Notification *n, string expected, shared_ptr *ps, shared_ptr s) { 108 | *ps = s; 109 | s->setReceiveCallback(std::tr1::bind(&receiveChecker, n, expected, _1)); 110 | s->start(); 111 | } 112 | 113 | // Code coverage - tests we can accept data from a newly connected client 114 | TEST(TCPTest, ListenConnectSendDataTest) { 115 | EventManager em; 116 | em.start(1); 117 | 118 | EventManager::WallTime t = EventManager::currentTime(); 119 | const char *data = "some test data"; 120 | uint64_t size = strlen(data); 121 | 122 | int port = 0; 123 | shared_ptr s = createListenSocket(&em, port); 124 | 125 | Notification n; 126 | shared_ptr ps; 127 | s->setAcceptCallback(std::tr1::bind(&acceptHandler, &n, string(data), &ps, _1)); 128 | 129 | shared_ptr c = TcpSocket::connect(&em, "127.0.0.1", port); 130 | ASSERT_TRUE(c != NULL); 131 | c->start(); 132 | 133 | ASSERT_FALSE(n.tryWait(t+0.001)); 134 | c->write(new IOBuffer((char *)&size, sizeof(size))); 135 | ASSERT_FALSE(n.tryWait(t+0.002)); 136 | c->write(new IOBuffer(data, size)); 137 | ASSERT_TRUE(n.tryWait(t+10.0)); 138 | ASSERT_TRUE(ps != NULL); 139 | 140 | c->disconnect(); 141 | ps->disconnect(); 142 | } 143 | 144 | void disconnectChecker(Notification *n) { 145 | n->signal(); 146 | } 147 | 148 | void acceptDisconnectHandler(Notification *n, shared_ptr *ps, shared_ptr s) { 149 | *ps = s; 150 | s->setDisconnectCallback(std::tr1::bind(&disconnectChecker, n)); 151 | s->start(); 152 | } 153 | 154 | // Code coverage - test disconnect handler. 155 | TEST(TCPTest, ListenConnectDisconnectTest) { 156 | EventManager em; 157 | em.start(4); 158 | 159 | EventManager::WallTime t = EventManager::currentTime(); 160 | const char *data = "some test data"; 161 | uint64_t size = strlen(data); 162 | 163 | int port = 0; 164 | shared_ptr s = createListenSocket(&em, port); 165 | 166 | Notification n; 167 | shared_ptr ps; 168 | s->setAcceptCallback(std::tr1::bind(&acceptDisconnectHandler, &n, &ps, _1)); 169 | //s->start(); 170 | 171 | shared_ptr c = TcpSocket::connect(&em, "127.0.0.1", port); 172 | ASSERT_TRUE(c != NULL); 173 | 174 | ASSERT_FALSE(n.tryWait(t+0.001)); 175 | c->disconnect(); 176 | ASSERT_TRUE(n.tryWait(t+0.4)); 177 | ASSERT_TRUE(ps != NULL); 178 | 179 | ps->disconnect(); 180 | } 181 | 182 | void acceptConnectionCallback(shared_ptr s) { 183 | // This does nothing but because we don't store the TcpSocket, it will cause 184 | // it to get destroyed, immediately disconnecting the endpoint. 185 | } 186 | 187 | // Order of operations - write before start. 188 | TEST(TCPTest, WriteBeforeStartedClient) { 189 | EventManager em; 190 | em.start(4); 191 | 192 | int port = 0; 193 | shared_ptr s = createListenSocket(&em, port); 194 | s->setAcceptCallback(std::tr1::bind(&acceptConnectionCallback, std::tr1::placeholders::_1)); 195 | 196 | shared_ptr t(TcpSocket::connect(&em, "127.0.0.1", port)); 197 | Notification n; 198 | t->setDisconnectCallback(std::tr1::bind(&Notification::signal, &n)); 199 | t->write(new IOBuffer("abcd", 4)); 200 | t->start(); 201 | LOG(INFO) << "Print " << t->isDisconnected(); 202 | n.wait(); 203 | } 204 | 205 | void FillWriteBufferReceive(int *counter, Notification *n, IOBuffer *buf) { 206 | int s = buf->size(); 207 | *counter -= s; 208 | buf->consume(s); 209 | CHECK_GE(*counter, 0); 210 | if (*counter == 0) { 211 | n->signal(); 212 | } 213 | } 214 | 215 | // Consume data and count bytes. 216 | void FillWriteBufferAccept(int *counter, Notification *n, shared_ptr *ps, shared_ptr s) { 217 | *ps = s; 218 | s->setReceiveCallback(std::tr1::bind(&FillWriteBufferReceive, 219 | counter, n, std::tr1::placeholders::_1)); 220 | s->start(); 221 | } 222 | 223 | // Fill write buffer - Big write 224 | TEST(TCPTest, FillWriteBuffer) { 225 | EventManager em; 226 | em.start(4); 227 | 228 | int counter = 0; 229 | Notification n; 230 | shared_ptr accept_sock; 231 | 232 | int port = 0; 233 | shared_ptr s = createListenSocket(&em, port); 234 | s->setAcceptCallback(std::tr1::bind(&FillWriteBufferAccept, &counter, &n, &accept_sock, std::tr1::placeholders::_1)); 235 | shared_ptr t(TcpSocket::connect(&em, "127.0.0.1", port)); 236 | 237 | const int kSize = 1024*1024; 238 | //const int kSize = 1024 * 10; 239 | counter = kSize; 240 | char *data = new char[kSize]; 241 | for (int i = 0; i < kSize; ++i) { 242 | data[i] = i; 243 | } 244 | t->write(new IOBuffer(data, kSize)); 245 | delete[] data; 246 | 247 | t->start(); 248 | n.wait(); 249 | t->disconnect(); 250 | } 251 | 252 | // Order of operations - write after disconnect (no effect) 253 | TEST(TCPTest, WriteAfterDisconnect) { 254 | EventManager em; 255 | em.start(4); 256 | 257 | int port = 0; 258 | shared_ptr s = createListenSocket(&em, port); 259 | shared_ptr t(TcpSocket::connect(&em, "127.0.0.1", port)); 260 | 261 | Notification n; 262 | 263 | const char data[] = "testdata"; 264 | t->setDisconnectCallback(std::tr1::bind(&Notification::signal, &n)); 265 | t->start(); 266 | n.wait(); 267 | t->write(new IOBuffer(data, sizeof(data))); 268 | } 269 | 270 | // TODO: Order of operations - disconnect after disconnect 271 | // TODO: Order of operations - start after disconnect 272 | // TODO: Order of operations - disconnect from receive handler 273 | // TODO: Order of operations - write from receive handler 274 | // TODO: Destruction - destroy while receiving 275 | // TODO: Destruction - destroy before start 276 | // TODO: Destruction - destroy before disconnect 277 | // TODO: Destruction - destroy after disconnect 278 | --------------------------------------------------------------------------------