├── golibmc ├── setup.cfg ├── tox.ini ├── misc ├── travis │ ├── benchmark.sh │ ├── gotest.sh │ ├── buildwheels.sh │ ├── cppcheck.sh │ ├── unittest.sh │ └── cpptest.sh ├── slow_server.sh ├── deploy_pypi.sh ├── build_wheels.sh ├── graph_all.sh ├── update_version.sh ├── gomcbench │ ├── README.md │ └── gomcbench_test.go ├── .cppcheck-supp ├── idle_timeout_memcached_server ├── generate_hash_dataset.py ├── versioning.py └── memcached_server ├── MANIFEST.in ├── Makefile ├── .gitignore ├── tests ├── test_size.cpp ├── shabby │ ├── infinite_set_get.py │ ├── independent_timeout.py │ ├── timeout_issue.py │ ├── run_test.sh │ ├── gevent_issue.py │ ├── reproduce_incomplete_data_block_bug_on_get_multi.py │ ├── gil_issue.py │ ├── reconnect_delay.py │ └── slow_memcached_server.py ├── CMakeLists.txt ├── golibmc_debug_client.go ├── debug_client.cpp ├── test_unix.cpp ├── test_ketama.cpp ├── test_parser.cpp ├── test_common.h ├── test_client_pool.cpp ├── test_client_pool.py ├── profile_client.cpp ├── test_hashkit.cpp ├── test_router.py └── resources │ └── server_port.csv ├── src ├── version.go ├── HashkitFnv.cpp ├── Utility.cpp ├── BufferWriter.cpp ├── Common.cpp ├── Result.cpp ├── DataBlock.cpp ├── HashkitCrc.cpp ├── ClientPool.cpp ├── HashkitKetama.cpp ├── c_client.cpp ├── Client.cpp └── HashkitMd5.cpp ├── .github └── workflows │ ├── cpp.yml │ ├── golang.yml │ ├── publish.yml │ └── python.yml ├── include ├── hashkit │ ├── hashkit.h │ ├── ketama.h │ └── md5.h ├── Utility.h ├── BufferWriter.h ├── Result.h ├── DataBlock.h ├── Keywords.h ├── Parser.h ├── Export.h ├── LockPool.h ├── ConnectionPool.h ├── Connection.h ├── BufferReader.h ├── Client.h ├── c_client.h ├── ClientPool.h ├── llvm │ └── type_traits.h └── Common.h ├── ext └── gtest │ └── CMakeLists.txt.in ├── CONTRIBUTING.md ├── LICENSE.txt ├── .travis.yml.bak ├── libmc └── __init__.py ├── CMakeLists.txt ├── setup.py └── README.rst /golibmc: -------------------------------------------------------------------------------- 1 | src -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | description-file = README.rst 3 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27,py35,py36,py37,py38 3 | platform = linux2|linux|darwin 4 | -------------------------------------------------------------------------------- /misc/travis/benchmark.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | pip install -e . 5 | python ./misc/runbench.py 6 | -------------------------------------------------------------------------------- /misc/travis/gotest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | echo "CXX=${CXX}" 5 | go version 6 | cd golibmc 7 | go test 8 | -------------------------------------------------------------------------------- /misc/slow_server.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | python tests/shabby/slow_memcached_server.py & 5 | pid=$! 6 | echo "pid of slow memcached server: $pid" 7 | 8 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.rst 2 | include LICENSE.txt 3 | recursive-include include *.h 4 | recursive-include src *.cpp 5 | recursive-include libmc *.py *.pyx 6 | prune tests 7 | prune misc 8 | prune build 9 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | cppcheck: 2 | ./misc/travis/cppcheck.sh 3 | 4 | cpptest: 5 | CC=gcc CXX=g++ CMAKE_BUILD_TYPE=Debug ./misc/travis/cpptest.sh 6 | 7 | gotest: 8 | cd golibmc; go test -v 9 | 10 | pytest: 11 | pytest tests/ 12 | -------------------------------------------------------------------------------- /misc/deploy_pypi.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | 4 | pip install --upgrade wheel twine 5 | python setup.py sdist 6 | twine upload \ 7 | --non-interactive \ 8 | --skip-existing \ 9 | --verbose \ 10 | --repository-url https://upload.pypi.org/legacy/ \ 11 | dist/* 12 | -------------------------------------------------------------------------------- /misc/travis/buildwheels.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | docker run --rm \ 5 | -v `pwd`:/io \ 6 | -e DIST_VERSIONS="cp27-cp27mu cp36-cp36m cp37-cp37m cp38-cp38" \ 7 | -e PLAT=$PLAT \ 8 | quay.io/pypa/$PLAT \ 9 | /io/misc/build_wheels.sh 10 | 11 | mkdir -p dist 12 | ls -lh wheelhouse/*${PLAT}.whl 13 | cp wheelhouse/*${PLAT}.whl dist/ 14 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # python 2 | *.pyc 3 | *.egg-info 4 | *.egg/ 5 | *.egg 6 | .eggs/ 7 | dist/ 8 | 9 | # Ignore CI build directory 10 | cmake-build-debug/ 11 | .idea/ 12 | build/ 13 | venv/ 14 | bench_venv/ 15 | _client.cpp 16 | _client.so 17 | _client.*.so 18 | .tox/ 19 | 20 | .cache/ 21 | .clang_complete 22 | *.sw[op] 23 | tests/resources/keys_*.txt 24 | .vscode/ 25 | 26 | /go.mod 27 | /go.sum 28 | -------------------------------------------------------------------------------- /misc/build_wheels.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | # Compile wheels 5 | for CPYVER in $DIST_VERSIONS; do 6 | "/opt/python/${CPYVER}/bin/pip" wheel /io/ -w wheelhouse/ 7 | done 8 | 9 | # Bundle external shared libraries into the wheels 10 | for whl in wheelhouse/*.whl; do 11 | auditwheel repair "$whl" --plat $PLAT -w /io/wheelhouse/ 12 | done 13 | 14 | chmod -R 0777 /io/wheelhouse/ 15 | -------------------------------------------------------------------------------- /tests/test_size.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include "BufferReader.h" 3 | #include "Result.h" 4 | #include "gtest/gtest.h" 5 | 6 | 7 | TEST(test_size, main) { 8 | 9 | #ifdef MC_USE_SMALL_VECTOR 10 | ASSERT_EQ(sizeof(douban::mc::io::TokenData), 48U); 11 | #else 12 | ASSERT_EQ(sizeof(douban::mc::io::TokenData), 24U); 13 | #endif 14 | ASSERT_EQ(sizeof(douban::mc::io::TokenData*), 8U); 15 | } 16 | -------------------------------------------------------------------------------- /misc/travis/cppcheck.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | # NOTE:(everpcpc) ubuntu:bionic with cppcheck 1.82 introduce a bug: 5 | # https://bugs.launchpad.net/ubuntu/+source/cppcheck/+bug/1763841 6 | # https://github.com/danmar/cppcheck/pull/1026 7 | cppcheck --enable=all --error-exitcode=1 \ 8 | -UMSG_MORE -UNI_MAXHOST -UNI_MAXSERV -UUIO_MAXIOV \ 9 | --suppressions-list=misc/.cppcheck-supp -Iinclude src tests 10 | -------------------------------------------------------------------------------- /misc/graph_all.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | gen_graph () { 4 | gprof2dot --format=callgrind --output=$2 $1 5 | dot -Tpng $2 -o $3 6 | 7 | echo Gen graph $3 8 | } 9 | 10 | for i in `ls callgrind.out.*`; do 11 | seq=${i##callgrind.out.} 12 | dot_filename=out.dot.$seq 13 | graph_filename=graph.$seq.png 14 | if ! [[ -f $graph_filename ]]; then 15 | gen_graph $i $dot_filename $graph_filename 16 | fi 17 | done 18 | -------------------------------------------------------------------------------- /misc/travis/unittest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -ex 3 | 4 | echo "CXX=${CXX}" 5 | python misc/generate_hash_dataset.py tests/resources/keys.txt 6 | mkdir -p build 7 | cd build 8 | cmake -DCMAKE_BUILD_TYPE=$CMAKE_BUILD_TYPE -DWITH_TESTING=ON .. 9 | make -j8 10 | valgrind --leak-check=full make test 11 | cd .. 12 | python setup.py build_ext --inplace 13 | python -m pytest tests 14 | pip install -e . 15 | ./tests/shabby/run_test.sh 16 | -------------------------------------------------------------------------------- /misc/travis/cpptest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -ex 3 | 4 | echo "CXX=${CXX}" 5 | ./misc/memcached_server startall &>/dev/null & 6 | python misc/generate_hash_dataset.py tests/resources/keys.txt &>/dev/null & 7 | mkdir -p build 8 | cd build 9 | cmake -DCMAKE_BUILD_TYPE=$CMAKE_BUILD_TYPE -DWITH_TESTING=ON .. &>/dev/null 10 | make -j8 &>/dev/null 11 | wait 12 | ARGS=-V make test 13 | cd .. 14 | ./misc/memcached_server stopall &>/dev/null & 15 | -------------------------------------------------------------------------------- /src/version.go: -------------------------------------------------------------------------------- 1 | package golibmc 2 | 3 | const _Version = "1.4.15" 4 | const _Author = "mckelvin" 5 | const _Email = "mckelvin@users.noreply.github.com" 6 | const _Date = "Fri Jun 7 06:16:00 2024 +0800" 7 | 8 | // Version of the package 9 | const Version = _Version 10 | 11 | // Author of the package 12 | const Author = _Author 13 | 14 | // Email of the author 15 | const Email = _Email 16 | 17 | // Date of the package 18 | const Date = _Date 19 | -------------------------------------------------------------------------------- /.github/workflows/cpp.yml: -------------------------------------------------------------------------------- 1 | name: Cpp 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | cppcheck: 11 | runs-on: ubuntu-22.04 12 | 13 | steps: 14 | - uses: actions/checkout@v2 15 | - name: Setup system dependencies 16 | run: | 17 | sudo apt-get update 18 | sudo apt-get -y install cppcheck 19 | - name: Run cppcheck 20 | run: | 21 | ./misc/travis/cppcheck.sh 22 | -------------------------------------------------------------------------------- /misc/update_version.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # ln -sf ../../misc/update_version.sh .git/hooks/pre-commit 3 | VERSIONING_SCRIPT="`pwd`/misc/versioning.py" 4 | 5 | VERSIONING_FILES="`pwd`/libmc/__init__.py `pwd`/src/version.go" 6 | for VERSIONING_FILE in $VERSIONING_FILES 7 | do 8 | TMPFILE=$VERSIONING_FILE".2" 9 | cat $VERSIONING_FILE | \ 10 | python2 $VERSIONING_SCRIPT --clean | \ 11 | python2 $VERSIONING_SCRIPT > $TMPFILE 12 | mv $TMPFILE $VERSIONING_FILE 13 | git add $VERSIONING_FILE 14 | done 15 | -------------------------------------------------------------------------------- /tests/shabby/infinite_set_get.py: -------------------------------------------------------------------------------- 1 | import time 2 | from libmc import Client 3 | 4 | 5 | def main(): 6 | mc = Client(['127.0.0.1:%d' % (21211 + i) for i in range(10)]) 7 | dct = {'test_key_%d' % i: 'i' * 1000 for i in range(10)} 8 | while True: 9 | print '%.2f: set_multi: %r' % (time.time(), mc.set_multi(dct)) 10 | print '%.2f: get_multi: %r' % ( 11 | time.time(), mc.get_multi(dct.keys()) == dct 12 | ) 13 | time.sleep(0.5) 14 | print 15 | 16 | 17 | if __name__ == '__main__': 18 | main() 19 | -------------------------------------------------------------------------------- /include/hashkit/hashkit.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include 4 | 5 | namespace douban { 6 | namespace mc { 7 | namespace hashkit { 8 | 9 | typedef uint32_t (*hash_function_t)(const char *key, size_t key_length); 10 | 11 | uint32_t hash_md5(const char *key, size_t key_length); 12 | uint32_t hash_fnv1_32(const char *key, size_t key_length); 13 | uint32_t hash_fnv1a_32(const char *key, size_t key_length); 14 | uint32_t hash_crc_32(const char *key, size_t key_length); 15 | 16 | } // namespace hashkit 17 | } // namespace mc 18 | } // namespace douban 19 | -------------------------------------------------------------------------------- /tests/shabby/independent_timeout.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | import libmc 3 | from slow_memcached_server import PORT, BLOCKING_SECONDS 4 | 5 | 6 | def main(): 7 | mc1 = libmc.Client(['localhost:%s' % PORT]) 8 | mc1.config(libmc.MC_POLL_TIMEOUT, BLOCKING_SECONDS * 1000 * 2) 9 | assert mc1.version(), "start slow_memcached_server first" 10 | mc1.config(libmc.MC_POLL_TIMEOUT, BLOCKING_SECONDS * 1000 / 2) 11 | assert not mc1.set('foo', 1) 12 | mc2 = libmc.Client(['localhost:%s' % PORT]) 13 | mc2.config(libmc.MC_POLL_TIMEOUT, BLOCKING_SECONDS * 1000 * 2) 14 | assert not mc1.set('foo', 1) 15 | 16 | 17 | if __name__ == '__main__': 18 | main() 19 | -------------------------------------------------------------------------------- /ext/gtest/CMakeLists.txt.in: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8.2) 2 | project(gtest_builder C CXX) 3 | 4 | include(ExternalProject) 5 | 6 | ExternalProject_Add(googletest 7 | GIT_REPOSITORY https://github.com/google/googletest.git 8 | GIT_TAG release-1.8.0 9 | SOURCE_DIR "${CMAKE_BINARY_DIR}/googletest-src" 10 | BINARY_DIR "${CMAKE_BINARY_DIR}/googletest-build" 11 | CMAKE_ARGS -DCMAKE_ARCHIVE_OUTPUT_DIRECTORY_DEBUG:PATH=DebugLibs 12 | -DCMAKE_ARCHIVE_OUTPUT_DIRECTORY_RELEASE:PATH=ReleaseLibs 13 | UPDATE_DISCONNECTED 1 14 | CONFIGURE_COMMAND "" 15 | BUILD_COMMAND "" 16 | INSTALL_COMMAND "" 17 | TEST_COMMAND "" 18 | ) 19 | -------------------------------------------------------------------------------- /tests/shabby/timeout_issue.py: -------------------------------------------------------------------------------- 1 | import time 2 | import libmc 3 | import slow_memcached_server 4 | 5 | def main(): 6 | print('libmc path: %s' % libmc.__file__) 7 | mc = libmc.Client(['localhost:%s' % slow_memcached_server.PORT]) 8 | mc.config(libmc.MC_POLL_TIMEOUT, 3000) 9 | assert mc.version(), "start slow_memcached_server first" 10 | EXPECTED_TIMEOUT = 300 11 | mc.config(libmc.MC_POLL_TIMEOUT, EXPECTED_TIMEOUT) # timeout in 300 ms 12 | 13 | t0 = time.time() 14 | assert not mc.set('foo', 1) 15 | t1 = time.time() 16 | error_threshold = 10 17 | assert abs(EXPECTED_TIMEOUT - (t1 - t0) * 1000) < error_threshold 18 | 19 | 20 | if __name__ == '__main__': 21 | main() 22 | -------------------------------------------------------------------------------- /include/Utility.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | #include "rapidjson/itoa.h" 7 | 8 | #define MC_MAX_KEY_LENGTH 250 9 | 10 | namespace douban { 11 | namespace mc { 12 | namespace utility { 13 | 14 | inline int int64ToCharArray(int64_t n, char s[]) { 15 | char *end = ::rapidjson::internal::i64toa(n, s); 16 | *end = '\0'; 17 | return static_cast(end - s); 18 | } 19 | 20 | // credit to The New Page of Injections Book: 21 | // Memcached Injections @ blackhat2014 [pdf](http://t.cn/RP0J10Z) 22 | bool isValidKey(const char* key, const size_t keylen); 23 | void fprintBuffer(std::FILE* file, const char *data_buffer_, const unsigned int length); 24 | 25 | } // namespace utility 26 | } // namespace mc 27 | } // namespace douban 28 | -------------------------------------------------------------------------------- /tests/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | file(GLOB TEST_SRC_FILES ${PROJECT_SOURCE_DIR}/tests/test_*.cpp) 2 | 3 | foreach(SRC ${TEST_SRC_FILES}) 4 | get_filename_component(test_name ${SRC} NAME_WE) 5 | set_source_files_properties(${SRC} PROPERTIES COMPILE_FLAGS "-fexceptions -std=c++17") 6 | add_executable(${test_name} ${SRC}) 7 | add_dependencies(${test_name} gtest) 8 | target_link_libraries(${test_name} 9 | gtest_main 10 | pthread 11 | mc 12 | ) 13 | add_test(${test_name} ${test_name}) 14 | endforeach(SRC) 15 | 16 | add_executable(profile_client profile_client.cpp) 17 | if(NOT APPLE) 18 | target_link_libraries(profile_client rt) 19 | endif(NOT APPLE) 20 | target_link_libraries(profile_client mc) 21 | 22 | add_executable(debug_client debug_client.cpp) 23 | if(NOT APPLE) 24 | target_link_libraries(debug_client rt) 25 | endif(NOT APPLE) 26 | target_link_libraries(debug_client mc) 27 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | Thanks for using libmc! Please feel free to file an issue 4 | if you have any question or meet any trouble. Pull Requests are 5 | more than welcomed. 6 | 7 | ## Before you open a Pull Request 8 | 9 | ### Get some hints on how to build 10 | Before you get started, you may need know some basic steps 11 | on how to 12 | [build the project](https://github.com/douban/libmc/wiki/Build-and-installation#for-libmc-developer) first. 13 | 14 | 15 | ### Change the version definitions before you commit (optional) 16 | To get the exact version of the library you're running, it's 17 | advised (but not mandatory) to change the **version** constants 18 | defined in libmc everytime when you submit a commit. 19 | There's an easy way to enable this behaviour: 20 | 21 | ln -sf ../../misc/update_version.sh .git/hooks/pre-commit 22 | 23 | You need to run this command once before you first commit. 24 | -------------------------------------------------------------------------------- /tests/shabby/run_test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | BASEDIR=`dirname $0` 4 | 5 | python -c 'import greenify; import gevent;' 6 | python $BASEDIR/slow_memcached_server.py & 7 | pid=$! 8 | echo "pid of slow memcached server: $pid" 9 | 10 | if [ `uname` = 'Linux' ]; then 11 | echo 12 | echo "=== test gevent support ===" 13 | echo 14 | python $BASEDIR/gevent_issue.py 15 | else 16 | echo 17 | echo "=== test gevent support (skip) ===" 18 | echo " currently greenify only works on Linux" 19 | fi 20 | 21 | 22 | echo 23 | echo "=== test GIL ===" 24 | echo 25 | python $BASEDIR/gil_issue.py 26 | echo 27 | echo "=== test timeout ===" 28 | echo 29 | python $BASEDIR/timeout_issue.py 30 | echo 31 | echo "=== test reconnect delay ===" 32 | echo 33 | python $BASEDIR/reconnect_delay.py 34 | echo 35 | echo "=== test independent timeout ===" 36 | echo 37 | python $BASEDIR/independent_timeout.py 38 | 39 | kill $pid || exit 0 40 | -------------------------------------------------------------------------------- /include/BufferWriter.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | 8 | 9 | namespace douban { 10 | namespace mc { 11 | namespace io { 12 | 13 | 14 | /** 15 | * iovec based BufferWriter 16 | * writev / sendmsg 17 | **/ 18 | class BufferWriter { 19 | public: 20 | BufferWriter(); 21 | ~BufferWriter(); 22 | void reset(); 23 | void reserve(size_t n); 24 | void takeBuffer(const char* const buf, size_t buf_len); 25 | void takeNumber(int64_t val); 26 | const struct iovec* const getReadPtr(size_t &n); 27 | void commitRead(size_t nSent); 28 | void rewind(); 29 | size_t msgIovlen(); 30 | const bool isRead(); 31 | 32 | protected: 33 | std::vector m_iovec; 34 | std::vector m_originalIovec; 35 | std::vector m_unsignedStringList; 36 | 37 | // the index of iovec vector we'll read next 38 | size_t m_readIdx; 39 | 40 | // the number of iovec left to read 41 | size_t m_msgIovlen; 42 | }; 43 | 44 | inline const bool BufferWriter::isRead() { 45 | return m_readIdx != 0; 46 | } 47 | 48 | } // namespace io 49 | } // namespace mc 50 | } // namespace douban 51 | -------------------------------------------------------------------------------- /.github/workflows/golang.yml: -------------------------------------------------------------------------------- 1 | name: Golang 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | 10 | jobs: 11 | gotest: 12 | runs-on: ubuntu-22.04 13 | strategy: 14 | matrix: 15 | gover: ["1.15"] 16 | compiler: ["gcc", "clang"] 17 | 18 | steps: 19 | - uses: actions/checkout@v2 20 | - name: Setup system dependencies 21 | run: | 22 | sudo apt-get update 23 | sudo apt-get -y install memcached g++ 24 | - name: Set up Golang 25 | uses: actions/setup-go@v2 26 | with: 27 | go-version: ${{ matrix.gover }} 28 | - name: Set up Python 29 | uses: actions/setup-python@v2 30 | with: 31 | python-version: 3.13 32 | - name: Start memcached servers 33 | run: ./misc/memcached_server start 34 | - name: Start slow memcached server 35 | run: ./misc/slow_server.sh 36 | - name: Run gotest 37 | run: | 38 | if [[ ${{ matrix.compiler }} = "gcc" ]]; then export CC=gcc CXX=g++; fi 39 | if [[ ${{ matrix.compiler }} = "clang" ]]; then export CC=clang CXX=clang++; fi 40 | ./misc/travis/gotest.sh 41 | - name: Stop memcached servers 42 | run: ./misc/memcached_server stop 43 | -------------------------------------------------------------------------------- /tests/shabby/gevent_issue.py: -------------------------------------------------------------------------------- 1 | import gevent 2 | import time 3 | import gevent.monkey 4 | import slow_memcached_server 5 | gevent.monkey.patch_time() 6 | 7 | import greenify 8 | greenify.greenify() 9 | import libmc 10 | for so_path in libmc.DYNAMIC_LIBRARIES: 11 | assert greenify.patch_lib(so_path) 12 | mc = libmc.Client(["127.0.0.1:%d" % slow_memcached_server.PORT]) 13 | mc.config(libmc._client.MC_POLL_TIMEOUT, 14 | slow_memcached_server.BLOCKING_SECONDS * 1000 * 2) # ms 15 | 16 | 17 | stack = [] 18 | 19 | 20 | def mc_sleep(): 21 | print('begin mc sleep') 22 | stack.append('mc_sleep_begin') 23 | assert mc.set('foo', 'bar') 24 | stack.append('mc_sleep_end') 25 | print('end mc sleep') 26 | 27 | 28 | def singer(): 29 | i = 0 30 | for i in range(6): 31 | i += 1 32 | print('[%d] Oh, jingle bells, jingle bells, Jingle all the way.' % i) 33 | stack.append('sing') 34 | time.sleep(0.5) 35 | 36 | 37 | def main(): 38 | assert len(mc.version()) == 1, "Run `python slow_memcached_server.py` first" 39 | mc_sleeper = gevent.spawn(mc_sleep) 40 | xmas_singer = gevent.spawn(singer) 41 | gevent.joinall([xmas_singer, mc_sleeper]) 42 | assert stack.index('mc_sleep_end') - stack.index('mc_sleep_begin') > 1 43 | 44 | 45 | if __name__ == '__main__': 46 | main() 47 | -------------------------------------------------------------------------------- /tests/shabby/reproduce_incomplete_data_block_bug_on_get_multi.py: -------------------------------------------------------------------------------- 1 | import libmc 2 | print libmc.__file__ 3 | 4 | # server pending = 0.5 5 | 6 | UNIT_PENDING_SECONDS = 500 # ms 7 | 8 | def main(): 9 | # NOTE start 2 slow memcached server first! 10 | mc = libmc.Client(['localhost:8965', 'localhost:8966']) 11 | mc.config(libmc.MC_POLL_TIMEOUT, int(UNIT_PENDING_SECONDS * 0.5)) 12 | 13 | bv = '1' * 512 14 | assert not mc.set('bv', bv) 15 | assert mc.get('bv') is None 16 | 17 | assert not mc.set('foo', 1) 18 | assert mc.get('stubs') is None 19 | 20 | mc.config(libmc.MC_POLL_TIMEOUT, int(UNIT_PENDING_SECONDS * 1.2)) 21 | assert mc.set('foo', 1) 22 | assert mc.get('stubs') == 'yes' 23 | 24 | assert mc.set('bv', bv) 25 | assert mc.get('bv') is None # this should be timed out 26 | 27 | assert mc.get_multi(['stubs']) == {'stubs': 'yes'} 28 | assert mc.get_multi(['stubs', 'foo']) == {'stubs': 'yes', 'foo': 1} 29 | assert not mc.get_multi(['bv']) # this should be timed out 30 | 31 | # NOTE: A C++ backtrace will be printed to stderr if reproduced 32 | assert mc.get_multi(['bv', 'stubs']) == {'stubs': 'yes'} # this should be timed out 33 | 34 | assert mc.set('foo', 1) 35 | assert mc.get_multi(['stubs', 'foo']) == {'stubs': 'yes', 'foo': 1} 36 | 37 | 38 | if __name__ == '__main__': 39 | main() 40 | -------------------------------------------------------------------------------- /src/HashkitFnv.cpp: -------------------------------------------------------------------------------- 1 | #include "hashkit/hashkit.h" 2 | 3 | // http://www.isthe.com/chongo/src/fnv/hash_32a.c 4 | 5 | namespace douban { 6 | namespace mc { 7 | namespace hashkit { 8 | 9 | static const uint32_t FNV_32_INIT = 2166136261UL; 10 | 11 | #if defined(NO_FNV_GCC_OPTIMIZATION) 12 | static const uint32_t FNV_32_PRIME = 16777619UL; 13 | #endif 14 | 15 | uint32_t hash_fnv1_32(const char *key, size_t key_length) { 16 | uint32_t hash = FNV_32_INIT; 17 | size_t x; 18 | const uint8_t* unsigned_key = reinterpret_cast(key); 19 | 20 | for (x = 0; x < key_length; x++) { 21 | uint32_t val = (uint32_t)unsigned_key[x]; 22 | #if defined(NO_FNV_GCC_OPTIMIZATION) 23 | hash *= FNV_32_PRIME; 24 | #else 25 | hash += (hash << 1) + (hash << 4) + (hash << 7) + (hash << 8) + (hash << 24); 26 | #endif 27 | hash ^= val; 28 | } 29 | 30 | return hash; 31 | } 32 | 33 | uint32_t hash_fnv1a_32(const char *key, size_t key_length) { 34 | uint32_t hash = FNV_32_INIT; 35 | size_t x; 36 | const uint8_t* unsigned_key = reinterpret_cast(key); 37 | 38 | for (x= 0; x < key_length; x++) { 39 | uint32_t val = (uint32_t)unsigned_key[x]; 40 | hash ^= val; 41 | #if defined(NO_FNV_GCC_OPTIMIZATION) 42 | hash *= FNV_32_PRIME; 43 | #else 44 | hash += (hash << 1) + (hash << 4) + (hash << 7) + (hash << 8) + (hash << 24); 45 | #endif 46 | } 47 | 48 | return hash; 49 | } 50 | 51 | } // namespace hashkit 52 | } // namespace mc 53 | } // namespace douban 54 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014-2015, Douban Inc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | 8 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 9 | 10 | 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 11 | 12 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 13 | -------------------------------------------------------------------------------- /tests/golibmc_debug_client.go: -------------------------------------------------------------------------------- 1 | // A standalone client using golibmc which aims to do ad-hoc trials. 2 | 3 | package main 4 | 5 | import ( 6 | "fmt" 7 | golibmc "github.com/douban/libmc/src" 8 | "time" 9 | ) 10 | 11 | func newClient() *golibmc.Client { 12 | noreply := false 13 | prefix := "" 14 | hashFunc := golibmc.HashMD5 15 | failover := false 16 | disableLock := false 17 | 18 | // The following is an arbitrary server:port which cannot reach, 19 | // so it will timeout when we connect 20 | SERVERS := []string{"1.1.1.1:1 mc-server-will-timeout"} 21 | 22 | // SERVERS := []string{"localhost:21211 mc-server-local"} 23 | 24 | CONNECT_TIMEOUT := 10 25 | // CONNECT_TIMEOUT := 1000 26 | POLL_TIMEOUT := 300 27 | RETRY_TIMEOUT := 5 28 | 29 | client := golibmc.New(SERVERS, noreply, prefix, hashFunc, failover, disableLock) 30 | client.ConfigTimeout(golibmc.ConnectTimeout, time.Duration(CONNECT_TIMEOUT)*time.Millisecond) 31 | client.ConfigTimeout(golibmc.PollTimeout, time.Duration(POLL_TIMEOUT)*time.Millisecond) 32 | client.ConfigTimeout(golibmc.RetryTimeout, time.Duration(RETRY_TIMEOUT)*time.Second) 33 | return client 34 | } 35 | 36 | func main() { 37 | client := newClient() 38 | 39 | key := "hello" 40 | val := "world" 41 | 42 | err := client.Set(&golibmc.Item{Key: key, Value: []byte(val)}) 43 | if err != nil { 44 | fmt.Printf("ERROR: client.Set: %v\n", err) 45 | } 46 | 47 | item, err2 := client.Get(key) 48 | if err2 != nil { 49 | fmt.Printf("ERROR: client.Get: %v\n", err2) 50 | } else { 51 | fmt.Println(string(item.Value)) 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /.github/workflows/publish.yml: -------------------------------------------------------------------------------- 1 | name: Publish 2 | 3 | on: 4 | push: 5 | tags: 6 | - v* 7 | 8 | jobs: 9 | wheel: 10 | name: Build wheels on ${{ matrix.os }} 11 | runs-on: ${{ matrix.os }} 12 | strategy: 13 | matrix: 14 | os: [ubuntu-22.04, macos-13, macos-latest] 15 | 16 | steps: 17 | - uses: actions/checkout@v2 18 | - uses: actions/setup-python@v3 19 | 20 | - name: Install cibuildwheel 21 | run: python -m pip install cibuildwheel==3.2.0 22 | 23 | - name: Build wheels 24 | env: 25 | CIBW_BEFORE_BUILD: pip install 'setuptools<72.2.0 ; implementation_name == "pypy"' 26 | MACOSX_DEPLOYMENT_TARGET: "10.13" 27 | CIBW_SKIP: "pp*-macosx_*" 28 | run: python -m cibuildwheel --output-dir wheelhouse 29 | - name: Check build result 30 | run: ls -lh wheelhouse/ 31 | - uses: actions/upload-artifact@v4 32 | with: 33 | name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }} 34 | path: ./wheelhouse/*.whl 35 | 36 | pypi: 37 | needs: [wheel] 38 | runs-on: ubuntu-latest 39 | steps: 40 | - uses: actions/download-artifact@v4 41 | with: 42 | pattern: cibw-* 43 | path: wheelhouse/ 44 | merge-multiple: true 45 | - name: Check download result 46 | run: ls -lht wheelhouse/ 47 | - name: Publish to PyPI 48 | uses: pypa/gh-action-pypi-publish@release/v1 49 | with: 50 | packages-dir: wheelhouse/ 51 | skip_existing: true 52 | user: __token__ 53 | password: ${{ secrets.PYPI_API_TOKEN }} 54 | -------------------------------------------------------------------------------- /tests/shabby/gil_issue.py: -------------------------------------------------------------------------------- 1 | # if GIL is released before sleep in C++, the following output is expected: 2 | # 3 | # $ PYTHONPATH=../../ python gil_issue.py 4 | # sleep in C++ for 5s, you can sing infinitely now, singer. 5 | # [1] Oh, jingle bells, jingle bells, Jingle all the way. 6 | # [2] Oh, jingle bells, jingle bells, Jingle all the way. 7 | # [3] Oh, jingle bells, jingle bells, Jingle all the way. 8 | # [4] Oh, jingle bells, jingle bells, Jingle all the way. 9 | # [5] Oh, jingle bells, jingle bells, Jingle all the way. 10 | # exit sleep 11 | # [6] Oh, jingle bells, jingle bells, Jingle all the way. 12 | # [7] Oh, jingle bells, jingle bells, Jingle all the way. 13 | # ^C 14 | 15 | import time 16 | import threading 17 | from libmc import Client 18 | 19 | stack = [] 20 | 21 | 22 | def singer(): 23 | i = 0 24 | while i < 6: 25 | i += 1 26 | print('[%d] Oh, jingle bells, jingle bells, Jingle all the way.' % i) 27 | stack.append('sing') 28 | time.sleep(0.5) 29 | 30 | 31 | def mc_sleeper(): 32 | s = 2 33 | print('sleep in C++ for %ds, you can sing infinitely now, singer.' % s) 34 | mc = Client(["127.0.0.1:21211"]) 35 | release_gil = True # toggle here 36 | stack.append('mc_sleep_begin') 37 | mc._sleep(s, release_gil) 38 | stack.append('mc_sleep_end') 39 | print('exit sleep') 40 | 41 | 42 | def main(): 43 | t = threading.Thread(target=singer) 44 | t.start() 45 | mc_sleeper() 46 | t.join() 47 | assert stack.index('mc_sleep_end') - stack.index('mc_sleep_begin') > 1 48 | 49 | 50 | if __name__ == '__main__': 51 | main() 52 | -------------------------------------------------------------------------------- /include/Result.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include "Export.h" 5 | #include "BufferReader.h" 6 | #ifdef MC_USE_SMALL_VECTOR 7 | #include "llvm/SmallVector.h" 8 | #endif 9 | 10 | namespace douban { 11 | namespace mc { 12 | namespace types { 13 | 14 | 15 | class RetrievalResult { 16 | public: 17 | RetrievalResult(); 18 | RetrievalResult(const RetrievalResult& other); 19 | ~RetrievalResult(); 20 | 21 | douban::mc::io::TokenData key; // 24B 22 | douban::mc::io::TokenData data_block; // 24B 23 | cas_unique_t cas_unique; // 8B 24 | uint32_t bytesRemain; // 4B. bytes remain to read, complete data if this is 0 25 | uint32_t bytes; // 4B 26 | flags_t flags; // 4B 27 | uint8_t key_len; // 1B 28 | retrieval_result_t* inner(); 29 | protected: 30 | retrieval_result_t m_inner; 31 | }; 32 | 33 | 34 | void delete_broadcast_result(broadcast_result_t* ptr); 35 | 36 | 37 | class LineResult { 38 | public: 39 | LineResult(); 40 | LineResult(const LineResult& other); 41 | ~LineResult(); 42 | douban::mc::io::TokenData line; 43 | size_t line_len; 44 | char* inner(size_t& n); 45 | protected: 46 | char* m_inner; 47 | }; 48 | 49 | 50 | #ifdef MC_USE_SMALL_VECTOR 51 | typedef llvm::SmallVector RetrievalResultList; 52 | #else 53 | typedef std::vector RetrievalResultList; 54 | #endif 55 | typedef std::vector MessageResultList; 56 | typedef std::vector LineResultList; 57 | typedef std::vector UnsignedResultList; 58 | 59 | 60 | } // namespace types 61 | } // namespace mc 62 | } // namespace douban 63 | -------------------------------------------------------------------------------- /misc/gomcbench/README.md: -------------------------------------------------------------------------------- 1 | # Benchmark for golibmc 2 | 3 | (compared with [bradfitz/gomemcache](https://github.com/bradfitz/gomemcache/)) 4 | 5 | ## How to run the benchmark 6 | 7 | go get github.com/bradfitz/gomemcache/memcache 8 | go get github.com/douban/libmc/golibmc 9 | cd misc/gomcbench 10 | go test -bench . 11 | 12 | ## Results 13 | 14 | result based on a MacBook Pro (Retina, 13-inch, Late 2013) w/ 8GB Mem, and a 2.4 GHz Intel Core i5 CPU. 15 | 16 | BenchmarkGolibmcGetSingleSmallValue-4 30000 46638 ns/op 17 | BenchmarkGomemcacheGetSingleSmallValue-4 20000 87720 ns/op 18 | 19 | BenchmarkGolibmcGetSingleLargeValue-4 20000 83470 ns/op 20 | BenchmarkGomemcacheGetSingleLargeValue-4 10000 110609 ns/op 21 | 22 | BenchmarkGolibmcGet10SmallValues-4 10000 137134 ns/op 23 | BenchmarkGomemcacheGet10SmallValues-4 5000 329235 ns/op 24 | 25 | BenchmarkGolibmcGet100SmallValues-4 3000 382400 ns/op 26 | BenchmarkGomemcacheGet100SmallValues-4 2000 894691 ns/op 27 | 28 | BenchmarkGolibmcGet1000SmallValues-4 500 2212252 ns/op 29 | BenchmarkGomemcacheGet1000SmallValues-4 200 6924752 ns/op 30 | 31 | BenchmarkGolibmcGet10LargeValues-4 5000 284501 ns/op 32 | BenchmarkGomemcacheGet10LargeValues-4 3000 546686 ns/op 33 | 34 | BenchmarkGolibmcGet100LargeValues-4 500 2892901 ns/op 35 | BenchmarkGomemcacheGet100LargeValues-4 500 3534152 ns/op 36 | 37 | BenchmarkGolibmcGet1000LargeValues-4 50 21692412 ns/op 38 | BenchmarkGomemcacheGet1000LargeValues-4 50 22605197 ns/op 39 | -------------------------------------------------------------------------------- /include/hashkit/ketama.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | #include "Connection.h" 7 | #include "hashkit/hashkit.h" 8 | 9 | namespace douban { 10 | namespace mc { 11 | namespace hashkit { 12 | 13 | 14 | typedef struct continuum_item_s { 15 | uint32_t hash_value; 16 | size_t conn_idx; 17 | douban::mc::Connection* conn; 18 | 19 | static struct compare_s { 20 | bool operator() (const struct continuum_item_s& left, const struct continuum_item_s& right) { 21 | return left.hash_value < right.hash_value; 22 | } 23 | } compare; 24 | } continuum_item_t; 25 | 26 | 27 | class KetamaSelector { 28 | public: 29 | KetamaSelector(); 30 | void setHashFunction(hash_function_t fn); 31 | void enableFailover(); 32 | void disableFailover(); 33 | 34 | void reset(); 35 | void addServers(douban::mc::Connection* conns, size_t nConns); 36 | 37 | int getServer(const char* key, size_t key_len, bool check_alive = true); 38 | douban::mc::Connection* getConn(const char* key, size_t key_len, bool check_alive = true); 39 | 40 | protected: 41 | std::vector::iterator getServerIt(const char* key, size_t key_len, 42 | bool check_alive); 43 | 44 | std::vector m_continuum; 45 | size_t m_nServers; 46 | bool m_useFailover; 47 | hash_function_t m_hashFunction; 48 | static const size_t s_pointerPerHash; 49 | static const size_t s_pointerPerServer; 50 | static const hash_function_t s_defaultHashFunction; 51 | 52 | #ifndef NDEBUG 53 | bool m_sorted; 54 | #endif 55 | }; 56 | 57 | } // namespace hashkit 58 | } // namespace mc 59 | } // namespace douban 60 | -------------------------------------------------------------------------------- /include/DataBlock.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include 3 | #include 4 | #include "Common.h" 5 | 6 | 7 | namespace douban { 8 | namespace mc { 9 | namespace io { 10 | 11 | class DataBlock { 12 | public: 13 | DataBlock(); 14 | DataBlock(const DataBlock&); 15 | ~DataBlock(); 16 | static void setMinCapacity(size_t len); 17 | static size_t minCapacity(); 18 | 19 | void init(size_t len); 20 | void reset(); 21 | inline char* operator[](size_t offset) { 22 | assert(offset < m_size); 23 | return m_data + offset; 24 | } 25 | 26 | inline char* at(size_t offset) { 27 | assert(offset < m_size); 28 | return m_data + offset; 29 | } 30 | 31 | inline const char* operator[](size_t offset) const { 32 | assert(offset < m_size); 33 | return m_data + offset; 34 | } 35 | 36 | void acquire(size_t len); 37 | void release(size_t len); 38 | size_t size(); 39 | size_t capacity(); 40 | bool reusable(); 41 | size_t nBytesRef(); 42 | size_t occupy(size_t len); 43 | char* getWritePtr(); 44 | size_t getWriteLeft(); 45 | size_t find(char c, size_t since = 0); 46 | size_t findNotNumeric(size_t since = 0); 47 | 48 | protected: 49 | char* m_data; 50 | size_t m_capacity; 51 | size_t m_size; 52 | size_t m_nBytesRef; 53 | static size_t s_minCapacity; // FIXME: bad design 54 | }; 55 | 56 | 57 | inline void DataBlock::acquire(size_t len) { 58 | this->m_nBytesRef += len; 59 | } 60 | 61 | inline void DataBlock::release(size_t len) { 62 | assert(this->m_nBytesRef >= len); 63 | this->m_nBytesRef -= len; 64 | } 65 | 66 | 67 | inline size_t DataBlock::size() { 68 | return m_size; 69 | } 70 | 71 | } // namespace io 72 | } // namespace mc 73 | } // namespace douban 74 | -------------------------------------------------------------------------------- /tests/debug_client.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include "Common.h" 4 | #include "Client.h" 5 | #include "test_common.h" 6 | 7 | using douban::mc::Client; 8 | using douban::mc::tests::newClient; 9 | 10 | 11 | void set_(Client* client, const char* const* keys, const size_t* key_lens, 12 | const flags_t* flags, const char* const* vals, const size_t* val_len, size_t n = 1) { 13 | 14 | const exptime_t exptime = 0; 15 | const int noreply = 0; 16 | 17 | message_result_t **m_results = NULL; 18 | size_t nResults = 0; 19 | 20 | client->set(keys, key_lens, flags, exptime, NULL, noreply, vals, val_len, 21 | n, &m_results, &nResults); 22 | client->destroyMessageResult(); 23 | } 24 | 25 | 26 | void get_(Client* client, const char* const * keys, const size_t* key_lens, size_t n = 1) { 27 | retrieval_result_t** r_results = NULL; 28 | size_t nResults = 0; 29 | client->get(keys, key_lens, n, &r_results, &nResults); 30 | client->destroyRetrievalResult(); 31 | } 32 | 33 | 34 | int main() { 35 | Client* client = newClient(20); 36 | 37 | while (true) { 38 | 39 | const char* key = "hello"; 40 | const size_t key_len = 5; 41 | 42 | const char* val = "world"; 43 | const size_t val_len = 5; 44 | 45 | flags_t flags = 0; 46 | set_(client, &key, &key_len, &flags, &val, &val_len, 1); 47 | usleep(3000000); 48 | 49 | // get_(client, &key, &key_len, 1); 50 | 51 | retrieval_result_t** r_results = NULL; 52 | size_t nResults = 0; 53 | 54 | err_code_t rv = client->get(&key, &key_len, 1, &r_results, &nResults); 55 | client->destroyRetrievalResult(); 56 | 57 | if (r_results!=NULL) { 58 | std::cout << r_results[0]->data_block; 59 | } 60 | 61 | std::cout << rv << std::endl; 62 | 63 | } 64 | delete client; 65 | 66 | return 0; 67 | } 68 | -------------------------------------------------------------------------------- /include/Keywords.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Result.h" 4 | 5 | namespace douban { 6 | namespace mc { 7 | namespace keywords { 8 | 9 | static const char kCR = '\r'; 10 | static const char kLF = '\n'; 11 | static const char kCRLF[] = "\r\n"; 12 | static const char kSPACE[] = " "; 13 | 14 | static const char kGET[] = "get"; 15 | static const char kGETS[] = "gets"; 16 | 17 | static const char kSET_[] = "set "; 18 | static const char kADD_[] = "add "; 19 | static const char kREPLACE_[] = "replace "; 20 | static const char kAPPEND_[] = "append "; 21 | static const char kPREPEND_[] = "prepend "; 22 | static const char kCAS_[] = "cas "; 23 | 24 | static const char kDELETE_[] = "delete "; 25 | static const char kTOUCH_[] = "touch "; 26 | 27 | static const char kINCR_[] = "incr "; 28 | static const char kDECR_[] = "decr "; 29 | 30 | static const char k_NOREPLY[] = " noreply"; 31 | 32 | static const char kVERSION[] = "version"; 33 | static const char kSTATS[] = "stats"; 34 | static const char kFLUSHALL[] = "flush_all"; 35 | 36 | static const char kQUIT[] = "quit"; 37 | 38 | 39 | // error messages for human 40 | static const char kSEND_ERROR[] = "send_error"; 41 | static const char kRECV_ERROR[] = "recv_error"; 42 | static const char kCONN_POLL_ERROR[] = "conn_poll_error"; 43 | static const char kPOLL_TIMEOUT_ERROR[] = "poll_timeout_error"; 44 | static const char kPOLL_ERROR[] = "poll_error"; 45 | static const char kSERVER_ERROR[] = "server_error"; 46 | static const char kPROGRAMMING_ERROR[] = "programming_error"; 47 | static const char kINVALID_KEY_ERROR[] = "invalid_key_error"; 48 | static const char kINCOMPLETE_BUFFER_ERROR[] = "incomplete_buffer_error"; 49 | 50 | static const char kUPDATE_SERVER[] = "update_server"; 51 | static const char kCONN_QUIT[] = "conn_quit"; 52 | 53 | } // namespace keywords 54 | } // namespace mc 55 | } // namespace douban 56 | -------------------------------------------------------------------------------- /src/Utility.cpp: -------------------------------------------------------------------------------- 1 | #include "Utility.h" 2 | #include "Common.h" 3 | 4 | namespace douban { 5 | namespace mc { 6 | namespace utility { 7 | 8 | bool isValidKey(const char* key, const size_t keylen) { 9 | 10 | #define HANDLE_BAD_MC_KEY() \ 11 | do { \ 12 | log_warn("invalid mc key of length %zu: \"%.*s\"", keylen, static_cast(keylen), key); \ 13 | return false; \ 14 | } while (0) 15 | 16 | if (keylen > MC_MAX_KEY_LENGTH) { 17 | HANDLE_BAD_MC_KEY(); 18 | } 19 | 20 | for (size_t i = 0; i < keylen; i++) { 21 | switch (key[i]) { 22 | case ' ': 23 | HANDLE_BAD_MC_KEY(); 24 | break; 25 | case '\r': 26 | HANDLE_BAD_MC_KEY(); 27 | break; 28 | case '\n': 29 | HANDLE_BAD_MC_KEY(); 30 | break; 31 | case 0: 32 | HANDLE_BAD_MC_KEY(); 33 | break; 34 | default: 35 | break; 36 | } 37 | } 38 | return true; 39 | } 40 | 41 | 42 | // credits for: https://gist.github.com/sergot/1333837 43 | void fprintBuffer(std::FILE* file, const char *data_buffer_, const unsigned int length) { 44 | const unsigned char* data_buffer = reinterpret_cast(data_buffer_); 45 | unsigned int i, j; 46 | for (i = 0; i < length; i++) { 47 | unsigned char byte = data_buffer[i]; 48 | fprintf(file, "%02x ", data_buffer[i]); 49 | if (((i%16) == 15) || (i == length-1)) { 50 | for (j = 0; j < 15-(i%16); j++) { 51 | fprintf(file, " "); 52 | } 53 | fprintf(file, "| "); 54 | for (j=(i-(i%16)); j <= i; j++) { 55 | byte = data_buffer[j]; 56 | if ((byte > 31) && (byte < 127)) { 57 | fprintf(file, "%c", byte); 58 | } else { 59 | fprintf(file, "."); 60 | } 61 | } 62 | fprintf(file, "\n"); 63 | } 64 | } 65 | } 66 | 67 | 68 | } // namespace utility 69 | } // namespace mc 70 | } // namespace douban 71 | -------------------------------------------------------------------------------- /misc/.cppcheck-supp: -------------------------------------------------------------------------------- 1 | *:include/rapidjson/itoa.h:141 2 | *:include/rapidjson/itoa.h:216 3 | *:include/llvm/SmallVector.h:55 4 | *:include/llvm/SmallVector.h:88 5 | *:include/llvm/SmallVector.h:134 6 | *:include/llvm/SmallVector.h:162 7 | *:include/llvm/SmallVector.h:164 8 | *:include/llvm/SmallVector.h:214 9 | *:include/llvm/SmallVector.h:515 10 | *:include/llvm/SmallVector.h:516 11 | *:include/llvm/SmallVector.h:546 12 | *:include/llvm/SmallVector.h:715 13 | *:include/llvm/SmallVector.h:718 14 | *:include/llvm/SmallVector.h:726 15 | *:include/llvm/SmallVector.h:735 16 | *:include/llvm/SmallVector.h:796 17 | constParameter:include/BufferReader.h:99 18 | unreadVariable:include/LockPool.h 19 | unusedFunction:src/Client.cpp:239 20 | unusedFunction:src/c_client.cpp:8 21 | unusedFunction:src/c_client.cpp:13 22 | unusedFunction:src/c_client.cpp:25 23 | unusedFunction:src/c_client.cpp:31 24 | unusedFunction:src/c_client.cpp:37 25 | unusedFunction:src/c_client.cpp:43 26 | unusedFunction:src/c_client.cpp:50 27 | unusedFunction:src/c_client.cpp:56 28 | unusedFunction:src/c_client.cpp:67 29 | unusedFunction:src/c_client.cpp:68 30 | unusedFunction:src/c_client.cpp:72 31 | unusedFunction:src/c_client.cpp:89 32 | unusedFunction:src/c_client.cpp:90 33 | unusedFunction:src/c_client.cpp:91 34 | unusedFunction:src/c_client.cpp:92 35 | unusedFunction:src/c_client.cpp:93 36 | unusedFunction:src/c_client.cpp:94 37 | unusedFunction:src/c_client.cpp:98 38 | unusedFunction:src/c_client.cpp:106 39 | unusedFunction:src/c_client.cpp:112 40 | unusedFunction:src/c_client.cpp:120 41 | unusedFunction:src/c_client.cpp:128 42 | unusedFunction:src/c_client.cpp:135 43 | unusedFunction:src/c_client.cpp:140 44 | unusedFunction:src/c_client.cpp:145 45 | unusedFunction:src/c_client.cpp:149 46 | unusedFunction:src/c_client.cpp:154 47 | unusedFunction:src/c_client.cpp:159 48 | unusedFunction:src/Utility.cpp:43 49 | *:src/Connection.cpp:32 50 | *:src/Connection.cpp:37 51 | -------------------------------------------------------------------------------- /tests/test_unix.cpp: -------------------------------------------------------------------------------- 1 | #include "Common.h" 2 | #include "Client.h" 3 | #include "test_common.h" 4 | 5 | #include 6 | #include 7 | #include "gtest/gtest.h" 8 | 9 | using douban::mc::Client; 10 | using douban::mc::tests::md5Client; 11 | using douban::mc::splitServerString; 12 | 13 | Client* newUnixClient() { 14 | const char * hosts[] = { "/tmp/env_mc_dev/var/run/unix_test.socket" }; 15 | const uint32_t ports[] = { 0 }; 16 | struct stat info; 17 | // fails if ../misc/memcached_server wasn't started with startall or unix 18 | EXPECT_EQ(stat(hosts[0], &info), 0); 19 | 20 | return md5Client(hosts, ports, 1); 21 | } 22 | 23 | TEST(test_unix, establish_connection) { 24 | Client* client = newUnixClient(); 25 | ASSERT_TRUE(client != NULL); 26 | delete client; 27 | } 28 | 29 | TEST(test_unix, host_parse_regression) { 30 | char test[] = "127.0.0.1:21211 testing"; 31 | server_string_split_t out = splitServerString(test); 32 | ASSERT_STREQ(out.host, "127.0.0.1"); 33 | ASSERT_STREQ(out.port, "21211"); 34 | ASSERT_STREQ(out.alias, "testing"); 35 | } 36 | 37 | TEST(test_unix, socket_path_spaces) { 38 | char test[] = "/tmp/spacey\\ path testing"; 39 | server_string_split_t out = splitServerString(test); 40 | ASSERT_STREQ(out.host, "/tmp/spacey\\ path"); 41 | ASSERT_EQ(out.port, nullptr); 42 | ASSERT_STREQ(out.alias, "testing"); 43 | } 44 | 45 | TEST(test_unix, socket_path_escaping) { 46 | char test[] = "/tmp/spicy\\\\ path testing"; 47 | server_string_split_t out = splitServerString(test); 48 | ASSERT_STREQ(out.host, "/tmp/spicy\\\\"); 49 | ASSERT_EQ(out.port, nullptr); 50 | ASSERT_STREQ(out.alias, "path"); 51 | } 52 | 53 | TEST(test_unix, alias_space_escaping) { 54 | char test[] = "/tmp/path testing\\ alias"; 55 | server_string_split_t out = splitServerString(test); 56 | ASSERT_STREQ(out.host, "/tmp/path"); 57 | ASSERT_EQ(out.port, nullptr); 58 | ASSERT_STREQ(out.alias, "testing\\ alias"); 59 | } 60 | -------------------------------------------------------------------------------- /tests/shabby/reconnect_delay.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import os 4 | import time 5 | import libmc 6 | import slow_memcached_server 7 | import subprocess 8 | 9 | 10 | def memcached_server_ctl(cmd, port): 11 | ctl_path = os.path.join( 12 | os.path.dirname(os.path.dirname(os.path.dirname( 13 | os.path.abspath(__file__) 14 | ))), 15 | 'misc', 'memcached_server' 16 | ) 17 | print(ctl_path) 18 | subprocess.check_call([ctl_path, cmd, str(port)]) 19 | 20 | 21 | def test_soft_server_error(): 22 | mc = libmc.Client(["127.0.0.1:%d" % slow_memcached_server.PORT]) 23 | mc.config(libmc._client.MC_POLL_TIMEOUT, 24 | slow_memcached_server.BLOCKING_SECONDS * 1000 * 2) # ms 25 | 26 | RETRY_TIMEOUT = 2 27 | mc.config(libmc.MC_RETRY_TIMEOUT, RETRY_TIMEOUT) 28 | 29 | assert mc.set('foo', 1) 30 | assert not mc.set(slow_memcached_server.KEY_SET_SERVER_ERROR.decode('utf8'), 1) 31 | assert mc.set('foo', 1) # back to live 32 | time.sleep(RETRY_TIMEOUT / 2) 33 | assert mc.set('foo', 1) # alive 34 | time.sleep(RETRY_TIMEOUT + 1) 35 | assert mc.set('foo', 1) # alive 36 | 37 | 38 | def test_hard_server_error(): 39 | normal_port = 21211 40 | mc = libmc.Client(["127.0.0.1:%d" % normal_port]) 41 | 42 | RETRY_TIMEOUT = 20 43 | mc.config(libmc.MC_RETRY_TIMEOUT, RETRY_TIMEOUT) 44 | 45 | assert mc.set('foo', 1) 46 | memcached_server_ctl('stop', normal_port) 47 | 48 | # I dont know why, 49 | # but the server is still connectable after stop. 50 | # add a sleep to wait for the server quit 51 | time.sleep(3) 52 | 53 | assert not mc.set('foo', 1) # fail 54 | memcached_server_ctl('start', normal_port) 55 | assert not mc.set('foo', 1) # still fail 56 | time.sleep(RETRY_TIMEOUT + 1) 57 | assert mc.set('foo', 1) # back to live 58 | 59 | 60 | def main(): 61 | test_soft_server_error() 62 | test_hard_server_error() 63 | 64 | 65 | if __name__ == '__main__': 66 | main() 67 | -------------------------------------------------------------------------------- /include/Parser.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | 6 | #include "Common.h" 7 | #include "BufferReader.h" 8 | #include "Result.h" 9 | 10 | 11 | namespace douban { 12 | namespace mc { 13 | 14 | typedef enum { 15 | MODE_UNDEFINED, 16 | MODE_END_STATE, 17 | MODE_COUNTING 18 | } ParserMode; 19 | 20 | 21 | class PacketParser { 22 | public: 23 | PacketParser(); 24 | explicit PacketParser(io::BufferReader* reader); 25 | ~PacketParser(); 26 | void setBufferReader(io::BufferReader* reader); 27 | void setMode(ParserMode md); 28 | void addRequestKey(const char* const key, const size_t len); 29 | std::vector* getRequestKeys(); 30 | struct iovec* currentRequestKey(); 31 | size_t requestKeyCount(); 32 | void process_packets(err_code_t &err); 33 | void reset(); 34 | void rewind(); 35 | 36 | types::RetrievalResultList* getRetrievalResults(); 37 | types::MessageResultList* getMessageResults(); 38 | types::LineResultList* getLineResults(); 39 | types::UnsignedResultList* getUnsignedResults(); 40 | 41 | protected: 42 | int start_state(err_code_t& err); 43 | bool canEndParse(); 44 | void processMessageResult(message_result_type tp); 45 | void processLineResult(err_code_t& err); 46 | 47 | 48 | std::vector m_requestKeys; 49 | io::BufferReader* m_buffer_reader; 50 | parser_state_t m_state; 51 | ParserMode m_mode; 52 | size_t m_expectedResultCount; 53 | size_t m_requestKeyIdx; 54 | 55 | types::RetrievalResultList m_retrievalResults; 56 | types::MessageResultList m_messageResults; 57 | types::LineResultList m_lineResults; 58 | types::UnsignedResultList m_unsignedResults; 59 | 60 | // mt means Member-Tmp-variable 61 | types::RetrievalResult* mt_kvPtr; 62 | }; 63 | 64 | 65 | inline bool PacketParser::canEndParse() { 66 | assert(m_mode == MODE_END_STATE || m_mode == MODE_COUNTING); 67 | return m_mode == MODE_END_STATE ? IS_END_STATE(m_state) : m_requestKeyIdx == m_requestKeys.size(); 68 | } 69 | 70 | } // namespace mc 71 | } // namespace douban 72 | -------------------------------------------------------------------------------- /.travis.yml.bak: -------------------------------------------------------------------------------- 1 | dist: xenial 2 | language: python 3 | os: linux 4 | python: 5 | - '2.7' 6 | - '3.6' 7 | - '3.7' 8 | - '3.8' 9 | cache: 10 | pip: true 11 | env: 12 | - TEST_SUITE=unittest CMAKE_BUILD_TYPE=Release PRJ_COMPILER=clang 13 | - TEST_SUITE=unittest CMAKE_BUILD_TYPE=Debug PRJ_COMPILER=clang 14 | - TEST_SUITE=unittest CMAKE_BUILD_TYPE=Release PRJ_COMPILER=gcc 15 | - TEST_SUITE=unittest CMAKE_BUILD_TYPE=Debug PRJ_COMPILER=gcc 16 | - TEST_SUITE=benchmark 17 | jobs: 18 | include: 19 | - env: TEST_SUITE=gotest PRJ_COMPILER=gcc 20 | - env: TEST_SUITE=gotest PRJ_COMPILER=clang 21 | - env: TEST_SUITE=cppcheck 22 | - env: TEST_SUITE=buildwheels PLAT=manylinux1_x86_64 23 | python: '3.7' 24 | services: 25 | - docker 26 | addons: 27 | apt: 28 | sources: 29 | - ubuntu-toolchain-r-test 30 | packages: 31 | - python-dev 32 | - python-numpy 33 | - valgrind 34 | - cppcheck 35 | - memcached 36 | - libmemcached11 37 | - g++ 38 | - golang 39 | install: 40 | - if [[ $TEST_SUITE = "unittest" || $TEST_SUITE = "benchmark" || $TEST_SUITE = "buildwheels" ]]; then pip install --upgrade pip setuptools future; fi 41 | - if [[ $TEST_SUITE = "unittest" || $TEST_SUITE = "benchmark" ]]; then pip install pytest greenify gevent; fi 42 | - if [[ $TEST_SUITE = "benchmark" ]]; then pip install python-memcached pylibmc; fi 43 | - if [[ $TEST_SUITE = "buildwheels" ]]; then docker pull quay.io/pypa/$PLAT; fi 44 | before_script: 45 | - if [[ $TEST_SUITE != "cppcheck" && $TEST_SUITE != "buildwheels" ]]; then ./misc/memcached_server start; fi 46 | script: 47 | - if [[ $PRJ_COMPILER = "gcc" ]]; then export CC=gcc CXX=g++; fi 48 | - if [[ $PRJ_COMPILER = "clang" ]]; then export CC=clang CXX=clang++; fi 49 | - "./misc/travis/$TEST_SUITE.sh" 50 | after_script: 51 | - if [[ $TEST_SUITE != "cppcheck" && $TEST_SUITE != "buildwheels" ]]; then ./misc/memcached_server stop; fi 52 | deploy: 53 | provider: script 54 | script: "./misc/deploy_pypi.sh" 55 | skip_cleanup: true 56 | on: 57 | tags: true 58 | repo: douban/libmc 59 | branch: master 60 | condition: $TEST_SUITE = "buildwheels" 61 | -------------------------------------------------------------------------------- /.github/workflows/python.yml: -------------------------------------------------------------------------------- 1 | name: Python 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | unittest: 11 | runs-on: ubuntu-22.04 12 | strategy: 13 | matrix: 14 | pyver: ["3.10", "3.11", "3.12", "3.13", "3.14"] 15 | compiler: ["gcc", "clang"] 16 | build_type: ["Debug", "Release"] 17 | 18 | steps: 19 | - uses: actions/checkout@v2 20 | - name: Setup system dependencies 21 | run: | 22 | sudo apt-get update 23 | sudo apt-get -y install valgrind memcached g++ 24 | - name: Set up Python 25 | uses: actions/setup-python@v2 26 | with: 27 | python-version: ${{ matrix.pyver }} 28 | - name: Install python dependencies 29 | run: | 30 | python -m pip install --upgrade pip 31 | pip install setuptools future pytest greenify gevent numpy 32 | - name: Start memcached servers 33 | run: ./misc/memcached_server startall 34 | - name: Run unittest 35 | run: | 36 | if [[ ${{ matrix.compiler }} = "gcc" ]]; then export CC=gcc CXX=g++; fi 37 | if [[ ${{ matrix.compiler }} = "clang" ]]; then export CC=clang CXX=clang++; fi 38 | ./misc/travis/unittest.sh 39 | - name: Stop memcached servers 40 | run: ./misc/memcached_server stopall 41 | 42 | benchmark: 43 | runs-on: ubuntu-22.04 44 | strategy: 45 | matrix: 46 | pyver: ["3.10", "3.11", "3.12", "3.13", "3.14"] 47 | 48 | steps: 49 | - uses: actions/checkout@v2 50 | - name: Setup system dependencies 51 | run: | 52 | sudo apt-get update 53 | sudo apt-get -y install memcached libmemcached-dev g++ 54 | - name: Set up Python 55 | uses: actions/setup-python@v2 56 | with: 57 | python-version: ${{ matrix.pyver }} 58 | - name: Install python dependencies 59 | run: | 60 | python -m pip install --upgrade pip 61 | pip install setuptools future python-memcached pylibmc 62 | - name: Start memcached servers 63 | run: ./misc/memcached_server start 64 | - name: Run benchmark 65 | run: | 66 | ./misc/travis/benchmark.sh 67 | - name: Stop memcached servers 68 | run: ./misc/memcached_server stop 69 | -------------------------------------------------------------------------------- /tests/test_ketama.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | #include "Common.h" 6 | #include "Connection.h" 7 | #include "hashkit/ketama.h" 8 | #include "gtest/gtest.h" 9 | #include "test_common.h" 10 | 11 | using std::atoi; 12 | using std::string; 13 | using std::getline; 14 | using std::ifstream; 15 | using std::stringstream; 16 | using douban::mc::hashkit::KetamaSelector; 17 | using douban::mc::tests::get_resource_path; 18 | using douban::mc::Connection; 19 | 20 | size_t wc(const char* path) { 21 | ifstream hintLineFile(path); 22 | size_t nLines = std::count( 23 | std::istreambuf_iterator(hintLineFile), 24 | std::istreambuf_iterator(), 25 | '\n' 26 | ); 27 | hintLineFile.close(); 28 | return nLines; 29 | } 30 | 31 | 32 | void load_servers(KetamaSelector& ks, Connection* conns, size_t nConns, const char* csv_path) { 33 | ifstream serverFile(csv_path); 34 | string line; 35 | ASSERT_TRUE(serverFile.good()); 36 | int i = 0; 37 | while (std::getline(serverFile, line)) { 38 | stringstream lineStream(line); 39 | string hostname = "", port_; 40 | getline(lineStream, hostname, ','); 41 | getline(lineStream, port_, ','); 42 | uint32_t port = static_cast(atoi(port_.c_str())); 43 | conns[i].init(hostname.c_str(), port); 44 | ++i; 45 | } 46 | ASSERT_EQ(nConns, i); 47 | ks.addServers(conns, i); 48 | 49 | serverFile.close(); 50 | } 51 | 52 | void valid_key_pool(KetamaSelector& ks, const char* csv_path) { 53 | ifstream key_pool(csv_path); 54 | string line; 55 | ASSERT_TRUE(key_pool.good()); 56 | while (std::getline(key_pool, line)) { 57 | stringstream lineStream(line); 58 | string key = "", idx_; 59 | getline(lineStream, key, ','); 60 | getline(lineStream, idx_, ','); 61 | int idx = atoi(idx_.c_str()); 62 | bool check_alive = false; 63 | ASSERT_EQ(ks.getServer(key.c_str(), key.size(), check_alive), idx); 64 | } 65 | key_pool.close(); 66 | } 67 | 68 | 69 | TEST(test_ketama, servers) { 70 | string csv_path = get_resource_path("server_port.csv"); 71 | size_t nServers = wc(csv_path.c_str()); 72 | ASSERT_EQ(nServers, 200); 73 | 74 | Connection* conns = new Connection[nServers]; 75 | KetamaSelector ks; 76 | load_servers(ks, conns, nServers, csv_path.c_str()); 77 | valid_key_pool(ks, get_resource_path("key_pool_idx.csv").c_str()); 78 | delete[] conns; 79 | } 80 | -------------------------------------------------------------------------------- /misc/idle_timeout_memcached_server: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # based on http://serverfault.com/a/360230/105615 3 | 4 | set -e 5 | 6 | PORTS="21211 21212 21213 21214 21215 21216 21217 21218 21219 21220 21221 21222 21223 21224 21225 21226 21227 21228 21229 21230" 7 | USER=memcached 8 | basedir="/tmp/env_mc_dev" 9 | mkdir -p "$basedir/var/log" 10 | mkdir -p "$basedir/var/run" 11 | 12 | RETVAL=0 13 | prog="memcached" 14 | cmd="`which memcached`" 15 | ip="127.0.0.1" 16 | memory=64 17 | threads=4 18 | 19 | 20 | function start() 21 | { 22 | port="$1" 23 | if [ `ps -ef | grep "$cmd" | grep -c $port` -ge 1 ]; then 24 | echo "Starting the memcached server on port '$port'... " 25 | else 26 | if [ ! -f $basedir/var/log/${port}.log ]; then 27 | mkdir -p $basedir/var/log 28 | touch $basedir/var/log/${port}.log 29 | fi 30 | $cmd -d -u $USER -l $ip -t $threads -m ${memory} -p $port -P $basedir/var/run/${port}.pid -o idle_timeout=1 > $basedir/var/log/${port}.log 2>&1 31 | echo "Starting the memcached server on port '$port'... " 32 | fi 33 | } 34 | 35 | function stop() 36 | { 37 | port="$1" 38 | if [ `ps -ef | grep "$cmd" | grep -c $port` -eq 0 ]; then 39 | echo $"Stopped the memcached server on port '$port'... " 40 | else 41 | kill -TERM `ps -ef | grep "$cmd" | grep $port | grep -v grep | awk '{ print $2 }'` 42 | echo "Stopping the memcached server on port '$port'... " 43 | fi 44 | rm -rf $basedir 45 | } 46 | 47 | case "$1" in 48 | start) 49 | if [ -n "$2" ]; then 50 | start $2 51 | else 52 | for port in $PORTS; do 53 | start $port & 54 | done 55 | wait 56 | fi 57 | ;; 58 | stop) 59 | if [ -n "$2" ]; then 60 | port="$2" 61 | stop $port 62 | else 63 | for port in $PORTS; do 64 | stop $port & 65 | done 66 | wait 67 | fi 68 | ;; 69 | restart) 70 | if [ -n "$2" ]; then 71 | stop $2 72 | start $2 73 | else 74 | for port in $PORTS; do 75 | stop $port & 76 | start $port & 77 | done 78 | wait 79 | fi 80 | ;; 81 | *) 82 | printf 'Usage: %s {start|stop|restart} \n' "$prog" 83 | exit 1 84 | ;; 85 | esac 86 | -------------------------------------------------------------------------------- /misc/generate_hash_dataset.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import binascii 4 | import hashlib 5 | import numpy as np 6 | 7 | from builtins import str as unicode 8 | 9 | FNV_32_INIT = 0x811c9dc5 10 | FNV_32_PRIME = 0x01000193 11 | 12 | def compute_crc_32(key): 13 | return np.uint32(binascii.crc32(key)) 14 | 15 | 16 | def compute_fnv1_32(key): 17 | hval = FNV_32_INIT 18 | fnv_32_prime = FNV_32_PRIME 19 | uint32_max = 2 ** 32 20 | for s in key: 21 | hval = (hval * fnv_32_prime) % uint32_max 22 | hval = hval ^ (ord(s) if isinstance(s, str) else s) 23 | return np.uint32(hval) 24 | 25 | 26 | def compute_fnv1a_32(key): 27 | hval = FNV_32_INIT 28 | fnv_32_prime = FNV_32_PRIME 29 | uint32_max = 2 ** 32 30 | for s in key: 31 | hval = hval ^ (ord(s) if isinstance(s, str) else s) 32 | hval = (hval * fnv_32_prime) % uint32_max 33 | return np.uint32(hval) 34 | 35 | 36 | def compute_md5(key): 37 | md5 = hashlib.md5() 38 | md5.update(key) 39 | digest = md5.digest() 40 | 41 | def convert(d): 42 | if isinstance(d, int): 43 | return np.uint32(d & 0xFF) 44 | else: 45 | return np.uint32(ord(d) & 0xFF) 46 | 47 | return ((convert(digest[3]) << 24) | 48 | (convert(digest[2]) << 16) | 49 | (convert(digest[1]) << 8) | 50 | (convert(digest[0]))) 51 | 52 | 53 | U32_HASH_FN_DICT = { 54 | 'crc_32': (compute_crc_32, []), 55 | 'fnv1_32': (compute_fnv1_32, []), 56 | 'fnv1a_32': (compute_fnv1a_32, []), 57 | 'md5': (compute_md5, []), 58 | } 59 | 60 | def main(argv): 61 | if not len(argv) == 2: 62 | print("usage: python %s in.txt" % argv[0]) 63 | return 1 64 | 65 | input_path = os.path.abspath(sys.argv[1]) 66 | 67 | with open(input_path, 'rb') as fhandler: 68 | for line in fhandler: 69 | key = line.strip() 70 | for hash_name, (hash_fn, hash_rst) in U32_HASH_FN_DICT.items(): 71 | hash_rst.append(hash_fn(key)) 72 | 73 | for hash_name, (hash_fn, hash_rst) in U32_HASH_FN_DICT.items(): 74 | prefix, dot_ext = os.path.splitext(input_path) 75 | out_path = '%s_%s%s' % (prefix, hash_name, dot_ext) 76 | with open(out_path, 'w') as fhandler: 77 | fhandler.writelines(("%s\r\n" % h for h in hash_rst)) 78 | print(out_path) 79 | 80 | 81 | if __name__ == '__main__': 82 | sys.exit(main(sys.argv)) 83 | -------------------------------------------------------------------------------- /include/Export.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #ifdef __cplusplus 5 | #include 6 | #else 7 | #include 8 | #endif 9 | 10 | 11 | typedef enum { 12 | // Client config options 13 | CFG_POLL_TIMEOUT = 0, 14 | CFG_CONNECT_TIMEOUT, 15 | CFG_RETRY_TIMEOUT, 16 | CFG_HASH_FUNCTION, 17 | CFG_MAX_RETRIES, 18 | CFG_SET_FAILOVER, 19 | 20 | // type separator to track number of Client config options to save 21 | CLIENT_CONFIG_OPTION_COUNT, 22 | 23 | // ClientPool config options 24 | CFG_INITIAL_CLIENTS, 25 | CFG_MAX_CLIENTS, 26 | CFG_MAX_GROWTH 27 | } config_options_t; 28 | 29 | 30 | typedef enum { 31 | OPT_HASH_MD5, 32 | OPT_HASH_FNV1_32, 33 | OPT_HASH_FNV1A_32, 34 | OPT_HASH_CRC_32, 35 | } hash_function_options_t; 36 | 37 | 38 | typedef enum { 39 | RET_SEND_ERR = -9, 40 | RET_RECV_ERR = -8, 41 | RET_CONN_POLL_ERR = -7, 42 | RET_POLL_TIMEOUT_ERR = -6, 43 | RET_POLL_ERR = -5, 44 | RET_MC_SERVER_ERR = -4, 45 | RET_PROGRAMMING_ERR = -3, 46 | RET_INVALID_KEY_ERR = -2, 47 | RET_INCOMPLETE_BUFFER_ERR = -1, 48 | RET_OK = 0 49 | } err_code_t; 50 | 51 | typedef struct { 52 | char* host; 53 | char* port; 54 | char* alias; 55 | } server_string_split_t; 56 | 57 | 58 | typedef int64_t exptime_t; 59 | typedef uint32_t flags_t; 60 | typedef uint64_t cas_unique_t; 61 | 62 | 63 | typedef struct { 64 | char* key; // 8B 65 | char* data_block; // 8B 66 | cas_unique_t cas_unique; // 8B 67 | uint32_t bytes; // 4B 68 | flags_t flags; // 4B 69 | uint8_t key_len; // 1B 70 | } retrieval_result_t; 71 | 72 | 73 | enum message_result_type { 74 | MSG_LIBMC_INVALID = -1, 75 | MSG_EXISTS = 0, 76 | MSG_OK, 77 | MSG_STORED, 78 | MSG_NOT_STORED, 79 | MSG_NOT_FOUND, 80 | MSG_DELETED, 81 | MSG_TOUCHED, 82 | }; 83 | 84 | 85 | typedef struct { 86 | enum message_result_type type_; 87 | char* key; 88 | size_t key_len; 89 | } message_result_t; 90 | 91 | 92 | typedef struct { 93 | char* key; 94 | size_t key_len; 95 | uint64_t value; 96 | } unsigned_result_t; 97 | 98 | 99 | // For flush_all command, we need to specify 100 | // {host} and {msg_type}, 101 | // for other broadcast commands, we need to specify 102 | // all fields except {msg_type} 103 | typedef struct { 104 | char* host; 105 | char** lines; 106 | size_t* line_lens; 107 | size_t len; 108 | enum message_result_type msg_type; // for flush_all command 109 | } broadcast_result_t; 110 | -------------------------------------------------------------------------------- /src/BufferWriter.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | #include "BufferWriter.h" 6 | #include "Utility.h" 7 | 8 | namespace douban{ 9 | namespace mc { 10 | namespace io { 11 | 12 | BufferWriter::BufferWriter() :m_readIdx(0), m_msgIovlen(0) { 13 | } 14 | 15 | 16 | BufferWriter::~BufferWriter() { 17 | reset(); 18 | } 19 | 20 | 21 | void BufferWriter::reset() { 22 | m_iovec.clear(); 23 | m_originalIovec.clear(); 24 | for (std::vector::const_iterator it = m_unsignedStringList.begin(); 25 | it != m_unsignedStringList.end(); ++it) { 26 | delete[] *it; 27 | } 28 | m_unsignedStringList.clear(); 29 | m_readIdx = 0; 30 | m_msgIovlen = 0; 31 | } 32 | 33 | 34 | void BufferWriter::reserve(size_t n) { 35 | m_iovec.reserve(n); 36 | } 37 | 38 | 39 | void BufferWriter::takeBuffer(const char* const buf, size_t buf_len) { 40 | struct iovec iov; 41 | iov.iov_base = const_cast(buf); 42 | iov.iov_len = buf_len; 43 | m_iovec.push_back(iov); 44 | ++m_msgIovlen; 45 | } 46 | 47 | 48 | void BufferWriter::takeNumber(int64_t val) { 49 | m_unsignedStringList.push_back(new char[32]); 50 | char* buf = m_unsignedStringList.back(); 51 | struct iovec iov; 52 | 53 | iov.iov_base = buf; 54 | iov.iov_len = douban::mc::utility::int64ToCharArray(val, buf); 55 | m_iovec.push_back(iov); 56 | ++m_msgIovlen; 57 | } 58 | 59 | 60 | const struct iovec* const BufferWriter::getReadPtr(size_t &n) { 61 | n = m_msgIovlen; 62 | if (n > 0) { 63 | return &m_iovec[m_readIdx]; 64 | } 65 | return NULL; 66 | } 67 | 68 | 69 | void BufferWriter::commitRead(size_t nSent) { 70 | while (m_msgIovlen > 0 && nSent >= m_iovec[m_readIdx].iov_len) { 71 | nSent -= m_iovec[m_readIdx].iov_len; 72 | ++m_readIdx; 73 | --m_msgIovlen; 74 | } 75 | 76 | if (nSent > 0) { 77 | if (m_originalIovec.empty()) { 78 | m_originalIovec = m_iovec; 79 | } 80 | struct iovec* iovPtr = &m_iovec[m_readIdx]; 81 | iovPtr->iov_base = static_cast(iovPtr->iov_base) + nSent; 82 | iovPtr->iov_len -= nSent; 83 | } 84 | } 85 | 86 | 87 | void BufferWriter::rewind() { 88 | m_readIdx = 0; 89 | m_msgIovlen = m_iovec.size(); 90 | if (!m_originalIovec.empty()) { 91 | m_iovec = m_originalIovec; 92 | } 93 | } 94 | 95 | 96 | size_t BufferWriter::msgIovlen() { 97 | return m_msgIovlen; 98 | } 99 | 100 | 101 | } // namespace io 102 | } // namespace mc 103 | } // namespace douban 104 | -------------------------------------------------------------------------------- /tests/test_parser.cpp: -------------------------------------------------------------------------------- 1 | #include "Common.h" 2 | #include "Result.h" 3 | #include "BufferReader.h" 4 | #include "Parser.h" 5 | #include 6 | #include "gtest/gtest.h" 7 | 8 | using douban::mc::types::RetrievalResult; 9 | 10 | using douban::mc::io::BufferReader; 11 | using douban::mc::io::DataBlock; 12 | using douban::mc::io::TokenData; 13 | using douban::mc::PacketParser; 14 | 15 | 16 | 17 | TEST(test_parser, empty_result) { 18 | err_code_t err; 19 | BufferReader reader; 20 | PacketParser parser; 21 | parser.setMode(douban::mc::MODE_END_STATE); 22 | parser.setBufferReader(&reader); 23 | int i = 0; 24 | char input_buffer[][5] = { 25 | "E", "ND\r\n" 26 | }; 27 | while (true) { 28 | if (i < 2) { 29 | reader.write(input_buffer[i], strlen(input_buffer[i])); 30 | ++i; 31 | } 32 | parser.process_packets(err); 33 | if (err == RET_INCOMPLETE_BUFFER_ERR) { 34 | continue; 35 | } 36 | break; 37 | } 38 | ASSERT_EQ(parser.getRetrievalResults()->size(), 0); 39 | } 40 | 41 | 42 | TEST(test_parser, multi_results) { 43 | err_code_t err; 44 | DataBlock::setMinCapacity(10); 45 | BufferReader reader; 46 | PacketParser parser; 47 | parser.setMode(douban::mc::MODE_END_STATE); 48 | parser.setBufferReader(&reader); 49 | size_t i = 0; 50 | 51 | char input_buffer[][100] = { 52 | "VALUE fo", "o 0 ", "14", "\r", "\n", "VALUE foo ", "0 14", "\r\n", 53 | "VALUE baz", " 1 ", "4 1024", "\r", "\n", "\r\n \"", "\r\n", 54 | "VALUE ba", "r 0 ", "4", "\r", "\n", "1234", "\r\n", 55 | "E", "ND\r\n" 56 | }; 57 | size_t n_input = 24; 58 | 59 | while (true) { 60 | if (i < n_input) { 61 | ASSERT_NO_THROW(reader.write(input_buffer[i], strlen(input_buffer[i]))); 62 | ++i; 63 | } 64 | parser.process_packets(err); 65 | if (err == RET_INCOMPLETE_BUFFER_ERR) { 66 | continue; 67 | } 68 | break; 69 | } 70 | RetrievalResult* res = NULL; 71 | retrieval_result_t* innerRes; 72 | ASSERT_EQ(parser.getRetrievalResults()->size(), 3); 73 | 74 | char keys[][4] = {"foo", "baz", "bar"}; 75 | size_t len_vals[] = {14, 4, 4}; 76 | char vals[][15] = {"VALUE foo 0 14", "\r\n \"", "1234"}; 77 | 78 | for (i = 0; i < 3; i++) { 79 | res = &((*parser.getRetrievalResults())[i]); 80 | innerRes = res->inner(); 81 | 82 | size_t len_key = 3; 83 | ASSERT_N_STREQ(innerRes->key, keys[i], len_key); 84 | size_t len_val = innerRes->bytes; 85 | ASSERT_EQ(len_val, len_vals[i]); 86 | if (len_val > 0) { 87 | ASSERT_N_STREQ(innerRes->data_block, vals[i], len_val); 88 | } 89 | } 90 | } 91 | 92 | // TODO test MODE_COUNTING 93 | -------------------------------------------------------------------------------- /src/Common.cpp: -------------------------------------------------------------------------------- 1 | #include "Common.h" 2 | #include "Keywords.h" 3 | 4 | #ifdef __GLIBC__ 5 | #include 6 | #include 7 | 8 | 9 | void printBacktrace() { 10 | void *trace_elems[20]; 11 | int trace_elem_count(backtrace(trace_elems, 20)); 12 | char **stack_syms(backtrace_symbols(trace_elems, trace_elem_count)); 13 | for ( int i = 0 ; i < trace_elem_count ; ++i ) { 14 | fprintf(stderr, " %s\n", stack_syms[i]); 15 | } 16 | free(stack_syms); 17 | } 18 | #else 19 | void printBacktrace() {} 20 | #endif 21 | 22 | namespace douban { 23 | namespace mc { 24 | 25 | const char* errCodeToString(err_code_t err) { 26 | switch (err) 27 | { 28 | case RET_SEND_ERR: 29 | return keywords::kSEND_ERROR; 30 | case RET_RECV_ERR: 31 | return keywords::kRECV_ERROR; 32 | case RET_CONN_POLL_ERR: 33 | return keywords::kCONN_POLL_ERROR; 34 | case RET_POLL_TIMEOUT_ERR: 35 | return keywords::kPOLL_TIMEOUT_ERROR; 36 | case RET_POLL_ERR: 37 | return keywords::kPOLL_ERROR; 38 | case RET_MC_SERVER_ERR: 39 | return keywords::kSERVER_ERROR; 40 | case RET_PROGRAMMING_ERR: 41 | return keywords::kPROGRAMMING_ERROR; 42 | case RET_INVALID_KEY_ERR: 43 | return keywords::kINVALID_KEY_ERROR; 44 | case RET_INCOMPLETE_BUFFER_ERR: 45 | return keywords::kINCOMPLETE_BUFFER_ERROR; 46 | case RET_OK: 47 | return "ok"; 48 | default: 49 | return "unknown"; 50 | } 51 | } 52 | 53 | bool isUnixSocket(const char* host) { 54 | // errors on the side of false negatives, allowing syntax expansion; 55 | // starting slash syntax is from libmemcached 56 | return host[0] == '/'; 57 | } 58 | 59 | // modifies input string and output pointers reference input 60 | server_string_split_t splitServerString(char* input) { 61 | bool escaped = false; 62 | server_string_split_t res = { input, NULL, NULL }; 63 | for (;; input++) { 64 | switch (*input) 65 | { 66 | case '\0': 67 | return res; 68 | case ':': 69 | if (res.alias == NULL) { 70 | *input = '\0'; 71 | if (res.port == NULL) { 72 | res.port = input + 1; 73 | } 74 | } 75 | escaped = false; 76 | continue; 77 | case ' ': 78 | if (!escaped) { 79 | *input = '\0'; 80 | if (res.alias == NULL) { 81 | res.alias = input + 1; 82 | continue; 83 | } else { 84 | return res; 85 | } 86 | } 87 | default: 88 | escaped = false; 89 | continue; 90 | case '\\': 91 | escaped ^= 1; 92 | } 93 | } 94 | } 95 | 96 | } // namespace mc 97 | } // namespace douban 98 | -------------------------------------------------------------------------------- /tests/test_common.h: -------------------------------------------------------------------------------- 1 | #include 2 | #include "Client.h" 3 | #include 4 | 5 | 6 | namespace douban { 7 | namespace mc { 8 | namespace tests { 9 | 10 | extern void gen_random(char *s, const int len); 11 | 12 | void gen_random(char *s, const int len) { 13 | static const char alphanum[] = 14 | "0123456789" 15 | "ABCDEFGHIJKLMNOPQRSTUVWXYZ" 16 | "abcdefghijklmnopqrstuvwxyz"; 17 | int i = 0; 18 | for (i = 0; i < len; ++i) { 19 | s[i] = alphanum[rand() % (sizeof(alphanum) - 1)]; 20 | } 21 | 22 | s[len] = 0; 23 | } 24 | 25 | 26 | mc::Client* md5Client(const char* const * hosts, const uint32_t* ports, const size_t n, 27 | const char* const * aliases = NULL) { 28 | mc::Client* client = new mc::Client(); 29 | client->config(CFG_HASH_FUNCTION, OPT_HASH_MD5); 30 | client->init(hosts, ports, n, aliases); 31 | broadcast_result_t* results; 32 | size_t nHosts; 33 | int ret = client->version(&results, &nHosts); 34 | client->destroyBroadcastResult(); 35 | if (ret != 0) { 36 | delete client; 37 | return NULL; 38 | } 39 | return client; 40 | } 41 | 42 | 43 | mc::Client* newClient(int n) { 44 | assert(n <= 20); 45 | const char * hosts[] = { 46 | "127.0.0.1", 47 | "127.0.0.1", 48 | "127.0.0.1", 49 | "127.0.0.1", 50 | "127.0.0.1", 51 | "127.0.0.1", 52 | "127.0.0.1", 53 | "127.0.0.1", 54 | "127.0.0.1", 55 | "127.0.0.1", 56 | "127.0.0.1", 57 | "127.0.0.1", 58 | "127.0.0.1", 59 | "127.0.0.1", 60 | "127.0.0.1", 61 | "127.0.0.1", 62 | "127.0.0.1", 63 | "127.0.0.1", 64 | "127.0.0.1", 65 | "127.0.0.1" 66 | }; 67 | const uint32_t ports[] = { 68 | 21211, 21212, 21213, 21214, 21215, 21216, 21217, 21218, 21219, 21220, 69 | 21221, 21222, 21223, 21224, 21225, 21226, 21227, 21228, 21229, 21230 70 | }; 71 | const char * aliases[] = { 72 | "alfa", 73 | "bravo", 74 | "charlie", 75 | "delta", 76 | "echo", 77 | NULL, 78 | "golf", 79 | "hotel", 80 | "india", 81 | "juliett", 82 | "kilo", 83 | "lima", 84 | "mike", 85 | "november", 86 | "oscar", 87 | "papa", 88 | "quebec", 89 | "romeo", 90 | "sierra", 91 | "tango" 92 | }; 93 | return md5Client(hosts, ports, n, aliases); 94 | } 95 | 96 | 97 | std::string get_resource_path(const char* basename) { 98 | std::string this_path(__FILE__); 99 | int pos = this_path.rfind("/"); 100 | return this_path.substr(0, pos) + "/resources/" + std::string(basename); 101 | } 102 | 103 | 104 | } // namespace tests 105 | } // namespace mc 106 | } // namespace douban 107 | -------------------------------------------------------------------------------- /src/Result.cpp: -------------------------------------------------------------------------------- 1 | #include "Result.h" 2 | #include "Common.h" 3 | 4 | namespace douban { 5 | namespace mc { 6 | namespace types { 7 | 8 | 9 | RetrievalResult::RetrievalResult() { 10 | this->cas_unique = 0; 11 | this->bytes = 0; 12 | this->bytesRemain = this->bytes + 1; 13 | this->flags = 0; 14 | this->key_len = 0; 15 | m_inner.key = NULL; 16 | m_inner.data_block = NULL; 17 | } 18 | 19 | RetrievalResult::RetrievalResult(const RetrievalResult& other) { 20 | copyTokenData(other.key, this->key); 21 | copyTokenData(other.data_block, this->data_block); 22 | 23 | this->cas_unique = other.cas_unique; 24 | this->bytesRemain = other.bytesRemain; 25 | this->bytes = other.bytes; 26 | this->flags = other.flags; 27 | this->key_len = other.key_len; 28 | this->m_inner.key = NULL; 29 | this->m_inner.data_block = NULL; 30 | } 31 | 32 | 33 | RetrievalResult::~RetrievalResult() { 34 | if (key.size() > 1) { // copy happened 35 | delete[] m_inner.key; 36 | } 37 | if (data_block.size() > 1) { 38 | delete[] m_inner.data_block; 39 | } 40 | freeTokenData(key); 41 | freeTokenData(data_block); 42 | } 43 | 44 | retrieval_result_t* RetrievalResult::inner() { 45 | if (m_inner.key == NULL) { 46 | m_inner.key = parseTokenData(this->key, this->key_len); 47 | } 48 | if (m_inner.data_block == NULL) { 49 | m_inner.data_block = parseTokenData(this->data_block, this->bytes); 50 | } 51 | m_inner.cas_unique = this->cas_unique; // 8B 52 | m_inner.bytes = this->bytes; // 4B 53 | m_inner.flags = this->flags; // 2B 54 | m_inner.key_len = this->key_len; // 1B 55 | return &m_inner; 56 | } 57 | 58 | 59 | LineResult::LineResult() { 60 | this->m_inner = NULL; 61 | this->line_len = 0; 62 | } 63 | 64 | LineResult::LineResult(const LineResult& other) { 65 | this->line_len = other.line_len; 66 | copyTokenData(other.line, this->line); 67 | this->m_inner = NULL; 68 | } 69 | 70 | 71 | LineResult::~LineResult() { 72 | if (this->line.size() > 1) { 73 | delete[] this->m_inner; 74 | } 75 | freeTokenData(this->line); 76 | } 77 | 78 | char* LineResult::inner(size_t& n) { 79 | if (this->m_inner == NULL) { 80 | this->m_inner = parseTokenData(this->line, this->line_len); 81 | } 82 | n = this->line_len - 1; // NOTE: LineResult is always ends with '\r', which should be ignored 83 | return this->m_inner; 84 | } 85 | 86 | 87 | void delete_broadcast_result(broadcast_result_t* ptr) { 88 | if (ptr->lines) { 89 | delete[] ptr->lines; 90 | ptr->lines = NULL; 91 | } 92 | if (ptr->line_lens) { 93 | delete[] ptr->line_lens; 94 | ptr->line_lens = NULL; 95 | } 96 | } 97 | 98 | 99 | } // namespace types 100 | } // namespace mc 101 | } // namespace douban 102 | -------------------------------------------------------------------------------- /libmc/__init__.py: -------------------------------------------------------------------------------- 1 | import os 2 | import functools 3 | from ._client import ( 4 | PyClient, ThreadUnsafe, 5 | encode_value, 6 | decode_value, 7 | PyClientPool, PyClientUnsafe as ClientUnsafe, 8 | 9 | MC_DEFAULT_EXPTIME, 10 | MC_POLL_TIMEOUT, 11 | MC_CONNECT_TIMEOUT, 12 | MC_RETRY_TIMEOUT, 13 | MC_SET_FAILOVER, 14 | MC_INITIAL_CLIENTS, 15 | MC_MAX_CLIENTS, 16 | MC_MAX_GROWTH, 17 | 18 | MC_HASH_MD5, 19 | MC_HASH_FNV1_32, 20 | MC_HASH_FNV1A_32, 21 | MC_HASH_CRC_32, 22 | 23 | MC_RETURN_SEND_ERR, 24 | MC_RETURN_RECV_ERR, 25 | MC_RETURN_CONN_POLL_ERR, 26 | MC_RETURN_POLL_TIMEOUT_ERR, 27 | MC_RETURN_POLL_ERR, 28 | MC_RETURN_MC_SERVER_ERR, 29 | MC_RETURN_PROGRAMMING_ERR, 30 | MC_RETURN_INVALID_KEY_ERR, 31 | MC_RETURN_INCOMPLETE_BUFFER_ERR, 32 | MC_RETURN_OK, 33 | __file__ as _libmc_so_file 34 | ) 35 | 36 | __VERSION__ = "1.4.15" 37 | __version__ = "1.4.15" 38 | __author__ = "mckelvin" 39 | __email__ = "mckelvin@users.noreply.github.com" 40 | __date__ = "Fri Jun 7 06:16:00 2024 +0800" 41 | 42 | 43 | class Client(PyClient): 44 | pass 45 | 46 | class ClientPool(PyClientPool): 47 | pass 48 | 49 | class ThreadedClient: 50 | def __init__(self, *args, **kwargs): 51 | self._client_pool = ClientPool(*args, **kwargs) 52 | 53 | def update_servers(self, servers): 54 | return self._client_pool.update_servers(servers) 55 | 56 | def config(self, opt, val): 57 | self._client_pool.config(opt, val) 58 | 59 | def __getattr__(self, key): 60 | if not hasattr(Client, key): 61 | raise AttributeError 62 | result = getattr(Client, key) 63 | if callable(result): 64 | @functools.wraps(result) 65 | def wrapper(*args, **kwargs): 66 | with self._client_pool.client() as mc: 67 | return getattr(mc, key)(*args, **kwargs) 68 | return wrapper 69 | return result 70 | 71 | 72 | DYNAMIC_LIBRARIES = [os.path.abspath(_libmc_so_file)] 73 | 74 | 75 | __all__ = [ 76 | 'Client', 'ThreadUnsafe', '__VERSION__', 'encode_value', 'decode_value', 77 | 'ClientUnsafe', 'ClientPool', 'ThreadedClient', 78 | 79 | 'MC_DEFAULT_EXPTIME', 'MC_POLL_TIMEOUT', 'MC_CONNECT_TIMEOUT', 80 | 'MC_RETRY_TIMEOUT', 'MC_SET_FAILOVER', 'MC_INITIAL_CLIENTS', 81 | 'MC_MAX_CLIENTS', 'MC_MAX_GROWTH', 82 | 83 | 'MC_HASH_MD5', 'MC_HASH_FNV1_32', 'MC_HASH_FNV1A_32', 'MC_HASH_CRC_32', 84 | 85 | 'MC_RETURN_SEND_ERR', 'MC_RETURN_RECV_ERR', 'MC_RETURN_CONN_POLL_ERR', 86 | 'MC_RETURN_POLL_TIMEOUT_ERR', 'MC_RETURN_POLL_ERR', 87 | 'MC_RETURN_MC_SERVER_ERR', 'MC_RETURN_PROGRAMMING_ERR', 88 | 'MC_RETURN_INVALID_KEY_ERR', 'MC_RETURN_INCOMPLETE_BUFFER_ERR', 89 | 'MC_RETURN_OK', 'DYNAMIC_LIBRARIES' 90 | ] 91 | -------------------------------------------------------------------------------- /src/DataBlock.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #include "DataBlock.h" 4 | #include "Common.h" 5 | 6 | namespace douban { 7 | namespace mc { 8 | namespace io { 9 | 10 | DataBlock::DataBlock() 11 | :m_data(NULL), m_capacity(0), m_size(0), m_nBytesRef(0) 12 | { } 13 | 14 | 15 | DataBlock::DataBlock(const DataBlock& other) { 16 | if (other.m_data != NULL) { 17 | log_err("copy constructor of DataBlock should never be " 18 | "called after initialization."); 19 | return; 20 | } 21 | assert(other.m_data == NULL && other.m_capacity == 0 && 22 | other.m_size == 0 && other.m_nBytesRef == 0); 23 | 24 | this->m_data = other.m_data; 25 | this->m_capacity = other.m_capacity; 26 | this->m_size = other.m_size; 27 | this->m_nBytesRef = other.m_nBytesRef; 28 | } 29 | 30 | 31 | size_t DataBlock::s_minCapacity(MIN_DATABLOCK_CAPACITY); 32 | 33 | 34 | DataBlock::~DataBlock() { 35 | delete[] m_data; 36 | } 37 | 38 | 39 | void DataBlock::init(size_t len) { 40 | if (m_data != NULL) { 41 | log_err("DataBlock(%p)::init should only be called once", this); 42 | return; 43 | } 44 | this->m_data = new char[len]; 45 | this->m_capacity = len; 46 | this->m_nBytesRef = 0; 47 | this->m_size = 0; 48 | } 49 | 50 | 51 | void DataBlock::setMinCapacity(size_t len) { 52 | log_warn("make sure this line is never called in production"); 53 | s_minCapacity = len; 54 | } 55 | 56 | 57 | size_t DataBlock::minCapacity() { 58 | return s_minCapacity; 59 | } 60 | 61 | 62 | void DataBlock::reset() { 63 | this->m_nBytesRef = 0; 64 | this->m_size = 0; 65 | } 66 | 67 | 68 | size_t DataBlock::capacity() { 69 | return m_capacity; 70 | } 71 | 72 | 73 | size_t DataBlock::occupy(size_t len) { 74 | assert(m_size + len <= m_capacity); 75 | this->acquire(len); 76 | return m_size += len; 77 | } 78 | 79 | 80 | char* DataBlock::getWritePtr() { 81 | if (m_size == m_capacity) { 82 | return NULL; 83 | } 84 | return m_data + m_size; 85 | } 86 | 87 | 88 | size_t DataBlock::getWriteLeft() { 89 | return m_capacity - m_size; 90 | } 91 | 92 | 93 | size_t DataBlock::find(char c, size_t since) { 94 | char *p = std::find(m_data + since, m_data + m_size, c); 95 | return p - m_data; 96 | } 97 | 98 | 99 | bool is_not_digit(int c) { 100 | return !('0' <= c && c <= '9'); 101 | } 102 | 103 | 104 | size_t DataBlock::findNotNumeric(size_t since) { 105 | char *p = std::find_if(m_data + since, m_data + m_size, is_not_digit); 106 | return p - m_data; 107 | } 108 | 109 | 110 | bool DataBlock::reusable() { 111 | return m_nBytesRef == 0; 112 | } 113 | 114 | 115 | size_t DataBlock::nBytesRef() { 116 | return m_nBytesRef; 117 | } 118 | 119 | } // namespace io 120 | } // namespace mc 121 | } // namespace douban 122 | -------------------------------------------------------------------------------- /include/LockPool.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | namespace douban { 11 | namespace mc { 12 | 13 | // https://stackoverflow.com/a/14792685/3476782 14 | class OrderedLock { 15 | std::queue m_fifo_locks; 16 | protected: 17 | std::mutex m_fifo_access; 18 | std::atomic m_locked; 19 | 20 | protected: 21 | OrderedLock() : m_locked(true) {}; 22 | std::unique_lock lock() { 23 | std::unique_lock acquire(m_fifo_access); 24 | if (m_locked) { 25 | std::condition_variable signal; 26 | m_fifo_locks.emplace(&signal); 27 | signal.wait(acquire); 28 | m_fifo_locks.pop(); 29 | } else { 30 | m_locked = true; 31 | } 32 | return acquire; 33 | } 34 | 35 | void unlock() { 36 | if (m_fifo_locks.empty()) { 37 | m_locked = false; 38 | } else { 39 | m_fifo_locks.front()->notify_all(); 40 | } 41 | } 42 | }; 43 | 44 | class LockPool : public OrderedLock { 45 | std::deque m_available; 46 | std::list m_muxes; 47 | std::list m_mux_mallocs; 48 | 49 | protected: 50 | std::deque m_thread_workers; 51 | 52 | LockPool() {} 53 | ~LockPool() { 54 | std::lock_guard freeing(m_fifo_access); 55 | for (auto worker : m_thread_workers) { 56 | std::lock_guard freeing_worker(*worker); 57 | } 58 | for (auto mem : m_muxes) { 59 | mem->std::mutex::~mutex(); 60 | } 61 | for (auto mem : m_mux_mallocs) { 62 | delete[] mem; 63 | } 64 | } 65 | 66 | void addWorkers(size_t n) { 67 | std::unique_lock growing_pool(m_fifo_access); 68 | const auto from = m_thread_workers.size(); 69 | const auto muxes = new std::mutex[n]; 70 | m_mux_mallocs.push_back(muxes); 71 | for (size_t i = 0; i < n; i++) { 72 | m_available.push_back(from + i); 73 | m_muxes.push_back(&muxes[i]); 74 | } 75 | // static_cast needed for some versions of C++ 76 | std::transform( 77 | muxes, muxes + n, std::back_inserter(m_thread_workers), 78 | static_cast(std::addressof)); 79 | unlock(); 80 | } 81 | 82 | int acquireWorker() { 83 | auto fifo_lock = lock(); 84 | const auto res = m_available.front(); 85 | m_available.pop_front(); 86 | if (!m_available.empty()) { 87 | unlock(); 88 | } 89 | return res; 90 | } 91 | 92 | void releaseWorker(int worker) { 93 | std::unique_lock growing_pool(m_fifo_access); 94 | m_available.push_front(worker); 95 | unlock(); 96 | } 97 | }; 98 | 99 | } // namespace mc 100 | } // namespace douban 101 | -------------------------------------------------------------------------------- /include/hashkit/md5.h: -------------------------------------------------------------------------------- 1 | /** 2 | * \file md5.h 3 | * 4 | * \brief MD5 message digest algorithm (hash function) 5 | * 6 | * Copyright (C) 2006-2013, Brainspark B.V. 7 | * 8 | * This file is part of PolarSSL (http://www.polarssl.org) 9 | * Lead Maintainer: Paul Bakker 10 | * 11 | * All rights reserved. 12 | * 13 | * This program is free software; you can redistribute it and/or modify 14 | * it under the terms of the GNU General Public License as published by 15 | * the Free Software Foundation; either version 2 of the License, or 16 | * (at your option) any later version. 17 | * 18 | * This program is distributed in the hope that it will be useful, 19 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 20 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 21 | * GNU General Public License for more details. 22 | * 23 | * You should have received a copy of the GNU General Public License along 24 | * with this program; if not, write to the Free Software Foundation, Inc., 25 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 26 | */ 27 | #ifndef POLARSSL_MD5_H 28 | #define POLARSSL_MD5_H 29 | 30 | #include 31 | #include 32 | 33 | namespace douban { 34 | namespace mc { 35 | namespace hashkit { 36 | 37 | /** 38 | * \brief MD5 context structure 39 | */ 40 | typedef struct 41 | { 42 | uint32_t total[2]; /*!< number of bytes processed */ 43 | uint32_t state[4]; /*!< intermediate digest state */ 44 | unsigned char buffer[64]; /*!< data block being processed */ 45 | 46 | unsigned char ipad[64]; /*!< HMAC: inner padding */ 47 | unsigned char opad[64]; /*!< HMAC: outer padding */ 48 | } 49 | md5_context; 50 | 51 | /** 52 | * \brief MD5 context setup 53 | * 54 | * \param ctx context to be initialized 55 | */ 56 | void md5_starts( md5_context *ctx ); 57 | 58 | /** 59 | * \brief MD5 process buffer 60 | * 61 | * \param ctx MD5 context 62 | * \param input buffer holding the data 63 | * \param ilen length of the input data 64 | */ 65 | void md5_update( md5_context *ctx, const unsigned char *input, size_t ilen ); 66 | 67 | /** 68 | * \brief MD5 final digest 69 | * 70 | * \param ctx MD5 context 71 | * \param output MD5 checksum result 72 | */ 73 | void md5_finish( md5_context *ctx, unsigned char output[16] ); 74 | 75 | /* Internal use */ 76 | void md5_process( md5_context *ctx, const unsigned char data[64] ); 77 | 78 | /** 79 | * \brief Output = MD5( input buffer ) 80 | * 81 | * \param input buffer holding the data 82 | * \param ilen length of the input data 83 | * \param output MD5 checksum result 84 | */ 85 | void md5( const unsigned char *input, size_t ilen, unsigned char output[16] ); 86 | 87 | } // namespace hashkit 88 | } // namespace mc 89 | } // namespace douban 90 | 91 | #endif /* md5.h */ 92 | -------------------------------------------------------------------------------- /include/ConnectionPool.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include "Common.h" 5 | #include "Connection.h" 6 | #include "hashkit/ketama.h" 7 | 8 | namespace douban { 9 | namespace mc { 10 | 11 | class ConnectionPool { 12 | public: 13 | ConnectionPool(); 14 | ~ConnectionPool(); 15 | void setHashFunction(hash_function_options_t fn_opt); 16 | int init(const char* const * hosts, const uint32_t* ports, const size_t n, 17 | const char* const * aliases = NULL); 18 | int updateServers(const char* const * hosts, const uint32_t* ports, const size_t n, 19 | const char* const * aliases = NULL); 20 | const char* getServerAddressByKey(const char* key, const size_t keyLen); 21 | const char* getRealtimeServerAddressByKey(const char* key, const size_t keyLen); 22 | void enableConsistentFailover(); 23 | void disableConsistentFailover(); 24 | void dispatchRetrieval(op_code_t op, const char* const* keys, const size_t* keyLens, 25 | size_t nKeys); 26 | void dispatchStorage(op_code_t op, 27 | const char* const* keys, const size_t* keyLens, 28 | const flags_t* flags, const exptime_t exptime, 29 | const cas_unique_t* cas_uniques, const bool noreply, 30 | const char* const* vals, const size_t* valLens, 31 | size_t nItems); 32 | void dispatchDeletion(const char* const* keys, const size_t* keyLens, 33 | const bool noreply, size_t nItems); 34 | void dispatchTouch(const char* const* keys, const size_t* keyLens, 35 | const exptime_t exptime, const bool noreply, size_t nItems); 36 | void dispatchIncrDecr(op_code_t op, const char* key, const size_t keyLen, 37 | const uint64_t delta, const bool noreply); 38 | void broadcastCommand(const char * const cmd, const size_t cmdLen, const bool noreply=false); 39 | 40 | err_code_t waitPoll(); 41 | 42 | void collectRetrievalResult(std::vector& results); 43 | void collectMessageResult(std::vector& results); 44 | void collectBroadcastResult(std::vector& results, bool isFlushAll=false); 45 | void collectUnsignedResult(std::vector& results); 46 | void reset(); 47 | void setPollTimeout(int timeout); 48 | void setConnectTimeout(int timeout); 49 | void setRetryTimeout(int timeout); 50 | void setMaxRetries(int max_retries); 51 | 52 | protected: 53 | void markDeadAll(pollfd_t* pollfds, const char* reason); 54 | void markDeadConn(Connection* conn, const char* reason, pollfd_t* fd_ptr); 55 | void rewindConn(Connection* conn, pollfd_t* fd_ptr); 56 | 57 | uint32_t m_nActiveConn; // wait for poll 58 | uint32_t m_nInvalidKey; 59 | std::vector m_activeConns; 60 | hashkit::KetamaSelector m_connSelector; 61 | Connection *m_conns; 62 | size_t m_nConns; 63 | int m_pollTimeout; 64 | }; 65 | 66 | } // namespace mc 67 | } // namespace douban 68 | -------------------------------------------------------------------------------- /tests/test_client_pool.cpp: -------------------------------------------------------------------------------- 1 | #include "ClientPool.h" 2 | #include "test_common.h" 3 | 4 | #include 5 | #include "gtest/gtest.h" 6 | 7 | using douban::mc::ClientPool; 8 | using douban::mc::tests::gen_random; 9 | 10 | const unsigned int n_ops = 5; 11 | const unsigned int data_size = 10; 12 | const unsigned int n_servers = 20; 13 | const unsigned int start_port = 21211; 14 | const char host[] = "127.0.0.1"; 15 | unsigned int n_threads = 8; 16 | 17 | void inner_test_loop(ClientPool* pool) { 18 | retrieval_result_t **r_results = NULL; 19 | message_result_t **m_results = NULL; 20 | size_t nResults = 0; 21 | flags_t flags[] = {}; 22 | size_t data_lens[] = {data_size}; 23 | exptime_t exptime = 0; 24 | char key[data_size + 1]; 25 | char value[data_size + 1]; 26 | const char* keys = &key[0]; 27 | const char* values = &value[0]; 28 | 29 | for (unsigned int j = 0; j < n_ops; j++) { 30 | gen_random(key, data_size); 31 | gen_random(value, data_size); 32 | auto c = pool->acquire(); 33 | c->set(&keys, data_lens, flags, exptime, NULL, 0, &values, data_lens, 1, &m_results, &nResults); 34 | c->destroyMessageResult(); 35 | c->get(&keys, data_lens, 1, &r_results, &nResults); 36 | EXPECT_EQ(nResults, 1); 37 | ASSERT_N_STREQ(r_results[0]->data_block, values, data_size); 38 | c->destroyRetrievalResult(); 39 | pool->release(c); 40 | } 41 | } 42 | 43 | bool check_availability(ClientPool* pool) { 44 | auto c = pool->acquire(); 45 | broadcast_result_t* results; 46 | size_t nHosts; 47 | int ret = c->version(&results, &nHosts); 48 | c->destroyBroadcastResult(); 49 | pool->release(c); 50 | return ret == 0; 51 | } 52 | 53 | TEST(test_client_pool, simple_set_get) { 54 | uint32_t ports[n_servers]; 55 | const char* hosts[n_servers]; 56 | for (unsigned int i = 0; i < n_servers; i++) { 57 | ports[i] = start_port + i; 58 | hosts[i] = host; 59 | } 60 | 61 | ClientPool* pool = new ClientPool(); 62 | pool->config(CFG_HASH_FUNCTION, OPT_HASH_FNV1A_32); 63 | pool->init(hosts, ports, n_servers); 64 | ASSERT_TRUE(check_availability(pool)); 65 | 66 | for (unsigned int j = 0; j < n_threads; j++) { 67 | inner_test_loop(pool); 68 | } 69 | 70 | delete pool; 71 | } 72 | 73 | TEST(test_client_pool, threaded_set_get) { 74 | uint32_t ports[n_servers]; 75 | const char* hosts[n_servers]; 76 | for (unsigned int i = 0; i < n_servers; i++) { 77 | ports[i] = start_port + i; 78 | hosts[i] = host; 79 | } 80 | 81 | std::thread* threads = new std::thread[n_threads]; 82 | ClientPool* pool = new ClientPool(); 83 | pool->config(CFG_HASH_FUNCTION, OPT_HASH_FNV1A_32); 84 | //pool->config(CFG_INITIAL_CLIENTS, 4); 85 | pool->init(hosts, ports, n_servers); 86 | ASSERT_TRUE(check_availability(pool)); 87 | 88 | for (unsigned int i = 0; i < n_threads; i++) { 89 | threads[i] = std::thread([&pool] { inner_test_loop(pool); }); 90 | } 91 | for (unsigned int i = 0; i < n_threads; i++) { 92 | threads[i].join(); 93 | } 94 | delete[] threads; 95 | delete pool; 96 | } 97 | -------------------------------------------------------------------------------- /misc/versioning.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # from: https://gist.githubusercontent.com/pkrusche/7369262/raw/5bf2dc8afb88d3fdde7be6d16ee4290db6735f37/versioning.py 3 | 4 | """ Git Versioning Script 5 | 6 | Will transform stdin to expand some keywords with git version/author/date information. 7 | 8 | Specify --clean to remove this information before commit. 9 | 10 | Setup: 11 | 12 | 1. Copy versioning.py into your git repository 13 | 14 | 2. Run: 15 | 16 | git config filter.versioning.smudge 'python versioning.py' 17 | git config filter.versioning.clean 'python versioning.py --clean' 18 | echo 'version.py filter=versioning' >> .gitattributes 19 | git add versioning.py 20 | 21 | 22 | 3. add a version.py file with this contents: 23 | 24 | __version__ = "" 25 | __author__ = "" 26 | __email__ = "" 27 | __date__ = "" 28 | 29 | or 30 | 31 | _Version = "" 32 | _Author = "" 33 | _Email = "" 34 | _Date = "" 35 | """ 36 | 37 | import sys 38 | import subprocess 39 | import re 40 | 41 | 42 | def main(): 43 | clean = False 44 | if len(sys.argv) > 1: 45 | if sys.argv[1] == '--clean': 46 | clean = True 47 | 48 | # initialise empty here. Otherwise: forkbomb through the git calls. 49 | subst_list = { 50 | "version": "", 51 | "date": "", 52 | # "author": "", 53 | # "email": "" 54 | } 55 | 56 | for line in sys.stdin: 57 | if not clean: 58 | subst_list = { 59 | # '--dirty' could be added to the following, too, 60 | # but is not supported everywhere 61 | "version": subprocess.check_output([ 62 | 'git', 'describe', '--always', '--tags' 63 | ]), 64 | "date": subprocess.check_output([ 65 | 'git', 'log', '--pretty=format:"%ad"', '-1' 66 | ]), 67 | # "author": subprocess.check_output([ 68 | # 'git', 'log', '--pretty=format:"%an"', '-1' 69 | # ]), 70 | # "email": subprocess.check_output([ 71 | # 'git', 'log', '--pretty=format:"%ae"', '-1' 72 | # ]) 73 | } 74 | for k, v in subst_list.iteritems(): 75 | v = re.sub(r'[\n\r\t"\']', "", v) 76 | rexp = "__%s__\s*=[\s'\"]+" % k 77 | line = re.sub(rexp, "__%s__ = \"%s\"\n" % (k, v), line) 78 | rexp = "_%s\s*=[\s'\"]+" % k.capitalize() 79 | line = re.sub( 80 | rexp, "_%s = \"%s\"\n" % (k.capitalize(), v), line 81 | ) 82 | sys.stdout.write(line) 83 | else: 84 | for k in subst_list: 85 | rexp = "__%s__\s*=.*" % k 86 | line = re.sub(rexp, "__%s__ = \"\"" % k, line) 87 | rexp = "_%s\s*=.*" % k.capitalize() 88 | line = re.sub(rexp, "_%s = \"\"" % k.capitalize(), line) 89 | sys.stdout.write(line) 90 | 91 | 92 | if __name__ == "__main__": 93 | main() 94 | -------------------------------------------------------------------------------- /CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8.2) 2 | project(libmc) 3 | set(CMAKE_MACOSX_RPATH 1) 4 | 5 | set (MC_VERSION_MAJOR 1) 6 | set (MC_VERSION_MINOR 4) 7 | set (MC_VERSION_PATCH 5) 8 | 9 | set (MC_VERSION ${MC_VERSION_MAJOR}.${MC_VERSION_MINOR}) 10 | set (MC_APIVERSION ${MC_VERSION}.${MC_VERSION_PATCH}) 11 | 12 | add_definitions(-DMC_USE_SMALL_VECTOR) 13 | 14 | if (NOT CMAKE_BUILD_TYPE) 15 | set(CMAKE_BUILD_TYPE "Release" CACHE STRING "Choose the type of build, options are: Debug Release." FORCE) 16 | endif (NOT CMAKE_BUILD_TYPE) 17 | 18 | set(CMAKE_CXX_FLAGS_COMMON "-Wall -fno-rtti -fno-exceptions -std=c++17") 19 | set(CMAKE_CXX_FLAGS_DEBUG "-DDEBUG -g2 ${CMAKE_CXX_FLAGS_COMMON}" CACHE STRING "CXX DEBUG FLAGS" FORCE) 20 | set(CMAKE_CXX_FLAGS_RELEASE "-DNDEBUG -O3 ${CMAKE_CXX_FLAGS_COMMON}" CACHE STRING "CXX RELEASE FLAGS" FORCE) 21 | set(CMAKE_INSTALL_INCLUDE include CACHE PATH "Output directory for header files") 22 | set(CMAKE_INSTALL_LIBDIR lib CACHE PATH "Output directory for libraries") 23 | 24 | include_directories(include) 25 | 26 | FILE( 27 | GLOB mc_HEADERS 28 | include/*.h 29 | include/hashkit/*.h 30 | include/llvm/*.h 31 | include/rapidjson/*.h 32 | ) 33 | FILE( 34 | GLOB mc_SOURCES 35 | src/*.cpp 36 | ) 37 | add_library(mc STATIC ${mc_SOURCES}) 38 | 39 | install( 40 | FILES include/c_client.h 41 | DESTINATION ${CMAKE_INSTALL_INCLUDE}/${CMAKE_PROJECT} 42 | ) 43 | install( 44 | TARGETS mc 45 | DESTINATION ${CMAKE_INSTALL_LIBDIR} 46 | EXPORT mc 47 | ) 48 | 49 | option(WITH_TESTING "Enable testing" OFF) 50 | if (WITH_TESTING) 51 | # Download and unpack googletest at configure time 52 | configure_file(ext/gtest/CMakeLists.txt.in ext/gtest/CMakeLists.txt) 53 | execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" . 54 | RESULT_VARIABLE result 55 | WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/ext/gtest ) 56 | if(result) 57 | message(FATAL_ERROR "CMake step for googletest failed: ${result}") 58 | endif() 59 | execute_process(COMMAND ${CMAKE_COMMAND} --build . 60 | RESULT_VARIABLE result 61 | WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/ext/gtest ) 62 | if(result) 63 | message(FATAL_ERROR "Build step for googletest failed: ${result}") 64 | endif() 65 | 66 | # Prevent overriding the parent project's compiler/linker 67 | # settings on Windows 68 | set(gtest_force_shared_crt ON CACHE BOOL "" FORCE) 69 | 70 | # Add googletest directly to our build. This defines 71 | # the gtest and gtest_main targets. 72 | add_subdirectory(${CMAKE_BINARY_DIR}/googletest-src 73 | ${CMAKE_BINARY_DIR}/googletest-build 74 | EXCLUDE_FROM_ALL) 75 | 76 | # The gtest/gtest_main targets carry header search path 77 | # dependencies automatically when using CMake 2.8.11 or 78 | # later. Otherwise we have to add them here ourselves. 79 | if (CMAKE_VERSION VERSION_LESS 2.8.11) 80 | include_directories("${gtest_SOURCE_DIR}/include") 81 | endif() 82 | 83 | # Now simply link against gtest or gtest_main as needed. Eg 84 | enable_testing() 85 | add_subdirectory(tests) 86 | endif (WITH_TESTING) 87 | -------------------------------------------------------------------------------- /include/Connection.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | 7 | #include 8 | 9 | #include "Common.h" 10 | #include "BufferReader.h" 11 | #include "BufferWriter.h" 12 | #include "Parser.h" 13 | #include "Result.h" 14 | 15 | 16 | namespace douban { 17 | namespace mc { 18 | 19 | class Connection { 20 | 21 | public: 22 | Connection(); 23 | ~Connection(); 24 | int init(const char* host, uint32_t port, const char* alias = NULL); 25 | int connect(); 26 | void close(); 27 | const bool alive(); 28 | bool tryReconnect(bool check_retries = true); 29 | void markDead(const char* reason, int delay = 0); 30 | int socketFd() const; 31 | 32 | const char* name(); 33 | const char* host(); 34 | const uint32_t port(); 35 | const bool hasAlias(); 36 | const bool isSent(); 37 | 38 | void takeBuffer(const char* const buf, size_t buf_len); 39 | void addRequestKey(const char* const key, const size_t len); 40 | size_t requestKeyCount(); 41 | void setParserMode(ParserMode md); 42 | void takeNumber(int64_t val); 43 | ssize_t send(); 44 | ssize_t recv(bool peek = false); 45 | void process(err_code_t& err); 46 | types::RetrievalResultList* getRetrievalResults(); 47 | types::MessageResultList* getMessageResults(); 48 | types::LineResultList* getLineResults(); 49 | types::UnsignedResultList* getUnsignedResults(); 50 | 51 | std::vector* getRequestKeys(); 52 | 53 | 54 | void reset(); 55 | void rewind(); 56 | void setRetryTimeout(int timeout); 57 | const int getRetryTimeout(); 58 | void setConnectTimeout(int timeout); 59 | void setMaxRetries(int max_retries); 60 | 61 | size_t m_counter; 62 | 63 | protected: 64 | int connectPoll(int fd, const sockaddr* ai_ptr, const socklen_t ai_addrlen); 65 | int unixSocketConnect(); 66 | 67 | char m_name[MC_NI_MAXHOST + 1 + MC_NI_MAXSERV]; 68 | char m_host[MC_NI_MAXHOST]; 69 | uint32_t m_port; 70 | 71 | int m_socketFd; 72 | bool m_alive; 73 | bool m_hasAlias; 74 | bool m_unixSocket; 75 | time_t m_deadUntil; 76 | io::BufferWriter* m_buffer_writer; // for send 77 | io::BufferReader* m_buffer_reader; // for recv 78 | PacketParser m_parser; 79 | 80 | int m_connectTimeout; 81 | int m_retryTimeout; 82 | 83 | int m_maxRetries; // max reconnect tries during one command 84 | int m_retires; 85 | 86 | private: 87 | Connection(const Connection& conn); 88 | }; 89 | 90 | 91 | inline const bool Connection::alive() { 92 | return m_alive; 93 | } 94 | 95 | inline const char* Connection::name() { 96 | return m_name; 97 | } 98 | 99 | inline const char* Connection::host() { 100 | return m_host; 101 | } 102 | 103 | inline const uint32_t Connection::port() { 104 | return m_port; 105 | } 106 | 107 | inline const bool Connection::hasAlias() { 108 | return m_hasAlias; 109 | } 110 | 111 | inline const bool Connection::isSent() { 112 | return m_buffer_writer->isRead(); 113 | } 114 | 115 | inline const int Connection::getRetryTimeout() { 116 | return m_retryTimeout; 117 | } 118 | 119 | 120 | } // namespace mc 121 | } // namespace douban 122 | -------------------------------------------------------------------------------- /include/BufferReader.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #include "Export.h" 10 | #include "Common.h" 11 | #include "DataBlock.h" 12 | 13 | #ifdef MC_USE_SMALL_VECTOR 14 | #include "llvm/SmallVector.h" 15 | #endif 16 | 17 | namespace douban { 18 | namespace mc { 19 | namespace io { 20 | 21 | typedef std::list DataBlockList; 22 | typedef DataBlockList::iterator DataBlockListIterator; 23 | 24 | 25 | struct DataCursor_s { 26 | DataBlockListIterator iterator; 27 | size_t offset; 28 | 29 | bool operator==(const DataCursor_s& other) const { 30 | return (this->iterator == other.iterator && this->offset == other.offset); 31 | } 32 | 33 | bool operator!=(const DataCursor_s& other) const { 34 | return (this->iterator != other.iterator || this->offset != other.offset); 35 | } 36 | }; 37 | typedef struct DataCursor_s DataCursor; 38 | 39 | 40 | typedef struct { 41 | DataBlockListIterator iterator; 42 | size_t offset; 43 | size_t size; 44 | } DataBlockSlice; 45 | 46 | #ifdef MC_USE_SMALL_VECTOR 47 | typedef llvm::SmallVector TokenData; 48 | #else 49 | typedef std::vector TokenData; 50 | #endif 51 | 52 | void freeTokenData(TokenData& td); 53 | char* parseTokenData(TokenData& td, size_t reserved); 54 | void copyTokenData(const TokenData& src, TokenData& dst); 55 | 56 | 57 | class BufferReader { 58 | public: 59 | BufferReader(); 60 | ~BufferReader(); 61 | void reset(); 62 | 63 | size_t prepareWriteBlock(size_t len); 64 | 65 | char* getWritePtr(); 66 | void commitWrite(size_t len); 67 | void write(char* ptr, size_t len, bool copying = true); 68 | 69 | size_t capacity(); 70 | size_t size(); 71 | size_t readLeft(); 72 | size_t nDataBlock(); 73 | size_t nBytesRef(); 74 | 75 | const char peek(err_code_t& err, size_t offset) const; 76 | 77 | size_t readUntil(err_code_t& err, char value, TokenData& tokenData); 78 | size_t skipUntil(err_code_t& err, char value); 79 | void readUnsigned(err_code_t& err, uint64_t& value); 80 | void readBytes(err_code_t& err, size_t len, TokenData& tokenData); 81 | void expectBytes(err_code_t& err, const char* str, size_t str_size); 82 | void skipBytes(err_code_t& err, size_t str_size); 83 | void setNextPreferedDataBlockSize(size_t n); 84 | size_t getNextPreferedDataBlockSize(); 85 | 86 | protected: 87 | const char charAtCursor(DataCursor& cur) const; 88 | 89 | DataBlockList m_dataBlockList; 90 | size_t m_capacity; 91 | size_t m_size; 92 | size_t m_readLeft; 93 | DataCursor m_blockReadCursor; 94 | DataBlockListIterator m_blockWriteIterator; 95 | size_t m_nextPreferedDataBlockSize; 96 | }; 97 | 98 | 99 | inline const char BufferReader::charAtCursor(DataCursor& cur) const { 100 | // NOTE: make sure cur is valid 101 | /** 102 | * DataBlock& db = *cur.iterator; 103 | * char* chrPtr = db[cur.offset]; 104 | * return *chrPtr; 105 | **/ 106 | return *((*cur.iterator)[cur.offset]); 107 | } 108 | 109 | 110 | inline size_t BufferReader::readLeft() { 111 | return m_readLeft; 112 | } 113 | 114 | } // namespace io 115 | } // namespace mc 116 | } // namespace douban 117 | -------------------------------------------------------------------------------- /include/Client.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include "Export.h" 5 | #include "Result.h" 6 | #include "ConnectionPool.h" 7 | 8 | 9 | namespace douban { 10 | namespace mc { 11 | 12 | class Client : public ConnectionPool { 13 | public: 14 | Client(); 15 | ~Client(); 16 | void config(config_options_t opt, int val); 17 | // retrieval commands 18 | void destroyRetrievalResult(); 19 | 20 | #define DECL_RETRIEVAL_CMD(M) \ 21 | err_code_t M(const char* const* keys, const size_t* keyLens, size_t nKeys, \ 22 | retrieval_result_t*** results, size_t* nResults); 23 | DECL_RETRIEVAL_CMD(get) 24 | DECL_RETRIEVAL_CMD(gets) 25 | #undef DECL_RETRIEVAL_CMD 26 | 27 | // storage commands 28 | void destroyMessageResult(); 29 | #define DECL_STORAGE_CMD(M) \ 30 | err_code_t M(const char* const* keys, const size_t* keyLens, \ 31 | const flags_t* flags, const exptime_t exptime, \ 32 | const cas_unique_t* cas_uniques, const bool noreply, \ 33 | const char* const* vals, const size_t* valLens, \ 34 | size_t nItems, message_result_t*** results, size_t* nResults) 35 | 36 | DECL_STORAGE_CMD(set); 37 | DECL_STORAGE_CMD(add); 38 | DECL_STORAGE_CMD(replace); 39 | DECL_STORAGE_CMD(append); 40 | DECL_STORAGE_CMD(prepend); 41 | DECL_STORAGE_CMD(cas); 42 | #undef DECL_STORAGE_CMD 43 | err_code_t _delete(const char* const* keys, const size_t* keyLens, 44 | const bool noreply, size_t nItems, 45 | message_result_t*** results, size_t* nResults); 46 | 47 | // broadcast commands 48 | void destroyBroadcastResult(); 49 | 50 | err_code_t version(broadcast_result_t** results, size_t* nHosts); 51 | err_code_t quit(); 52 | err_code_t stats(broadcast_result_t** results, size_t* nHosts); 53 | err_code_t flushAll(broadcast_result_t** results, size_t* nHosts); 54 | 55 | // touch 56 | err_code_t touch(const char* const* keys, const size_t* keyLens, 57 | const exptime_t exptime, const bool noreply, size_t nItems, 58 | message_result_t*** results, size_t* nResults); 59 | 60 | // incr / decr 61 | void destroyUnsignedResult(); 62 | err_code_t incr(const char* key, const size_t keyLen, const uint64_t delta, 63 | const bool noreply, 64 | unsigned_result_t** result, size_t* nResults); 65 | err_code_t decr(const char* key, const size_t keyLen, const uint64_t delta, 66 | const bool noreply, 67 | unsigned_result_t** result, size_t* nResults); 68 | 69 | inline void toggleFlushAllFeature(bool enabled) { 70 | m_flushAllEnabled = enabled; 71 | } 72 | void _sleep(uint32_t seconds); // check GIL in Python 73 | 74 | protected: 75 | void collectRetrievalResult(retrieval_result_t*** results, size_t* nResults); 76 | void collectMessageResult(message_result_t*** results, size_t* nResults); 77 | void collectBroadcastResult(broadcast_result_t** results, size_t* nHosts, bool isFlushAll=false); 78 | void collectUnsignedResult(unsigned_result_t** results, size_t* nResults); 79 | 80 | std::vector m_outRetrievalResultPtrs; 81 | std::vector m_outMessageResultPtrs; 82 | std::vector m_outBroadcastResultPtrs; 83 | std::vector m_outUnsignedResultPtrs; 84 | 85 | bool m_flushAllEnabled; 86 | }; 87 | 88 | } // namespace mc 89 | } // namespace douban 90 | -------------------------------------------------------------------------------- /include/c_client.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | #include "Export.h" 7 | 8 | #ifdef __cplusplus 9 | extern "C" { 10 | #endif 11 | 12 | void* client_create(); 13 | void client_init(void* client, const char* const * hosts, const uint32_t* ports, 14 | size_t n, const char* const * aliases, const int failover); 15 | void client_config(void* client, config_options_t opt, int val); 16 | void client_destroy(void* client); 17 | 18 | const char* client_get_server_address_by_key(void* client, const char* key, const size_t key_len); 19 | const char* client_get_realtime_server_address_by_key(void* client, const char* key, 20 | const size_t key_len); 21 | 22 | err_code_t client_version(void* client, broadcast_result_t** results, size_t* n_hosts); 23 | void client_destroy_broadcast_result(void* client); 24 | 25 | #define DECL_RETRIEVAL_CMD(M) \ 26 | err_code_t client_##M(void* client, const char* const* keys, const size_t* key_lens, \ 27 | size_t nKeys, retrieval_result_t*** results, size_t* n_results) 28 | DECL_RETRIEVAL_CMD(get); 29 | DECL_RETRIEVAL_CMD(gets); 30 | #undef DECL_RETRIEVAL_CMD 31 | 32 | void client_destroy_retrieval_result(void* client); 33 | 34 | #define DECL_STORAGE_CMD(M) \ 35 | err_code_t client_##M(void* client, const char* const* keys, const size_t* key_lens, \ 36 | const flags_t* flags, const exptime_t exptime, \ 37 | const cas_unique_t* cas_uniques, const bool noreply, \ 38 | const char* const* vals, const size_t* val_lens, \ 39 | size_t nItems, message_result_t*** results, size_t* n_results) 40 | DECL_STORAGE_CMD(set); 41 | DECL_STORAGE_CMD(add); 42 | DECL_STORAGE_CMD(replace); 43 | DECL_STORAGE_CMD(append); 44 | DECL_STORAGE_CMD(prepend); 45 | DECL_STORAGE_CMD(cas); 46 | #undef DECL_STORAGE_CMD 47 | 48 | err_code_t client_touch(void* client, const char* const* keys, const size_t* key_lens, 49 | const exptime_t exptime, const bool noreply, size_t n_items, 50 | message_result_t*** results, size_t* n_results); 51 | void client_destroy_message_result(void* client); 52 | 53 | err_code_t client_delete(void*client, const char* const* keys, const size_t* key_lens, 54 | const bool noreply, size_t n_items, 55 | message_result_t*** results, size_t* n_results); 56 | 57 | err_code_t client_incr(void* client, const char* key, const size_t keyLen, 58 | const uint64_t delta, const bool noreply, 59 | unsigned_result_t** results, size_t* n_results); 60 | err_code_t client_decr(void* client, const char* key, const size_t keyLen, 61 | const uint64_t delta, const bool noreply, 62 | unsigned_result_t** results, size_t* n_results); 63 | void client_destroy_unsigned_result(void* client); 64 | 65 | err_code_t client_stats(void* client, broadcast_result_t** results, size_t* n_servers); 66 | void client_toggle_flush_all_feature(void* client, bool enabled); 67 | err_code_t client_flush_all(void* client, broadcast_result_t** results, size_t* n_servers); 68 | err_code_t client_quit(void* client); 69 | 70 | const char* err_code_to_string(err_code_t err); 71 | server_string_split_t splitServerString(char* input); 72 | #ifdef __cplusplus 73 | } 74 | #endif 75 | -------------------------------------------------------------------------------- /tests/test_client_pool.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | import unittest 3 | import threading 4 | import functools 5 | import os 6 | from libmc import ClientPool, ThreadedClient, MC_MAX_CLIENTS 7 | 8 | def setup_loging(f): 9 | g = None 10 | 11 | @functools.wraps(f) 12 | def wrapper(*args, **kwargs): 13 | return g(*args, **kwargs) 14 | 15 | @functools.wraps(f) 16 | def begin(*args, **kwargs): 17 | nonlocal g 18 | with open("/tmp/debug.log", "w+") as fp: 19 | fp.write("") 20 | g = f 21 | return wrapper(*args, **kwargs) 22 | 23 | g = begin 24 | return wrapper 25 | 26 | @functools.wraps(print) 27 | @setup_loging 28 | def threaded_print(*args, **kwargs): 29 | with open('/tmp/debug.log', 'a+') as fp: 30 | print(*args, **kwargs, file=fp) 31 | 32 | class ClientOps: 33 | nthreads=8 34 | ops = 100 35 | 36 | def tid(self, mc): 37 | return (os.getpid(), threading.current_thread().native_id) 38 | 39 | def client_misc(self, mc, i=0): 40 | tid = self.tid(mc) + (i,) 41 | tid = "_".join(map(str, tid)) 42 | f, t = 'foo_' + tid, 'tuiche_' + tid 43 | mc.get_multi([f, t]) 44 | mc.delete(f) 45 | mc.delete(t) 46 | assert mc.get(f) is None 47 | assert mc.get(t) is None 48 | 49 | mc.set(f, 'biu') 50 | mc.set(t, 'bb') 51 | assert mc.get(f) == 'biu' 52 | assert mc.get(t) == 'bb' 53 | assert (mc.get_multi([f, t]) == 54 | {f: 'biu', t: 'bb'}) 55 | mc.set_multi({f: 1024, t: '8964'}) 56 | assert (mc.get_multi([f, t]) == 57 | {f: 1024, t: '8964'}) 58 | 59 | def client_threads(self, target): 60 | errs = [] 61 | def passthrough(args): 62 | _, e, tb, t = args 63 | if hasattr(e, "add_note"): 64 | e.add_note("Occurred in thread " + str(t)) 65 | errs.append(e.with_traceback(tb)) 66 | 67 | threading.excepthook = passthrough 68 | ts = [threading.Thread(target=target) for i in range(self.nthreads)] 69 | for t in ts: 70 | t.start() 71 | 72 | for t in ts: 73 | t.join() 74 | 75 | if errs: 76 | e = errs[0] 77 | if hasattr(e, "add_note"): 78 | e.add_note(f"Along with {len(errs)} errors in other threads") 79 | raise e 80 | 81 | class ThreadedSingleServerCase(unittest.TestCase, ClientOps): 82 | def setUp(self): 83 | self.pool = ClientPool(["127.0.0.1:21211"]) 84 | 85 | def misc(self): 86 | for i in range(self.ops): 87 | self.test_pool_client_misc(i) 88 | 89 | def test_pool_client_misc(self, i=0): 90 | with self.pool.client() as mc: 91 | self.client_misc(mc, i) 92 | 93 | def test_acquire(self): 94 | with self.pool.client() as mc: 95 | pass 96 | 97 | def test_pool_client_threaded(self): 98 | self.client_threads(self.misc) 99 | 100 | class ThreadedClientOps(ClientOps): 101 | def misc(self): 102 | for i in range(self.ops): 103 | self.client_misc(self.imp, i) 104 | 105 | 106 | class ThreadedClientWrapperCheck(unittest.TestCase, ThreadedClientOps): 107 | def setUp(self): 108 | self.imp = ThreadedClient(["127.0.0.1:21211"]) 109 | 110 | def test_many_threads(self): 111 | self.client_threads(self.misc) 112 | 113 | -------------------------------------------------------------------------------- /misc/memcached_server: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # based on http://serverfault.com/a/360230/105615 3 | 4 | set -e 5 | 6 | PORTS="21211 21212 21213 21214 21215 21216 21217 21218 21219 21220 21221 21222 21223 21224 21225 21226 21227 21228 21229 21230" 7 | USER=memcached 8 | basedir="/tmp/env_mc_dev" 9 | mkdir -p "$basedir/var/log" 10 | mkdir -p "$basedir/var/run" 11 | 12 | RETVAL=0 13 | prog="memcached" 14 | cmd="`which memcached`" 15 | ip="127.0.0.1" 16 | memory=64 17 | threads=4 18 | 19 | 20 | function start() 21 | { 22 | port="$1" 23 | if [ `ps -ef | grep "$cmd" | grep -c $port` -ge 1 ]; then 24 | echo "Starting the memcached server on port '$port'... " 25 | else 26 | if [ ! -f $basedir/var/log/${port}.log ]; then 27 | mkdir -p $basedir/var/log 28 | touch $basedir/var/log/${port}.log 29 | fi 30 | $cmd -d -u $USER -l $ip -t $threads -m ${memory} -p $port -P $basedir/var/run/${port}.pid > $basedir/var/log/${port}.log 2>&1 31 | echo "Starting the memcached server on port '$port'... " 32 | fi 33 | } 34 | 35 | function unix() 36 | { 37 | name="${1:-unix_test}" 38 | if [ ! -f $basedir/var/log/${name}.log ]; then 39 | mkdir -p $basedir/var/log 40 | touch $basedir/var/log/${name}.log 41 | fi 42 | mkdir -p $basedir/var/run 43 | $cmd -d -u $USER -s $basedir/var/run/${name}.socket -t $threads -m ${memory} -P $basedir/var/run/${name}.pid > $basedir/var/log/${name}.log 2>&1 44 | echo "Starting the memcached server on '$basedir/var/run/${name}.socket'... " 45 | } 46 | 47 | function stop() 48 | { 49 | port="$1" 50 | if [ `ps -ef | grep "$cmd" | grep -c $port` -eq 0 ]; then 51 | echo $"Stopped the memcached server on port '$port'... " 52 | else 53 | kill -TERM `ps -ef | grep "$cmd" | grep $port | grep -v grep | awk '{ print $2 }'` 54 | echo "Stopping the memcached server on port '$port'... " 55 | fi 56 | } 57 | 58 | case "$1" in 59 | start) 60 | if [ -n "$2" ]; then 61 | start $2 62 | else 63 | for port in $PORTS; do 64 | start $port & 65 | done 66 | wait 67 | fi 68 | ;; 69 | stop) 70 | if [ -n "$2" ]; then 71 | port="$2" 72 | stop $port 73 | else 74 | for port in $PORTS; do 75 | stop $port & 76 | done 77 | wait 78 | rm -rf $basedir 79 | fi 80 | ;; 81 | restart) 82 | if [ -n "$2" ]; then 83 | stop $2 84 | start $2 85 | else 86 | for port in $PORTS; do 87 | stop $port & 88 | start $port & 89 | done 90 | wait 91 | fi 92 | ;; 93 | unix) 94 | shift 95 | unix $@ 96 | ;; 97 | startall) 98 | unix & 99 | for port in $PORTS; do 100 | start $port & 101 | done 102 | wait 103 | ;; 104 | stopall) 105 | if [ `ls $basedir/var/run/ | grep -c .pid` -ge 1 ]; then 106 | names="`basename $basedir/var/run/*.pid | cut -d. -f1`" 107 | for name in $names; do 108 | stop $name & 109 | done 110 | fi 111 | wait 112 | rm -rf $basedir 113 | ;; 114 | *) 115 | printf 'Usage: %s {start|stop|restart} \n' "$prog" 116 | exit 1 117 | ;; 118 | esac 119 | -------------------------------------------------------------------------------- /src/HashkitCrc.cpp: -------------------------------------------------------------------------------- 1 | #include "hashkit/hashkit.h" 2 | 3 | 4 | namespace douban { 5 | namespace mc { 6 | namespace hashkit { 7 | 8 | static const unsigned int crc32tab[256] = { 9 | 0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 10 | 0x076dc419, 0x706af48f, 0xe963a535, 0x9e6495a3, 11 | 0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, 12 | 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, 13 | 0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de, 14 | 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7, 15 | 0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 16 | 0x14015c4f, 0x63066cd9, 0xfa0f3d63, 0x8d080df5, 17 | 0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, 18 | 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b, 19 | 0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 20 | 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59, 21 | 0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 22 | 0x21b4f4b5, 0x56b3c423, 0xcfba9599, 0xb8bda50f, 23 | 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, 24 | 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d, 25 | 0x76dc4190, 0x01db7106, 0x98d220bc, 0xefd5102a, 26 | 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433, 27 | 0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 28 | 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01, 29 | 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 30 | 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 31 | 0x65b0d9c6, 0x12b7e950, 0x8bbeb8ea, 0xfcb9887c, 32 | 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65, 33 | 0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 34 | 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb, 35 | 0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, 36 | 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, 37 | 0x5005713c, 0x270241aa, 0xbe0b1010, 0xc90c2086, 38 | 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f, 39 | 0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 40 | 0x59b33d17, 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad, 41 | 0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, 42 | 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, 43 | 0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8, 44 | 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1, 45 | 0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 46 | 0xf762575d, 0x806567cb, 0x196c3671, 0x6e6b06e7, 47 | 0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, 48 | 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5, 49 | 0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 50 | 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b, 51 | 0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 52 | 0xdf60efc3, 0xa867df55, 0x316e8eef, 0x4669be79, 53 | 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, 54 | 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 55 | 0xc5ba3bbe, 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 56 | 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d, 57 | 0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 58 | 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713, 59 | 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 60 | 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 61 | 0x86d3d2d4, 0xf1d4e242, 0x68ddb3f8, 0x1fda836e, 62 | 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777, 63 | 0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 64 | 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45, 65 | 0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 66 | 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db, 67 | 0xaed16a4a, 0xd9d65adc, 0x40df0b66, 0x37d83bf0, 68 | 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9, 69 | 0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 70 | 0xbad03605, 0xcdd70693, 0x54de5729, 0x23d967bf, 71 | 0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, 72 | 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d, 73 | }; 74 | 75 | uint32_t hash_crc_32(const char* key, size_t key_length) { 76 | uint32_t crc; 77 | unsigned int i; 78 | 79 | const uint8_t* unsigned_key = reinterpret_cast(key); 80 | crc = ~0; 81 | 82 | for (i = 0; i < key_length; i++) { 83 | crc = (crc >> 8) ^ crc32tab[(crc ^ (unsigned_key[i])) & 0xff]; 84 | } 85 | 86 | return (~crc); 87 | } 88 | 89 | 90 | } // namespace hashkit 91 | } // namespace mc 92 | } // namespace douban 93 | -------------------------------------------------------------------------------- /include/ClientPool.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | 7 | #include "Client.h" 8 | #include "LockPool.h" 9 | 10 | namespace douban { 11 | namespace mc { 12 | 13 | template 14 | void duplicate_strings(const char* const * strs, const size_t n, 15 | std::deque >& out, std::vector& refs) { 16 | out.resize(n); 17 | refs.resize(n); 18 | for (size_t i = 0; i < n; i++) { 19 | if (strs == NULL || strs[i] == NULL) { 20 | out[i][0] = '\0'; 21 | refs[i] = NULL; 22 | continue; 23 | } 24 | std::snprintf(out[i].data(), N, "%s", strs[i]); 25 | refs[i] = out[i].data(); 26 | } 27 | } 28 | 29 | class irange { 30 | int i; 31 | 32 | public: 33 | using value_type = int; 34 | using pointer = const int*; 35 | using reference = const int&; 36 | using difference_type = int; 37 | using iterator_category = std::random_access_iterator_tag; 38 | 39 | explicit irange(int i) : i(i) {} 40 | 41 | reference operator*() const { return i; } 42 | pointer operator->() const { return &i; } 43 | value_type operator[](int n) const { return i + n; } 44 | friend bool operator< (const irange& lhs, const irange& rhs) { return lhs.i < rhs.i; } 45 | friend bool operator> (const irange& lhs, const irange& rhs) { return rhs < lhs; } 46 | friend bool operator<=(const irange& lhs, const irange& rhs) { return !(lhs > rhs); } 47 | friend bool operator>=(const irange& lhs, const irange& rhs) { return !(lhs < rhs); } 48 | friend bool operator==(const irange& lhs, const irange& rhs) { return lhs.i == rhs.i; } 49 | friend bool operator!=(const irange& lhs, const irange& rhs) { return !(lhs == rhs); } 50 | irange& operator++() { ++i; return *this; } 51 | irange& operator--() { --i; return *this; } 52 | irange operator++(int) { irange tmp = *this; ++tmp; return tmp; } 53 | irange operator--(int) { irange tmp = *this; --tmp; return tmp; } 54 | irange& operator+=(difference_type n) { i += n; return *this; } 55 | irange& operator-=(difference_type n) { i -= n; return *this; } 56 | friend irange operator+(const irange& lhs, difference_type n) { irange tmp = lhs; tmp += n; return tmp; } 57 | friend irange operator+(difference_type n, const irange& rhs) { return rhs + n; } 58 | friend irange operator-(const irange& lhs, difference_type n) { irange tmp = lhs; tmp -= n; return tmp; } 59 | friend difference_type operator-(const irange& lhs, const irange& rhs) { return lhs.i - rhs.i; } 60 | }; 61 | 62 | typedef struct { 63 | Client c; 64 | int index; 65 | } IndexedClient; 66 | 67 | class ClientPool : LockPool { 68 | public: 69 | ClientPool(); 70 | ~ClientPool(); 71 | void config(config_options_t opt, int val); 72 | int init(const char* const * hosts, const uint32_t* ports, 73 | const size_t n, const char* const * aliases = NULL); 74 | int updateServers(const char* const * hosts, const uint32_t* ports, 75 | const size_t n, const char* const * aliases = NULL); 76 | IndexedClient* _acquire(); 77 | void _release(const IndexedClient* idx); 78 | Client* acquire(); 79 | void release(const Client* ref); 80 | 81 | private: 82 | int growPool(size_t by); 83 | int setup(Client* c); 84 | inline bool shouldGrowUnsafe(); 85 | int autoGrow(); 86 | 87 | bool m_opt_changed[CLIENT_CONFIG_OPTION_COUNT]; 88 | int m_opt_value[CLIENT_CONFIG_OPTION_COUNT]; 89 | std::deque m_clients; 90 | size_t m_initial_clients; 91 | size_t m_max_clients; 92 | size_t m_max_growth; 93 | 94 | std::deque > m_hosts_data; 95 | std::deque > m_aliases_data; 96 | std::vector m_ports; 97 | 98 | std::vector m_hosts; 99 | std::vector m_aliases; 100 | 101 | std::mutex m_pool_lock; 102 | mutable std::shared_mutex m_acquiring_growth; 103 | }; 104 | 105 | } // namespace mc 106 | } // namespace douban 107 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | import re 4 | import sys 5 | import pkg_resources 6 | import platform 7 | from distutils.sysconfig import get_config_var 8 | from distutils.version import LooseVersion 9 | from glob import glob 10 | from setuptools import setup, Extension 11 | 12 | sources = (glob("src/*.cpp") + ["libmc/_client.pyx"]) 13 | include_dirs = ["include"] 14 | 15 | COMPILER_FLAGS = [ 16 | "-fno-strict-aliasing", 17 | "-fno-exceptions", 18 | "-fno-rtti", 19 | "-Wall", 20 | "-DMC_USE_SMALL_VECTOR", 21 | "-O3", 22 | "-DNDEBUG", 23 | "-std=c++17", 24 | ] 25 | 26 | 27 | def is_installed(requirement): 28 | try: 29 | pkg_resources.require(requirement) 30 | except pkg_resources.ResolutionError: 31 | return False 32 | else: 33 | return True 34 | 35 | 36 | # For mac, ensure extensions are built for macos 10.9 when compiling on a 37 | # 10.9 system or above, overriding distuitls behaviour which is to target 38 | # the version that python was built for. This may be overridden by setting 39 | # MACOSX_DEPLOYMENT_TARGET before calling setup.py 40 | # https://github.com/pandas-dev/pandas/issues/23424#issuecomment-446393981 41 | if sys.platform == 'darwin' and 'MACOSX_DEPLOYMENT_TARGET' not in os.environ: 42 | current_system = LooseVersion(platform.mac_ver()[0]) 43 | python_target = LooseVersion(get_config_var('MACOSX_DEPLOYMENT_TARGET')) 44 | if python_target < LooseVersion('10.9') and current_system >= LooseVersion('10.9'): 45 | os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.9' 46 | 47 | # Resolving Cython dependency via 'setup_requires' requires setuptools >= 18.0: 48 | # https://github.com/pypa/setuptools/commit/a811c089a4362c0dc6c4a84b708953dde9ebdbf8 49 | setuptools_req = "setuptools >= 18.0" 50 | if not is_installed(setuptools_req): 51 | import textwrap 52 | print( 53 | textwrap.dedent( 54 | """ 55 | setuptools >= 18.0 is required, and the dependency cannot be 56 | automatically resolved with the version of setuptools that is 57 | currently installed (%s). 58 | 59 | you can upgrade setuptools: 60 | $ pip install -U setuptools 61 | """ % pkg_resources.get_distribution("setuptools").version), 62 | file=sys.stderr) 63 | exit(1) 64 | 65 | 66 | def find_version(*file_paths): 67 | with open(os.path.join(*file_paths)) as fhandler: 68 | version_file = fhandler.read() 69 | version_match = re.search(r"^__VERSION__ = ['\"]([^'\"]*)['\"]", version_file, re.M) 70 | if version_match: 71 | return version_match.group(1) 72 | raise RuntimeError("Unable to find version string.") 73 | 74 | 75 | setup( 76 | name="libmc", 77 | packages=["libmc"], 78 | version=find_version("libmc", "__init__.py"), 79 | license="BSD License", 80 | description="Fast and light-weight memcached client for C++/Python", 81 | author="PAN, Myautsai", 82 | author_email="myautsai@gmail.com", 83 | url="https://github.com/douban/libmc", 84 | keywords=["memcached", "memcache", "client"], 85 | long_description=open("README.rst").read(), 86 | classifiers=[ 87 | "Programming Language :: C++", 88 | "Programming Language :: Python", 89 | "Programming Language :: Python :: 3", 90 | "Programming Language :: Python :: 3.10", 91 | "Programming Language :: Python :: 3.11", 92 | "Programming Language :: Python :: 3.12", 93 | "Programming Language :: Python :: 3.13", 94 | "Programming Language :: Python :: 3.14", 95 | "Programming Language :: Python :: Implementation :: CPython", 96 | "Development Status :: 5 - Production/Stable", 97 | "Operating System :: POSIX :: Linux", 98 | "Intended Audience :: Developers", 99 | "License :: OSI Approved :: BSD License", 100 | "Topic :: Software Development :: Libraries", 101 | ], 102 | setup_requires=[ 103 | # Support for the basestring type is new in Cython 0.20. 104 | 'Cython >= 0.20', 105 | ], 106 | ext_modules=[ 107 | Extension( 108 | "libmc._client", 109 | sources, 110 | include_dirs=include_dirs, 111 | language="c++", 112 | extra_compile_args=COMPILER_FLAGS, 113 | ) 114 | ], 115 | tests_require=[ 116 | "pytest", 117 | "future", 118 | "numpy", 119 | ]) 120 | -------------------------------------------------------------------------------- /include/llvm/type_traits.h: -------------------------------------------------------------------------------- 1 | //===- llvm/Support/type_traits.h - Simplfied type traits -------*- C++ -*-===// 2 | // 3 | // The LLVM Compiler Infrastructure 4 | // 5 | // This file is distributed under the University of Illinois Open Source 6 | // License. See LICENSE.TXT for details. 7 | // 8 | //===----------------------------------------------------------------------===// 9 | // 10 | // This file provides a template class that determines if a type is a class or 11 | // not. The basic mechanism, based on using the pointer to member function of 12 | // a zero argument to a function was "boosted" from the boost type_traits 13 | // library. See http://www.boost.org/ for all the gory details. 14 | // 15 | //===----------------------------------------------------------------------===// 16 | 17 | #ifndef LLVM_SUPPORT_TYPE_TRAITS_H 18 | #define LLVM_SUPPORT_TYPE_TRAITS_H 19 | 20 | #include 21 | 22 | // This is actually the conforming implementation which works with abstract 23 | // classes. However, enough compilers have trouble with it that most will use 24 | // the one in boost/type_traits/object_traits.hpp. This implementation actually 25 | // works with VC7.0, but other interactions seem to fail when we use it. 26 | 27 | namespace llvm { 28 | 29 | namespace dont_use 30 | { 31 | // These two functions should never be used. They are helpers to 32 | // the is_class template below. They cannot be located inside 33 | // is_class because doing so causes at least GCC to think that 34 | // the value of the "value" enumerator is not constant. Placing 35 | // them out here (for some strange reason) allows the sizeof 36 | // operator against them to magically be constant. This is 37 | // important to make the is_class::value idiom zero cost. it 38 | // evaluates to a constant 1 or 0 depending on whether the 39 | // parameter T is a class or not (respectively). 40 | template char is_class_helper(void(T::*)()); 41 | template double is_class_helper(...); 42 | } 43 | 44 | template 45 | struct is_class 46 | { 47 | // is_class<> metafunction due to Paul Mensonides (leavings@attbi.com). For 48 | // more details: 49 | // http://groups.google.com/groups?hl=en&selm=000001c1cc83%24e154d5e0%247772e50c%40c161550a&rnum=1 50 | public: 51 | enum { value = sizeof(char) == sizeof(dont_use::is_class_helper(0)) }; 52 | }; 53 | 54 | 55 | /// isPodLike - This is a type trait that is used to determine whether a given 56 | /// type can be copied around with memcpy instead of running ctors etc. 57 | template 58 | struct isPodLike { 59 | // If we don't know anything else, we can (at least) assume that all non-class 60 | // types are PODs. 61 | static const bool value = !is_class::value; 62 | }; 63 | 64 | // std::pair's are pod-like if their elements are. 65 | template 66 | struct isPodLike > { 67 | static const bool value = isPodLike::value & isPodLike::value; 68 | }; 69 | 70 | 71 | /// \brief Metafunction that determines whether the two given types are 72 | /// equivalent. 73 | template 74 | struct is_same { 75 | static const bool value = false; 76 | }; 77 | 78 | template 79 | struct is_same { 80 | static const bool value = true; 81 | }; 82 | 83 | // enable_if_c - Enable/disable a template based on a metafunction 84 | template 85 | struct enable_if_c { 86 | typedef T type; 87 | }; 88 | 89 | template struct enable_if_c { }; 90 | 91 | // enable_if - Enable/disable a template based on a metafunction 92 | template 93 | struct enable_if : public enable_if_c { }; 94 | 95 | namespace dont_use { 96 | template char base_of_helper(const volatile Base*); 97 | template double base_of_helper(...); 98 | } 99 | 100 | /// is_base_of - Metafunction to determine whether one type is a base class of 101 | /// (or identical to) another type. 102 | template 103 | struct is_base_of { 104 | static const bool value 105 | = is_class::value && is_class::value && 106 | sizeof(char) == sizeof(dont_use::base_of_helper((Derived*)0)); 107 | }; 108 | 109 | // remove_pointer - Metafunction to turn Foo* into Foo. Defined in 110 | // C++0x [meta.trans.ptr]. 111 | template struct remove_pointer { typedef T type; }; 112 | template struct remove_pointer { typedef T type; }; 113 | template struct remove_pointer { typedef T type; }; 114 | template struct remove_pointer { typedef T type; }; 115 | template struct remove_pointer { 116 | typedef T type; }; 117 | 118 | template 119 | struct conditional { typedef T type; }; 120 | 121 | template 122 | struct conditional { typedef F type; }; 123 | 124 | } 125 | 126 | #endif 127 | -------------------------------------------------------------------------------- /misc/gomcbench/gomcbench_test.go: -------------------------------------------------------------------------------- 1 | package gomcbench 2 | 3 | import ( 4 | "fmt" 5 | "strings" 6 | "testing" 7 | 8 | "github.com/bradfitz/gomemcache/memcache" 9 | golibmc "github.com/douban/libmc/src" 10 | ) 11 | 12 | const NSERVERS int = 10 13 | const key1 string = "google" 14 | 15 | var smallValue []byte = []byte("google") // 6B 16 | var largeValue []byte = []byte(strings.Repeat("A", 10240)) // 10 KB 17 | 18 | func getGolibmcClient(nServers int) *golibmc.Client { 19 | servers := make([]string, nServers) 20 | for i := 0; i < nServers; i++ { 21 | servers[i] = fmt.Sprintf("localhost:%d", 21211+i) 22 | } 23 | noreply := false 24 | prefix := "" 25 | hashFunc := golibmc.HashCRC32 26 | failover := false 27 | disableLock := false 28 | return golibmc.New(servers, noreply, prefix, hashFunc, failover, disableLock) 29 | } 30 | 31 | func getGomemcacheClient(nServers int) *memcache.Client { 32 | servers := make([]string, nServers) 33 | for i := 0; i < nServers; i++ { 34 | servers[i] = fmt.Sprintf("localhost:%d", 21211+i) 35 | } 36 | 37 | return memcache.New(servers...) 38 | } 39 | 40 | func benchmarkGolibmcGetSingleValue(nTimes int, value []byte) { 41 | mc := getGolibmcClient(NSERVERS) 42 | item := golibmc.Item{ 43 | Key: key1, 44 | Value: value, 45 | } 46 | mc.Set(&item) 47 | for i := 0; i < nTimes; i++ { 48 | mc.Get(key1) 49 | } 50 | } 51 | 52 | func benchmarkGomemcacheGetSingleValue(nTimes int, value []byte) { 53 | mc := getGomemcacheClient(NSERVERS) 54 | item := memcache.Item{ 55 | Key: key1, 56 | Value: value, 57 | } 58 | mc.Set(&item) 59 | for i := 0; i < nTimes; i++ { 60 | mc.Get(key1) 61 | } 62 | } 63 | 64 | func benchmarkGolibmcGetMultiValues(nTimes int, nItems int, value []byte) { 65 | mc := getGolibmcClient(NSERVERS) 66 | keys := make([]string, nItems) 67 | items := make([]*golibmc.Item, nItems) 68 | for i := 0; i < nItems; i++ { 69 | key := fmt.Sprintf("test_multi_key_%d", i) 70 | items[i] = &golibmc.Item{ 71 | Key: key, 72 | Value: value, 73 | } 74 | keys[i] = key 75 | } 76 | 77 | mc.SetMulti(items) 78 | for i := 0; i < nTimes; i++ { 79 | mc.GetMulti(keys) 80 | } 81 | } 82 | 83 | func benchmarkGomemcacheGetMultiValues(nTimes int, nItems int, value []byte) { 84 | mc := getGomemcacheClient(NSERVERS) 85 | keys := make([]string, nItems) 86 | for i := 0; i < nItems; i++ { 87 | key := fmt.Sprintf("test_multi_key_%d", i) 88 | item := &memcache.Item{ 89 | Key: key, 90 | Value: value, 91 | } 92 | keys[i] = key 93 | mc.Set(item) 94 | } 95 | 96 | for i := 0; i < nTimes; i++ { 97 | mc.GetMulti(keys) 98 | } 99 | } 100 | 101 | // get small value 102 | 103 | func BenchmarkGolibmcGetSingleSmallValue(b *testing.B) { 104 | benchmarkGolibmcGetSingleValue(b.N, smallValue) 105 | } 106 | 107 | func BenchmarkGomemcacheGetSingleSmallValue(b *testing.B) { 108 | benchmarkGomemcacheGetSingleValue(b.N, smallValue) 109 | } 110 | 111 | // get large value 112 | 113 | func BenchmarkGolibmcGetSingleLargeValue(b *testing.B) { 114 | benchmarkGolibmcGetSingleValue(b.N, largeValue) 115 | } 116 | 117 | func BenchmarkGomemcacheGetSingleLargeValue(b *testing.B) { 118 | benchmarkGomemcacheGetSingleValue(b.N, largeValue) 119 | } 120 | 121 | // get multi(10) small value 122 | 123 | func BenchmarkGolibmcGet10SmallValues(b *testing.B) { 124 | benchmarkGolibmcGetMultiValues(b.N, 10, smallValue) 125 | } 126 | 127 | func BenchmarkGomemcacheGet10SmallValues(b *testing.B) { 128 | benchmarkGomemcacheGetMultiValues(b.N, 10, smallValue) 129 | } 130 | 131 | // get multi(100) small value 132 | 133 | func BenchmarkGolibmcGet100SmallValues(b *testing.B) { 134 | benchmarkGolibmcGetMultiValues(b.N, 100, smallValue) 135 | } 136 | 137 | func BenchmarkGomemcacheGet100SmallValues(b *testing.B) { 138 | benchmarkGomemcacheGetMultiValues(b.N, 100, smallValue) 139 | } 140 | 141 | // get multi(1000) small value 142 | 143 | func BenchmarkGolibmcGet1000SmallValues(b *testing.B) { 144 | benchmarkGolibmcGetMultiValues(b.N, 1000, smallValue) 145 | } 146 | 147 | func BenchmarkGomemcacheGet1000SmallValues(b *testing.B) { 148 | benchmarkGomemcacheGetMultiValues(b.N, 1000, smallValue) 149 | } 150 | 151 | // get multi(10) large value 152 | 153 | func BenchmarkGolibmcGet10LargeValues(b *testing.B) { 154 | benchmarkGolibmcGetMultiValues(b.N, 10, largeValue) 155 | } 156 | 157 | func BenchmarkGomemcacheGet10LargeValues(b *testing.B) { 158 | benchmarkGomemcacheGetMultiValues(b.N, 10, largeValue) 159 | } 160 | 161 | // get multi(100) large value 162 | 163 | func BenchmarkGolibmcGet100LargeValues(b *testing.B) { 164 | benchmarkGolibmcGetMultiValues(b.N, 100, largeValue) 165 | } 166 | 167 | func BenchmarkGomemcacheGet100LargeValues(b *testing.B) { 168 | benchmarkGomemcacheGetMultiValues(b.N, 100, largeValue) 169 | } 170 | 171 | // get multi(1000) large value 172 | 173 | func BenchmarkGolibmcGet1000LargeValues(b *testing.B) { 174 | benchmarkGolibmcGetMultiValues(b.N, 1000, largeValue) 175 | } 176 | 177 | func BenchmarkGomemcacheGet1000LargeValues(b *testing.B) { 178 | benchmarkGomemcacheGetMultiValues(b.N, 1000, largeValue) 179 | } 180 | -------------------------------------------------------------------------------- /src/ClientPool.cpp: -------------------------------------------------------------------------------- 1 | //#include 2 | #include 3 | #include "ClientPool.h" 4 | 5 | namespace douban { 6 | namespace mc { 7 | 8 | // default max of 4 clients to match memcached's default of 4 worker threads 9 | ClientPool::ClientPool(): m_initial_clients(1), m_max_clients(4), m_max_growth(4) { 10 | memset(m_opt_changed, false, sizeof m_opt_changed); 11 | memset(m_opt_value, 0, sizeof m_opt_value); 12 | } 13 | 14 | ClientPool::~ClientPool() { 15 | } 16 | 17 | void ClientPool::config(config_options_t opt, int val) { 18 | std::lock_guard config_pool(m_pool_lock); 19 | if (opt < CLIENT_CONFIG_OPTION_COUNT) { 20 | m_opt_changed[opt] = true; 21 | m_opt_value[opt] = val; 22 | for (auto &client : m_clients) { 23 | client.c.config(opt, val); 24 | } 25 | return; 26 | } 27 | std::unique_lock initializing(m_acquiring_growth); 28 | switch (opt) { 29 | case CFG_INITIAL_CLIENTS: 30 | m_initial_clients = val; 31 | if (m_initial_clients > m_max_clients) { 32 | m_max_clients = m_initial_clients; 33 | } 34 | if (m_clients.size() < m_initial_clients) { 35 | growPool(m_initial_clients); 36 | } 37 | break; 38 | case CFG_MAX_CLIENTS: 39 | m_max_clients = val; 40 | break; 41 | case CFG_MAX_GROWTH: 42 | m_max_growth = val; 43 | break; 44 | default: 45 | break; 46 | } 47 | } 48 | 49 | int ClientPool::init(const char* const * hosts, const uint32_t* ports, 50 | const size_t n, const char* const * aliases) { 51 | updateServers(hosts, ports, n, aliases); 52 | std::unique_lock initializing(m_acquiring_growth); 53 | std::lock_guard config_pool(m_pool_lock); 54 | return growPool(m_initial_clients); 55 | } 56 | 57 | int ClientPool::updateServers(const char* const* hosts, const uint32_t* ports, 58 | const size_t n, const char* const* aliases) { 59 | std::lock_guard updating_clients(m_pool_lock); 60 | duplicate_strings(hosts, n, m_hosts_data, m_hosts); 61 | duplicate_strings(aliases, n, m_aliases_data, m_aliases); 62 | 63 | m_ports.resize(n); 64 | std::copy(ports, ports + n, m_ports.begin()); 65 | 66 | std::atomic rv = 0; 67 | std::lock_guard updating(m_fifo_access); 68 | std::for_each(irange(0), irange(m_clients.size()), 69 | //std::for_each(std::execution::par_unseq, irange(0), irange(m_clients.size()), 70 | [this, &rv](int i) { 71 | std::lock_guard updating_worker(*m_thread_workers[i]); 72 | const int err = m_clients[i].c.updateServers( 73 | m_hosts.data(), m_ports.data(), m_hosts.size(), m_aliases.data()); 74 | if (err != 0) { 75 | rv.store(err, std::memory_order_relaxed); 76 | } 77 | }); 78 | return rv; 79 | } 80 | 81 | int ClientPool::setup(Client* c) { 82 | for (int i = 0; i < CLIENT_CONFIG_OPTION_COUNT; i++) { 83 | if (m_opt_changed[i]) { 84 | c->config(static_cast(i), m_opt_value[i]); 85 | } 86 | } 87 | return c->init(m_hosts.data(), m_ports.data(), m_hosts.size(), m_aliases.data()); 88 | } 89 | 90 | // needs to hold both m_acquiring_growth and m_pool_lock 91 | int ClientPool::growPool(size_t by) { 92 | assert(by > 0); 93 | size_t from = m_clients.size(); 94 | m_clients.resize(from + by); 95 | std::atomic rv = 0; 96 | std::for_each(irange(from), irange(from + by), 97 | //std::for_each(std::execution::par_unseq, irange(from), irange(from + by), 98 | [this, &rv](int i) { 99 | const int err = setup(&m_clients[i].c); 100 | m_clients[i].index = i; 101 | if (err != 0) { 102 | rv.store(err, std::memory_order_relaxed); 103 | } 104 | }); 105 | // adds workers with non-zero return values 106 | // if changed, acquire should probably raise rather than hang 107 | addWorkers(by); 108 | return rv; 109 | } 110 | 111 | inline bool ClientPool::shouldGrowUnsafe() { 112 | return m_clients.size() < m_max_clients && m_locked; 113 | } 114 | 115 | int ClientPool::autoGrow() { 116 | std::unique_lock growing(m_acquiring_growth); 117 | if (shouldGrowUnsafe()) { 118 | std::lock_guard growing_pool(m_pool_lock); 119 | return growPool(MIN(m_max_clients - m_clients.size(), 120 | MIN(m_max_growth, m_clients.size()))); 121 | } 122 | return 0; 123 | } 124 | 125 | IndexedClient* ClientPool::_acquire() { 126 | m_acquiring_growth.lock_shared(); 127 | const auto growing = shouldGrowUnsafe(); 128 | m_acquiring_growth.unlock_shared(); 129 | if (growing) { 130 | std::thread acquire_overflow(&ClientPool::autoGrow, this); 131 | acquire_overflow.detach(); 132 | } 133 | 134 | int idx = acquireWorker(); 135 | m_thread_workers[idx]->lock(); 136 | return &m_clients[idx]; 137 | } 138 | 139 | void ClientPool::_release(const IndexedClient* idx) { 140 | std::mutex* const * mux = &m_thread_workers[idx->index]; 141 | (**mux).unlock(); 142 | releaseWorker(idx->index); 143 | } 144 | 145 | Client* ClientPool::acquire() { 146 | return &_acquire()->c; 147 | } 148 | 149 | void ClientPool::release(const Client* ref) { 150 | // C std 6.7.2.1-13 151 | auto idx = reinterpret_cast(ref); 152 | return _release(idx); 153 | } 154 | 155 | } // namespace mc 156 | } // namespace douban 157 | -------------------------------------------------------------------------------- /tests/profile_client.cpp: -------------------------------------------------------------------------------- 1 | // make profile_client && valgrind --tool=callgrind ./tests/profile_client && qcachegrind callgrind.out.9109 2 | #include 3 | #include "Common.h" 4 | #include "Client.h" 5 | #include "test_common.h" 6 | 7 | #ifndef __MACH__ 8 | #include 9 | #else 10 | #include 11 | #include 12 | #endif 13 | 14 | using douban::mc::Client; 15 | using douban::mc::tests::newClient; 16 | using douban::mc::tests::gen_random; 17 | 18 | 19 | static double getCPUTime() { 20 | timespec ts; 21 | #ifdef __MACH__ // OS X does not have clock_gettime, use clock_get_time 22 | clock_serv_t cclock; 23 | mach_timespec_t mts; 24 | host_get_clock_service(mach_host_self(), CALENDAR_CLOCK, &cclock); 25 | clock_get_time(cclock, &mts); 26 | mach_port_deallocate(mach_task_self(), cclock); 27 | ts.tv_sec = mts.tv_sec; 28 | ts.tv_nsec = mts.tv_nsec; 29 | #else 30 | clock_gettime(CLOCK_REALTIME, &ts); 31 | #endif 32 | return static_cast(ts.tv_sec) + 1e-9 * static_cast(ts.tv_nsec); 33 | } 34 | 35 | void set_(Client* client, const char* const* keys, const size_t* key_lens, 36 | const flags_t* flags, const char* const* vals, const size_t* val_len, size_t n = 1) { 37 | 38 | const exptime_t exptime = 0; 39 | const int noreply = 0; 40 | 41 | message_result_t **m_results = NULL; 42 | size_t nResults = 0; 43 | 44 | client->set(keys, key_lens, flags, exptime, NULL, noreply, vals, val_len, 45 | n, &m_results, &nResults); 46 | client->destroyMessageResult(); 47 | } 48 | 49 | 50 | void get_(Client* client, const char* const * keys, const size_t* key_lens, size_t n = 1) { 51 | retrieval_result_t** r_results = NULL; 52 | size_t nResults = 0; 53 | client->get(keys, key_lens, n, &r_results, &nResults); 54 | client->destroyRetrievalResult(); 55 | } 56 | 57 | static const int N_LOOP = 10000; 58 | 59 | 60 | #define DEFINE_PROFILE_SET_GET(M, N) \ 61 | void profile_set_get_##M(Client* client, const char* key, const size_t key_len, const char* val) { \ 62 | size_t mm = (M); \ 63 | flags_t flags = 0; \ 64 | set_(client, &key, &key_len, &flags, &val, &mm, 1); \ 65 | for (int i = 0; i < (N); i++) { \ 66 | get_(client, &key, &key_len, 1); \ 67 | } \ 68 | } 69 | 70 | DEFINE_PROFILE_SET_GET(1000, N_LOOP) 71 | DEFINE_PROFILE_SET_GET(4000, N_LOOP) 72 | DEFINE_PROFILE_SET_GET(8000, N_LOOP) 73 | DEFINE_PROFILE_SET_GET(20000, N_LOOP) 74 | DEFINE_PROFILE_SET_GET(512000, 1) 75 | DEFINE_PROFILE_SET_GET(1000000, 1) 76 | 77 | 78 | // FIXME: ##NITEM##VAL_LEN 79 | #define DEFINE_PROFILE_SET_GET_MULTI(NITEM, VAL_LEN, N) \ 80 | void profile_set_get_multi_##NITEM##VAL_LEN(Client* client, const char* const * keys, \ 81 | const size_t* key_lens, const char* const * vals) { \ 82 | size_t val_lens[(NITEM)]; \ 83 | flags_t flags[(NITEM)]; \ 84 | for (int i = 0; i < (NITEM); i++) { \ 85 | val_lens[i] = (VAL_LEN); \ 86 | flags[i] = 0; \ 87 | } \ 88 | \ 89 | set_(client, keys, key_lens, flags, vals, val_lens, (NITEM)); \ 90 | for (int i = 0; i < (N); i++) { \ 91 | get_(client, keys, key_lens, (NITEM)); \ 92 | } \ 93 | } 94 | 95 | DEFINE_PROFILE_SET_GET_MULTI(10, 100, 1000) 96 | DEFINE_PROFILE_SET_GET_MULTI(10, 1000, 1000) 97 | DEFINE_PROFILE_SET_GET_MULTI(100, 100, 1000) 98 | 99 | 100 | static const int N_ITEMS = 1000; 101 | static const int KEY_MAX = 200; 102 | static const int VAL_MAX = 1000000; 103 | static const int VAL_MAX2 = 1000; 104 | 105 | int main() { 106 | Client* client = newClient(20); 107 | 108 | const char key[] = "test_foo"; 109 | const size_t key_len = 8; 110 | char val_[VAL_MAX + 1]; 111 | gen_random(val_, VAL_MAX); 112 | const char* val = val_; 113 | 114 | char **keys = new char*[N_ITEMS]; 115 | size_t* key_lens = new size_t[N_ITEMS]; 116 | char **vals = new char*[N_ITEMS]; 117 | size_t* val_lens = new size_t[N_ITEMS]; 118 | 119 | for (int i = 0; i < N_ITEMS; i++) { 120 | keys[i] = new char[KEY_MAX + 1]; 121 | key_lens[i] = snprintf(keys[i], KEY_MAX, "test_profile_key_%d", i); 122 | vals[i] = new char[VAL_MAX2 + 1]; 123 | gen_random(vals[i], VAL_MAX2); 124 | val_lens[i] = VAL_MAX2; 125 | } 126 | 127 | #define TIMEIT(REPR) do { \ 128 | double t0 = getCPUTime(); \ 129 | (REPR); \ 130 | double t1 = getCPUTime(); \ 131 | (void) t0; \ 132 | (void) t1; \ 133 | log_info(#REPR" in %.3f s", t1 - t0); \ 134 | } while (0) 135 | 136 | TIMEIT(profile_set_get_1000(client, key, key_len, val)); 137 | TIMEIT(profile_set_get_4000(client, key, key_len, val)); 138 | TIMEIT(profile_set_get_8000(client, key, key_len, val)); 139 | TIMEIT(profile_set_get_20000(client, key, key_len, val)); 140 | TIMEIT(profile_set_get_512000(client, key, key_len, val)); 141 | TIMEIT(profile_set_get_1000000(client, key, key_len, val)); 142 | TIMEIT(profile_set_get_multi_10100(client, keys, key_lens, vals)); 143 | TIMEIT(profile_set_get_multi_101000(client, keys, key_lens, vals)); 144 | TIMEIT(profile_set_get_multi_100100(client, keys, key_lens, vals)); 145 | 146 | for (int i = 0; i < N_ITEMS; i++) { 147 | delete[] keys[i]; 148 | delete[] vals[i]; 149 | } 150 | delete[] keys; 151 | delete[] key_lens; 152 | delete[] vals; 153 | delete[] val_lens; 154 | 155 | delete client; 156 | return 0; 157 | } 158 | -------------------------------------------------------------------------------- /src/HashkitKetama.cpp: -------------------------------------------------------------------------------- 1 | #include "hashkit/ketama.h" 2 | #include 3 | #include 4 | #include "Common.h" 5 | 6 | 7 | using douban::mc::Connection; 8 | 9 | namespace douban { 10 | namespace mc { 11 | namespace hashkit { 12 | 13 | 14 | const size_t KetamaSelector::s_pointerPerHash = 1; 15 | const size_t KetamaSelector::s_pointerPerServer = 100; 16 | const hash_function_t KetamaSelector::s_defaultHashFunction = &hash_md5; 17 | 18 | KetamaSelector::KetamaSelector() 19 | :m_nServers(0), m_useFailover(false), m_hashFunction(NULL) 20 | #ifndef NDEBUG 21 | , m_sorted(false) 22 | #endif 23 | { 24 | } 25 | 26 | void KetamaSelector::setHashFunction(hash_function_t fn) { 27 | m_hashFunction = fn; 28 | } 29 | 30 | void KetamaSelector::enableFailover() { 31 | m_useFailover = true; 32 | } 33 | 34 | void KetamaSelector::disableFailover() { 35 | m_useFailover = false; 36 | } 37 | 38 | void KetamaSelector::reset() { 39 | m_continuum.clear(); 40 | m_nServers = 0; 41 | } 42 | 43 | void KetamaSelector::addServers(Connection* conns, size_t nConns) { 44 | 45 | // from: libmemcached/libmemcached/hosts.cc +303 46 | char sort_host[MC_NI_MAXHOST + 1 + MC_NI_MAXSERV + 1 + MC_NI_MAXSERV]= ""; 47 | for (size_t i = 0; i < nConns; i++) { 48 | Connection* conn = &conns[i]; 49 | int sort_host_len = 0; 50 | for (size_t pointer_idx= 0; pointer_idx < s_pointerPerServer / s_pointerPerHash; 51 | pointer_idx++) { 52 | if (conn->hasAlias()) { 53 | sort_host_len = snprintf(sort_host, sizeof(sort_host), "%s-%zu", 54 | conn->name(), pointer_idx); 55 | } else { 56 | if (conn->port() != MC_DEFAULT_PORT) { 57 | sort_host_len = snprintf(sort_host, sizeof(sort_host), "%s:%u-%zu", 58 | conn->host(), conn->port(), pointer_idx); 59 | } else { 60 | sort_host_len = snprintf(sort_host, sizeof(sort_host), "%s-%zu", 61 | conn->host(), pointer_idx); 62 | } 63 | } 64 | continuum_item_t item; 65 | // Equivalent to `MEMCACHED_BEHAVIOR_KETAMA_HASH` behavior in libmemcached, 66 | // but here it always use hash_md5. 67 | item.hash_value = hash_md5(sort_host, sort_host_len); 68 | item.conn_idx = i; 69 | item.conn = conn; 70 | m_continuum.push_back(item); 71 | } 72 | } 73 | 74 | m_nServers = nConns; 75 | 76 | std::sort(m_continuum.begin(), m_continuum.end(), continuum_item_t::compare); 77 | #ifndef NDEBUG 78 | m_sorted = true; 79 | #endif 80 | } 81 | 82 | 83 | std::vector::iterator KetamaSelector::getServerIt(const char* key, size_t key_len, 84 | bool check_alive) { 85 | #ifndef NDEBUG 86 | if (!m_sorted) { 87 | return m_continuum.end(); 88 | } 89 | #endif 90 | std::vector::iterator it = m_continuum.end(); 91 | switch (m_nServers) { 92 | case 0: 93 | return m_continuum.end(); 94 | break; 95 | case 1: 96 | it = m_continuum.begin(); 97 | break; 98 | default: 99 | continuum_item_t target_item; 100 | 101 | if (m_hashFunction == NULL) { 102 | m_hashFunction = s_defaultHashFunction; 103 | log_warn("hash function is not specified, use hash_md5"); 104 | } 105 | target_item.hash_value = m_hashFunction(key, key_len); 106 | target_item.conn_idx = 0; 107 | target_item.conn = NULL; 108 | it = std::lower_bound(m_continuum.begin(), m_continuum.end(), target_item, 109 | continuum_item_t::compare); 110 | break; 111 | } 112 | 113 | if (it == m_continuum.end()) { 114 | it = m_continuum.begin(); 115 | } 116 | Connection* origin_conn = it->conn; 117 | 118 | bool is_alive = true; 119 | if (check_alive && origin_conn != NULL) { 120 | is_alive = origin_conn->tryReconnect(false); 121 | } 122 | 123 | if (!is_alive) { 124 | if (m_useFailover) { 125 | size_t max_iter = m_continuum.size(); 126 | do { 127 | ++it; 128 | if (it == m_continuum.end()) { 129 | it = m_continuum.begin(); 130 | } 131 | if (it->conn != origin_conn && it->conn->tryReconnect(false)) { 132 | break; 133 | } 134 | } while (--max_iter); 135 | if (max_iter == 0) { 136 | log_warn("no server is avaliable(alive) for key: \"%.*s\"", static_cast(key_len), key); 137 | return m_continuum.end(); 138 | } 139 | } else { 140 | return m_continuum.end(); 141 | } 142 | } 143 | 144 | return it; 145 | } 146 | 147 | 148 | int KetamaSelector::getServer(const char* key, size_t key_len, bool check_alive) { 149 | std::vector::iterator it = getServerIt(key, key_len, check_alive); 150 | if (it == m_continuum.end()) { 151 | return -1; 152 | } 153 | return static_cast(it->conn_idx); 154 | } 155 | 156 | Connection* KetamaSelector::getConn(const char* key, size_t key_len, bool check_alive) { 157 | std::vector::iterator it = getServerIt(key, key_len, check_alive); 158 | if (it == m_continuum.end()) { 159 | return NULL; 160 | } 161 | return it->conn; 162 | } 163 | 164 | 165 | } // namespace hashkit 166 | } // namespace mc 167 | } // namespace douban 168 | -------------------------------------------------------------------------------- /tests/shabby/slow_memcached_server.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import sys 4 | import time 5 | import socket 6 | 7 | import socketserver as SocketServer 8 | 9 | PORT = 0x2305 10 | BLOCKING_SECONDS = 0.5 # seconds 11 | 12 | 13 | DAYS30 = 3600 * 24 * 30 14 | KEY_GET_SERVER_ERROR = b'gimme_get_server_error' 15 | KEY_SET_SERVER_ERROR = b'gimme_set_server_error' 16 | KEY_SERVER_DOWN = b'biubiubiu' 17 | 18 | memcached = None 19 | 20 | 21 | class Server(SocketServer.ThreadingMixIn, SocketServer.TCPServer): 22 | allow_reuse_address = True 23 | 24 | 25 | def is_valid_key(key): 26 | if len(key) > 250: 27 | return False 28 | if any(c in key for c in (b' ', b'\0', b'\r', b'\n')): 29 | return False 30 | return True 31 | 32 | 33 | class MemcachedProvider(object): 34 | 35 | def __init__(self): 36 | self._store = { 37 | 'stubs': (0, time.time() + DAYS30, 3, 'yes') 38 | } 39 | 40 | def process_get(self, request): 41 | assert request.startswith(b'get ') 42 | request = request.rstrip() 43 | response = b'' 44 | for key in request.split(b' ')[1:]: 45 | if key == KEY_GET_SERVER_ERROR: 46 | return b'SERVER_ERROR\r\n' 47 | if not is_valid_key(key): 48 | return b'CLIENT_ERROR invalid key\r\n' 49 | if key not in self._store: 50 | continue 51 | flags, exptime, bytes_, data_block = self._store[key] 52 | if time.time() > exptime: 53 | del self._store[key] 54 | continue 55 | response += ( 56 | b'VALUE %s %d %d\r\n%s\r\n' % ( 57 | key, flags, bytes_, data_block 58 | ) 59 | ) 60 | response += b'END\r\n' 61 | 62 | return response 63 | 64 | def process_set(self, request): 65 | t0 = time.time() 66 | response = b'' 67 | while request: 68 | assert request.startswith(b'set ') 69 | pos = request.find(b'\r\n') 70 | assert pos > 0 71 | set_meta = request[:pos] 72 | request = request[pos + 2:] 73 | 74 | key, flags, exptime, bytes_ = set_meta.split(b' ')[1:] 75 | if key == KEY_SERVER_DOWN: 76 | print("I'm shot. Dying.") 77 | return None 78 | 79 | if key == KEY_SET_SERVER_ERROR: 80 | return b'SERVER_ERROR\r\n' 81 | if not is_valid_key(key): 82 | response += b'CLIENT_ERROR invalid key\r\n' 83 | continue 84 | flags = int(flags) 85 | exptime = int(exptime) 86 | bytes_ = int(bytes_) 87 | if exptime == 0: 88 | exptime = DAYS30 89 | 90 | if exptime <= DAYS30: 91 | exptime += t0 92 | 93 | assert len(request) >= bytes_ + 2 94 | data_block = request[:bytes_] 95 | request = request[bytes_ + 2:] 96 | self._store[key] = flags, exptime, bytes_, data_block 97 | response += b'STORED\r\n' 98 | 99 | return response 100 | 101 | def process_version(self, request): 102 | return b'VERSION 1.0.24-shabby\r\n' 103 | 104 | def process_rest(self, request): 105 | print('unexpected request: %s' % request) 106 | return b'ERROR\r\n' 107 | 108 | def __getattr__(self, name): 109 | if name.startswith('process_'): 110 | return self.process_rest 111 | raise AttributeError() 112 | 113 | 114 | class Handler(SocketServer.BaseRequestHandler): 115 | 116 | mcp = MemcachedProvider() 117 | 118 | def handle(self): 119 | while True: 120 | req = b'' 121 | while req[-2:] != b'\r\n': 122 | try: 123 | req += self.request.recv(8192) 124 | except socket.error as ex: 125 | if ex.errno == 54: 126 | pass 127 | else: 128 | raise ex 129 | 130 | print('> %s' % req[:-2]) 131 | if not req.strip() or req.strip() == b'quit': 132 | break 133 | pos_space = req.find(b' ') 134 | if pos_space > 0: 135 | meth = req[:pos_space] 136 | else: 137 | meth = req.rstrip() 138 | 139 | res = getattr(self.mcp, 'process_%s' % meth.decode('ascii'), )(req) 140 | if res is None: 141 | memcached.shutdown() 142 | return 143 | if meth != 'version': 144 | time.sleep(BLOCKING_SECONDS) 145 | 146 | n_sent = 0 147 | while n_sent != len(res): 148 | n_sent += self.request.send(res[n_sent:n_sent+100]) 149 | if n_sent != len(res): 150 | print('sleep and send more') 151 | time.sleep(BLOCKING_SECONDS * 2) 152 | 153 | print('< %s' % res[:-2]) 154 | 155 | 156 | def main(argv): 157 | global memcached 158 | port = PORT 159 | if len(argv) == 2: 160 | port = int(argv[1]) 161 | 162 | memcached = Server(("", port), Handler) 163 | print('serve at tcp://%s:%s' % ('0.0.0.0', port)) 164 | try: 165 | memcached.serve_forever() 166 | except KeyboardInterrupt: 167 | return 168 | finally: 169 | memcached.server_close() 170 | 171 | 172 | if __name__ == '__main__': 173 | main(sys.argv) 174 | 175 | -------------------------------------------------------------------------------- /tests/test_hashkit.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include "Common.h" 7 | #include "hashkit/md5.h" 8 | #include "hashkit/hashkit.h" 9 | #include "gtest/gtest.h" 10 | #include "test_common.h" 11 | 12 | 13 | using std::atoi; 14 | using std::string; 15 | using std::getline; 16 | using std::ifstream; 17 | using std::stringstream; 18 | using douban::mc::tests::get_resource_path; 19 | 20 | 21 | static const char UTF8_ZHONG[] = "\xe4\xb8\xad"; // "中" in UTF-8 22 | 23 | 24 | void test_md5_hexdigest(const char key[], size_t key_len, char hexdigest[33]) { 25 | const unsigned char* input = reinterpret_cast(key); 26 | unsigned char out[16]; 27 | douban::mc::hashkit::md5(input, key_len, out); 28 | int i = 0; 29 | char hexdigest2[33] = {0}; 30 | for (i = 0; i < 16; i++) { 31 | snprintf(hexdigest2 + i * 2, 3, "%02x", out[i]); 32 | } 33 | ASSERT_N_STREQ(hexdigest, hexdigest2, 32); 34 | } 35 | 36 | 37 | TEST(hashkit, md5_string) { 38 | test_md5_hexdigest("", 0U, CSTR("d41d8cd98f00b204e9800998ecf8427e")); 39 | test_md5_hexdigest("123456", 6U, CSTR("e10adc3949ba59abbe56e057f20f883e")); 40 | test_md5_hexdigest("abcdef", 6U, CSTR("e80b5017098950fc58aad83c8c14978e")); 41 | test_md5_hexdigest(UTF8_ZHONG, 3U, CSTR("aed1dfbc31703955e64806b799b67645")); 42 | } 43 | 44 | 45 | TEST(hashkit, fnv1_32) { 46 | // test data are generated using: 47 | // http://www.isthe.com/chongo/src/fnv/fnv-5.0.3.tar.gz 48 | // echo -ne "abcdef" | ./fnv132 49 | 50 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32("", 0U), 0x811c9dc5U); 51 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32("123456", 6U), 0xeb008bb8U); 52 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32("abcdef", 6U), 0x9f2d4718); 53 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32(UTF8_ZHONG, 3U), 0x6cd2e676); 54 | 55 | // SOLVED! What is the shortest binary data set for which the 32-bit FNV-1 hash is 0? 56 | // 32-bit solution by Russ Cox on 2011-Mar-02: 57 | // There are 254 solutions of length 5, ranging from 58 | // FNV32_1(01 47 6c 10 f3) = 0 59 | // to 60 | // FNV32_1(fd 45 41 08 a0) = 0 61 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32("\x01\x47\x6c\x10\xf3", 5U), 0U); 62 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32("\xfd\x45\x41\x08\xa0", 5U), 0U); 63 | } 64 | 65 | 66 | TEST(hashkit, fnv1a_32) { 67 | // test data are generated using: 68 | // http://www.isthe.com/chongo/src/fnv/fnv-5.0.3.tar.gz 69 | // echo -ne "abcdef" | ./fnv1a32 70 | 71 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32("", 0U), 0x811c9dc5U); 72 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32("123456", 6U), 0x9995b6aaU); 73 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32("abcdef", 6U), 0xff478a2aU); 74 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32(UTF8_ZHONG, 3U), 0xb846d3f4U); 75 | 76 | // SOLVED! What is the shortest binary data set for which the 32-bit FNV-1a hash is 0? 77 | // 32-bit solution by Russ Cox on 2011-Mar-02: 78 | // The solutions of length 4 are: 79 | // 80 | // FNV32_1a(cc 24 31 c4) = 0 81 | // FNV32_1a(e0 4d 9f cb) = 0 82 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32("\xcc\x24\x31\xc4", 4U), 0U); 83 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32("\xe0\x4d\x9f\xcb", 4U), 0U); 84 | } 85 | 86 | 87 | TEST(hashkit, crc_32) { 88 | // test data are generated via: 89 | // hex(np.uint32(zlib.crc32(key))) 90 | ASSERT_EQ(douban::mc::hashkit::hash_crc_32("", 0U), 0x0U); 91 | ASSERT_EQ(douban::mc::hashkit::hash_crc_32("123456", 6U), 0x972d361U); 92 | ASSERT_EQ(douban::mc::hashkit::hash_crc_32("abcdef", 6U), 0x4b8e39efU); 93 | ASSERT_EQ(douban::mc::hashkit::hash_crc_32(UTF8_ZHONG, 3U), 0xd5d15a8bU); 94 | } 95 | 96 | 97 | TEST(hashkit, mass) { 98 | std::string keys_path = get_resource_path("keys.txt"); 99 | std::string keys_crc_32_path = get_resource_path("keys_crc_32.txt"); 100 | std::string keys_fnv1_32_path = get_resource_path("keys_fnv1_32.txt"); 101 | std::string keys_fnv1a_32_path = get_resource_path("keys_fnv1a_32.txt"); 102 | std::string keys_md5_path = get_resource_path("keys_md5.txt"); 103 | 104 | ifstream keys_stream(keys_path.c_str()); 105 | ifstream keys_crc_32_stream(keys_crc_32_path.c_str()); 106 | ifstream keys_fnv1_32_stream(keys_fnv1_32_path.c_str()); 107 | ifstream keys_fnv1a_32_stream(keys_fnv1a_32_path.c_str()); 108 | ifstream keys_md5_stream(keys_md5_path.c_str()); 109 | 110 | string key_line; 111 | uint32_t key_crc_32, key_fnv1_32, key_fnv1a_32, key_md5; 112 | ASSERT_TRUE(keys_stream.good()); 113 | ASSERT_TRUE(keys_crc_32_stream.good()); 114 | ASSERT_TRUE(keys_fnv1_32_stream.good()); 115 | ASSERT_TRUE(keys_fnv1a_32_stream.good()); 116 | ASSERT_TRUE(keys_md5_stream.good()); 117 | 118 | while (getline(keys_stream, key_line)) { 119 | keys_crc_32_stream >> key_crc_32; 120 | keys_fnv1_32_stream >> key_fnv1_32; 121 | keys_fnv1a_32_stream >> key_fnv1a_32; 122 | keys_md5_stream >> key_md5; 123 | 124 | ASSERT_EQ(douban::mc::hashkit::hash_crc_32(key_line.c_str(), key_line.length()), key_crc_32); 125 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1_32(key_line.c_str(), key_line.length()), key_fnv1_32); 126 | ASSERT_EQ(douban::mc::hashkit::hash_fnv1a_32(key_line.c_str(), key_line.length()), key_fnv1a_32); 127 | ASSERT_EQ(douban::mc::hashkit::hash_md5(key_line.c_str(), key_line.length()), key_md5); 128 | } 129 | keys_stream.close(); 130 | keys_crc_32_stream.close(); 131 | keys_fnv1_32_stream.close(); 132 | keys_fnv1a_32_stream.close(); 133 | keys_md5_stream.close(); 134 | } 135 | -------------------------------------------------------------------------------- /tests/test_router.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import os 4 | import unittest 5 | import libmc 6 | from libmc import Client 7 | 8 | 9 | RES_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'resources') 10 | 11 | 12 | class HashRouterCase(unittest.TestCase): 13 | """ 14 | Assume all memecached are accessible and won't establish any connections 15 | """ 16 | 17 | def test_md5_router(self): 18 | server_list = ['localhost', 'myhost:11211', '127.0.0.1:11212', 19 | 'myhost:11213'] 20 | mc = Client(server_list) 21 | 22 | md5_mc = Client(server_list, hash_fn=libmc.MC_HASH_MD5) 23 | rs = { 24 | 'test:10000': 'localhost:11211', 25 | 'test:20000': '127.0.0.1:11212', 26 | 'test:30000': '127.0.0.1:11212', 27 | 'test:40000': '127.0.0.1:11212', 28 | 'test:50000': '127.0.0.1:11212', 29 | 'test:60000': 'myhost:11213', 30 | 'test:70000': '127.0.0.1:11212', 31 | 'test:80000': '127.0.0.1:11212', 32 | 'test:90000': '127.0.0.1:11212', 33 | } 34 | for k in rs: 35 | self.assertEqual(mc.get_host_by_key(k), rs[k]) 36 | self.assertEqual(md5_mc.get_host_by_key(k), rs[k]) 37 | 38 | for addr in server_list: 39 | ps = addr.split(':') 40 | 41 | if len(ps) == 1: 42 | hostname = ps[0] 43 | port = 11211 44 | else: 45 | hostname = ps[0] 46 | port = int(ps[1]) 47 | 48 | if port == 11211: 49 | key = '%s-10' % hostname 50 | else: 51 | key = '%s:%s-10' % (hostname, port) 52 | 53 | self.assertEqual(mc.get_host_by_key(key), 54 | '%s:%s' % (hostname, port)) 55 | self.assertEqual(md5_mc.get_host_by_key(key), 56 | '%s:%s' % (hostname, port)) 57 | 58 | def test_md5_router_mass(self): 59 | with open(os.path.join(RES_DIR, 'server_port.csv')) as fhandler: 60 | server_list = [':'.join(addr.strip().split(',')) 61 | for addr in fhandler.readlines()] 62 | mc = Client(server_list, hash_fn=libmc.MC_HASH_MD5) 63 | with open(os.path.join(RES_DIR, 'key_pool_idx.csv')) as fhandler: 64 | for line in fhandler: 65 | key, idx_ = line.strip().split(',') 66 | idx = int(idx_) 67 | assert mc.get_host_by_key(key) == server_list[idx] 68 | 69 | def test_fnv1a_32_router(self): 70 | fnv1a_32_mc = Client([ 71 | 'localhost', 'myhost:11211', '127.0.0.1:11212', 'myhost:11213' 72 | ], hash_fn=libmc.MC_HASH_FNV1A_32) 73 | rs = { 74 | 'test:10000': '127.0.0.1:11212', 75 | 'test:20000': '127.0.0.1:11212', 76 | 'test:30000': 'myhost:11213', 77 | 'test:40000': 'myhost:11211', 78 | 'test:50000': 'myhost:11211', 79 | 'test:60000': '127.0.0.1:11212', 80 | 'test:70000': '127.0.0.1:11212', 81 | 'test:80000': 'myhost:11213', 82 | 'test:90000': 'localhost:11211' 83 | } 84 | 85 | for k in rs: 86 | self.assertEqual(fnv1a_32_mc.get_host_by_key(k), rs[k]) 87 | 88 | def test_crc_32_router(self): 89 | crc_32_mc = Client([ 90 | 'localhost', 'myhost:11211', '127.0.0.1:11212', 'myhost:11213' 91 | ], hash_fn=libmc.MC_HASH_CRC_32) 92 | rs = { 93 | 'test:10000': '127.0.0.1:11212', 94 | 'test:20000': '127.0.0.1:11212', 95 | 'test:30000': '127.0.0.1:11212', 96 | 'test:40000': 'myhost:11213', 97 | 'test:50000': 'myhost:11211', 98 | 'test:60000': 'localhost:11211', 99 | 'test:70000': 'myhost:11213', 100 | 'test:80000': 'myhost:11211', 101 | 'test:90000': 'localhost:11211' 102 | } 103 | for k in rs: 104 | self.assertEqual(crc_32_mc.get_host_by_key(k), rs[k]) 105 | 106 | 107 | class HashRouteRealtimeCase(unittest.TestCase): 108 | """ 109 | Will try to connect to corresponding memcached server and 110 | may failover accordingly. 111 | """ 112 | 113 | def test_realtime_host_w_failover(self): 114 | servers = ["127.0.0.1:21211", "127.0.0.1:21212"] 115 | mc = Client(servers) 116 | failover_mc = Client(servers, failover=True) 117 | hosts_visited = set() 118 | for i in range(1000): 119 | key = "test:realtime:route:%d" % i 120 | h1 = mc.get_host_by_key(key) 121 | h2 = mc.get_realtime_host_by_key(key) 122 | assert h1 is not None 123 | assert h1 == h2 124 | 125 | h3 = failover_mc.get_host_by_key(key) 126 | h4 = failover_mc.get_realtime_host_by_key(key) 127 | assert h3 == h1 128 | assert h4 == h1 129 | hosts_visited.add(h1) 130 | assert len(hosts_visited) == len(servers) 131 | 132 | def test_none_host(self): 133 | existed_server = "127.0.0.1:21211" 134 | not_existed_server = "127.0.0.1:1" 135 | servers = [existed_server, not_existed_server] 136 | mc = Client(servers) 137 | failover_mc = Client(servers, failover=True) 138 | 139 | hosts_visited = set() 140 | for i in range(1000): 141 | key = "test:realtime:route:%d" % i 142 | h1 = mc.get_host_by_key(key) 143 | h2 = mc.get_realtime_host_by_key(key) 144 | if h1 == existed_server: 145 | assert h1 == h2 146 | elif h1 == not_existed_server: 147 | assert h2 is None 148 | 149 | h3 = failover_mc.get_host_by_key(key) 150 | h4 = failover_mc.get_realtime_host_by_key(key) 151 | assert h3 == h1 152 | assert h4 == existed_server 153 | hosts_visited.add(h1) 154 | assert len(hosts_visited) == len(servers) 155 | -------------------------------------------------------------------------------- /include/Common.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include 4 | #include 5 | #include 6 | 7 | #include 8 | #include 9 | #include 10 | #include 11 | 12 | #include "Export.h" 13 | 14 | #define PROJECT_NAME "libmc" 15 | #define MC_DEFAULT_PORT 11211 16 | #define MC_DEFAULT_POLL_TIMEOUT 300 17 | #define MC_DEFAULT_CONNECT_TIMEOUT 10 18 | #define MC_DEFAULT_RETRY_TIMEOUT 5 19 | #define MC_DEFAULT_MAX_RETRIES 0 20 | 21 | 22 | #ifdef UIO_MAXIOV 23 | #define MC_UIO_MAXIOV UIO_MAXIOV 24 | #else 25 | #define MC_UIO_MAXIOV 1024 26 | #endif 27 | 28 | 29 | #ifdef NI_MAXHOST 30 | #define MC_NI_MAXHOST NI_MAXHOST 31 | #else 32 | #define MC_NI_MAXHOST 1025 33 | #endif 34 | 35 | #ifdef NI_MAXSERV 36 | #define MC_NI_MAXSERV NI_MAXSERV 37 | #else 38 | #define MC_NI_MAXSERV 32 39 | #endif 40 | 41 | #ifdef MSG_MORE 42 | #define MC_MSG_MORE MSG_MORE 43 | #else 44 | #define MC_MSG_MORE 0 45 | #endif 46 | 47 | 48 | #define MIN_DATABLOCK_CAPACITY 8192 49 | #define MIN(A, B) (((A) > (B)) ? (B) : (A)) 50 | #define MAX(A, B) (((A) < (B)) ? (B) : (A)) 51 | #define CSTR(STR) (const_cast(STR)) 52 | #define ASSERT_N_STREQ(S1, S2, N) do {ASSERT_TRUE(0 == std::strncmp((S1), (S2), (N)));} while (0) 53 | 54 | #if defined __cplusplus 55 | # define __VOID_CAST static_cast 56 | #else 57 | # define __VOID_CAST (void) 58 | #endif 59 | 60 | void printBacktrace(); 61 | 62 | #define _mc_clean_errno() (errno == 0 ? "None" : strerror(errno)) 63 | 64 | #define _mc_output_stderr(LEVEL, FORMAT, ...) \ 65 | fprintf( \ 66 | stderr, \ 67 | "[" PROJECT_NAME "] [" #LEVEL "] [%s:%d] " FORMAT "\n", \ 68 | __FILE__, __LINE__, ##__VA_ARGS__ \ 69 | ) 70 | 71 | 72 | #define MC_LOG_LEVEL_ERROR 1 73 | #define MC_LOG_LEVEL_WARNING 2 74 | #define MC_LOG_LEVEL_INFO 3 75 | #define MC_LOG_LEVEL_DEBUG 4 76 | 77 | #define MC_LOG_LEVEL MC_LOG_LEVEL_ERROR 78 | 79 | 80 | #ifdef NDEBUG 81 | 82 | #define log_debug_if(cond, M, ...) __VOID_CAST(0) 83 | #define log_debug(M, ...) __VOID_CAST(0) 84 | 85 | #define _ASSERTION_FAILED(cond) do { \ 86 | _mc_output_stderr(FATAL, "failed assertion `%s'" , #cond); \ 87 | printBacktrace(); \ 88 | } while (0) 89 | 90 | #else 91 | 92 | #if MC_LOG_LEVEL >= MC_LOG_LEVEL_DEBUG 93 | #define log_debug(M, ...) _mc_output_stderr(DEBUG, "[E: %s] " M, _mc_clean_errno(), ##__VA_ARGS__) 94 | #else 95 | #define log_debug(M, ...) __VOID_CAST(0) 96 | #endif 97 | #define _ASSERTION_FAILED(cond) do { \ 98 | _mc_output_stderr(FATAL, "failed assertion `%s'" , #cond); \ 99 | printBacktrace(); \ 100 | abort(); \ 101 | } while (0) 102 | 103 | #define log_debug_if(cond, M, ...) do { \ 104 | if (cond) log_debug(M, ##__VA_ARGS__); \ 105 | } while (0) 106 | 107 | #endif // NDEBUG 108 | 109 | #define ASSERT(cond) if (!(cond)) _ASSERTION_FAILED(cond) 110 | 111 | #define NOT_REACHED() ASSERT(0) 112 | 113 | #if MC_LOG_LEVEL >= MC_LOG_LEVEL_INFO 114 | #define log_info(M, ...) _mc_output_stderr(INFO, M, ##__VA_ARGS__) 115 | #else 116 | #define log_info(M, ...) __VOID_CAST(0) 117 | #endif 118 | 119 | #define log_info_if(cond, M, ...) do { \ 120 | if (cond) log_info(M, ##__VA_ARGS__); \ 121 | } while (0) 122 | 123 | #if MC_LOG_LEVEL >= MC_LOG_LEVEL_WARNING 124 | #define log_warn(M, ...) _mc_output_stderr(WARN, "[E: %s] " M, _mc_clean_errno(), ##__VA_ARGS__) 125 | #else 126 | #define log_warn(M, ...) __VOID_CAST(0) 127 | #endif 128 | 129 | #define log_warn_if(cond, M, ...) do { \ 130 | if (cond) log_warn(M, ##__VA_ARGS__); \ 131 | } while (0) 132 | 133 | #if MC_LOG_LEVEL >= MC_LOG_LEVEL_ERROR 134 | #define log_err(M, ...) _mc_output_stderr(ERROR, "[E: %s] " M, _mc_clean_errno(), ##__VA_ARGS__) 135 | #else 136 | #define log_err(M, ...) __VOID_CAST(0) 137 | #endif 138 | 139 | #define log_err_if(cond, M, ...) do { \ 140 | if (cond) log_err(M, ##__VA_ARGS__); \ 141 | } while (0) 142 | 143 | 144 | namespace douban { 145 | namespace mc { 146 | 147 | typedef struct pollfd pollfd_t; 148 | 149 | // FSM: finite state machine 150 | typedef enum { 151 | FSM_START, 152 | 153 | // END 154 | FSM_END, // got "END\r\n" 155 | 156 | // ERROR 157 | FSM_ERROR, // got "ERROR\r\n" 158 | 159 | // VALUE [ ]\r\n 160 | // \r\n 161 | FSM_GET_START, // got "VALUE " 162 | FSM_GET_KEY, // got "key " 163 | FSM_GET_FLAGS, // got "flags " 164 | FSM_GET_BYTES, // got "bytes " or "bytes\r" 165 | FSM_GET_CAS, // got "cas\r" or got "bytes\r" 166 | FSM_GET_VALUE_REMAINING, // not got + "\r\n" 167 | 168 | // VERSION 169 | FSM_VER_START, // got "VERSION " 170 | 171 | // STAT 172 | FSM_STAT_START, // got "STAT " 173 | 174 | // [0-9] // INCR/DECR 175 | FSM_INCR_DECR_START, // got [0-9] 176 | FSM_INCR_DECR_REMAINING, // not got "\r\n" 177 | } parser_state_t; 178 | 179 | #define IS_END_STATE(st) ((st) == FSM_END or (st) == FSM_ERROR) 180 | 181 | 182 | typedef enum { 183 | // storage commands 184 | // [ noreply]\r\n 185 | // cas [ noreply]\r\n 186 | // \r\n 187 | // -> 188 | // text msg 189 | SET_OP, 190 | ADD_OP, 191 | REPLACE_OP, 192 | APPEND_OP, 193 | PREPEND_OP, 194 | CAS_OP, 195 | 196 | // retrieval commands 197 | // get *\r\n 198 | // gets *\r\n 199 | // -> 200 | // VALUE []\r\n 201 | // \r\n 202 | // "END\r\n" 203 | GET_OP, 204 | GETS_OP, 205 | 206 | // incr/decr [ noreply]\r\n 207 | // -> 208 | // \r\n or "NOT_FOUND\r\n" 209 | INCR_OP, 210 | DECR_OP, 211 | 212 | // touch [ noreply]\r\n 213 | // -> 214 | // text msg 215 | TOUCH_OP, 216 | 217 | // delete [ noreply]\r\n 218 | // -> 219 | // text msg 220 | DELETE_OP, 221 | 222 | STATS_OP, 223 | FLUSHALL_OP, 224 | VERSION_OP, 225 | QUIT_OP, 226 | } op_code_t; 227 | 228 | const char* errCodeToString(err_code_t err); 229 | bool isUnixSocket(const char* host); 230 | server_string_split_t splitServerString(char* input); 231 | 232 | } // namespace mc 233 | } // namespace douban 234 | -------------------------------------------------------------------------------- /src/c_client.cpp: -------------------------------------------------------------------------------- 1 | #include "c_client.h" 2 | #include "Client.h" 3 | 4 | 5 | using douban::mc::Client; 6 | 7 | 8 | void* client_create() { 9 | return new Client(); 10 | } 11 | 12 | 13 | void client_init(void* client, const char* const * hosts, const uint32_t* ports, 14 | size_t n, const char* const * aliases, const int failover) { 15 | douban::mc::Client* c = static_cast(client); 16 | c->init(hosts, ports, n, aliases); 17 | if (failover) { 18 | c->enableConsistentFailover(); 19 | } else { 20 | c->disableConsistentFailover(); 21 | } 22 | } 23 | 24 | 25 | void client_config(void* client, config_options_t opt, int val) { 26 | douban::mc::Client* c = static_cast(client); 27 | return c->config(opt, val); 28 | } 29 | 30 | 31 | void client_destroy(void* client) { 32 | douban::mc::Client* c = static_cast(client); 33 | delete c; 34 | } 35 | 36 | 37 | const char* client_get_server_address_by_key(void* client, const char* key, const size_t key_len) { 38 | douban::mc::Client* c = static_cast(client); 39 | return c->getServerAddressByKey(key, key_len); 40 | } 41 | 42 | 43 | const char* client_get_realtime_server_address_by_key(void* client, const char* key, 44 | const size_t key_len) { 45 | douban::mc::Client* c = static_cast(client); 46 | return c->getRealtimeServerAddressByKey(key, key_len); 47 | } 48 | 49 | 50 | err_code_t client_version(void* client, broadcast_result_t** results, size_t* n_hosts) { 51 | douban::mc::Client* c = static_cast(client); 52 | return c->version(results, n_hosts); 53 | } 54 | 55 | 56 | void client_destroy_broadcast_result(void* client) { 57 | douban::mc::Client* c = static_cast(client); 58 | return c->destroyBroadcastResult(); 59 | } 60 | 61 | #define IMPL_RETRIEVAL_CMD(M) \ 62 | err_code_t client_##M(void* client, const char* const* keys, const size_t* key_lens, \ 63 | size_t n_keys, retrieval_result_t*** results, size_t* n_results) { \ 64 | douban::mc::Client* c = static_cast(client); \ 65 | return c->M(keys, key_lens, n_keys, results, n_results); \ 66 | } 67 | IMPL_RETRIEVAL_CMD(get) 68 | IMPL_RETRIEVAL_CMD(gets) 69 | #undef IMPL_RETRIEVAL_CMD 70 | 71 | 72 | void client_destroy_retrieval_result(void* client) { 73 | douban::mc::Client* c = static_cast(client); 74 | return c->destroyRetrievalResult(); 75 | } 76 | 77 | 78 | #define IMPL_STORAGE_CMD(M) \ 79 | err_code_t client_##M(void* client, const char* const* keys, const size_t* key_lens, \ 80 | const flags_t* flags, const exptime_t exptime, \ 81 | const cas_unique_t* cas_uniques, const bool noreply, \ 82 | const char* const* vals, const size_t* val_lens, \ 83 | size_t nItems, message_result_t*** results, size_t* n_results) { \ 84 | douban::mc::Client* c = static_cast(client); \ 85 | return c->M(keys, key_lens, flags, exptime, cas_uniques, \ 86 | noreply, vals, val_lens, nItems, results, n_results); \ 87 | } 88 | 89 | IMPL_STORAGE_CMD(set) 90 | IMPL_STORAGE_CMD(add) 91 | IMPL_STORAGE_CMD(replace) 92 | IMPL_STORAGE_CMD(append) 93 | IMPL_STORAGE_CMD(prepend) 94 | IMPL_STORAGE_CMD(cas) 95 | #undef IMPL_STORAGE_CMD 96 | 97 | 98 | err_code_t client_touch(void* client, const char* const* keys, const size_t* key_lens, 99 | const exptime_t exptime, const bool noreply, size_t n_items, 100 | message_result_t*** results, size_t* n_results) { 101 | douban::mc::Client* c = static_cast(client); 102 | return c->touch(keys, key_lens, exptime, noreply, n_items, results, n_results); 103 | } 104 | 105 | 106 | void client_destroy_message_result(void* client) { 107 | douban::mc::Client* c = static_cast(client); 108 | return c->destroyMessageResult(); 109 | } 110 | 111 | 112 | err_code_t client_delete(void*client, const char* const* keys, const size_t* key_lens, 113 | const bool noreply, size_t n_items, 114 | message_result_t*** results, size_t* n_results) { 115 | douban::mc::Client* c = static_cast(client); 116 | return c->_delete(keys, key_lens, noreply, n_items, results, n_results); 117 | } 118 | 119 | 120 | err_code_t client_incr(void* client, const char* key, const size_t keyLen, 121 | const uint64_t delta, const bool noreply, 122 | unsigned_result_t** results, size_t* n_results) { 123 | douban::mc::Client* c = static_cast(client); 124 | return c->incr(key, keyLen, delta, noreply, results, n_results); 125 | } 126 | 127 | 128 | err_code_t client_decr(void* client, const char* key, const size_t keyLen, 129 | const uint64_t delta, const bool noreply, 130 | unsigned_result_t** results, size_t* n_results) { 131 | douban::mc::Client* c = static_cast(client); 132 | return c->decr(key, keyLen, delta, noreply, results, n_results); 133 | } 134 | 135 | void client_destroy_unsigned_result(void* client) { 136 | douban::mc::Client* c = static_cast(client); 137 | return c->destroyUnsignedResult(); 138 | } 139 | 140 | err_code_t client_stats(void* client, broadcast_result_t** results, size_t* n_servers) { 141 | douban::mc::Client* c = static_cast(client); 142 | return c->stats(results, n_servers); 143 | } 144 | 145 | void client_toggle_flush_all_feature(void* client, bool enabled) { 146 | douban::mc::Client* c = static_cast(client); 147 | return c->toggleFlushAllFeature(enabled); 148 | } 149 | err_code_t client_flush_all(void* client, broadcast_result_t** results, size_t* n_servers) { 150 | douban::mc::Client* c = static_cast(client); 151 | return c->flushAll(results, n_servers); 152 | } 153 | 154 | err_code_t client_quit(void* client) { 155 | douban::mc::Client* c = static_cast(client); 156 | return c->quit(); 157 | } 158 | 159 | const char* err_code_to_string(err_code_t err) { 160 | return douban::mc::errCodeToString(err); 161 | } 162 | 163 | server_string_split_t splitServerString(char* input) { 164 | return douban::mc::splitServerString(input); 165 | } 166 | -------------------------------------------------------------------------------- /tests/resources/server_port.csv: -------------------------------------------------------------------------------- 1 | gPGttJQAMQWjtorqmWcNbHcW,50765 2 | jqJUEPSxlJ,10989 3 | EKRNBPYDHEoff,29952 4 | cRKSFBODvgJPnQRBpElvw,53038 5 | NhoiPAtmDwPBrpMXMvSYdBYYuTZXgK,31491 6 | ExK,28484 7 | VBBZEynjEHOuDoXUuxoRd,25783 8 | RBnmZTQqn,43988 9 | cgZfykaKuZNZubkAblcZQTCQhhSmwTVAthTOdFfZbZlIlcGIti,64145 10 | ElCMREIVaYYcVSEXkImNdEnog,12212 11 | tgYsgAdeGrBKtbCyfeRFMUDiFfF,35825 12 | CWTO,42 13 | oXCGtmZMIbqtpEedcyhOuQgxQdNaLjSoVzldmZwcGNxOrH,17215 14 | tsQfvlzWOviB,59597 15 | OrVZagMRRaONsCFuBcqbmxDFwSWXSOOJVtPYTk,10699 16 | PuZvNECnStwmUpWyfmpB,63881 17 | CzuuNOkPFliAmwElNDZqzGZzYkiqywJsndueXmkGChWWB,60871 18 | qLTZjVylBvVwAOSUDaAhAjpPezfUxy,36780 19 | cYUzzuZmCKfdwRBVwPvCefszdrAgW,23261 20 | GtPwIWkynVpzwxerQDSvZKJidzylsxvQSWPRtcRzPR,45937 21 | rWySZIxX,50919 22 | VjgahqolrURVYAZTxFCUhyhxCfmKcax,14100 23 | QpPIRFgwzCiA,8349 24 | yGCOlxvkAyTBqJlyNfkyWHffbEDzXt,35975 25 | UyyvpebTgZyqVXWdcenQMbOVHweQCPcFURea,27867 26 | rMLRzHxTVPLGhar,7179 27 | PRrMpvI,29840 28 | OfqLrjPxGi,21029 29 | RRMazsXQQcVIHNnK,9097 30 | sTUDqDuOkBgXzbQYUajlptmnjwBdwFfPHJWX,45187 31 | KZNLlOEp,57345 32 | zSFeBM,23269 33 | KCYGpKxNhdLpguFlTSZhxPmkqDGseOev,59116 34 | VDNFeKjIz,2110 35 | sclvUYNRHr,23829 36 | kPomxQdIcDIUYKuxkWajLsycP,36468 37 | TgkLHLEHZWzegC,272 38 | kSeltTjqBPOcLTvZgsPpPJuFcibkofFySSKgUxybRHYoC,47362 39 | nWvUUBsLGWAyMSeekYLfTEIxasMYzkByqnz,45371 40 | AiUSwcAFjJLgXGYRyqvwebwRdLBiIupMDlxdC,15456 41 | HPnRPrxp,47340 42 | TawyzGEdCbyZOLppFhJFocsfuXElHQZ,65470 43 | QsthbmpJluAHEBDbOYPVVAwnGqFNdQUCJKaA,1747 44 | AEjMHMEYdPluTUkaXPkAJbrVZVwACWJfz,15596 45 | zCChHSupDIvmywrkrObXBhlwYoizoYjRZfh,45598 46 | sVVNcPpNDHlLAuAQovGAuweLhycvwVWoRETUbjqwf,64512 47 | AJSoojXBDLpfWIBEKbmBihXtjdvGcODtZnmgMl,3164 48 | quFLCGDWNhSigZRxps,54176 49 | Q,23821 50 | EGBtOuKagUkgMm,56875 51 | lVtSiBdggGXMstMZXzlFlK,24394 52 | PeptPx,50457 53 | d,5598 54 | JrOHMYozWNroZRez,3494 55 | dPDplciTtkKvSyGJwIfRgVgHNFMxckneHhoGTJ,6263 56 | DEAVdDFeaeAlhUVHQCZGfHQZ,22682 57 | uLXeGaaRASN,2436 58 | hMhTFCsiwPcKqj,440 59 | GsWSUAPeputumirrDSapUPCCjZ,2432 60 | vlDBysLzLXbHmvwCDsqoQsODsMcFJLzpKhsZolPxl,6693 61 | bJwrAwIkmRuh,28414 62 | omhjHwzhtpyVCzEXPYeLo,48558 63 | GgDlsqrPzcGKEv,942 64 | EIutQUZtPkTFcSteEBFWzPWSyM,59120 65 | uiMMcKMpavuUAyQVDhgmQNAozUDIMrv,8218 66 | bJHzKUeV,29685 67 | yxnXQOLTYLbuAKUxVoyzVsDUZLZUX,16196 68 | xalUYLAyHaN,17967 69 | esdwfEsfrunfXDpjjHcNtBDzSVCtDdngrftYgmRqvEe,31389 70 | uFLZzzXnLGMmOi,35363 71 | alrcBTCauDeSFymEKriQptiPTHVlMvjdNqZzPy,17196 72 | ijGPUnchVSPVfALJFrcotMlfNMjkkuc,23906 73 | QEmjwQUJgF,18707 74 | oOCzyi,64742 75 | gNg,64056 76 | WOEdWLm,7878 77 | rAKmSmqLSCaFskrzMFBxUoiJQixNKSgHBsjGbTDqM,22952 78 | rlfmKWyYkRRtqybL,62633 79 | aBwZDATmyVCoPwbTduJiQTSvdOXtKcvdwppsdDlVTc,24586 80 | pQKMGxFjnTkfnEuTovZFAaufyzHSMQxNZdJUdHNkvWGbNzaT,58476 81 | OnHPkLPdZKcuKGNKfrZOzdximUrJOEAZEIrZqZIth,2083 82 | wXaEifQcCPutMlBPheUnGfKECuWcTXQtmiKuNhtcqvx,18175 83 | WHFrpTASnGIkrtATgiTxsflFvLOKJlJiXzKnVuNJw,23796 84 | YzkaqWEB,58674 85 | rZItZQglkZtDCyliMiBlDuLC,46248 86 | gwkAdmhkN,62787 87 | QinknNTmHWPtAGSJEdRBYyFHdpYU,62123 88 | THLYirRvMmzYISlsYSYKcLXSXPpeUQbzBpGbhu,26531 89 | MtIBpoKbtXWeeBHWVNNvtdbQXbFPPeUwZCHLxnucTgWXDiIW,24151 90 | ZDwqnEZLGbUAWMmGtrxm,32697 91 | ObEykVZPTempCxWOZCSxESrzgMlVKF,54394 92 | xxhyTdWRhAkSUnzpfiKtGPnvvgZrgL,36605 93 | gIbhrsYfWfCZYuIgEBXdLFIWwAWXNwmRYZyNwuFLbiXEIJeHJ,45033 94 | lwz,11070 95 | HxjzyZINsmYvaDPgWmJcjXSdDvUHjNNgHXi,62253 96 | knZVwJmlxGmmSRPiSyKWywaVuOQgFbZjtlzHUDe,50432 97 | QcZJOlCpbulMKjROffSUnJTkpckVUpeee,45194 98 | sYrfpEFyIwYkPOoZpOKhM,6132 99 | wHIaUoGNFmijxUGtjPQZuiPmFTTmLZcNSDoAhqPKfhMxSkwO,49052 100 | aVaeEOVXyVSAjvugvUZheQgNXNcqmflm,48403 101 | wgnleOgUkqZ,11211 102 | enyvuDhahUMbvDvGUVWKPk,11211 103 | VychTJxhlKBScqsPj,11211 104 | qYVAnyocdvnZsXPWxyfUGsNRkgRChW,11211 105 | OSdUPyAdjJKIBKbq,11211 106 | ZCmfUdIHHPuDcZTbrURGrfyVjr,11211 107 | yrCQHbtzegvxDcDOgCFRwFEhZwZeXloSrpaaIXW,11211 108 | DZIwGtuYlZiNxUqpaFYzztJftLqMhqSj,11211 109 | HHzWfmumskiGBSlJrstqhQMmSNVfnWcqUlsoCBwio,11211 110 | EBFEikIrQMWfCnYYumZuDtiDLPdlpzrIHOVnQRwR,11211 111 | XTUamIjWLJyMMcd,11211 112 | ycWbgGNqrxnibnybdSjgkzd,11211 113 | fSVJuOrnmbPnFdTBykuusYLY,11211 114 | MjpLZcxrbPZFLMRsMcpHoREHQJKcCQdioIkYrmFjmsUfNx,11211 115 | UhzAKbfErYHtXUzREbDiwpNdunQgJswTVohOAJttBb,11211 116 | DBgyrddLPmpypqFDHOkTJOoeohMPJf,11211 117 | bZiHHWapTEeWDeDAPTyHmkYdxCuLBpUfCwBhnOYotrWSWgf,11211 118 | ErpLeygOtYa,11211 119 | GDtHHDwQSTZEiRxuQIiLSDKDNeSNhHZqsZxDoJjbMkR,11211 120 | rCsYDXmYpsgAjAeRMyFQpqMQRSbMhorWknRnqLHipOrd,11211 121 | PRlFWSMMoVOXOJIpOxXQnqWZMBsqsTxFGUeZtMdRKaP,11211 122 | DXimFdnPHqCMsGCGcllR,11211 123 | XarrkZ,11211 124 | FyESOiRJOWWgTelVpJPxGbsXuSJZzdWzTcMWzJttDlu,11211 125 | FapnKHDLUmkPXIGcrYTSMFtfVCejcZAMOwKNq,11211 126 | pksyvadzPGRNDeLWiWMXlviEuloIyFF,11211 127 | cpmLfErOyqoTkq,11211 128 | KuhFVJdxLnCkhBQbWxljkw,11211 129 | ivxCR,11211 130 | rrKNqOdmfxmDqYkfgvzfcJxiFUuyUfQdGCwDiHoVKOSx,11211 131 | OlgFbzMGRiyvxBdzwLobfyJOPhHdGycNmSIW,11211 132 | GMSCwOiewnRlaPEahj,11211 133 | qKIUVdTydJodoLk,11211 134 | QK,11211 135 | JCcMQZjZIUKrRqrUEbopUmIRDuwSlyskSGLP,11211 136 | vbgHmGGHNTyOVRYMeMwwFhMKWItZAUbygKzwMd,11211 137 | kRFMARyUnzGR,11211 138 | iLqyBU,11211 139 | LSOcGeDwmlKCdW,11211 140 | UrdCOzNETURIsJMgzywfsbLpnJrfura,11211 141 | fCwkBCIiVBWvNCmqASjlDgKxacAIINDUM,11211 142 | YTMCCRaGKVPMzhGJsIbbbHDuPRs,11211 143 | dJUTOEVhoVcIlDoXQwSmqqAXairobWPuzuBrWhuzRH,11211 144 | N,11211 145 | xnBGXYytFMOCOBdUjdShCVZKFZJUfnI,11211 146 | GdZopcUJkzPOAfUfAXQsVrpnKdl,11211 147 | NuJCuCGXOzVpLPssOFMspFaWrzHARxEemOcn,11211 148 | uhISFBmRTktufXwaJfaWwc,11211 149 | TNSUVtCIcOLnugDtmFToaktdsunkfYcljV,11211 150 | pqLfCQuzEdUFIevnRyWhJiYEGWxt,11211 151 | rWuAZEJiGxglYA,11211 152 | gaJrFVLTMiYTPqiPLerDtdYgkzTrTNBJidBMXfGFqlcfLIAJi,11211 153 | YjRJYQBrLcUepmqKORZXHDI,11211 154 | NIKLjSImcjvvYKEQyDoqtybNcfBoviL,11211 155 | mhGaXuQ,11211 156 | jO,11211 157 | WgiTBAzTZSvysgVvaAKsXUiTdtl,11211 158 | aOvIfjDgFnjlVRbhzKQc,11211 159 | ToXgdWYfvqrgsGOFXmhjdFeoSzuZdWBwXgKsGTkv,11211 160 | LuqfdPGIAwWyVHn,11211 161 | HfZth,11211 162 | ufxRoPtGbZYlTdKabJwMxzB,11211 163 | ZRuZkbyirGSnBzbubhXhyYeuQWqskTxHqLimgaJoAGtmd,11211 164 | sebKYXAebHgyLgWKnTSHfRrjNLIzxTOu,11211 165 | MYUhCAMOLSHuSzWvGPTGvYlc,11211 166 | dHEHOBFjYjV,11211 167 | KrjiLg,11211 168 | EvTzfNkM,11211 169 | HtiSwgFiWUnHnDpklyfIyFnyfSuwqHmZStSUsjcwodNumc,11211 170 | wARAQfCPhkQskGsHOsnJTjULDsHvdHgAVJozpEwqDAHc,11211 171 | tWCOkCBGYfOoXbrTgwekNKDhZevfyrRlRDdgg,11211 172 | KeBOrsFhexVPwjWQyPDcYDbWjnpgOLcMTURPMEbIgwR,11211 173 | xhQwJhSJwhpcvgftibK,11211 174 | EYVRqnUAkLmYmXLbOurfkMkGVCKnueGaDMGuMxsPo,11211 175 | aIDULhyjaCuI,11211 176 | ecfQoikyCVQerzjRgEnRxrLVHMvsQmrpVEPYjSTL,11211 177 | rMVbneBPwWxbBWTTZLlmunkWRSfTgJQOpcwrqs,11211 178 | jpxucRkQPaUxctjpEiWNNZP,11211 179 | tNdzPuLcNiArKamqzAisOlCaSgSoS,11211 180 | H,11211 181 | NETXwosIiOYcUdKnCehaMG,11211 182 | NWDFJJxoEEUXRITesOfwvEZcmIDwoWs,11211 183 | YxWtZWrmicHiOUZYxeCdonuLsqB,11211 184 | uxZigpIfPGujVDow,11211 185 | MShbXGVXRvXlwvOYXvrCvrFGypgNyMuCignDPbYntDsatUjxPJ,11211 186 | pKclRBjmJgequiWaOgvIUpsgOLVeMhhPYHxXKHislVafawoR,11211 187 | IaygsbWvMVEBcQcZQd,11211 188 | Ts,11211 189 | bktznywQAhlyJBnSzw,11211 190 | ItXBFpdC,11211 191 | FTAiP,11211 192 | ttoHrxjCrsqpuFZpIkQhjnzydhwdwQUtOaWLffLpNL,11211 193 | wcHrGXDgfLhsYrWPwYIcSEULSLqIQSaPGbATzFKJfJNbnLXn,11211 194 | wRPAZwPnxIqqOLPVO,11211 195 | jEdiiLTclHbKqriBPRpZIKOdBDsk,11211 196 | bRKzjQSUtZXYZBhefBlo,11211 197 | ZPItRfqyknQCQrWfFgfIRLOKOPsYfuYASrKQ,11211 198 | lGBtGNvcaoWnkspuUQeEsZqtaJxlpDyTCpNo,11211 199 | xcAmcnnHccschvIYmvYw,11211 200 | bRsbiFQnbjsyvjerecvGgsocFAnPcRMULYdHybEKLOul,11211 201 | -------------------------------------------------------------------------------- /src/Client.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | #include "Common.h" 5 | #include "Client.h" 6 | #include "Keywords.h" 7 | 8 | namespace douban { 9 | namespace mc { 10 | 11 | Client::Client(): m_flushAllEnabled(false) { 12 | } 13 | 14 | 15 | Client::~Client() { 16 | } 17 | 18 | 19 | void Client::config(config_options_t opt, int val) { 20 | switch (opt) { 21 | case CFG_POLL_TIMEOUT: 22 | setPollTimeout(val); 23 | break; 24 | case CFG_CONNECT_TIMEOUT: 25 | setConnectTimeout(val); 26 | break; 27 | case CFG_RETRY_TIMEOUT: 28 | setRetryTimeout(val); 29 | break; 30 | case CFG_HASH_FUNCTION: 31 | ConnectionPool::setHashFunction(static_cast(val)); 32 | break; 33 | case CFG_MAX_RETRIES: 34 | setMaxRetries(val); 35 | break; 36 | case CFG_SET_FAILOVER: 37 | assert(val == 0 || val == 1); 38 | if (val == 0) { 39 | disableConsistentFailover(); 40 | } else { 41 | enableConsistentFailover(); 42 | } 43 | default: 44 | break; 45 | } 46 | } 47 | 48 | 49 | err_code_t Client::get(const char* const* keys, const size_t* keyLens, size_t nKeys, 50 | retrieval_result_t*** results, size_t* nResults) { 51 | dispatchRetrieval(GET_OP, keys, keyLens, nKeys); 52 | err_code_t rv = waitPoll(); 53 | collectRetrievalResult(results, nResults); 54 | return rv; 55 | } 56 | 57 | 58 | err_code_t Client::gets(const char* const* keys, const size_t* keyLens, size_t nKeys, 59 | retrieval_result_t*** results, size_t* nResults) { 60 | dispatchRetrieval(GETS_OP, keys, keyLens, nKeys); 61 | err_code_t rv = waitPoll(); 62 | collectRetrievalResult(results, nResults); 63 | return rv; 64 | } 65 | 66 | 67 | void Client::collectRetrievalResult(retrieval_result_t*** results, size_t* nResults) { 68 | assert(m_outRetrievalResultPtrs.empty()); 69 | ConnectionPool::collectRetrievalResult(m_outRetrievalResultPtrs); 70 | *nResults = m_outRetrievalResultPtrs.size(); 71 | if (*nResults == 0) { 72 | *results = NULL; 73 | } else { 74 | *results = &m_outRetrievalResultPtrs.front(); 75 | } 76 | } 77 | 78 | 79 | void Client::destroyRetrievalResult() { 80 | ConnectionPool::reset(); 81 | m_outRetrievalResultPtrs.clear(); 82 | } 83 | 84 | 85 | void Client::collectMessageResult(message_result_t*** results, size_t* nResults) { 86 | assert(m_outMessageResultPtrs.empty()); 87 | ConnectionPool::collectMessageResult(m_outMessageResultPtrs); 88 | *nResults = m_outMessageResultPtrs.size(); 89 | 90 | if (*nResults == 0) { 91 | *results = NULL; 92 | } else { 93 | *results = &m_outMessageResultPtrs.front(); 94 | } 95 | } 96 | 97 | 98 | void Client::destroyMessageResult() { 99 | ConnectionPool::reset(); 100 | m_outMessageResultPtrs.clear(); 101 | } 102 | 103 | 104 | #define IMPL_STORAGE_CMD(M, O) \ 105 | err_code_t Client::M(const char* const* keys, const size_t* keyLens, \ 106 | const flags_t* flags, const exptime_t exptime, \ 107 | const cas_unique_t* cas_uniques, const bool noreply, \ 108 | const char* const* vals, const size_t* valLens, \ 109 | size_t nItems, message_result_t*** results, size_t* nResults) { \ 110 | dispatchStorage((O), keys, keyLens, flags, exptime, cas_uniques, noreply, vals, \ 111 | valLens, nItems); \ 112 | err_code_t rv = waitPoll(); \ 113 | collectMessageResult(results, nResults); \ 114 | return rv;\ 115 | } 116 | 117 | IMPL_STORAGE_CMD(set, SET_OP) 118 | IMPL_STORAGE_CMD(add, ADD_OP) 119 | IMPL_STORAGE_CMD(replace, REPLACE_OP) 120 | IMPL_STORAGE_CMD(append, APPEND_OP) 121 | IMPL_STORAGE_CMD(prepend, PREPEND_OP) 122 | IMPL_STORAGE_CMD(cas, CAS_OP) 123 | #undef IMPL_STORAGE_CMD 124 | 125 | err_code_t Client::_delete(const char* const* keys, const size_t* keyLens, 126 | const bool noreply, size_t nItems, 127 | message_result_t*** results, size_t* nResults) { 128 | dispatchDeletion(keys, keyLens, noreply, nItems); 129 | err_code_t rv = waitPoll(); 130 | collectMessageResult(results, nResults); 131 | return rv; 132 | } 133 | 134 | 135 | void Client::collectBroadcastResult(broadcast_result_t** results, size_t* nHosts, bool isFlushAll) { 136 | assert(m_outBroadcastResultPtrs.empty()); 137 | *nHosts = m_nConns; 138 | ConnectionPool::collectBroadcastResult(m_outBroadcastResultPtrs, isFlushAll); 139 | *results = &m_outBroadcastResultPtrs.front(); 140 | } 141 | 142 | 143 | void Client::destroyBroadcastResult() { 144 | ConnectionPool::reset(); 145 | for (std::vector::iterator it = m_outBroadcastResultPtrs.begin(); 146 | it != m_outBroadcastResultPtrs.end(); ++it) { 147 | types::delete_broadcast_result(&(*it)); 148 | } 149 | m_outBroadcastResultPtrs.clear(); 150 | } 151 | 152 | 153 | err_code_t Client::version(broadcast_result_t** results, size_t* nHosts) { 154 | broadcastCommand(keywords::kVERSION, 7); 155 | err_code_t rv = waitPoll(); 156 | collectBroadcastResult(results, nHosts); 157 | return rv; 158 | } 159 | 160 | 161 | err_code_t Client::quit() { 162 | broadcastCommand(keywords::kQUIT, 4, true); 163 | err_code_t rv = waitPoll(); 164 | markDeadAll(NULL, keywords::kCONN_QUIT); 165 | return rv; 166 | } 167 | 168 | 169 | err_code_t Client::stats(broadcast_result_t** results, size_t* nHosts) { 170 | broadcastCommand(keywords::kSTATS, 5); 171 | err_code_t rv = waitPoll(); 172 | collectBroadcastResult(results, nHosts); 173 | return rv; 174 | } 175 | 176 | err_code_t Client::flushAll(broadcast_result_t** results, size_t* nHosts) { 177 | if (!m_flushAllEnabled) { 178 | *results = NULL; 179 | *nHosts = 0; 180 | return RET_PROGRAMMING_ERR; 181 | } 182 | 183 | broadcastCommand(keywords::kFLUSHALL, 9); 184 | err_code_t rv = waitPoll(); 185 | collectBroadcastResult(results, nHosts, true); 186 | return rv; 187 | } 188 | 189 | 190 | err_code_t Client::touch(const char* const* keys, const size_t* keyLens, 191 | const exptime_t exptime, const bool noreply, size_t nItems, 192 | message_result_t*** results, size_t* nResults) { 193 | dispatchTouch(keys, keyLens, exptime, noreply, nItems); 194 | err_code_t rv = waitPoll(); 195 | collectMessageResult(results, nResults); 196 | return rv; 197 | } 198 | 199 | 200 | void Client::collectUnsignedResult(unsigned_result_t** results, size_t* nResults) { 201 | 202 | assert(m_outUnsignedResultPtrs.empty()); 203 | ConnectionPool::collectUnsignedResult(m_outUnsignedResultPtrs); 204 | *nResults = m_outUnsignedResultPtrs.size(); 205 | 206 | if (*nResults == 0) { 207 | *results = NULL; 208 | } else { 209 | *results = m_outUnsignedResultPtrs.front(); 210 | } 211 | } 212 | 213 | err_code_t Client::incr(const char* key, const size_t keyLen, const uint64_t delta, 214 | const bool noreply, 215 | unsigned_result_t** results, size_t* nResults) { 216 | dispatchIncrDecr(INCR_OP, key, keyLen, delta, noreply); 217 | err_code_t rv = waitPoll(); 218 | collectUnsignedResult(results, nResults); 219 | return rv; 220 | } 221 | 222 | 223 | err_code_t Client::decr(const char* key, const size_t keyLen, const uint64_t delta, 224 | const bool noreply, 225 | unsigned_result_t** results, size_t* nResults) { 226 | dispatchIncrDecr(DECR_OP, key, keyLen, delta, noreply); 227 | err_code_t rv = waitPoll(); 228 | collectUnsignedResult(results, nResults); 229 | return rv; 230 | } 231 | 232 | 233 | void Client::destroyUnsignedResult() { 234 | ConnectionPool::reset(); 235 | m_outUnsignedResultPtrs.clear(); 236 | } 237 | 238 | 239 | void Client::_sleep(uint32_t seconds) { 240 | usleep(seconds * 1000000); 241 | } 242 | 243 | } // namespace mc 244 | } // namespace douban 245 | -------------------------------------------------------------------------------- /src/HashkitMd5.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | * RFC 1321 compliant MD5 implementation 3 | * 4 | * Copyright (C) 2006-2013, Brainspark B.V. 5 | * 6 | * This file is part of PolarSSL (http://www.polarssl.org) 7 | * Lead Maintainer: Paul Bakker 8 | * 9 | * All rights reserved. 10 | * 11 | * This program is free software; you can redistribute it and/or modify 12 | * it under the terms of the GNU General Public License as published by 13 | * the Free Software Foundation; either version 2 of the License, or 14 | * (at your option) any later version. 15 | * 16 | * This program is distributed in the hope that it will be useful, 17 | * but WITHOUT ANY WARRANTY; without even the implied warranty of 18 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 19 | * GNU General Public License for more details. 20 | * 21 | * You should have received a copy of the GNU General Public License along 22 | * with this program; if not, write to the Free Software Foundation, Inc., 23 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 24 | */ 25 | /* 26 | * The MD5 algorithm was designed by Ron Rivest in 1991. 27 | * 28 | * http://www.ietf.org/rfc/rfc1321.txt 29 | */ 30 | 31 | #include "hashkit/md5.h" 32 | #include "hashkit/hashkit.h" 33 | #include 34 | 35 | namespace douban { 36 | namespace mc { 37 | namespace hashkit { 38 | 39 | /* 40 | * 32-bit integer manipulation macros (little endian) 41 | */ 42 | #ifndef GET_UINT32_LE 43 | #define GET_UINT32_LE(n,b,i) \ 44 | { \ 45 | (n) = ( (uint32_t) (b)[(i) ] ) \ 46 | | ( (uint32_t) (b)[(i) + 1] << 8 ) \ 47 | | ( (uint32_t) (b)[(i) + 2] << 16 ) \ 48 | | ( (uint32_t) (b)[(i) + 3] << 24 ); \ 49 | } 50 | #endif 51 | 52 | #ifndef PUT_UINT32_LE 53 | #define PUT_UINT32_LE(n,b,i) \ 54 | { \ 55 | (b)[(i) ] = (unsigned char) ( (n) ); \ 56 | (b)[(i) + 1] = (unsigned char) ( (n) >> 8 ); \ 57 | (b)[(i) + 2] = (unsigned char) ( (n) >> 16 ); \ 58 | (b)[(i) + 3] = (unsigned char) ( (n) >> 24 ); \ 59 | } 60 | #endif 61 | 62 | /* 63 | * MD5 context setup 64 | */ 65 | void md5_starts( md5_context *ctx ) 66 | { 67 | ctx->total[0] = 0; 68 | ctx->total[1] = 0; 69 | 70 | ctx->state[0] = 0x67452301; 71 | ctx->state[1] = 0xEFCDAB89; 72 | ctx->state[2] = 0x98BADCFE; 73 | ctx->state[3] = 0x10325476; 74 | } 75 | 76 | void md5_process( md5_context *ctx, const unsigned char data[64] ) 77 | { 78 | uint32_t X[16], A, B, C, D; 79 | 80 | GET_UINT32_LE( X[ 0], data, 0 ); 81 | GET_UINT32_LE( X[ 1], data, 4 ); 82 | GET_UINT32_LE( X[ 2], data, 8 ); 83 | GET_UINT32_LE( X[ 3], data, 12 ); 84 | GET_UINT32_LE( X[ 4], data, 16 ); 85 | GET_UINT32_LE( X[ 5], data, 20 ); 86 | GET_UINT32_LE( X[ 6], data, 24 ); 87 | GET_UINT32_LE( X[ 7], data, 28 ); 88 | GET_UINT32_LE( X[ 8], data, 32 ); 89 | GET_UINT32_LE( X[ 9], data, 36 ); 90 | GET_UINT32_LE( X[10], data, 40 ); 91 | GET_UINT32_LE( X[11], data, 44 ); 92 | GET_UINT32_LE( X[12], data, 48 ); 93 | GET_UINT32_LE( X[13], data, 52 ); 94 | GET_UINT32_LE( X[14], data, 56 ); 95 | GET_UINT32_LE( X[15], data, 60 ); 96 | 97 | #define S(x,n) ((x << n) | ((x & 0xFFFFFFFF) >> (32 - n))) 98 | 99 | #define P(a,b,c,d,k,s,t) \ 100 | { \ 101 | a += F(b,c,d) + X[k] + t; a = S(a,s) + b; \ 102 | } 103 | 104 | A = ctx->state[0]; 105 | B = ctx->state[1]; 106 | C = ctx->state[2]; 107 | D = ctx->state[3]; 108 | 109 | #define F(x,y,z) (z ^ (x & (y ^ z))) 110 | 111 | P( A, B, C, D, 0, 7, 0xD76AA478 ); 112 | P( D, A, B, C, 1, 12, 0xE8C7B756 ); 113 | P( C, D, A, B, 2, 17, 0x242070DB ); 114 | P( B, C, D, A, 3, 22, 0xC1BDCEEE ); 115 | P( A, B, C, D, 4, 7, 0xF57C0FAF ); 116 | P( D, A, B, C, 5, 12, 0x4787C62A ); 117 | P( C, D, A, B, 6, 17, 0xA8304613 ); 118 | P( B, C, D, A, 7, 22, 0xFD469501 ); 119 | P( A, B, C, D, 8, 7, 0x698098D8 ); 120 | P( D, A, B, C, 9, 12, 0x8B44F7AF ); 121 | P( C, D, A, B, 10, 17, 0xFFFF5BB1 ); 122 | P( B, C, D, A, 11, 22, 0x895CD7BE ); 123 | P( A, B, C, D, 12, 7, 0x6B901122 ); 124 | P( D, A, B, C, 13, 12, 0xFD987193 ); 125 | P( C, D, A, B, 14, 17, 0xA679438E ); 126 | P( B, C, D, A, 15, 22, 0x49B40821 ); 127 | 128 | #undef F 129 | 130 | #define F(x,y,z) (y ^ (z & (x ^ y))) 131 | 132 | P( A, B, C, D, 1, 5, 0xF61E2562 ); 133 | P( D, A, B, C, 6, 9, 0xC040B340 ); 134 | P( C, D, A, B, 11, 14, 0x265E5A51 ); 135 | P( B, C, D, A, 0, 20, 0xE9B6C7AA ); 136 | P( A, B, C, D, 5, 5, 0xD62F105D ); 137 | P( D, A, B, C, 10, 9, 0x02441453 ); 138 | P( C, D, A, B, 15, 14, 0xD8A1E681 ); 139 | P( B, C, D, A, 4, 20, 0xE7D3FBC8 ); 140 | P( A, B, C, D, 9, 5, 0x21E1CDE6 ); 141 | P( D, A, B, C, 14, 9, 0xC33707D6 ); 142 | P( C, D, A, B, 3, 14, 0xF4D50D87 ); 143 | P( B, C, D, A, 8, 20, 0x455A14ED ); 144 | P( A, B, C, D, 13, 5, 0xA9E3E905 ); 145 | P( D, A, B, C, 2, 9, 0xFCEFA3F8 ); 146 | P( C, D, A, B, 7, 14, 0x676F02D9 ); 147 | P( B, C, D, A, 12, 20, 0x8D2A4C8A ); 148 | 149 | #undef F 150 | 151 | #define F(x,y,z) (x ^ y ^ z) 152 | 153 | P( A, B, C, D, 5, 4, 0xFFFA3942 ); 154 | P( D, A, B, C, 8, 11, 0x8771F681 ); 155 | P( C, D, A, B, 11, 16, 0x6D9D6122 ); 156 | P( B, C, D, A, 14, 23, 0xFDE5380C ); 157 | P( A, B, C, D, 1, 4, 0xA4BEEA44 ); 158 | P( D, A, B, C, 4, 11, 0x4BDECFA9 ); 159 | P( C, D, A, B, 7, 16, 0xF6BB4B60 ); 160 | P( B, C, D, A, 10, 23, 0xBEBFBC70 ); 161 | P( A, B, C, D, 13, 4, 0x289B7EC6 ); 162 | P( D, A, B, C, 0, 11, 0xEAA127FA ); 163 | P( C, D, A, B, 3, 16, 0xD4EF3085 ); 164 | P( B, C, D, A, 6, 23, 0x04881D05 ); 165 | P( A, B, C, D, 9, 4, 0xD9D4D039 ); 166 | P( D, A, B, C, 12, 11, 0xE6DB99E5 ); 167 | P( C, D, A, B, 15, 16, 0x1FA27CF8 ); 168 | P( B, C, D, A, 2, 23, 0xC4AC5665 ); 169 | 170 | #undef F 171 | 172 | #define F(x,y,z) (y ^ (x | ~z)) 173 | 174 | P( A, B, C, D, 0, 6, 0xF4292244 ); 175 | P( D, A, B, C, 7, 10, 0x432AFF97 ); 176 | P( C, D, A, B, 14, 15, 0xAB9423A7 ); 177 | P( B, C, D, A, 5, 21, 0xFC93A039 ); 178 | P( A, B, C, D, 12, 6, 0x655B59C3 ); 179 | P( D, A, B, C, 3, 10, 0x8F0CCC92 ); 180 | P( C, D, A, B, 10, 15, 0xFFEFF47D ); 181 | P( B, C, D, A, 1, 21, 0x85845DD1 ); 182 | P( A, B, C, D, 8, 6, 0x6FA87E4F ); 183 | P( D, A, B, C, 15, 10, 0xFE2CE6E0 ); 184 | P( C, D, A, B, 6, 15, 0xA3014314 ); 185 | P( B, C, D, A, 13, 21, 0x4E0811A1 ); 186 | P( A, B, C, D, 4, 6, 0xF7537E82 ); 187 | P( D, A, B, C, 11, 10, 0xBD3AF235 ); 188 | P( C, D, A, B, 2, 15, 0x2AD7D2BB ); 189 | P( B, C, D, A, 9, 21, 0xEB86D391 ); 190 | 191 | #undef F 192 | 193 | ctx->state[0] += A; 194 | ctx->state[1] += B; 195 | ctx->state[2] += C; 196 | ctx->state[3] += D; 197 | } 198 | 199 | /* 200 | * MD5 process buffer 201 | */ 202 | void md5_update( md5_context *ctx, const unsigned char *input, size_t ilen ) 203 | { 204 | size_t fill; 205 | uint32_t left; 206 | 207 | if( ilen == 0 ) 208 | return; 209 | 210 | left = ctx->total[0] & 0x3F; 211 | fill = 64 - left; 212 | 213 | ctx->total[0] += (uint32_t) ilen; 214 | ctx->total[0] &= 0xFFFFFFFF; 215 | 216 | if( ctx->total[0] < (uint32_t) ilen ) 217 | ctx->total[1]++; 218 | 219 | if( left && ilen >= fill ) 220 | { 221 | memcpy( (void *) (ctx->buffer + left), input, fill ); 222 | md5_process( ctx, ctx->buffer ); 223 | input += fill; 224 | ilen -= fill; 225 | left = 0; 226 | } 227 | 228 | while( ilen >= 64 ) 229 | { 230 | md5_process( ctx, input ); 231 | input += 64; 232 | ilen -= 64; 233 | } 234 | 235 | if( ilen > 0 ) 236 | { 237 | memcpy( (void *) (ctx->buffer + left), input, ilen ); 238 | } 239 | } 240 | 241 | static const unsigned char md5_padding[64] = 242 | { 243 | 0x80, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 244 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 245 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 246 | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 247 | }; 248 | 249 | /* 250 | * MD5 final digest 251 | */ 252 | void md5_finish( md5_context *ctx, unsigned char output[16] ) 253 | { 254 | uint32_t last, padn; 255 | uint32_t high, low; 256 | unsigned char msglen[8]; 257 | 258 | high = ( ctx->total[0] >> 29 ) 259 | | ( ctx->total[1] << 3 ); 260 | low = ( ctx->total[0] << 3 ); 261 | 262 | PUT_UINT32_LE( low, msglen, 0 ); 263 | PUT_UINT32_LE( high, msglen, 4 ); 264 | 265 | last = ctx->total[0] & 0x3F; 266 | padn = ( last < 56 ) ? ( 56 - last ) : ( 120 - last ); 267 | 268 | md5_update( ctx, md5_padding, padn ); 269 | md5_update( ctx, msglen, 8 ); 270 | 271 | PUT_UINT32_LE( ctx->state[0], output, 0 ); 272 | PUT_UINT32_LE( ctx->state[1], output, 4 ); 273 | PUT_UINT32_LE( ctx->state[2], output, 8 ); 274 | PUT_UINT32_LE( ctx->state[3], output, 12 ); 275 | } 276 | 277 | /* 278 | * output = MD5( input buffer ) 279 | */ 280 | void md5( const unsigned char *input, size_t ilen, unsigned char output[16] ) 281 | { 282 | md5_context ctx; 283 | 284 | md5_starts( &ctx ); 285 | md5_update( &ctx, input, ilen ); 286 | md5_finish( &ctx, output ); 287 | 288 | memset( &ctx, 0, sizeof( md5_context ) ); 289 | } 290 | 291 | 292 | uint32_t hash_md5(const char *key, size_t key_length) 293 | { 294 | unsigned char results[16]; 295 | 296 | md5((unsigned char*)key, (size_t)key_length, results); 297 | 298 | return ((uint32_t) (results[3] & 0xFF) << 24) 299 | | ((uint32_t) (results[2] & 0xFF) << 16) 300 | | ((uint32_t) (results[1] & 0xFF) << 8) 301 | | (results[0] & 0xFF); 302 | } 303 | 304 | 305 | } // namespace hashkit 306 | } // namespace mc 307 | } // namespace douban 308 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | libmc 2 | ===== 3 | 4 | |build_go| |build_py| 5 | |status| |pypiv| |pyversions| |wheel| |license| 6 | 7 | libmc is a memcached client library for Python without any other 8 | dependencies at runtime. It's mainly written in C++ and Cython and 9 | can be considered a drop in replacement for libmemcached and 10 | `python-libmemcached `__. 11 | 12 | libmc is developed and maintained by Douban Inc. Currently, it is 13 | working in a production environment, powering all web traffic on 14 | `douban.com `__ 15 | (`english wiki `__). 16 | 17 | Build and Installation 18 | ---------------------- 19 | 20 | For users: 21 | 22 | :: 23 | 24 | pip install libmc 25 | 26 | Usage: 27 | 28 | .. code:: python 29 | 30 | import libmc 31 | 32 | mc = libmc.Client(['localhost:11211', 'localhost:11212']) 33 | mc.set('foo', 'bar') 34 | assert mc.get('foo') == 'bar' 35 | 36 | Under the hood 37 | -------------- 38 | 39 | Under the hood, libmc consists of 2 parts: an internal, fully-functional 40 | memcached client implementation in C++ and a Cython wrapper around that 41 | implementation. Dynamic memory allocation and memory-copy are slow, so 42 | we've tried our best to avoid them. libmc also supports the ``set_multi`` 43 | command, which is not natively supported by the `memcached 44 | protocol `__. 45 | Some techniques have been applied to make ``set_multi`` command extremely fast 46 | in libmc (compared to similiar libraries). 47 | 48 | Configuration 49 | ------------- 50 | 51 | .. code:: python 52 | 53 | import libmc 54 | from libmc import ( 55 | MC_HASH_MD5, MC_POLL_TIMEOUT, MC_CONNECT_TIMEOUT, MC_RETRY_TIMEOUT 56 | ) 57 | 58 | mc = libmc.Client( 59 | [ 60 | 'localhost:11211', 61 | 'localhost:11212', 62 | 'remote_host', 63 | 'remote_host mc.mike', 64 | 'remote_host:11213 mc.oscar' 65 | ], 66 | do_split=True, 67 | comp_threshold=0, 68 | noreply=False, 69 | prefix=None, 70 | hash_fn=MC_HASH_MD5, 71 | failover=False 72 | ) 73 | 74 | mc.config(MC_POLL_TIMEOUT, 100) # 100 ms 75 | mc.config(MC_CONNECT_TIMEOUT, 300) # 300 ms 76 | mc.config(MC_RETRY_TIMEOUT, 5) # 5 s 77 | 78 | 79 | - ``servers``: a list of memcached server addresses. Each address 80 | should be formated as ``hostname[:port] [alias]``, where ``port`` and 81 | ``alias`` are optional. If ``port`` is not given, the default port ``11211`` 82 | will be used. If given, ``alias`` will be used to compute the server hash, 83 | which would otherwise be computed based on ``host`` and ``port`` 84 | (i.e. whichever portion is given). 85 | - ``do_split``: splits large values (up to 10MB) into chunks (<1MB). The 86 | memcached server implementation will not store items larger than 1MB, 87 | however in some environments it is beneficial to shard up to 10MB of data. 88 | Attempts to store more than that are ignored. Default: ``True``. 89 | - ``comp_threshold``: compresses large values using zlib. If 90 | ``buffer length > comp_threshold > 0`` (in bytes), the buffer will be 91 | compressed. If ``comp_threshold == 0``, the string buffer will never be 92 | compressed. Default: ``0`` 93 | - ``noreply``: controls memcached's 94 | ``noreply`` `feature `__. 95 | Default: ``False`` 96 | - ``prefix``: The key prefix. default: ``''`` 97 | - ``hash_fn``: hashing function for keys. possible values: 98 | 99 | - ``MC_HASH_MD5`` 100 | - ``MC_HASH_FNV1_32`` 101 | - ``MC_HASH_FNV1A_32`` 102 | - ``MC_HASH_CRC_32`` 103 | 104 | default: ``MC_HASH_MD5`` 105 | 106 | **NOTE:** fnv1\_32, fnv1a\_32, crc\_32 implementations in libmc are 107 | per each spec, but they're not compatible with corresponding 108 | implementions in libmemcached. 109 | 110 | - ``failover``: Whether to failover to next server when current server 111 | is not available. Default: ``False`` 112 | 113 | - ``MC_POLL_TIMEOUT`` Timeout parameter used during set/get procedure. 114 | Default: ``300`` ms 115 | - ``MC_CONNECT_TIMEOUT`` Timeout parameter used when connecting to 116 | memcached server in the initial phase. Default: ``100`` ms 117 | - ``MC_RETRY_TIMEOUT`` When a server is not available due to server-end 118 | error, libmc will try to establish the broken connection in every 119 | ``MC_RETRY_TIMEOUT`` s until the connection is back to live. Default: 120 | ``5`` s 121 | 122 | **NOTE:** The hashing algorithm for host mapping on continuum is always 123 | md5. 124 | 125 | Contributing to libmc 126 | --------------------- 127 | 128 | Feel free to send a **Pull Request**. For feature requests or any 129 | questions, please open an **Issue**. 130 | 131 | For **SECURITY DISCLOSURE**, please disclose the information responsibly 132 | by sending an email to security@douban.com directly instead of creating 133 | a GitHub issue. 134 | 135 | FAQ 136 | --- 137 | 138 | Does libmc support PHP? 139 | ^^^^^^^^^^^^^^^^^^^^^^^ 140 | 141 | No, but, if you like, you can write a wrapper for PHP based on the C++ 142 | implementation. 143 | 144 | Is Memcached binary protocol supported ? 145 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 146 | 147 | No. Only Memcached ASCII protocol is supported currently. 148 | 149 | Why reinventing the wheel? 150 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ 151 | 152 | Before libmc, we were using 153 | `python-libmemcached `__, 154 | which is a python extention for 155 | `libmemcached `__. 156 | libmemcached is quite weird and buggy. After nearly one decade, there're 157 | still some unsolved bugs. 158 | 159 | Is libmc thread-safe ? 160 | ^^^^^^^^^^^^^^^^^^^^^^ 161 | 162 | Yes. ``libmc.ThreadedClient`` is a thread-safe client implementation. To hold 163 | access for more than one request, ``libmc.ClientPool`` can be used with Python 164 | ``with`` statements. ``libmc.Client``, however, is a single-threaded memcached 165 | client. If you initialize a standard client in one thread but reuse that in 166 | another thread, a Python ``ThreadUnsafe`` Exception will be raised. 167 | 168 | Is libmc compatible with gevent? 169 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 170 | 171 | Yes, with the help of `greenify `__, 172 | libmc is friendly to gevent. Read ``tests/shabby/gevent_issue.py`` for 173 | details. ``libmc.ThreadedClient`` and ``libmc.ClientPool`` are not compatible. 174 | [#]_ 175 | 176 | **Notice:** 177 | 178 | ``gevent.monkey.patch_all()`` will override 179 | ``threading.current_thread().ident`` to Greenlet's ID, 180 | this will cause libmc to throw a ThreadUnSafe error 181 | or run into dead lock, you should only patch the things 182 | that you need, e.g. 183 | 184 | .. code:: python 185 | 186 | from gevent import monkey 187 | monkey.patch_socket() 188 | 189 | Acknowledgments 190 | --------------- 191 | 192 | - Thanks to `@fahrenheit2539 `__ and 193 | the llvm project for the standalone. 194 | `SmallVector `__ 195 | implementation. 196 | - Thanks to `@miloyip `__ for the high 197 | performance `i64toa `__ 198 | implementation. 199 | - Thanks to `Ivan Novikov `__ for the 200 | research in `THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED 201 | INJECTIONS `__. 202 | - Thanks to the PolarSSL project for the md5 implementation. 203 | - Thanks to `@lericson `__ for the `benchmark 204 | script in 205 | pylibmc `__. 206 | - Thanks to the libmemcached project and some other projects possibly 207 | not mentioned here. 208 | 209 | Contributors 210 | ------------ 211 | 212 | - `@mckelvin `__ 213 | - `@zzl0 `__ 214 | - `@windreamer `__ 215 | - `@lembacon `__ 216 | - `@seansay `__ 217 | - `@mosasiru `__ 218 | - `@jumpeiMano `__ 219 | 220 | 221 | Who is using 222 | ------------ 223 | 224 | - `豆瓣 `__ 225 | - `下厨房 `__ 226 | - `Some other projects on GitHub `__ 227 | - Want to add your company/organization name here? 228 | Please feel free to send a PR! 229 | 230 | Documentation 231 | ------------- 232 | 233 | https://github.com/douban/libmc/wiki 234 | 235 | Footnotes 236 | --------- 237 | 238 | .. [#] In order to use a single executable for multiple greenlet contexts, 239 | gevent has to `copy thread memory 240 | `__ 241 | to and from the same stack space. This doesn't affect Python references, 242 | which are handed off through gevent, but makes it impossible for shared 243 | libraries to pass memory addresses across greenlets, which is required for 244 | the worker pool. 245 | 246 | 247 | LICENSE 248 | ------- 249 | 250 | Copyright (c) 2014-2020, Douban Inc. All rights reserved. 251 | 252 | Licensed under a BSD license: 253 | https://github.com/douban/libmc/blob/master/LICENSE.txt 254 | 255 | .. |build_go| image:: https://github.com/douban/libmc/actions/workflows/golang.yml/badge.svg 256 | :target: https://github.com/douban/libmc/actions/workflows/golang.yml 257 | 258 | .. |build_py| image:: https://github.com/douban/libmc/actions/workflows/python.yml/badge.svg 259 | :target: https://github.com/douban/libmc/actions/workflows/python.yml 260 | 261 | .. |pypiv| image:: https://img.shields.io/pypi/v/libmc 262 | :target: https://pypi.org/project/libmc/ 263 | 264 | .. |status| image:: https://img.shields.io/pypi/status/libmc 265 | .. |pyversions| image:: https://img.shields.io/pypi/pyversions/libmc 266 | .. |wheel| image:: https://img.shields.io/pypi/wheel/libmc 267 | .. |license| image:: https://img.shields.io/pypi/l/libmc?color=blue 268 | --------------------------------------------------------------------------------