├── .gitignore ├── .gitmodules ├── .hgignore ├── .hgtags ├── .runner ├── .travis.sh ├── .travis.yml ├── LICENSE ├── MANIFEST.in ├── Makefile ├── NOTICE ├── README.md ├── RELNOTES.md ├── build ├── publish └── pyenv-setup ├── commands.py ├── docsrc ├── Makefile ├── _templates │ └── layout.html ├── advanced.rst ├── bucket.rst ├── client.rst ├── conf.py ├── datatypes.rst ├── index.rst ├── make.bat ├── object.rst ├── query.rst └── security.rst ├── make.ps1 ├── riak ├── __init__.py ├── benchmark.py ├── benchmarks │ ├── multiget.py │ └── timeseries.py ├── bucket.py ├── client │ ├── __init__.py │ ├── index_page.py │ ├── multi.py │ ├── operations.py │ └── transport.py ├── codecs │ ├── __init__.py │ ├── http.py │ ├── pbuf.py │ ├── ttb.py │ └── util.py ├── content.py ├── datatypes │ ├── __init__.py │ ├── counter.py │ ├── datatype.py │ ├── errors.py │ ├── flag.py │ ├── hll.py │ ├── map.py │ ├── register.py │ ├── set.py │ └── types.py ├── erl_src │ ├── riak_kv_test_backend.beam │ ├── riak_kv_test_backend.erl │ ├── riak_search_test_backend.beam │ └── riak_search_test_backend.erl ├── mapreduce.py ├── multidict.py ├── node.py ├── pb │ ├── __init__.py │ ├── messages.py │ ├── riak_dt_pb2.py │ ├── riak_kv_pb2.py │ ├── riak_pb2.py │ ├── riak_search_pb2.py │ ├── riak_ts_pb2.py │ └── riak_yokozuna_pb2.py ├── resolver.py ├── riak_error.py ├── riak_object.py ├── security.py ├── table.py ├── test_server.py ├── tests │ ├── __init__.py │ ├── base.py │ ├── comparison.py │ ├── pool-grinder.py │ ├── suite.py │ ├── test_2i.py │ ├── test_btypes.py │ ├── test_client.py │ ├── test_comparison.py │ ├── test_datatypes.py │ ├── test_datetime.py │ ├── test_feature_detection.py │ ├── test_filters.py │ ├── test_kv.py │ ├── test_mapreduce.py │ ├── test_misc.py │ ├── test_pool.py │ ├── test_search.py │ ├── test_security.py │ ├── test_server_test.py │ ├── test_timeseries_pbuf.py │ ├── test_timeseries_ttb.py │ ├── test_util.py │ ├── test_yokozuna.py │ └── yz_setup.py ├── transports │ ├── __init__.py │ ├── feature_detect.py │ ├── http │ │ ├── __init__.py │ │ ├── connection.py │ │ ├── resources.py │ │ ├── search.py │ │ ├── stream.py │ │ └── transport.py │ ├── pool.py │ ├── security.py │ ├── tcp │ │ ├── __init__.py │ │ ├── connection.py │ │ ├── stream.py │ │ └── transport.py │ └── transport.py ├── ts_object.py ├── tz.py └── util.py ├── setup.cfg ├── setup.py ├── tox.ini └── version.py /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | #*# 3 | _build/ 4 | build/ 5 | .coverage 6 | dist/ 7 | docsrc/doctrees/ 8 | *.egg 9 | .eggs/ 10 | envs/ 11 | .idea/ 12 | py-build/ 13 | *.pyc 14 | __pycache__/ 15 | .python-version 16 | README.rst 17 | riak-*/ 18 | riak.egg-info/ 19 | .*.swp 20 | .tox/ 21 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "riak_pb"] 2 | path = riak_pb 3 | url = git://github.com/basho/riak_pb.git 4 | [submodule "tools"] 5 | path = tools 6 | url = git://github.com/basho/riak-client-tools.git 7 | [submodule "docs"] 8 | path = docs 9 | url = https://github.com/basho/riak-python-client.git 10 | branch = gh-pages 11 | -------------------------------------------------------------------------------- /.hgignore: -------------------------------------------------------------------------------- 1 | syntax: glob 2 | *.pyc 3 | docs 4 | riak.egg-info 5 | build 6 | *.html 7 | .*.swp 8 | dist/* -------------------------------------------------------------------------------- /.hgtags: -------------------------------------------------------------------------------- 1 | c8c596bfdd48a9e28de4c66911af592bf490f76b riak-0.10 2 | aafe6f3135beba5bfd12be313af068b78e6d6712 riak-python-client-1.0.0 3 | cab37c98ec948378b0ef905001272640d72a8d0c riak-python-client-1.0.1 4 | cab37c98ec948378b0ef905001272640d72a8d0c riak-python-client-1.0.1 5 | 0000000000000000000000000000000000000000 riak-python-client-1.0.1 6 | 0000000000000000000000000000000000000000 riak-python-client-1.0.1 7 | c22f9779ee7104d54c4a178edbf1db0b80731603 riak-python-client-1.0.1 8 | 9dc6141a5087919ae05f19d21eae8e03d949734c riak-python-client-1.1.0 9 | 6f4f58e0c3811be43ec7f9e098fa7492cdd23c59 riak-python-client-1.1.1 10 | -------------------------------------------------------------------------------- /.runner: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -o errexit 4 | set -o nounset 5 | 6 | have_tox='false' 7 | if hash tox 2>/dev/null 8 | then 9 | echo '[INFO] tox command present, will use that to run tests' 10 | have_tox='true' 11 | fi 12 | 13 | have_py2='false' 14 | if hash python2 2>/dev/null 15 | then 16 | have_py2='true' 17 | fi 18 | 19 | have_py3='false' 20 | if hash python3 2>/dev/null 21 | then 22 | have_py3='true' 23 | fi 24 | 25 | have_riak_admin='false' 26 | if hash riak-admin 2>/dev/null 27 | then 28 | have_riak_admin='true' 29 | $riak_admin='riak-admin' 30 | else 31 | set +o nounset 32 | 33 | if [[ -x $RIAK_ADMIN ]] 34 | then 35 | have_riak_admin='true' 36 | riak_admin="$RIAK_ADMIN" 37 | elif [[ -x $RIAK_DIR/bin/riak-admin ]] 38 | then 39 | have_riak_admin='true' 40 | riak_admin="$RIAK_DIR/bin/riak-admin" 41 | fi 42 | 43 | set -o nounset 44 | fi 45 | 46 | function lint 47 | { 48 | if ! hash flake8 2>/dev/null 49 | then 50 | pip install --upgrade flake8 51 | fi 52 | flake8 --exclude=riak/pb riak *.py 53 | } 54 | 55 | function run_tests 56 | { 57 | local protocol="${1:-pbc}" 58 | export RIAK_TEST_PROTOCOL="$protocol" 59 | if [[ $have_tox == 'true' ]] 60 | then 61 | tox 62 | else 63 | if [[ $have_py2 == 'true' ]] 64 | then 65 | python2 setup.py test 66 | fi 67 | if [[ $have_py3 == 'true' ]] 68 | then 69 | python3 setup.py test 70 | fi 71 | fi 72 | } 73 | 74 | function run_tests_each_protocol 75 | { 76 | for protocol in pbc http 77 | do 78 | run_tests "$protocol" 79 | done 80 | } 81 | 82 | function export_host_environment_vars 83 | { 84 | local riak_test_host="${RIAK_TEST_HOST:-localhost}" 85 | local -i riak_test_pb_port="${RIAK_TEST_PB_PORT:-8087}" 86 | local -i riak_test_http_port="${RIAK_TEST_HTTP_PORT:-8098}" 87 | export RIAK_TEST_HOST="$riak_test_host" 88 | export RIAK_TEST_PB_PORT="$riak_test_pb_port" 89 | export RIAK_TEST_HTTP_PORT="$riak_test_http_port" 90 | } 91 | 92 | function export_test_environment_vars 93 | { 94 | export RUN_BTYPES=1 95 | export RUN_CLIENT=1 96 | export RUN_DATATYPES=1 97 | export RUN_INDEXES=1 98 | export RUN_KV=1 99 | export RUN_MAPREDUCE=1 100 | export RUN_RESOLVE=1 101 | export RUN_TIMESERIES=1 102 | export RUN_YZ=1 103 | } 104 | 105 | function unexport_test_environment_vars 106 | { 107 | export RUN_BTYPES=0 108 | export RUN_CLIENT=0 109 | export RUN_DATATYPES=0 110 | export RUN_INDEXES=0 111 | export RUN_KV=0 112 | export RUN_MAPREDUCE=0 113 | export RUN_RESOLVE=0 114 | export RUN_TIMESERIES=0 115 | export RUN_YZ=0 116 | } 117 | 118 | function security_test 119 | { 120 | if [[ $have_riak_admin == 'true' ]] 121 | then 122 | export_host_environment_vars 123 | unexport_test_environment_vars 124 | export RUN_SECURITY=1 125 | $riak_admin security enable 126 | run_tests 'pbc' 127 | else 128 | echo '[ERROR] riak-admin must be in PATH, RIAK_ADMIN var set to path, or RIAK_DIR set.' 1>&2 129 | exit 1 130 | fi 131 | } 132 | 133 | function integration_test 134 | { 135 | export_host_environment_vars 136 | export_test_environment_vars 137 | run_tests_each_protocol 138 | } 139 | 140 | function timeseries_test 141 | { 142 | unexport_test_environment_vars 143 | export RUN_TIMESERIES=1 144 | run_tests_each_protocol 145 | } 146 | 147 | arg="${1:-lint}" 148 | case "$arg" in 149 | 'lint') 150 | lint;; 151 | 'unit-test') 152 | run_tests;; 153 | 'integration-test') 154 | integration_test;; 155 | 'security-test') 156 | security_test;; 157 | 'timeseries-test') 158 | timeseries_test;; 159 | *) 160 | echo "[ERROR] unknown argument: '$arg'" 1>&2 161 | exit 1;; 162 | esac 163 | -------------------------------------------------------------------------------- /.travis.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -o errexit 3 | 4 | flake8 --ignore E123,E126,E226,E722,E741 --exclude=riak/pb riak *.py 5 | 6 | sudo riak-admin security disable 7 | 8 | python setup.py test 9 | 10 | sudo riak-admin security enable 11 | 12 | if [[ $RIAK_TEST_PROTOCOL == 'pbc' ]] 13 | then 14 | export RUN_SECURITY=1 15 | python setup.py test --test-suite riak.tests.test_security 16 | else 17 | echo '[INFO]: security tests run on PB protocol only' 18 | fi 19 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: required 2 | dist: trusty 3 | language: python 4 | python: 5 | - '2.7' 6 | - '3.6' 7 | - nightly 8 | addons: 9 | hosts: 10 | - riak-test 11 | install: 12 | - pip install --upgrade pip setuptools flake8 13 | before_script: 14 | - jdk_switcher use oraclejdk8 15 | - sudo ./tools/travis-ci/riak-install -d "$RIAK_DOWNLOAD_URL" 16 | - sudo ./tools/setup-riak -s 17 | env: 18 | matrix: 19 | - RIAK_TEST_PROTOCOL=pbc RIAK_DOWNLOAD_URL=http://s3.amazonaws.com/downloads.basho.com/riak/2.0/2.0.7/ubuntu/trusty/riak_2.0.7-1_amd64.deb 20 | - RIAK_TEST_PROTOCOL=http RIAK_DOWNLOAD_URL=http://s3.amazonaws.com/downloads.basho.com/riak/2.0/2.0.7/ubuntu/trusty/riak_2.0.7-1_amd64.deb 21 | - RIAK_TEST_PROTOCOL=pbc RIAK_DOWNLOAD_URL=http://s3.amazonaws.com/downloads.basho.com/riak/2.2/2.2.0/ubuntu/trusty/riak_2.2.0-1_amd64.deb 22 | - RIAK_TEST_PROTOCOL=http RIAK_DOWNLOAD_URL=http://s3.amazonaws.com/downloads.basho.com/riak/2.2/2.2.0/ubuntu/trusty/riak_2.2.0-1_amd64.deb 23 | global: 24 | - RIAK_TEST_PB_PORT=8087 25 | - RIAK_TEST_HTTP_PORT=8098 26 | - RUN_BTYPES=1 27 | - RUN_CLIENT=1 28 | - RUN_MAPREDUCE=1 29 | - RUN_KV=1 30 | - RUN_RESOLVE=1 31 | - RUN_YZ=1 32 | - RUN_DATATYPES=1 33 | - RUN_INDEXES=1 34 | - RUN_SECURITY=0 35 | script: 36 | - ./.travis.sh 37 | notifications: 38 | slack: 39 | secure: kU1XcvTAliCWKuYpMWEMbD4qkbmlnWGLAIKbBQjtIh5ZRzISgjdUFzGcC31eHoQFv12LQdp5KAFj0Y1FyEvLxi0W8VeWKpsBGc06ntuECaN9MNHRBzKKclrTMGTfpBWZ5IO17XSUu2lKaNz6GDGRkiZA+sxYAVPfZSXY3u86IuY= 40 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include docs/* 2 | include riak/erl_src/* 3 | include README.md 4 | include README.rst 5 | include LICENSE 6 | include RELNOTES.md 7 | include version.py 8 | include commands.py 9 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | unexport LANG 2 | unexport LC_ADDRESS 3 | unexport LC_COLLATE 4 | unexport LC_CTYPE 5 | unexport LC_IDENTIFICATION 6 | unexport LC_MEASUREMENT 7 | unexport LC_MESSAGES 8 | unexport LC_MONETARY 9 | unexport LC_NAME 10 | unexport LC_NUMERIC 11 | unexport LC_PAPER 12 | unexport LC_TELEPHONE 13 | unexport LC_TIME 14 | 15 | PANDOC_VERSION := $(shell pandoc --version) 16 | PROTOC_VERSION := $(shell protoc --version) 17 | 18 | PROJDIR := $(realpath $(CURDIR)) 19 | DOCSRC := $(PROJDIR)/docsrc 20 | DOCTREES := $(DOCSRC)/doctrees 21 | DOCSDIR := $(PROJDIR)/docs 22 | 23 | PYPI_REPOSITORY ?= pypi 24 | 25 | all: lint test 26 | 27 | .PHONY: lint 28 | lint: 29 | $(PROJDIR)/.runner lint 30 | 31 | .PHONY: docs 32 | docs: 33 | sphinx-build -b html -d $(DOCTREES) $(DOCSRC) $(DOCSDIR) 34 | @echo "The HTML pages are in $(DOCSDIR)" 35 | 36 | .PHONY: pb_clean 37 | pb_clean: 38 | @echo "==> Python (clean)" 39 | @rm -rf riak/pb/*_pb2.py riak/pb/*.pyc riak/pb/__pycache__ __pycache__ py-build 40 | 41 | .PHONY: pb_compile 42 | pb_compile: pb_clean 43 | ifeq ($(PROTOC_VERSION),) 44 | $(error The protoc command is required to parse proto files) 45 | endif 46 | ifneq ($(PROTOC_VERSION),libprotoc 2.5.0) 47 | $(error protoc must be version 2.5.0) 48 | endif 49 | @echo "==> Python (compile)" 50 | @protoc -Iriak_pb/src --python_out=riak/pb riak_pb/src/*.proto 51 | @python setup.py build_messages 52 | 53 | .PHONY: test_sdist 54 | test_sdist: 55 | @python setup.py sdist 56 | 57 | .PHONY: release_sdist 58 | release_sdist: 59 | ifeq ($(VERSION),) 60 | $(error VERSION must be set to build a release and deploy this package) 61 | endif 62 | ifeq ($(PANDOC_VERSION),) 63 | $(error The pandoc command is required to correctly convert README.md to rst format) 64 | endif 65 | ifeq ($(RELEASE_GPG_KEYNAME),) 66 | $(error RELEASE_GPG_KEYNAME must be set to build a release and deploy this package) 67 | endif 68 | ifeq ("$(wildcard $(PROJDIR)/.python-version)","") 69 | $(error expected $(PROJDIR)/.python-version to exist. Run $(PROJDIR)/build/pyenv-setup) 70 | endif 71 | @python -c 'import pypandoc' 72 | @echo "==> Python tagging version $(VERSION)" 73 | @$(PROJDIR)/build/publish $(VERSION) validate 74 | @git tag --sign -a "$(VERSION)" -m "riak-python-client $(VERSION)" --local-user "$(RELEASE_GPG_KEYNAME)" 75 | @git push --tags 76 | @echo "==> pypi repository: $(PYPI_REPOSITORY)" 77 | @echo "==> Python (sdist)" 78 | @python setup.py sdist upload --repository $(PYPI_REPOSITORY) --show-response --sign --identity $(RELEASE_GPG_KEYNAME) 79 | @$(PROJDIR)/build/publish $(VERSION) 80 | 81 | .PHONY: release 82 | release: release_sdist 83 | ifeq ($(RELEASE_GPG_KEYNAME),) 84 | $(error RELEASE_GPG_KEYNAME must be set to build a release and deploy this package) 85 | endif 86 | ifeq ("$(wildcard $(PROJDIR)/.python-version)","") 87 | $(error expected $(PROJDIR)/.python-version to exist. Run $(PROJDIR)/build/pyenv-setup) 88 | endif 89 | @echo "==> pypi repository: $(PYPI_REPOSITORY)" 90 | @echo "==> Python 2.7 (bdist_egg)" 91 | @python2.7 setup.py build --build-base=py-build/2.7 bdist_egg upload --repository $(PYPI_REPOSITORY) --show-response --sign --identity $(RELEASE_GPG_KEYNAME) 92 | @echo "==> Python 3.3 (bdist_egg)" 93 | @python3.3 setup.py build --build-base=py-build/3.3 bdist_egg upload --repository $(PYPI_REPOSITORY) --show-response --sign --identity $(RELEASE_GPG_KEYNAME) 94 | @echo "==> Python 3.4 (bdist_egg)" 95 | @python3.4 setup.py build --build-base=py-build/3.4 bdist_egg upload --repository $(PYPI_REPOSITORY) --show-response --sign --identity $(RELEASE_GPG_KEYNAME) 96 | @echo "==> Python 3.5 (bdist_egg)" 97 | @python3.5 setup.py build --build-base=py-build/3.5 bdist_egg upload --repository $(PYPI_REPOSITORY) --show-response --sign --identity $(RELEASE_GPG_KEYNAME) 98 | 99 | .PHONY: unit-test 100 | unit-test: 101 | @$(PROJDIR)/.runner unit-test 102 | 103 | .PHONY: integration-test 104 | integration-test: 105 | @$(PROJDIR)/.runner integration-test 106 | 107 | .PHONY: security-test 108 | security-test: 109 | @$(PROJDIR)/.runner security-test 110 | 111 | .PHONY: timeseries-test 112 | timeseries-test: 113 | @$(PROJDIR)/.runner timeseries-test 114 | 115 | .PHONY: test 116 | test: integration-test 117 | 118 | .PHONY: help 119 | help: 120 | @echo '' 121 | @echo ' Targets: 122 | @echo ' ------------------------------------------------------------' 123 | @echo ' lint - Run linter (flake8) ' 124 | @echo ' test - Run all tests ' 125 | @echo ' unit-test - Run unit tests ' 126 | @echo ' integration-test - Run integration tests ' 127 | @echo ' security-test - Run integration tests (security enabled) ' 128 | @echo ' timeseries-test - Run timeseries integration tests ' 129 | @echo ' ------------------------------------------------------------' 130 | @echo '' 131 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Riak Python Client 2 | Copyright 2010-present Basho Technologies, Inc. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Python Client for Riak 2 | 3 | ## Build Status 4 | 5 | [![Build Status](https://travis-ci.org/basho/riak-python-client.svg?branch=master)](https://travis-ci.org/basho/riak-python-client) 6 | 7 | ## Documentation 8 | 9 | [Documentation for the Riak Python Client Library](http://basho.github.io/riak-python-client/index.html) is available [here](http://basho.github.io/riak-python-client/index.html). 10 | 11 | Documentation for Riak is available [here](http://docs.basho.com/riak/latest). 12 | 13 | ## Repository Cloning 14 | 15 | *NOTE*: please clone this repository using the `--recursive` argument to `git clone` or follow the clone with `git submodule update --init`. This repository uses two submodules. 16 | 17 | # Installation 18 | 19 | The recommended versions of Python for use with this client are Python `2.7.8` (or greater, `2.7.11` as of `2016-06-21`), `3.3.x`, `3.4.x` and `3.5.x`. The latest version from each series should be preferred. Older versions of the Python `2.7.X` and `3.X` series should be used with caution as they are not covered by integration tests. 20 | 21 | ## Riak TS (Timeseries) 22 | 23 | You must use version `2.7.11`, `3.4.4` or `3.5.1` (or greater within a version series). Otherwise you will be affected by [this Python bug](https://bugs.python.org/issue23517). 24 | 25 | ## From Source 26 | 27 | ```sh 28 | python setup.py install 29 | ``` 30 | 31 | There are additional dependencies on Python packages `setuptools` and `protobuf`. 32 | 33 | ## From PyPI 34 | 35 | Official packages are signed and published to [PyPI](https://pypi.python.org/pypi/riak). 36 | 37 | To install from [PyPI](https://pypi.python.org/pypi/riak) directly you can use `pip`. 38 | 39 | ```sh 40 | pip install riak 41 | ``` 42 | 43 | # Testing 44 | 45 | ## Unit Tests 46 | 47 | Unit tests will be executed via `tox` if it is in your `PATH`, otherwise by the `python2` and (if available), `python3` executables: 48 | 49 | ```sh 50 | make unit-test 51 | ``` 52 | 53 | ## Integration Tests 54 | 55 | You have two options to run Riak locally - either build from source, or use a pre-installed Riak package. 56 | 57 | ### Source 58 | 59 | To setup the default test configuration, build a Riak node from a clone of `github.com/basho/riak`: 60 | 61 | ```sh 62 | # check out latest release tag 63 | git checkout riak-2.1.4 64 | make locked-deps 65 | make rel 66 | ``` 67 | 68 | [Source build documentation](http://docs.basho.com/riak/kv/latest/setup/installing/source/). 69 | 70 | When building from source, the protocol buffers port will be `8087` and HTTP will be `8098`. 71 | 72 | ### Package 73 | 74 | Install using your platform's package manager ([docs](http://docs.basho.com/riak/kv/latest/setup/installing/)) 75 | 76 | When installing from a package, the protocol buffers port will be `8087` and HTTP will be `8098`. 77 | 78 | ### Running Integration Tests 79 | 80 | * Ensure you've initialized this repo's submodules: 81 | 82 | ```sh 83 | git submodule update --init 84 | ``` 85 | 86 | * Run the following: 87 | 88 | ```sh 89 | ./tools/setup-riak 90 | make integration-test 91 | ``` 92 | 93 | 94 | Contributors 95 | -------------------------- 96 | 97 | * Andrew Thompson 98 | * Andy Gross 99 | * Armon Dadgar 100 | * Brett Hazen 101 | * Brett Hoerner 102 | * Brian Roach 103 | * Bryan Fink 104 | * Daniel Lindsley 105 | * Daniel Néri 106 | * Daniel Reverri 107 | * [Dan Root](https://github.com/daroot) 108 | * [David Basden](https://github.com/dbasden) 109 | * [David Delassus](https://github.com/linkdd) 110 | * David Koblas 111 | * Dmitry Rozhkov 112 | * Eric Florenzano 113 | * Eric Moritz 114 | * Filip de Waard 115 | * Gilles Devaux 116 | * Greg Nelson 117 | * Gregory Burd 118 | * Greg Stein 119 | * Ian Plosker 120 | * Jayson Baird 121 | * Jeffrey Massung 122 | * Jon Meredith 123 | * Josip Lisec 124 | * Justin Sheehy 125 | * Kevin Smith 126 | * [Luke Bakken](https://github.com/lukebakken) 127 | * Mark Erdmann 128 | * Mark Phillips 129 | * Mathias Meyer 130 | * Matt Heitzenroder 131 | * [Matt Lohier](https://github.com/aquam8) 132 | * Mikhail Sobolev 133 | * Reid Draper 134 | * Russell Brown 135 | * Rusty Klophaus 136 | * Rusty Klophaus 137 | * Scott Lystig Fritchie 138 | * Sean Cribbs 139 | * Shuhao Wu 140 | * Silas Sewell 141 | * Socrates Lee 142 | * Soren Hansen 143 | * Sreejith Kesavan 144 | * Timothée Peignier 145 | * [`tobixx`](https://github.com/tobixx) 146 | * [Tin Tvrtković](https://github.com/Tinche) 147 | * [Vitaly Shestovskiy](https://github.com/lamp0chka) 148 | * William Kral 149 | * [Yasser Souri](https://github.com/yassersouri) 150 | -------------------------------------------------------------------------------- /build/publish: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -o errexit 4 | set -o nounset 5 | 6 | declare -r debug='false' 7 | declare -r tmpfile_file="/tmp/publish.$$.tmpfiles" 8 | 9 | function make_temp_file 10 | { 11 | local template="${1:-publish.$$.XXXXXX}" 12 | if [[ $template != *XXXXXX ]] 13 | then 14 | template="$template.XXXXXX" 15 | fi 16 | local tmp=$(mktemp -t "$template") 17 | echo "$tmp" >> "$tmpfile_file" 18 | echo "$tmp" 19 | } 20 | 21 | function now 22 | { 23 | date '+%Y-%m-%d %H:%M:%S' 24 | } 25 | 26 | function pwarn 27 | { 28 | echo "$(now) [warning]: $@" 1>&2 29 | } 30 | 31 | function perr 32 | { 33 | echo "$(now) [error]: $@" 1>&2 34 | } 35 | 36 | function pinfo 37 | { 38 | echo "$(now) [info]: $@" 39 | } 40 | 41 | function pdebug 42 | { 43 | if [[ $debug == 'true' ]] 44 | then 45 | echo "$(now) [debug]: $@" 46 | fi 47 | } 48 | 49 | function errexit 50 | { 51 | perr "$@" 52 | exit 1 53 | } 54 | 55 | function onexit 56 | { 57 | if [[ -f $tmpfile_file ]] 58 | then 59 | for tmpfile in $(< $tmpfile_file) 60 | do 61 | pdebug "removing temp file $tmpfile" 62 | rm -f $tmpfile 63 | done 64 | rm -f $tmpfile_file 65 | fi 66 | } 67 | 68 | function gh_publish { 69 | if [[ -z $version_string ]] 70 | then 71 | errexit 'gh_publish: version_string required' 72 | fi 73 | 74 | # NB: no 'v' here at start of version_string 75 | local -r package_name="riak-$version_string.tar.gz" 76 | local -r package="./dist/riak-$version_string.tar.gz" 77 | if [[ ! -s $package ]] 78 | then 79 | errexit "gh_publish: expected to find $package in dist/" 80 | fi 81 | 82 | # NB: we use a X.Y.Z tag 83 | local -r release_json="{ 84 | \"tag_name\" : \"$version_string\", 85 | \"name\" : \"Riak Python Client $version_string\", 86 | \"body\" : \"riak-python-client $version_string\nhttps://github.com/basho/riak-python-client/blob/master/RELNOTES.md\", 87 | \"draft\" : false, 88 | \"prerelease\" : $is_prerelease 89 | }" 90 | 91 | pdebug "Release JSON: $release_json" 92 | 93 | local curl_content_file="$(make_temp_file)" 94 | local curl_stdout_file="$(make_temp_file)" 95 | local curl_stderr_file="$(make_temp_file)" 96 | 97 | curl -4so $curl_content_file -w '%{http_code}' -XPOST \ 98 | -H "Authorization: token $(< $github_api_key_file)" -H 'Content-type: application/json' \ 99 | 'https://api.github.com/repos/basho/riak-python-client/releases' -d "$release_json" 1> "$curl_stdout_file" 2> "$curl_stderr_file" 100 | if [[ $? != 0 ]] 101 | then 102 | errexit "curl error exited with code: '$?' see '$curl_stderr_file'" 103 | fi 104 | 105 | local -i curl_rslt="$(< $curl_stdout_file)" 106 | if (( curl_rslt == 422 )) 107 | then 108 | pwarn "Release in GitHub already exists! (http code: '$curl_rslt')" 109 | curl -4so $curl_content_file -w '%{http_code}' -XGET \ 110 | -H "Authorization: token $(< $github_api_key_file)" -H 'Content-type: application/json' \ 111 | "https://api.github.com/repos/basho/riak-python-client/releases/tags/$version_string" 1> "$curl_stdout_file" 2> "$curl_stderr_file" 112 | if [[ $? != 0 ]] 113 | then 114 | errexit "curl error exited with code: '$?' see '$curl_stderr_file'" 115 | fi 116 | elif (( curl_rslt != 201 )) 117 | then 118 | errexit "Creating release in GitHub failed with http code '$curl_rslt'" 119 | fi 120 | 121 | if [[ ! -s $curl_content_file ]] 122 | then 123 | errexit 'no release info to parse for asset uploads' 124 | fi 125 | 126 | # "upload_url": "https://uploads.github.com/repos/basho/riak-python-client/releases/1115734/assets{?name,label}" 127 | # https://uploads.github.com/repos/basho/riak-python-client/releases/1115734/assets{?name,label} 128 | local -r upload_url_with_name=$(perl -ne 'print qq($1\n) and exit if /"upload_url"[ :]+"(https:\/\/[^"]+)"/' "$curl_content_file") 129 | local -r upload_url="${upload_url_with_name/\{?name,label\}/?name=$package_name}" 130 | 131 | local curl_content_file="$(make_temp_file)" 132 | local curl_stdout_file="$(make_temp_file)" 133 | local curl_stderr_file="$(make_temp_file)" 134 | 135 | curl -4so $curl_content_file -w '%{http_code}' -XPOST \ 136 | -H "Authorization: token $(< $github_api_key_file)" -H 'Content-type: application/x-compressed, application/x-tar' \ 137 | "$upload_url" --data-binary "@$package" 1> "$curl_stdout_file" 2> "$curl_stderr_file" 138 | if [[ $? != 0 ]] 139 | then 140 | errexit "curl error exited with code: '$?' see '$curl_stderr_file'" 141 | fi 142 | 143 | curl_rslt="$(< $curl_stdout_file)" 144 | if (( curl_rslt != 201 )) 145 | then 146 | errexit "Uploading release assets to GitHub failed with http code '$curl_rslt'" 147 | fi 148 | } 149 | 150 | trap onexit EXIT 151 | 152 | declare -r version_string="${1:-unknown}" 153 | 154 | # https://www.python.org/dev/peps/pep-0440/ 155 | if [[ ! $version_string =~ ^[0-9].[0-9].[0-9]([abcr]+[0-9]+)?$ ]] 156 | then 157 | errexit 'first argument must be valid version string in X.Y.Z, X.Y.ZaN, X.Y.ZbN or X.Y.ZrcN format' 158 | fi 159 | 160 | is_prerelease='false' 161 | if [[ $version_string =~ ^[0-9].[0-9].[0-9][abcr]+[0-9]+$ ]] 162 | then 163 | pinfo "publishing pre-release version: $version_string" 164 | is_prerelease='true' 165 | else 166 | pinfo "publishing version $version_string" 167 | fi 168 | 169 | declare -r current_branch="$(git rev-parse --abbrev-ref HEAD)" 170 | 171 | declare -r github_api_key_file="$HOME/.ghapi" 172 | if [[ ! -s $github_api_key_file ]] 173 | then 174 | errexit "please save your GitHub API token in $github_api_key_file" 175 | fi 176 | 177 | # Validate commands 178 | if ! hash curl 2>/dev/null 179 | then 180 | errexit "'curl' must be in your PATH" 181 | fi 182 | 183 | validate=${2:-''} 184 | if [[ $validate == 'validate' ]] 185 | then 186 | exit 0 187 | fi 188 | 189 | gh_publish 190 | -------------------------------------------------------------------------------- /build/pyenv-setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | unset PYENV_VERSION 4 | 5 | if [[ ! -d $PYENV_ROOT ]] 6 | then 7 | export PYENV_ROOT="$HOME/.pyenv" 8 | fi 9 | 10 | declare -r PROJDIR="$PWD" 11 | if [[ ! -s $PROJDIR/riak/__init__.py ]] 12 | then 13 | echo "[ERROR] script must be run from the clone of github.com/basho/riak-python-client" 1>&2 14 | exit 1 15 | fi 16 | 17 | rm -f $PROJDIR/.python-version 18 | 19 | # Install pyenv if it's missing 20 | if [[ ! -d $PYENV_ROOT ]] 21 | then 22 | git clone 'https://github.com/yyuu/pyenv.git' $PYENV_ROOT 23 | else 24 | (cd $PYENV_ROOT && git fetch --all) 25 | fi 26 | 27 | (cd $PYENV_ROOT && git checkout $(git describe --tags $(git rev-list --tags --max-count=1))) 28 | 29 | declare -r pyenv_alias_dir="$PYENV_ROOT/plugins/pyenv-alias" 30 | if [[ ! -d $pyenv_alias_dir ]] 31 | then 32 | git clone 'https://github.com/s1341/pyenv-alias.git' $pyenv_alias_dir 33 | else 34 | (cd $pyenv_alias_dir && git pull origin master) 35 | fi 36 | 37 | # Add pyenv root to PATH 38 | # and initialize pyenv 39 | if [[ $PATH != */.pyenv* ]] 40 | then 41 | echo "[INFO] adding $PYENV_ROOT/bin to PATH" 42 | export PATH="$PYENV_ROOT/bin:$PATH" 43 | fi 44 | 45 | if [[ $(type -t pyenv) != 'function' ]] 46 | then 47 | echo "[INFO] init pyenv" 48 | eval "$(pyenv init -)" 49 | fi 50 | 51 | do_pip_upgrades='false' 52 | 53 | # NB: 2.7.8 is special-cased 54 | for pyver in 2.7 3.3 3.4 3.5 3.6 55 | do 56 | riak_py_alias="riak_$pyver" 57 | if ! pyenv versions | fgrep -v 'riak_2.7.8' | fgrep -q "$riak_py_alias" 58 | then 59 | # Need to install it 60 | do_pip_upgrades='true' 61 | 62 | declare -i pymaj="${pyver%.*}" 63 | declare -i pymin="${pyver#*.}" 64 | pyver_latest="$(pyenv install --list | grep -E "^[[:space:]]+$pymaj\\.$pymin\\.[[:digit:]]+\$" | tail -n1 | sed -e 's/[[:space:]]//g')" 65 | 66 | echo "[INFO] installing Python $pyver_latest" 67 | VERSION_ALIAS="$riak_py_alias" pyenv install "$pyver_latest" 68 | fi 69 | done 70 | 71 | if ! pyenv versions | fgrep -q 'riak_2.7.8' 72 | then 73 | # Need to install it 74 | do_pip_upgrades='true' 75 | 76 | echo "[INFO] installing Python 2.7.8" 77 | VERSION_ALIAS='riak_2.7.8' pyenv install '2.7.8' 78 | fi 79 | 80 | pushd $PROJDIR 81 | pyenv local 'riak_3.6' 'riak_3.5' 'riak_3.4' 'riak_3.3' 'riak_2.7' 'riak_2.7.8' 82 | 83 | pyenv rehash 84 | 85 | if [[ $do_pip_upgrades == 'true' ]] 86 | then 87 | for PY in $(pyenv versions --bare --skip-aliases | grep '^riak_') 88 | do 89 | echo "[INFO] $PY - upgrading pip / setuptools" 90 | PYENV_VERSION="$PY" pip install --upgrade pip setuptools 91 | done 92 | fi 93 | 94 | python_version="$(python --version)" 95 | if [[ $python_version == Python\ 3* ]] 96 | then 97 | pip install --ignore-installed tox 98 | if ! pip show --quiet tox 99 | then 100 | echo "[ERROR] install of 'tox' failed" 1>&2 101 | popd 102 | exit 1 103 | fi 104 | pyenv rehash 105 | else 106 | echo "[ERROR] expected Python 3 to be 'python' at this point" 1>&2 107 | popd 108 | exit 1 109 | fi 110 | 111 | popd 112 | -------------------------------------------------------------------------------- /docsrc/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # Internal variables. 11 | PAPEROPT_a4 = -D latex_paper_size=a4 12 | PAPEROPT_letter = -D latex_paper_size=letter 13 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 14 | 15 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest 16 | 17 | help: 18 | @echo "Please use \`make ' where is one of" 19 | @echo " html to make standalone HTML files" 20 | @echo " dirhtml to make HTML files named index.html in directories" 21 | @echo " singlehtml to make a single large HTML file" 22 | @echo " pickle to make pickle files" 23 | @echo " json to make JSON files" 24 | @echo " htmlhelp to make HTML files and a HTML help project" 25 | @echo " qthelp to make HTML files and a qthelp project" 26 | @echo " devhelp to make HTML files and a Devhelp project" 27 | @echo " epub to make an epub" 28 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 29 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 30 | @echo " text to make text files" 31 | @echo " man to make manual pages" 32 | @echo " changes to make an overview of all changed/added/deprecated items" 33 | @echo " linkcheck to check all external links for integrity" 34 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 35 | 36 | clean: 37 | -rm -rf $(BUILDDIR)/* 38 | 39 | html: 40 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 41 | @echo 42 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 43 | 44 | dirhtml: 45 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 46 | @echo 47 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 48 | 49 | singlehtml: 50 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 51 | @echo 52 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 53 | 54 | pickle: 55 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 56 | @echo 57 | @echo "Build finished; now you can process the pickle files." 58 | 59 | json: 60 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 61 | @echo 62 | @echo "Build finished; now you can process the JSON files." 63 | 64 | htmlhelp: 65 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 66 | @echo 67 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 68 | ".hhp project file in $(BUILDDIR)/htmlhelp." 69 | 70 | qthelp: 71 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 72 | @echo 73 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 74 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 75 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/RiakPythonbinding.qhcp" 76 | @echo "To view the help file:" 77 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/RiakPythonbinding.qhc" 78 | 79 | devhelp: 80 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 81 | @echo 82 | @echo "Build finished." 83 | @echo "To view the help file:" 84 | @echo "# mkdir -p $$HOME/.local/share/devhelp/RiakPythonbinding" 85 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/RiakPythonbinding" 86 | @echo "# devhelp" 87 | 88 | epub: 89 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 90 | @echo 91 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 92 | 93 | latex: 94 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 95 | @echo 96 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 97 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 98 | "(use \`make latexpdf' here to do that automatically)." 99 | 100 | latexpdf: 101 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 102 | @echo "Running LaTeX files through pdflatex..." 103 | make -C $(BUILDDIR)/latex all-pdf 104 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 105 | 106 | text: 107 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 108 | @echo 109 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 110 | 111 | man: 112 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 113 | @echo 114 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 115 | 116 | changes: 117 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 118 | @echo 119 | @echo "The overview file is in $(BUILDDIR)/changes." 120 | 121 | linkcheck: 122 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 123 | @echo 124 | @echo "Link check complete; look for any errors in the above output " \ 125 | "or in $(BUILDDIR)/linkcheck/output.txt." 126 | 127 | doctest: 128 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 129 | @echo "Testing of doctests in the sources finished, look at the " \ 130 | "results in $(BUILDDIR)/doctest/output.txt." 131 | -------------------------------------------------------------------------------- /docsrc/_templates/layout.html: -------------------------------------------------------------------------------- 1 | {% extends "!layout.html" %} 2 | 3 | {% block sidebarrel %}{% endblock %} 4 | 5 | {%- block content %} 6 | {{ navBar() }} 7 |
8 | 20 | {% block body %}{% endblock %} 21 |
 
22 | 34 |
35 | {%- endblock %} 36 | 37 | {% set css_files = css_files + ['_static/custom.css'] %} 38 | -------------------------------------------------------------------------------- /docsrc/advanced.rst: -------------------------------------------------------------------------------- 1 | ========================== 2 | Advanced Usage & Internals 3 | ========================== 4 | 5 | This page contains documentation for aspects of library internals that 6 | you will rarely need to interact with, but are important for 7 | understanding how it works and development purposes. 8 | 9 | --------------- 10 | Connection pool 11 | --------------- 12 | 13 | .. currentmodule:: riak.transports.pool 14 | 15 | .. autoclass:: Resource 16 | :members: 17 | 18 | .. autoclass:: Pool 19 | :members: 20 | 21 | .. autoclass:: PoolIterator 22 | 23 | .. autoexception:: BadResource 24 | .. autoexception:: ConnectionClosed 25 | 26 | ----------- 27 | Retry logic 28 | ----------- 29 | 30 | .. currentmodule:: riak.client.transport 31 | 32 | .. autoclass:: RiakClientTransport 33 | :members: 34 | :private-members: 35 | 36 | .. autofunction:: _is_retryable 37 | 38 | .. autofunction:: retryable 39 | 40 | .. autofunction:: retryableHttpOnly 41 | 42 | ------------------- 43 | Multiget / Multiput 44 | ------------------- 45 | 46 | .. currentmodule:: riak.client.multi 47 | 48 | .. autodata:: POOL_SIZE 49 | 50 | .. autoclass:: Task 51 | .. autoclass:: PutTask 52 | 53 | .. autoclass:: MultiGetPool 54 | :members: 55 | :private-members: 56 | 57 | .. autofunction:: multiget 58 | 59 | .. autoclass:: MultiPutPool 60 | :members: 61 | :private-members: 62 | 63 | .. autofunction:: multiput 64 | 65 | --------- 66 | Datatypes 67 | --------- 68 | 69 | .. currentmodule:: riak.datatypes 70 | 71 | ^^^^^^^^^^^^^^^^^^ 72 | Datatype internals 73 | ^^^^^^^^^^^^^^^^^^ 74 | 75 | .. automethod:: Datatype.to_op 76 | .. automethod:: Datatype._check_type 77 | .. automethod:: Datatype._coerce_value 78 | .. automethod:: Datatype._default_value 79 | .. automethod:: Datatype._post_init 80 | .. automethod:: Datatype._require_context 81 | .. autoattribute:: Datatype.type_name 82 | .. autoattribute:: Datatype._type_error_msg 83 | 84 | ^^^^^^^^^^^^ 85 | TypedMapView 86 | ^^^^^^^^^^^^ 87 | 88 | .. autoclass:: riak.datatypes.map.TypedMapView 89 | :members: 90 | :special-members: 91 | 92 | ^^^^^^^^^^^^^^ 93 | TYPES constant 94 | ^^^^^^^^^^^^^^ 95 | 96 | .. autodata:: TYPES 97 | 98 | ---------- 99 | Transports 100 | ---------- 101 | 102 | .. currentmodule:: riak.transports.transport 103 | 104 | .. autoclass:: Transport 105 | :members: 106 | :private-members: 107 | 108 | .. currentmodule:: riak.transports.feature_detect 109 | 110 | .. autoclass:: FeatureDetection 111 | :members: 112 | :private-members: 113 | 114 | ^^^^^^^^^^^^^^^^ 115 | Security helpers 116 | ^^^^^^^^^^^^^^^^ 117 | 118 | .. currentmodule:: riak.transports.security 119 | 120 | .. autofunction:: verify_cb 121 | .. autofunction:: configure_context 122 | 123 | .. autoclass:: RiakWrappedSocket 124 | .. autoclass:: fileobject 125 | 126 | .. automethod:: riak.security.SecurityCreds._check_revoked_cert 127 | .. automethod:: riak.security.SecurityCreds._has_credential 128 | 129 | ^^^^^^^^^^^^^^ 130 | HTTP Transport 131 | ^^^^^^^^^^^^^^ 132 | 133 | .. currentmodule:: riak.transports.http 134 | 135 | .. autoclass:: HttpPool 136 | 137 | .. autofunction:: is_retryable 138 | 139 | .. autoclass:: HttpTransport 140 | :members: 141 | 142 | ^^^^^^^^^^^^^ 143 | TCP Transport 144 | ^^^^^^^^^^^^^ 145 | 146 | .. currentmodule:: riak.transports.tcp 147 | 148 | .. autoclass:: TcpPool 149 | 150 | .. autofunction:: is_retryable 151 | 152 | .. autoclass:: TcpTransport 153 | :members: 154 | 155 | --------- 156 | Utilities 157 | --------- 158 | 159 | ^^^^^^^^^^^^^^^^^^ 160 | Link wrapper class 161 | ^^^^^^^^^^^^^^^^^^ 162 | 163 | .. autoclass:: riak.mapreduce.RiakLink 164 | 165 | ^^^^^^^^^^^^^^^^^ 166 | Multi-valued Dict 167 | ^^^^^^^^^^^^^^^^^ 168 | 169 | .. currentmodule:: riak.multidict 170 | 171 | .. autoclass:: MultiDict 172 | 173 | .. automethod:: add 174 | .. automethod:: getall 175 | .. automethod:: getone 176 | .. automethod:: mixed 177 | .. automethod:: dict_of_lists 178 | 179 | ^^^^^^^^^^^^^^^^^^ 180 | Micro-benchmarking 181 | ^^^^^^^^^^^^^^^^^^ 182 | 183 | .. currentmodule:: riak.benchmark 184 | 185 | .. autofunction:: measure 186 | 187 | .. autofunction:: measure_with_rehearsal 188 | 189 | .. autoclass:: Benchmark 190 | :members: 191 | 192 | ^^^^^^^^^^^^^ 193 | Miscellaneous 194 | ^^^^^^^^^^^^^ 195 | 196 | .. currentmodule:: riak.util 197 | 198 | .. autofunction:: quacks_like_dict 199 | 200 | .. autofunction:: deep_merge 201 | 202 | .. autofunction:: deprecated 203 | 204 | .. autoclass:: lazy_property 205 | 206 | ------------------ 207 | distutils commands 208 | ------------------ 209 | 210 | .. automodule:: commands 211 | :members: 212 | :undoc-members: 213 | 214 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 215 | Version extraction (``version`` module) 216 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 217 | 218 | .. automodule:: version 219 | -------------------------------------------------------------------------------- /docsrc/bucket.rst: -------------------------------------------------------------------------------- 1 | .. _bucket_types: 2 | 3 | ====================== 4 | Buckets & Bucket Types 5 | ====================== 6 | 7 | .. currentmodule:: riak.bucket 8 | 9 | **Buckets** are both namespaces for the key-value pairs you store in 10 | Riak, and containers for properties that apply to that namespace. In 11 | older versions of Riak, this was the only logical organization 12 | available. Now a higher-level collection called a **Bucket Type** can 13 | group buckets together. They allow for efficiently setting properties 14 | on a group of buckets at the same time. 15 | 16 | Unlike buckets, Bucket Types must be `explicitly created 17 | `_ 18 | and activated before being used:: 19 | 20 | riak-admin bucket-type create n_equals_1 '{"props":{"n_val":1}}' 21 | riak-admin bucket-type activate n_equals_1 22 | 23 | Bucket Type creation and activation is only supported via the 24 | ``riak-admin bucket-type`` command-line tool. Riak 2.0 does not 25 | include an API to perform these actions, but the Python client *can* 26 | :meth:`retrieve ` and :meth:`set 27 | ` bucket-type properties. 28 | 29 | If Bucket Types are not specified, the *default* bucket 30 | type is used. These buckets should be created via the :meth:`bucket() 31 | ` method on the client object, like so:: 32 | 33 | import riak 34 | 35 | client = riak.RiakClient() 36 | mybucket = client.bucket('mybucket') 37 | 38 | Buckets with a user-specified Bucket Type can also be created via the same 39 | :meth:`bucket()` method with 40 | an additional parameter or explicitly via 41 | :meth:`bucket_type()`:: 42 | 43 | othertype = client.bucket_type('othertype') 44 | otherbucket = othertype.bucket('otherbucket') 45 | 46 | # Alternate way to get a bucket within a bucket-type 47 | mybucket = client.bucket('mybucket', bucket_type='mybuckettype') 48 | 49 | For more detailed discussion, see `Using Bucket Types 50 | `_. 51 | 52 | -------------- 53 | Bucket objects 54 | -------------- 55 | 56 | .. autoclass:: RiakBucket 57 | 58 | .. attribute:: name 59 | 60 | The name of the bucket, a string. 61 | 62 | .. attribute:: bucket_type 63 | 64 | The parent :class:`BucketType` for the bucket. 65 | 66 | .. autoattribute:: resolver 67 | 68 | ----------------- 69 | Bucket properties 70 | ----------------- 71 | 72 | Bucket properties are flags and defaults that apply to all keys in the 73 | bucket. 74 | 75 | .. automethod:: RiakBucket.get_properties 76 | .. automethod:: RiakBucket.set_properties 77 | .. automethod:: RiakBucket.clear_properties 78 | .. automethod:: RiakBucket.get_property 79 | .. automethod:: RiakBucket.set_property 80 | 81 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 82 | Shortcuts for common properties 83 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 84 | 85 | Some of the most commonly-used bucket properties are exposed as object 86 | properties as well. The getters and setters simply call 87 | :meth:`RiakBucket.get_property` and :meth:`RiakBucket.set_property` 88 | respectively. 89 | 90 | .. autoattribute:: RiakBucket.n_val 91 | .. autoattribute:: RiakBucket.allow_mult 92 | .. autoattribute:: RiakBucket.r 93 | .. autoattribute:: RiakBucket.pr 94 | .. autoattribute:: RiakBucket.w 95 | .. autoattribute:: RiakBucket.dw 96 | .. autoattribute:: RiakBucket.pw 97 | .. autoattribute:: RiakBucket.rw 98 | 99 | ----------------- 100 | Working with keys 101 | ----------------- 102 | 103 | The primary purpose of buckets is to act as namespaces for keys. As 104 | such, you can use the bucket object to create, fetch and delete 105 | :class:`objects `. 106 | 107 | .. automethod:: RiakBucket.new 108 | .. automethod:: RiakBucket.new_from_file 109 | .. automethod:: RiakBucket.get 110 | .. automethod:: RiakBucket.multiget 111 | .. automethod:: RiakBucket.delete 112 | 113 | 114 | ---------------- 115 | Query operations 116 | ---------------- 117 | 118 | .. automethod:: RiakBucket.search 119 | .. automethod:: RiakBucket.get_index 120 | .. automethod:: RiakBucket.stream_index 121 | .. automethod:: RiakBucket.paginate_index 122 | .. automethod:: RiakBucket.paginate_stream_index 123 | 124 | 125 | ------------- 126 | Serialization 127 | ------------- 128 | 129 | Similar to :class:`RiakClient `, buckets can 130 | register custom transformation functions for media-types. When 131 | undefined on the bucket, :meth:`RiakBucket.get_encoder` and 132 | :meth:`RiakBucket.get_decoder` will delegate to the client associated 133 | with the bucket. 134 | 135 | .. automethod:: RiakBucket.get_encoder 136 | .. automethod:: RiakBucket.set_encoder 137 | .. automethod:: RiakBucket.get_decoder 138 | .. automethod:: RiakBucket.set_decoder 139 | 140 | ------------ 141 | Listing keys 142 | ------------ 143 | 144 | Shortcuts for :meth:`RiakClient.get_keys() 145 | ` and 146 | :meth:`RiakClient.stream_keys() 147 | ` are exposed on the bucket 148 | object. The same admonitions for these operations apply. 149 | 150 | .. automethod:: RiakBucket.get_keys 151 | .. automethod:: RiakBucket.stream_keys 152 | 153 | ------------------- 154 | Bucket Type objects 155 | ------------------- 156 | 157 | .. autoclass:: BucketType 158 | 159 | .. attribute:: name 160 | 161 | The name of the Bucket Type, a string. 162 | 163 | .. automethod:: BucketType.is_default 164 | 165 | .. automethod:: BucketType.bucket 166 | 167 | ---------------------- 168 | Bucket Type properties 169 | ---------------------- 170 | 171 | Bucket Type properties are flags and defaults that apply to all buckets in the 172 | Bucket Type. 173 | 174 | .. automethod:: BucketType.get_properties 175 | .. automethod:: BucketType.set_properties 176 | .. automethod:: BucketType.get_property 177 | .. automethod:: BucketType.set_property 178 | .. attribute:: BucketType.datatype 179 | 180 | The assigned datatype for this bucket type, if present. 181 | 182 | :rtype: None or str 183 | 184 | --------------- 185 | Listing buckets 186 | --------------- 187 | 188 | Shortcuts for :meth:`RiakClient.get_buckets() 189 | ` and 190 | :meth:`RiakClient.stream_buckets() 191 | ` are exposed on the bucket 192 | type object. This is similar to `Listing keys`_ on buckets. 193 | 194 | .. automethod:: BucketType.get_buckets 195 | .. automethod:: BucketType.stream_buckets 196 | 197 | ------------------- 198 | Deprecated Features 199 | ------------------- 200 | 201 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 202 | Shortcuts for Riak Search 1.0 203 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 204 | 205 | When Riak Search 1.0 is enabled on the server, you can toggle which 206 | buckets have automatic indexing turned on using the ``search`` bucket 207 | property (and on older versions, the ``precommit`` property). These 208 | methods simplify interacting with that configuration. 209 | 210 | .. automethod:: RiakBucket.search_enabled 211 | .. automethod:: RiakBucket.enable_search 212 | .. automethod:: RiakBucket.disable_search 213 | 214 | ^^^^^^^^^^^^^^^ 215 | Legacy Counters 216 | ^^^^^^^^^^^^^^^ 217 | 218 | The :meth:`~RiakBucket.get_counter` and 219 | :meth:`~RiakBucket.update_counter`. See :ref:`legacy_counters` for 220 | more details. 221 | 222 | .. warning:: Legacy counters are incompatible with Bucket Types. 223 | 224 | .. automethod:: RiakBucket.get_counter 225 | .. automethod:: RiakBucket.update_counter 226 | -------------------------------------------------------------------------------- /docsrc/client.rst: -------------------------------------------------------------------------------- 1 | ==================== 2 | Client & Connections 3 | ==================== 4 | 5 | To connect to a Riak cluster, you must create a 6 | :py:class:`~riak.client.RiakClient` object. The default configuration 7 | connects to a single Riak node on ``localhost`` with the default 8 | ports. The below instantiation statements are all equivalent:: 9 | 10 | from riak import RiakClient, RiakNode 11 | 12 | RiakClient() 13 | RiakClient(protocol='http', host='127.0.0.1', http_port=8098) 14 | RiakClient(nodes=[{'host':'127.0.0.1','http_port':8098}]) 15 | RiakClient(protocol='http', nodes=[RiakNode()]) 16 | 17 | 18 | .. note:: Connections are not established until you attempt to perform 19 | an operation. If the host or port are incorrect, you will not get 20 | an error raised immediately. 21 | 22 | The client maintains a connection pool behind the scenes, one for each 23 | protocol. Connections are opened as-needed; a random node is selected 24 | when a new connection is requested. 25 | 26 | -------------- 27 | Client objects 28 | -------------- 29 | 30 | .. currentmodule:: riak.client 31 | .. autoclass:: RiakClient 32 | 33 | .. autoattribute:: PROTOCOLS 34 | 35 | Prior to Riak 2.0 the ``'https'`` protocol was also an option, but now 36 | secure connections are handled by the :ref:`security-label` feature. 37 | 38 | .. autoattribute:: protocol 39 | .. autoattribute:: client_id 40 | .. autoattribute:: resolver 41 | .. attribute:: nodes 42 | 43 | The list of :class:`nodes ` that this 44 | client will connect to. It is best not to modify this property 45 | directly, as it is not thread-safe. 46 | 47 | ^^^^^ 48 | Nodes 49 | ^^^^^ 50 | 51 | The :attr:`nodes ` attribute of ``RiakClient`` objects is 52 | a list of ``RiakNode`` objects. If you include multiple host 53 | specifications in the ``RiakClient`` constructor, they will be turned 54 | into this type. 55 | 56 | .. autoclass:: riak.node.RiakNode 57 | :members: 58 | 59 | ^^^^^^^^^^^ 60 | Retry logic 61 | ^^^^^^^^^^^ 62 | 63 | Some operations that fail because of network errors or Riak node 64 | failure may be safely retried on another node, and the client will do 65 | so automatically. The items below can be used to configure this 66 | behavior. 67 | 68 | .. autoattribute:: RiakClient.retries 69 | 70 | .. automethod:: RiakClient.retry_count 71 | 72 | .. autodata:: riak.client.transport.DEFAULT_RETRY_COUNT 73 | 74 | ----------------------- 75 | Client-level Operations 76 | ----------------------- 77 | 78 | Some operations are not scoped by buckets or bucket types and can be 79 | performed on the client directly: 80 | 81 | .. automethod:: RiakClient.ping 82 | .. automethod:: RiakClient.get_buckets 83 | .. automethod:: RiakClient.stream_buckets 84 | 85 | ---------------------------------- 86 | Accessing Bucket Types and Buckets 87 | ---------------------------------- 88 | 89 | Most client operations are on :py:class:`bucket type objects 90 | `, the :py:class:`bucket objects 91 | ` they contain or keys within those buckets. Use the 92 | ``bucket_type`` or ``bucket`` methods for creating bucket types and buckets 93 | that will proxy operations to the called client. 94 | 95 | .. automethod:: RiakClient.bucket_type 96 | .. automethod:: RiakClient.bucket 97 | 98 | ---------------------- 99 | Bucket Type Operations 100 | ---------------------- 101 | 102 | .. automethod:: RiakClient.get_bucket_type_props 103 | .. automethod:: RiakClient.set_bucket_type_props 104 | 105 | ----------------- 106 | Bucket Operations 107 | ----------------- 108 | 109 | .. automethod:: RiakClient.get_bucket_props 110 | .. automethod:: RiakClient.set_bucket_props 111 | .. automethod:: RiakClient.clear_bucket_props 112 | .. automethod:: RiakClient.get_keys 113 | .. automethod:: RiakClient.stream_keys 114 | 115 | -------------------- 116 | Key-level Operations 117 | -------------------- 118 | 119 | .. automethod:: RiakClient.get 120 | .. automethod:: RiakClient.put 121 | .. automethod:: RiakClient.delete 122 | .. automethod:: RiakClient.multiget 123 | .. automethod:: RiakClient.fetch_datatype 124 | .. automethod:: RiakClient.update_datatype 125 | 126 | -------------------- 127 | Timeseries Operations 128 | -------------------- 129 | 130 | .. automethod:: RiakClient.ts_describe 131 | .. automethod:: RiakClient.ts_get 132 | .. automethod:: RiakClient.ts_put 133 | .. automethod:: RiakClient.ts_delete 134 | .. automethod:: RiakClient.ts_query 135 | .. automethod:: RiakClient.ts_stream_keys 136 | 137 | ---------------- 138 | Query Operations 139 | ---------------- 140 | 141 | .. automethod:: RiakClient.mapred 142 | .. automethod:: RiakClient.stream_mapred 143 | .. automethod:: RiakClient.get_index 144 | .. automethod:: RiakClient.stream_index 145 | .. automethod:: RiakClient.fulltext_search 146 | .. automethod:: RiakClient.paginate_index 147 | .. automethod:: RiakClient.paginate_stream_index 148 | 149 | ----------------------------- 150 | Search Maintenance Operations 151 | ----------------------------- 152 | 153 | .. automethod:: RiakClient.create_search_schema 154 | .. automethod:: RiakClient.get_search_schema 155 | .. automethod:: RiakClient.create_search_index 156 | .. automethod:: RiakClient.get_search_index 157 | .. automethod:: RiakClient.delete_search_index 158 | .. automethod:: RiakClient.list_search_indexes 159 | 160 | ------------- 161 | Serialization 162 | ------------- 163 | 164 | The client supports automatic transformation of Riak responses into 165 | Python types if encoders and decoders are registered for the 166 | media-types. Supported by default are ``application/json`` and 167 | ``text/plain``. 168 | 169 | .. autofunction:: default_encoder 170 | .. automethod:: RiakClient.get_encoder 171 | .. automethod:: RiakClient.set_encoder 172 | .. automethod:: RiakClient.get_decoder 173 | .. automethod:: RiakClient.set_decoder 174 | 175 | ------------------- 176 | Deprecated Features 177 | ------------------- 178 | 179 | ^^^^^^^^^^^^^^^^ 180 | Full-text search 181 | ^^^^^^^^^^^^^^^^ 182 | 183 | The original version of Riak Search has been replaced by :ref:`yz-label`, 184 | which is full-blown Solr integration with Riak. 185 | 186 | If Riak Search 1.0 is enabled, you can query an index via the bucket's 187 | :meth:`~riak.bucket.RiakBucket.search` method:: 188 | 189 | bucket.enable_search() 190 | bucket.new("one", data={'value':'one'}, 191 | content_type="application/json").store() 192 | 193 | bucket.search('value=one') 194 | 195 | To manually add and remove documents from an index (without an 196 | associated key), use the :class:`~riak.client.RiakClient` 197 | :meth:`~riak.client.RiakClient.fulltext_add` and 198 | :meth:`~riak.client.RiakClient.fulltext_delete` methods directly. 199 | 200 | .. automethod:: RiakClient.fulltext_add 201 | .. automethod:: RiakClient.fulltext_delete 202 | 203 | .. _legacy_counters: 204 | 205 | ^^^^^^^^^^^^^^^ 206 | Legacy Counters 207 | ^^^^^^^^^^^^^^^ 208 | 209 | The first Data Type introduced in Riak 1.4 were `counters`. These pre-date 210 | :ref:`Bucket Types ` and the current implementation. 211 | Rather than returning objects, the counter operations 212 | act directly on the value of the counter. Legacy counters are deprecated 213 | as of Riak 2.0. Please use :py:class:`~riak.datatypes.Counter` instead. 214 | 215 | .. warning:: Legacy counters are incompatible with Bucket Types. 216 | 217 | .. automethod:: RiakClient.get_counter 218 | .. automethod:: RiakClient.update_counter 219 | -------------------------------------------------------------------------------- /docsrc/index.rst: -------------------------------------------------------------------------------- 1 | Riak Python Client 2 | ================== 3 | 4 | Tutorial 5 | -------- 6 | 7 | The tutorial documentation has been converted to the `Basho Docs`_ as 8 | the `Taste of Riak: Python`_. The old tutorial_ that used to live here 9 | has been moved to the `Github Wiki`_ and is likely out-of-date. 10 | 11 | .. _`Basho Docs`: http://docs.basho.com/ 12 | .. _`Taste of Riak: Python`: 13 | http://docs.basho.com/riak/latest/dev/taste-of-riak/python/ 14 | .. _tutorial: 15 | https://github.com/basho/riak-python-client/wiki/Tutorial-%28old%29 16 | .. _`Github Wiki`: https://github.com/basho/riak-python-client/wiki 17 | 18 | Installation 19 | ------------ 20 | 21 | #. Ensure Riak installed & running. (``riak ping``) 22 | #. Install the Python client: 23 | 24 | #. If you use Pip_, ``pip install riak``. 25 | #. If you use easy_install_, run ``easy_install riak``. 26 | #. You can download the package off PyPI_, extract it and run 27 | ``python setup.py install``. 28 | 29 | .. _Pip: http://pip.openplans.org/ 30 | .. _easy_install: http://pypi.python.org/pypi/setuptools 31 | .. _PyPI: http://pypi.python.org/pypi/riak/ 32 | 33 | Development 34 | ----------- 35 | 36 | All development is done on Github_. Use Issues_ to report 37 | problems or submit contributions. 38 | 39 | .. _Github: https://github.com/basho/riak-python-client/ 40 | .. _Issues: https://github.com/basho/riak-python-client/issues 41 | 42 | 43 | Indices and tables 44 | ------------------ 45 | 46 | * :ref:`genindex` 47 | * :ref:`search` 48 | 49 | Contents 50 | -------- 51 | 52 | .. toctree:: 53 | :maxdepth: 1 54 | 55 | client 56 | bucket 57 | object 58 | datatypes 59 | query 60 | security 61 | advanced 62 | -------------------------------------------------------------------------------- /docsrc/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=_build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . 10 | if NOT "%PAPER%" == "" ( 11 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 12 | ) 13 | 14 | if "%1" == "" goto help 15 | 16 | if "%1" == "help" ( 17 | :help 18 | echo.Please use `make ^` where ^ is one of 19 | echo. html to make standalone HTML files 20 | echo. dirhtml to make HTML files named index.html in directories 21 | echo. singlehtml to make a single large HTML file 22 | echo. pickle to make pickle files 23 | echo. json to make JSON files 24 | echo. htmlhelp to make HTML files and a HTML help project 25 | echo. qthelp to make HTML files and a qthelp project 26 | echo. devhelp to make HTML files and a Devhelp project 27 | echo. epub to make an epub 28 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 29 | echo. text to make text files 30 | echo. man to make manual pages 31 | echo. changes to make an overview over all changed/added/deprecated items 32 | echo. linkcheck to check all external links for integrity 33 | echo. doctest to run all doctests embedded in the documentation if enabled 34 | goto end 35 | ) 36 | 37 | if "%1" == "clean" ( 38 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 39 | del /q /s %BUILDDIR%\* 40 | goto end 41 | ) 42 | 43 | if "%1" == "html" ( 44 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 45 | echo. 46 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 47 | goto end 48 | ) 49 | 50 | if "%1" == "dirhtml" ( 51 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 52 | echo. 53 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 54 | goto end 55 | ) 56 | 57 | if "%1" == "singlehtml" ( 58 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 59 | echo. 60 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 61 | goto end 62 | ) 63 | 64 | if "%1" == "pickle" ( 65 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 66 | echo. 67 | echo.Build finished; now you can process the pickle files. 68 | goto end 69 | ) 70 | 71 | if "%1" == "json" ( 72 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 73 | echo. 74 | echo.Build finished; now you can process the JSON files. 75 | goto end 76 | ) 77 | 78 | if "%1" == "htmlhelp" ( 79 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 80 | echo. 81 | echo.Build finished; now you can run HTML Help Workshop with the ^ 82 | .hhp project file in %BUILDDIR%/htmlhelp. 83 | goto end 84 | ) 85 | 86 | if "%1" == "qthelp" ( 87 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 88 | echo. 89 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 90 | .qhcp project file in %BUILDDIR%/qthelp, like this: 91 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\RiakPythonbinding.qhcp 92 | echo.To view the help file: 93 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\RiakPythonbinding.ghc 94 | goto end 95 | ) 96 | 97 | if "%1" == "devhelp" ( 98 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 99 | echo. 100 | echo.Build finished. 101 | goto end 102 | ) 103 | 104 | if "%1" == "epub" ( 105 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 106 | echo. 107 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 108 | goto end 109 | ) 110 | 111 | if "%1" == "latex" ( 112 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 113 | echo. 114 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 115 | goto end 116 | ) 117 | 118 | if "%1" == "text" ( 119 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 120 | echo. 121 | echo.Build finished. The text files are in %BUILDDIR%/text. 122 | goto end 123 | ) 124 | 125 | if "%1" == "man" ( 126 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 127 | echo. 128 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 129 | goto end 130 | ) 131 | 132 | if "%1" == "changes" ( 133 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 134 | echo. 135 | echo.The overview file is in %BUILDDIR%/changes. 136 | goto end 137 | ) 138 | 139 | if "%1" == "linkcheck" ( 140 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 141 | echo. 142 | echo.Link check complete; look for any errors in the above output ^ 143 | or in %BUILDDIR%/linkcheck/output.txt. 144 | goto end 145 | ) 146 | 147 | if "%1" == "doctest" ( 148 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 149 | echo. 150 | echo.Testing of doctests in the sources finished, look at the ^ 151 | results in %BUILDDIR%/doctest/output.txt. 152 | goto end 153 | ) 154 | 155 | :end 156 | -------------------------------------------------------------------------------- /docsrc/object.rst: -------------------------------------------------------------------------------- 1 | ================ 2 | Values & Objects 3 | ================ 4 | 5 | .. currentmodule:: riak.riak_object 6 | 7 | Keys in Riak are namespaced into :class:`buckets 8 | `, and their associated values are represented 9 | by :class:`objects `, not to be confused with Python 10 | "objects". A :class:`RiakObject` is a container for the key, the 11 | :ref:`vclock`, the value(s) and any metadata associated with the 12 | value(s). 13 | 14 | Values may also be :class:`datatypes `, but 15 | are not discussed here. 16 | 17 | ---------- 18 | RiakObject 19 | ---------- 20 | 21 | .. autoclass:: RiakObject 22 | 23 | .. attribute:: key 24 | 25 | The key of this object, a string. If not present, the server 26 | will generate a key the first time this object is stored. 27 | 28 | .. attribute:: bucket 29 | 30 | The :class:`bucket ` to which this 31 | object belongs. 32 | 33 | .. autoattribute:: resolver 34 | .. attribute:: vclock 35 | 36 | The :ref:`vclock` for this object. 37 | 38 | .. autoattribute:: exists 39 | 40 | .. _vclock: 41 | 42 | ^^^^^^^^^^^^ 43 | Vector clock 44 | ^^^^^^^^^^^^ 45 | 46 | Vector clocks are Riak's means of tracking the relationships between 47 | writes to a key. It is best practice to fetch the latest version of a 48 | key before attempting to modify or overwrite the value; if you do not, 49 | you may create :ref:`siblings` or lose data! The content of a vector 50 | clock is essentially opaque to the user. 51 | 52 | .. autoclass:: VClock 53 | 54 | ----------- 55 | Persistence 56 | ----------- 57 | 58 | Fetching, storing, and deleting keys are the bread-and-butter of Riak. 59 | 60 | .. automethod:: RiakObject.store 61 | .. automethod:: RiakObject.reload 62 | .. automethod:: RiakObject.delete 63 | 64 | .. _object_accessors: 65 | 66 | ------------------ 67 | Value and Metadata 68 | ------------------ 69 | 70 | Unless you have enabled :ref:`siblings` via the :attr:`allow_mult 71 | ` bucket property, you can 72 | inspect and manipulate the value and metadata of an object directly using these 73 | properties and methods: 74 | 75 | .. autoattribute:: RiakObject.data 76 | .. autoattribute:: RiakObject.encoded_data 77 | .. autoattribute:: RiakObject.content_type 78 | .. autoattribute:: RiakObject.charset 79 | .. autoattribute:: RiakObject.content_encoding 80 | .. autoattribute:: RiakObject.last_modified 81 | .. autoattribute:: RiakObject.etag 82 | .. autoattribute:: RiakObject.usermeta 83 | .. autoattribute:: RiakObject.links 84 | .. autoattribute:: RiakObject.indexes 85 | .. automethod:: RiakObject.add_index 86 | .. automethod:: RiakObject.remove_index 87 | .. automethod:: RiakObject.set_index 88 | .. automethod:: RiakObject.add_link 89 | 90 | .. _siblings: 91 | 92 | -------- 93 | Siblings 94 | -------- 95 | 96 | Because Riak's consistency model is "eventual" (and not linearizable), 97 | there is no way for it to disambiguate writes that happen 98 | concurrently. The :ref:`vclock` helps establish a 99 | "happens after" relationships so that concurrent writes can be 100 | detected, but with the exception of :ref:`datatypes`, Riak has no way 101 | to determine which write has the correct value. 102 | 103 | Instead, when :attr:`allow_mult ` 104 | is ``True``, Riak keeps all writes that appear to be concurrent. Thus, 105 | the contents of a key's value may, in fact, be multiple values, which 106 | are called "siblings". Siblings are modeled in :class:`RiakContent 107 | ` objects, which contain all of the same 108 | :ref:`object_accessors` methods and attributes as the parent object. 109 | 110 | .. autoattribute:: RiakObject.siblings 111 | 112 | .. autoclass:: riak.content.RiakContent 113 | 114 | You do not typically have to create :class:`RiakContent 115 | ` objects yourself, but they will be created 116 | for you when :meth:`fetching ` objects from Riak. 117 | 118 | .. note:: The :ref:`object_accessors` accessors on :class:`RiakObject` 119 | are actually proxied to the first sibling when the object has only 120 | one. 121 | 122 | 123 | ^^^^^^^^^^^^^^^^^^^^^^^ 124 | Conflicts and Resolvers 125 | ^^^^^^^^^^^^^^^^^^^^^^^ 126 | 127 | When an object is *not* in conflict, it has only one sibling. When it 128 | is in conflict, you will have to resolve the conflict before it can be 129 | written again. How you choose to resolve the conflict is up to you, 130 | but you can automate the process using a :attr:`resolver 131 | ` function. 132 | 133 | .. autofunction:: riak.resolver.default_resolver 134 | .. autofunction:: riak.resolver.last_written_resolver 135 | 136 | If you do not supply a resolver function, or your resolver leaves 137 | multiple siblings present, accessing the :ref:`object_accessors` will 138 | result in a :exc:`ConflictError ` being raised. 139 | 140 | .. autoexception:: riak.ConflictError 141 | -------------------------------------------------------------------------------- /make.ps1: -------------------------------------------------------------------------------- 1 | Set-StrictMode -Version Latest 2 | $ErrorActionPreference = 'Stop' 3 | 4 | $env:RIAK_TEST_HOST = 'riak-test' 5 | $env:RIAK_TEST_PROTOCOL = 'pbc' 6 | $env:RIAK_TEST_PB_PORT = 10017 7 | $env:RUN_DATATYPES = 1 8 | $env:RUN_INDEXES = 1 9 | $env:RUN_POOL = 1 10 | $env:RUN_YZ = 1 11 | 12 | flake8 --exclude=riak/pb riak commands.py setup.py version.py 13 | if ($LastExitCode -ne 0) { 14 | throw 'flake8 failed!' 15 | } 16 | 17 | python setup.py test 18 | if ($LastExitCode -ne 0) { 19 | throw 'python tests failed!' 20 | } 21 | -------------------------------------------------------------------------------- /riak/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """ 16 | The Riak API for Python allows you to connect to a Riak instance, 17 | create, modify, and delete Riak objects, add and remove links from 18 | Riak objects, run Javascript (and Erlang) based Map/Reduce 19 | operations, and run Linkwalking operations. 20 | """ 21 | 22 | from riak.riak_error import RiakError, ConflictError, ListError 23 | from riak.client import RiakClient 24 | from riak.bucket import RiakBucket, BucketType 25 | from riak.table import Table 26 | from riak.node import RiakNode 27 | from riak.riak_object import RiakObject 28 | from riak.mapreduce import RiakKeyFilter, RiakMapReduce, RiakLink 29 | 30 | 31 | __all__ = ['RiakBucket', 'Table', 'BucketType', 'RiakNode', 32 | 'RiakObject', 'RiakClient', 'RiakMapReduce', 'RiakKeyFilter', 33 | 'RiakLink', 'RiakError', 'ConflictError', 'ListError', 34 | 'ONE', 'ALL', 'QUORUM', 'key_filter', 35 | 'disable_list_exceptions'] 36 | 37 | ONE = "one" 38 | ALL = "all" 39 | QUORUM = "quorum" 40 | 41 | key_filter = RiakKeyFilter() 42 | 43 | """ 44 | Set to true to allow listing operations 45 | """ 46 | disable_list_exceptions = False 47 | -------------------------------------------------------------------------------- /riak/benchmark.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import print_function 16 | 17 | import os 18 | import gc 19 | import sys 20 | import traceback 21 | 22 | __all__ = ['measure', 'measure_with_rehearsal'] 23 | 24 | 25 | def measure_with_rehearsal(): 26 | """ 27 | Runs a benchmark when used as an iterator, injecting a garbage 28 | collection between iterations. Example:: 29 | 30 | for b in riak.benchmark.measure_with_rehearsal(): 31 | with b.report("pow"): 32 | for _ in range(10000): 33 | math.pow(2,10000) 34 | with b.report("factorial"): 35 | for i in range(100): 36 | math.factorial(i) 37 | """ 38 | return Benchmark(True) 39 | 40 | 41 | def measure(): 42 | """ 43 | Runs a benchmark once when used as a context manager. Example:: 44 | 45 | with riak.benchmark.measure() as b: 46 | with b.report("pow"): 47 | for _ in range(10000): 48 | math.pow(2,10000) 49 | with b.report("factorial"): 50 | for i in range(100): 51 | math.factorial(i) 52 | """ 53 | return Benchmark() 54 | 55 | 56 | class Benchmark(object): 57 | """ 58 | A benchmarking run, which may consist of multiple steps. See 59 | measure_with_rehearsal() and measure() for examples. 60 | """ 61 | def __init__(self, rehearse=False): 62 | """ 63 | Creates a new benchmark reporter. 64 | 65 | :param rehearse: whether to run twice to take counter the effects 66 | of garbage collection 67 | :type rehearse: boolean 68 | """ 69 | self.rehearse = rehearse 70 | if rehearse: 71 | self.count = 2 72 | else: 73 | self.count = 1 74 | self._report = None 75 | 76 | def __enter__(self): 77 | if self.rehearse: 78 | raise ValueError("measure_with_rehearsal() cannot be used in with " 79 | "statements, use measure() or the for..in " 80 | "statement") 81 | print_header() 82 | self._report = BenchmarkReport() 83 | self._report.__enter__() 84 | return self 85 | 86 | def __exit__(self, exc_type, exc_val, exc_tb): 87 | if self._report: 88 | return self._report.__exit__(exc_type, exc_val, exc_tb) 89 | else: 90 | print 91 | return True 92 | 93 | def __iter__(self): 94 | return self 95 | 96 | def next(self): 97 | """ 98 | Runs the next iteration of the benchmark. 99 | """ 100 | if self.count == 0: 101 | raise StopIteration 102 | elif self.count > 1: 103 | print_rehearsal_header() 104 | else: 105 | if self.rehearse: 106 | gc.collect() 107 | print("-" * 59) 108 | print() 109 | print_header() 110 | 111 | self.count -= 1 112 | return self 113 | 114 | def __next__(self): 115 | # Python 3.x Version 116 | return self.next() 117 | 118 | def report(self, name): 119 | """ 120 | Returns a report for the current step of the benchmark. 121 | """ 122 | self._report = None 123 | return BenchmarkReport(name) 124 | 125 | 126 | def print_rehearsal_header(): 127 | """ 128 | Prints the header for the rehearsal phase of a benchmark. 129 | """ 130 | print 131 | print("Rehearsal -------------------------------------------------") 132 | 133 | 134 | def print_report(label, user, system, real): 135 | """ 136 | Prints the report of one step of a benchmark. 137 | """ 138 | print("{:<12s} {:12f} {:12f} ( {:12f} )".format(label, 139 | user, 140 | system, 141 | real)) 142 | 143 | 144 | def print_header(): 145 | """ 146 | Prints the header for the normal phase of a benchmark. 147 | """ 148 | print("{:<12s} {:<12s} {:<12s} ( {:<12s} )" 149 | .format('', 'user', 'system', 'real')) 150 | 151 | 152 | class BenchmarkReport(object): 153 | """ 154 | A labeled step in a benchmark. Acts as a context-manager, printing 155 | its timing results when the context exits. 156 | """ 157 | def __init__(self, name='benchmark'): 158 | self.name = name 159 | self.start = None 160 | 161 | def __enter__(self): 162 | self.start = os.times() 163 | return self 164 | 165 | def __exit__(self, exc_type, exc_val, exc_tb): 166 | if not exc_type: 167 | user1, system1, _, _, real1 = self.start 168 | user2, system2, _, _, real2 = os.times() 169 | print_report(self.name, user2 - user1, system2 - system1, 170 | real2 - real1) 171 | elif exc_type is KeyboardInterrupt: 172 | return False 173 | else: 174 | msg = "EXCEPTION! type: %r val: %r" % (exc_type, exc_val) 175 | print(msg, file=sys.stderr) 176 | traceback.print_tb(exc_tb) 177 | return True if exc_type is None else False 178 | -------------------------------------------------------------------------------- /riak/benchmarks/multiget.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import binascii 16 | import os 17 | 18 | import riak.benchmark as benchmark 19 | 20 | from riak import RiakClient 21 | from multiprocessing import cpu_count 22 | 23 | nodes = [ 24 | ('riak-test', 8098, 8087), 25 | # ('riak-test', 10018, 10017), 26 | # ('riak-test', 10028, 10027), 27 | # ('riak-test', 10038, 10037), 28 | # ('riak-test', 10048, 10047), 29 | # ('riak-test', 10058, 10057), 30 | ] 31 | client = RiakClient( 32 | nodes=nodes, 33 | protocol='pbc', 34 | multiget_pool_size=128) 35 | 36 | bkeys = [('default', 'multiget', str(key)) for key in range(10000)] 37 | 38 | data = binascii.b2a_hex(os.urandom(1024)) 39 | 40 | print("Benchmarking multiget:") 41 | print(" CPUs: {0}".format(cpu_count())) 42 | print(" Threads: {0}".format(client._multiget_pool._size)) 43 | print(" Keys: {0}".format(len(bkeys))) 44 | print() 45 | 46 | with benchmark.measure() as b: 47 | with b.report('populate'): 48 | for _, bucket, key in bkeys: 49 | client.bucket(bucket).new(key, encoded_data=data, 50 | content_type='text/plain' 51 | ).store() 52 | for b in benchmark.measure_with_rehearsal(): 53 | # client.protocol = 'http' 54 | # with b.report('http seq'): 55 | # for _, bucket, key in bkeys: 56 | # client.bucket(bucket).get(key) 57 | # with b.report('http multi'): 58 | # client.multiget(bkeys) 59 | 60 | client.protocol = 'pbc' 61 | with b.report('pbc seq'): 62 | for _, bucket, key in bkeys: 63 | client.bucket(bucket).get(key) 64 | with b.report('pbc multi'): 65 | client.multiget(bkeys) 66 | -------------------------------------------------------------------------------- /riak/benchmarks/timeseries.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import datetime 16 | import random 17 | import sys 18 | 19 | import riak.benchmark as benchmark 20 | 21 | from multiprocessing import cpu_count 22 | from riak import RiakClient 23 | 24 | # logger = logging.getLogger() 25 | # logger.level = logging.DEBUG 26 | # logger.addHandler(logging.StreamHandler(sys.stdout)) 27 | 28 | # batch sizes 8, 16, 32, 64, 128, 256 29 | if len(sys.argv) != 3: 30 | raise AssertionError( 31 | 'first arg is batch size, second arg is true / false' 32 | 'for use_ttb') 33 | 34 | rowcount = 32768 35 | batchsz = int(sys.argv[1]) 36 | if rowcount % batchsz != 0: 37 | raise AssertionError('rowcount must be divisible by batchsz') 38 | use_ttb = sys.argv[2].lower() == 'true' 39 | 40 | epoch = datetime.datetime.utcfromtimestamp(0) 41 | onesec = datetime.timedelta(0, 1) 42 | 43 | weather = ['typhoon', 'hurricane', 'rain', 'wind', 'snow'] 44 | rows = [] 45 | for i in range(rowcount): 46 | ts = datetime.datetime(2016, 1, 1, 12, 0, 0) + \ 47 | datetime.timedelta(seconds=i) 48 | family_idx = i % batchsz 49 | series_idx = i % batchsz 50 | family = 'hash{:d}'.format(family_idx) 51 | series = 'user{:d}'.format(series_idx) 52 | w = weather[i % len(weather)] 53 | temp = (i % 100) + random.random() 54 | row = [family, series, ts, w, temp] 55 | key = [family, series, ts] 56 | rows.append(row) 57 | 58 | print("Benchmarking timeseries:") 59 | print(" Use TTB: {}".format(use_ttb)) 60 | print("Batch Size: {}".format(batchsz)) 61 | print(" CPUs: {}".format(cpu_count())) 62 | print(" Rows: {}".format(len(rows))) 63 | print() 64 | 65 | tbl = 'GeoCheckin' 66 | h = 'riak-test' 67 | n = [ 68 | {'host': h, 'pb_port': 10017}, 69 | {'host': h, 'pb_port': 10027}, 70 | {'host': h, 'pb_port': 10037}, 71 | {'host': h, 'pb_port': 10047}, 72 | {'host': h, 'pb_port': 10057} 73 | ] 74 | client = RiakClient(nodes=n, protocol='pbc', 75 | transport_options={'use_ttb': use_ttb}) 76 | table = client.table(tbl) 77 | 78 | with benchmark.measure() as b: 79 | for i in (1, 2, 3): 80 | with b.report('populate-%d' % i): 81 | for i in range(0, rowcount, batchsz): 82 | x = i 83 | y = i + batchsz 84 | r = rows[x:y] 85 | ts_obj = table.new(r) 86 | result = ts_obj.store() 87 | if result is not True: 88 | raise AssertionError("expected success") 89 | -------------------------------------------------------------------------------- /riak/client/index_page.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from collections import namedtuple, Sequence 16 | 17 | 18 | CONTINUATION = namedtuple('Continuation', ['c']) 19 | 20 | 21 | class IndexPage(Sequence, object): 22 | """ 23 | Encapsulates a single page of results from a secondary index 24 | query, with the ability to iterate over results (if not streamed), 25 | capture the page marker (continuation), and automatically fetch 26 | the next page. 27 | 28 | While users will interact with this object, it will be created 29 | automatically by the client and does not need to be instantiated 30 | elsewhere. 31 | """ 32 | def __init__(self, client, bucket, index, startkey, endkey, return_terms, 33 | max_results, term_regex): 34 | self.client = client 35 | self.bucket = bucket 36 | self.index = index 37 | self.startkey = startkey 38 | self.endkey = endkey 39 | self.return_terms = return_terms 40 | self.max_results = max_results 41 | self.results = None 42 | self.stream = False 43 | self.term_regex = term_regex 44 | 45 | continuation = None 46 | """ 47 | The opaque page marker that is used when fetching the next chunk 48 | of results. The user can simply call :meth:`next_page` to do so, 49 | or pass this to the :meth:`~riak.client.RiakClient.get_index` 50 | method using the ``continuation`` option. 51 | """ 52 | 53 | def __iter__(self): 54 | """ 55 | Emulates the iterator interface. When streaming, this means 56 | delegating to the stream, otherwise iterating over the 57 | existing result set. 58 | """ 59 | if self.results is None: 60 | raise ValueError("No index results to iterate") 61 | 62 | try: 63 | for result in self.results: 64 | if self.stream and isinstance(result, CONTINUATION): 65 | self.continuation = result.c 66 | else: 67 | yield self._inject_term(result) 68 | finally: 69 | if self.stream: 70 | self.results.close() 71 | 72 | def __len__(self): 73 | """ 74 | Returns the length of the captured results. 75 | """ 76 | if self._has_results(): 77 | return len(self.results) 78 | else: 79 | raise ValueError("Streamed index page has no length") 80 | 81 | def __getitem__(self, index): 82 | """ 83 | Fetches an item by index from the captured results. 84 | """ 85 | if self._has_results(): 86 | return self.results[index] 87 | else: 88 | raise ValueError("Streamed index page has no entries") 89 | 90 | def __eq__(self, other): 91 | """ 92 | An IndexPage can pretend to be equal to a list when it has 93 | captured results by simply comparing the internal results to 94 | the passed list. Otherwise the other object needs to be an 95 | equivalent IndexPage. 96 | """ 97 | if isinstance(other, list) and self._has_results(): 98 | return self._inject_term(self.results) == other 99 | elif isinstance(other, IndexPage): 100 | return other.__dict__ == self.__dict__ 101 | else: 102 | return False 103 | 104 | def __ne__(self, other): 105 | """ 106 | Converse of __eq__. 107 | """ 108 | return not self.__eq__(other) 109 | 110 | def has_next_page(self): 111 | """ 112 | Whether there is another page available, i.e. the response 113 | included a continuation. 114 | """ 115 | return self.continuation is not None 116 | 117 | def next_page(self, timeout=None, stream=None): 118 | """ 119 | Fetches the next page using the same parameters as the 120 | original query. 121 | 122 | Note that if streaming was used before, it will be used again 123 | unless overridden. 124 | 125 | :param stream: whether to enable streaming. `True` enables, 126 | `False` disables, `None` uses previous value. 127 | :type stream: boolean 128 | :param timeout: a timeout value in milliseconds, or 'infinity' 129 | :type timeout: int 130 | """ 131 | if not self.continuation: 132 | raise ValueError("Cannot get next index page, no continuation") 133 | 134 | if stream is not None: 135 | self.stream = stream 136 | 137 | args = {'bucket': self.bucket, 138 | 'index': self.index, 139 | 'startkey': self.startkey, 140 | 'endkey': self.endkey, 141 | 'return_terms': self.return_terms, 142 | 'max_results': self.max_results, 143 | 'continuation': self.continuation, 144 | 'timeout': timeout, 145 | 'term_regex': self.term_regex} 146 | 147 | if self.stream: 148 | return self.client.stream_index(**args) 149 | else: 150 | return self.client.get_index(**args) 151 | 152 | def _has_results(self): 153 | """ 154 | When not streaming, have results been assigned? 155 | """ 156 | return not (self.stream or self.results is None) 157 | 158 | def _should_inject_term(self, term): 159 | """ 160 | The index term should be injected when using an equality query 161 | and the return terms option. If the term is already a tuple, 162 | it can be skipped. 163 | """ 164 | return self.return_terms and not self.endkey 165 | 166 | def _inject_term(self, result): 167 | """ 168 | Upgrades a result (streamed or not) to include the index term 169 | when an equality query is used with return_terms. 170 | """ 171 | if self._should_inject_term(result): 172 | if type(result) is list: 173 | return [(self.startkey, r) for r in result] 174 | else: 175 | return (self.startkey, result) 176 | else: 177 | return result 178 | 179 | def __repr__(self): 180 | return "<{!s} {!r}>".format(self.__class__.__name__, self.__dict__) 181 | 182 | def close(self): 183 | if self.stream: 184 | self.results.close() 185 | -------------------------------------------------------------------------------- /riak/codecs/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import collections 16 | 17 | import riak.pb.messages 18 | 19 | from riak import RiakError 20 | from riak.codecs.util import parse_pbuf_msg 21 | from riak.util import bytes_to_str 22 | 23 | Msg = collections.namedtuple('Msg', 24 | ['msg_code', 'data', 'resp_code']) 25 | 26 | 27 | class Codec(object): 28 | def parse_msg(self): 29 | raise NotImplementedError('parse_msg not implemented') 30 | 31 | def maybe_incorrect_code(self, resp_code, expect=None): 32 | if expect and resp_code != expect: 33 | raise RiakError("unexpected message code: %d, expected %d" 34 | % (resp_code, expect)) 35 | 36 | def maybe_riak_error(self, msg_code, data=None): 37 | if msg_code == riak.pb.messages.MSG_CODE_ERROR_RESP: 38 | if data is None: 39 | raise RiakError('no error provided!') 40 | else: 41 | err = parse_pbuf_msg(msg_code, data) 42 | raise RiakError(bytes_to_str(err.errmsg)) 43 | -------------------------------------------------------------------------------- /riak/codecs/util.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import riak.pb.messages 16 | 17 | 18 | def parse_pbuf_msg(msg_code, data): 19 | pbclass = riak.pb.messages.MESSAGE_CLASSES.get(msg_code, None) 20 | if pbclass is None: 21 | return None 22 | pbo = pbclass() 23 | pbo.ParseFromString(data) 24 | return pbo 25 | -------------------------------------------------------------------------------- /riak/content.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from riak import RiakError 16 | from six import string_types 17 | 18 | 19 | class RiakContent(object): 20 | """ 21 | The RiakContent holds the metadata and value of a single sibling 22 | within a RiakObject. RiakObjects that have more than one sibling 23 | are considered to be in conflict. 24 | """ 25 | def __init__(self, robject, data=None, encoded_data=None, charset=None, 26 | content_type='application/json', content_encoding=None, 27 | last_modified=None, etag=None, usermeta=None, links=None, 28 | indexes=None, exists=False): 29 | self._robject = robject 30 | self._data = data 31 | self._encoded_data = encoded_data 32 | self.charset = charset 33 | self.content_type = content_type 34 | self.content_encoding = content_encoding 35 | self.last_modified = last_modified 36 | self.etag = etag 37 | self.usermeta = usermeta or {} 38 | self.links = links or [] 39 | self.indexes = indexes or set() 40 | self.exists = exists 41 | 42 | def _get_data(self): 43 | if self._encoded_data is not None and self._data is None: 44 | self._data = self._deserialize(self._encoded_data) 45 | self._encoded_data = None 46 | return self._data 47 | 48 | def _set_data(self, value): 49 | self._encoded_data = None 50 | self._data = value 51 | 52 | data = property(_get_data, _set_data, doc=""" 53 | The data stored in this object, as Python objects. For the raw 54 | data, use the `encoded_data` property. If unset, accessing 55 | this property will result in decoding the `encoded_data` 56 | property into Python values. The decoding is dependent on the 57 | `content_type` property and the bucket's registered decoders. 58 | :type mixed """) 59 | 60 | def _get_encoded_data(self): 61 | if self._data is not None and self._encoded_data is None: 62 | self._encoded_data = self._serialize(self._data) 63 | self._data = None 64 | return self._encoded_data 65 | 66 | def _set_encoded_data(self, value): 67 | self._data = None 68 | self._encoded_data = value 69 | 70 | encoded_data = property(_get_encoded_data, _set_encoded_data, doc=""" 71 | The raw data stored in this object, essentially the encoded 72 | form of the `data` property. If unset, accessing this property 73 | will result in encoding the `data` property into a string. The 74 | encoding is dependent on the `content_type` property and the 75 | bucket's registered encoders. 76 | :type str""") 77 | 78 | def _serialize(self, value): 79 | encoder = self._robject.bucket.get_encoder(self.content_type) 80 | if encoder: 81 | return encoder(value) 82 | elif isinstance(value, string_types): 83 | return value.encode() 84 | else: 85 | raise TypeError('No encoder for non-string data ' 86 | 'with content type "{0}"'. 87 | format(self.content_type)) 88 | 89 | def _deserialize(self, value): 90 | if not value: 91 | return value 92 | decoder = self._robject.bucket.get_decoder(self.content_type) 93 | if decoder: 94 | return decoder(value) 95 | else: 96 | raise TypeError('No decoder for content type "{0}"'. 97 | format(self.content_type)) 98 | 99 | def add_index(self, field, value): 100 | """ 101 | add_index(field, value) 102 | 103 | Tag this object with the specified field/value pair for 104 | indexing. 105 | 106 | :param field: The index field. 107 | :type field: string 108 | :param value: The index value. 109 | :type value: string or integer 110 | :rtype: :class:`RiakObject ` 111 | """ 112 | if field[-4:] not in ("_bin", "_int"): 113 | raise RiakError("Riak 2i fields must end with either '_bin'" 114 | " or '_int'.") 115 | 116 | self.indexes.add((field, value)) 117 | 118 | return self._robject 119 | 120 | def remove_index(self, field=None, value=None): 121 | """ 122 | remove_index(field=None, value=None) 123 | 124 | Remove the specified field/value pair as an index on this 125 | object. 126 | 127 | :param field: The index field. 128 | :type field: string 129 | :param value: The index value. 130 | :type value: string or integer 131 | :rtype: :class:`RiakObject ` 132 | """ 133 | if not field and not value: 134 | self.indexes.clear() 135 | elif field and not value: 136 | for index in [x for x in self.indexes if x[0] == field]: 137 | self.indexes.remove(index) 138 | elif field and value: 139 | self.indexes.remove((field, value)) 140 | else: 141 | raise RiakError("Cannot pass value without a field" 142 | " name while removing index") 143 | 144 | return self._robject 145 | 146 | remove_indexes = remove_index 147 | 148 | def set_index(self, field, value): 149 | """ 150 | set_index(field, value) 151 | 152 | Works like :meth:`add_index`, but ensures that there is only 153 | one index on given field. If other found, then removes it 154 | first. 155 | 156 | :param field: The index field. 157 | :type field: string 158 | :param value: The index value. 159 | :type value: string or integer 160 | :rtype: :class:`RiakObject ` 161 | """ 162 | to_rem = set((x for x in self.indexes if x[0] == field)) 163 | self.indexes.difference_update(to_rem) 164 | return self.add_index(field, value) 165 | 166 | def add_link(self, obj, tag=None): 167 | """ 168 | add_link(obj, tag=None) 169 | 170 | Add a link to a RiakObject. 171 | 172 | :param obj: Either a RiakObject or 3 item link tuple consisting 173 | of (bucket, key, tag). 174 | :type obj: mixed 175 | :param tag: Optional link tag. Defaults to bucket name. It is ignored 176 | if ``obj`` is a 3 item link tuple. 177 | :type tag: string 178 | :rtype: :class:`RiakObject ` 179 | """ 180 | if isinstance(obj, tuple): 181 | newlink = obj 182 | else: 183 | newlink = (obj.bucket.name, obj.key, tag) 184 | 185 | self.links.append(newlink) 186 | return self._robject 187 | -------------------------------------------------------------------------------- /riak/datatypes/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from .types import TYPES 16 | from .datatype import Datatype 17 | from .counter import Counter 18 | from .flag import Flag 19 | from .register import Register 20 | from .set import Set 21 | from .map import Map 22 | from .errors import ContextRequired 23 | from .hll import Hll 24 | 25 | 26 | __all__ = ['Datatype', 'TYPES', 'ContextRequired', 27 | 'Flag', 'Counter', 'Register', 'Set', 'Map', 'Hll'] 28 | -------------------------------------------------------------------------------- /riak/datatypes/counter.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import six 16 | 17 | from riak.datatypes.datatype import Datatype 18 | from riak.datatypes import TYPES 19 | 20 | 21 | class Counter(Datatype): 22 | """ 23 | A convergent datatype that represents a counter which can be 24 | incremented or decremented. This type can stand on its own or be 25 | embedded within a :class:`~riak.datatypes.Map`. 26 | """ 27 | 28 | type_name = 'counter' 29 | _type_error_msg = "Counters can only be integers" 30 | 31 | def _post_init(self): 32 | self._increment = 0 33 | 34 | def _default_value(self): 35 | return 0 36 | 37 | @Datatype.modified.getter 38 | def modified(self): 39 | """ 40 | Whether this counter has staged increments. 41 | """ 42 | return self._increment is not 0 43 | 44 | def to_op(self): 45 | """ 46 | Extracts the mutation operation from the counter 47 | :rtype: int, None 48 | """ 49 | if not self._increment == 0: 50 | return ('increment', self._increment) 51 | 52 | def increment(self, amount=1): 53 | """ 54 | Increments the counter by one or the given amount. 55 | 56 | :param amount: the amount to increment the counter 57 | :type amount: int 58 | """ 59 | self._raise_if_badtype(amount) 60 | self._increment += amount 61 | 62 | def decrement(self, amount=1): 63 | """ 64 | Decrements the counter by one or the given amount. 65 | 66 | :param amount: the amount to decrement the counter 67 | :type amount: int 68 | """ 69 | self._raise_if_badtype(amount) 70 | self._increment -= amount 71 | 72 | def _check_type(self, new_value): 73 | return isinstance(new_value, six.integer_types) 74 | 75 | 76 | TYPES['counter'] = Counter 77 | -------------------------------------------------------------------------------- /riak/datatypes/errors.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from riak import RiakError 16 | 17 | 18 | class ContextRequired(RiakError): 19 | """ 20 | This exception is raised when removals of map fields and set 21 | entries are attempted and the datatype hasn't been initialized 22 | with a context. 23 | """ 24 | 25 | _default_message = ("A context is required for remove operations, " 26 | "fetch the datatype first") 27 | 28 | def __init__(self, message=None): 29 | super(ContextRequired, self).__init__(message or 30 | self._default_message) 31 | -------------------------------------------------------------------------------- /riak/datatypes/flag.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from riak.datatypes.datatype import Datatype 16 | from riak.datatypes import TYPES 17 | 18 | 19 | class Flag(Datatype): 20 | """ 21 | A convergent datatype that represents a boolean value that can be 22 | enabled or disabled, and may only be embedded in :class:`Map` 23 | instances. 24 | """ 25 | 26 | type_name = 'flag' 27 | _type_error_msg = "Flags can only be booleans" 28 | 29 | def _post_init(self): 30 | self._op = None 31 | 32 | def _default_value(self): 33 | return False 34 | 35 | @Datatype.modified.getter 36 | def modified(self): 37 | """ 38 | Whether this flag has staged toggles. 39 | """ 40 | return self._op is not None 41 | 42 | def enable(self): 43 | """ 44 | Turns the flag on, effectively setting its value to ``True``. 45 | """ 46 | self._op = 'enable' 47 | 48 | def disable(self): 49 | """ 50 | Turns the flag off, effectively setting its value to ``False``. 51 | """ 52 | self._require_context() 53 | self._op = 'disable' 54 | 55 | def to_op(self): 56 | """ 57 | Extracts the mutation operation from the flag. 58 | 59 | :rtype: bool, None 60 | """ 61 | return self._op 62 | 63 | def _check_type(self, new_value): 64 | return isinstance(new_value, bool) 65 | 66 | 67 | TYPES['flag'] = Flag 68 | -------------------------------------------------------------------------------- /riak/datatypes/hll.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import six 16 | 17 | from .datatype import Datatype 18 | from riak.datatypes import TYPES 19 | 20 | __all__ = ['Hll'] 21 | 22 | 23 | class Hll(Datatype): 24 | """A convergent datatype representing a HyperLogLog set. 25 | Currently strings are the only supported value type. 26 | Example:: 27 | 28 | myhll.add('barista') 29 | myhll.add('roaster') 30 | myhll.add('brewer') 31 | """ 32 | 33 | type_name = 'hll' 34 | _type_error_msg = 'Hlls can only be integers' 35 | 36 | def _post_init(self): 37 | self._adds = set() 38 | 39 | def _default_value(self): 40 | return 0 41 | 42 | @Datatype.modified.getter 43 | def modified(self): 44 | """ 45 | Whether this HyperLogLog has staged adds. 46 | """ 47 | return len(self._adds) > 0 48 | 49 | def to_op(self): 50 | """ 51 | Extracts the modification operation from the Hll. 52 | 53 | :rtype: dict, None 54 | """ 55 | if not self._adds: 56 | return None 57 | changes = {} 58 | if self._adds: 59 | changes['adds'] = list(self._adds) 60 | return changes 61 | 62 | def add(self, element): 63 | """ 64 | Adds an element to the HyperLogLog. Datatype cardinality will 65 | be updated when the object is saved. 66 | 67 | :param element: the element to add 68 | :type element: str 69 | """ 70 | if not isinstance(element, six.string_types): 71 | raise TypeError("Hll elements can only be strings") 72 | self._adds.add(element) 73 | 74 | def _coerce_value(self, new_value): 75 | return int(new_value) 76 | 77 | def _check_type(self, new_value): 78 | return isinstance(new_value, six.integer_types) 79 | 80 | 81 | TYPES['hll'] = Hll 82 | -------------------------------------------------------------------------------- /riak/datatypes/register.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from collections import Sized 16 | from riak.datatypes.datatype import Datatype 17 | from six import string_types 18 | from riak.datatypes import TYPES 19 | 20 | 21 | class Register(Sized, Datatype): 22 | """ 23 | A convergent datatype that represents an opaque string that is set 24 | with last-write-wins semantics, and may only be embedded in 25 | :class:`~riak.datatypes.Map` instances. 26 | """ 27 | 28 | type_name = 'register' 29 | _type_error_msg = "Registers can only be strings" 30 | 31 | def _post_init(self): 32 | self._new_value = None 33 | 34 | def _default_value(self): 35 | return "" 36 | 37 | @Datatype.value.getter 38 | def value(self): 39 | """ 40 | Returns a copy of the original value of the register. 41 | 42 | :rtype: str 43 | """ 44 | return self._value[:] 45 | 46 | @Datatype.modified.getter 47 | def modified(self): 48 | """ 49 | Whether this register has staged assignment. 50 | """ 51 | return self._new_value is not None 52 | 53 | def to_op(self): 54 | """ 55 | Extracts the mutation operation from the register. 56 | 57 | :rtype: str, None 58 | """ 59 | if self._new_value is not None: 60 | return ('assign', self._new_value) 61 | 62 | def assign(self, new_value): 63 | """ 64 | Assigns a new value to the register. 65 | 66 | :param new_value: the new value for the register 67 | :type new_value: str 68 | """ 69 | self._raise_if_badtype(new_value) 70 | self._new_value = new_value 71 | 72 | def __len__(self): 73 | return len(self.value) 74 | 75 | def _check_type(self, new_value): 76 | return isinstance(new_value, string_types) 77 | 78 | 79 | TYPES['register'] = Register 80 | -------------------------------------------------------------------------------- /riak/datatypes/set.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import collections 16 | 17 | from .datatype import Datatype 18 | from six import string_types 19 | from riak.datatypes import TYPES 20 | 21 | __all__ = ['Set'] 22 | 23 | 24 | class Set(collections.Set, Datatype): 25 | """A convergent datatype representing a Set with observed-remove 26 | semantics. Currently strings are the only supported value type. 27 | Example:: 28 | 29 | myset.add('barista') 30 | myset.add('roaster') 31 | myset.add('brewer') 32 | 33 | Likewise they can simply be removed:: 34 | 35 | myset.discard('barista') 36 | 37 | This datatype also implements the `Set ABC 38 | `_, meaning it 39 | supports ``len()``, ``in``, and iteration. 40 | 41 | """ 42 | 43 | type_name = 'set' 44 | _type_error_msg = "Sets can only be iterables of strings" 45 | 46 | def _post_init(self): 47 | self._adds = set() 48 | self._removes = set() 49 | 50 | def _default_value(self): 51 | return frozenset() 52 | 53 | @Datatype.modified.getter 54 | def modified(self): 55 | """ 56 | Whether this set has staged adds or removes. 57 | """ 58 | return len(self._removes | self._adds) > 0 59 | 60 | def to_op(self): 61 | """ 62 | Extracts the modification operation from the set. 63 | 64 | :rtype: dict, None 65 | """ 66 | if not self._adds and not self._removes: 67 | return None 68 | changes = {} 69 | if self._adds: 70 | changes['adds'] = list(self._adds) 71 | if self._removes: 72 | changes['removes'] = list(self._removes) 73 | return changes 74 | 75 | # collections.Set API, operates only on the immutable version 76 | def __contains__(self, element): 77 | return element in self.value 78 | 79 | def __iter__(self): 80 | return iter(self.value) 81 | 82 | def __len__(self): 83 | return len(self.value) 84 | 85 | # Sort of like collections.MutableSet API, without the additional 86 | # methods. 87 | def add(self, element): 88 | """ 89 | Adds an element to the set. 90 | 91 | .. note: You may add elements that already exist in the set. 92 | This may be used as an "assertion" that the element is a 93 | member. 94 | 95 | :param element: the element to add 96 | :type element: str 97 | """ 98 | _check_element(element) 99 | self._adds.add(element) 100 | 101 | def discard(self, element): 102 | """ 103 | Removes an element from the set. 104 | 105 | .. note: You may remove elements from the set that are not 106 | present, but a context from the server is required. 107 | 108 | :param element: the element to remove 109 | :type element: str 110 | """ 111 | _check_element(element) 112 | self._require_context() 113 | self._removes.add(element) 114 | 115 | def _coerce_value(self, new_value): 116 | return frozenset(new_value) 117 | 118 | def _check_type(self, new_value): 119 | if not isinstance(new_value, collections.Iterable): 120 | return False 121 | for element in new_value: 122 | if not isinstance(element, string_types): 123 | return False 124 | return True 125 | 126 | 127 | def _check_element(element): 128 | if not isinstance(element, string_types): 129 | raise TypeError("Set elements can only be strings") 130 | 131 | 132 | TYPES['set'] = Set 133 | -------------------------------------------------------------------------------- /riak/datatypes/types.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | #: A dict from :attr:`type names ` to the 16 | #: class that implements them. This is used inside :class:`Map` to 17 | #: initialize new values. 18 | TYPES = {} 19 | -------------------------------------------------------------------------------- /riak/erl_src/riak_kv_test_backend.beam: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/basho/riak-python-client/91de13a16607cdf553d1a194e762734e3bec4231/riak/erl_src/riak_kv_test_backend.beam -------------------------------------------------------------------------------- /riak/erl_src/riak_search_test_backend.beam: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/basho/riak-python-client/91de13a16607cdf553d1a194e762734e3bec4231/riak/erl_src/riak_search_test_backend.beam -------------------------------------------------------------------------------- /riak/erl_src/riak_search_test_backend.erl: -------------------------------------------------------------------------------- 1 | %% ------------------------------------------------------------------- 2 | %% 3 | %% Copyright (c) 2007-2010 Basho Technologies, Inc. All Rights Reserved. 4 | %% 5 | %% ------------------------------------------------------------------- 6 | 7 | -module(riak_search_test_backend). 8 | -behavior(riak_search_backend). 9 | 10 | -export([ 11 | reset/0, 12 | start/2, 13 | stop/1, 14 | index/2, 15 | delete/2, 16 | stream/6, 17 | range/8, 18 | info/5, 19 | fold/3, 20 | is_empty/1, 21 | drop/1 22 | ]). 23 | -export([ 24 | stream_results/3 25 | ]). 26 | 27 | -include_lib("riak_search/include/riak_search.hrl"). 28 | -define(T(P), list_to_atom("rs" ++ integer_to_list(P))). 29 | -record(state, {partition, table}). 30 | 31 | reset() -> 32 | {ok, Ring} = riak_core_ring_manager:get_my_ring(), 33 | [ catch ets:delete_all_objects(?T(P)) || 34 | P <- riak_core_ring:my_indices(Ring) ], 35 | riak_search_config:clear(), 36 | ok. 37 | 38 | start(Partition, _Config) -> 39 | Table = ets:new(?T(Partition), 40 | [named_table, public, ordered_set]), 41 | {ok, #state{partition=Partition, table=Table}}. 42 | 43 | stop(State) -> 44 | maybe_delete(State). 45 | 46 | index(IFTVPKList, #state{table=Table}=State) -> 47 | lists:foreach( 48 | fun({I, F, T, V, P, K}) -> 49 | Key = {b(I), b(F), b(T), b(V)}, 50 | case ets:lookup(Table, Key) of 51 | [{_, _, ExistingKeyClock}] -> 52 | if ExistingKeyClock > K -> 53 | %% stored data is newer 54 | ok; 55 | true -> 56 | %% stored data is older 57 | ets:update_element(Table, Key, 58 | [{2, P},{3, K}]) 59 | end; 60 | [] -> 61 | ets:insert(Table, {Key, P, K}) 62 | end 63 | end, 64 | IFTVPKList), 65 | {reply, {indexed, node()}, State}. 66 | 67 | delete(IFTVKList, State) -> 68 | Table = State#state.table, 69 | lists:foreach(fun(IFTVK) -> delete_fun(IFTVK, Table) end, IFTVKList), 70 | {reply, {deleted, node()}, State}. 71 | 72 | delete_fun({I, F, T, V, K}, Table) -> 73 | Key = {b(I), b(F), b(T), b(V)}, 74 | case ets:lookup(Table, Key) of 75 | [{Key, _Props, ExistingKeyClock}] -> 76 | if ExistingKeyClock > K -> 77 | %% stored data is newer 78 | ok; 79 | true -> 80 | %% stored data is older 81 | ets:delete(Table, Key) 82 | end; 83 | [] -> 84 | ok 85 | end; 86 | delete_fun({I, F, T, V, _P, K}, Table) -> 87 | %% copied idea from merge_index_backend 88 | %% other operations include Props, though delete shouldn't 89 | delete_fun({I, F, T, V, K}, Table). 90 | 91 | info(Index, Field, Term, Sender, State) -> 92 | Count = ets:select_count(State#state.table, 93 | [{{{b(Index), b(Field), b(Term), '_'}, 94 | '_', '_'}, 95 | [],[true]}]), 96 | riak_search_backend:info_response(Sender, [{Term, node(), Count}]), 97 | noreply. 98 | 99 | -define(STREAM_SIZE, 100). 100 | 101 | range(Index, Field, StartTerm, EndTerm, _Size, FilterFun, Sender, State) -> 102 | ST = b(StartTerm), 103 | ET = b(EndTerm), 104 | spawn(riak_search_ets_backend, stream_results, 105 | [Sender, 106 | FilterFun, 107 | ets:select(State#state.table, 108 | [{{{b(Index), b(Field), '$1', '$2'}, '$3', '_'}, 109 | [{'>=', '$1', ST}, {'=<', '$1', ET}], 110 | [{{'$2', '$3'}}]}], 111 | ?STREAM_SIZE)]), 112 | noreply. 113 | 114 | stream(Index, Field, Term, FilterFun, Sender, State) -> 115 | spawn(riak_search_ets_backend, stream_results, 116 | [Sender, 117 | FilterFun, 118 | ets:select(State#state.table, 119 | [{{{b(Index), b(Field), b(Term), '$1'}, '$2', '_'}, 120 | [], [{{'$1', '$2'}}]}], 121 | ?STREAM_SIZE)]), 122 | noreply. 123 | 124 | stream_results(Sender, FilterFun, {Results0, Continuation}) -> 125 | case lists:filter(fun({V,P}) -> FilterFun(V, P) end, Results0) of 126 | [] -> 127 | ok; 128 | Results -> 129 | riak_search_backend:response_results(Sender, Results) 130 | end, 131 | stream_results(Sender, FilterFun, ets:select(Continuation)); 132 | stream_results(Sender, _, '$end_of_table') -> 133 | riak_search_backend:response_done(Sender). 134 | 135 | fold(FoldFun, Acc, State) -> 136 | Fun = fun({{I,F,T,V},P,K}, {OuterAcc, {{I,{F,T}},InnerAcc}}) -> 137 | %% same IFT, just accumulate doc/props/clock 138 | {OuterAcc, {{I,{F,T}},[{V,P,K}|InnerAcc]}}; 139 | ({{I,F,T,V},P,K}, {OuterAcc, {FoldKey, VPKList}}) -> 140 | %% finished a string of IFT, send it off 141 | %% (sorted order is assumed) 142 | NewOuterAcc = FoldFun(FoldKey, VPKList, OuterAcc), 143 | {NewOuterAcc, {{I,{F,T}},[{V,P,K}]}}; 144 | ({{I,F,T,V},P,K}, {OuterAcc, undefined}) -> 145 | %% first round through the fold - just start building 146 | {OuterAcc, {{I,{F,T}},[{V,P,K}]}} 147 | end, 148 | {OuterAcc0, Final} = ets:foldl(Fun, {Acc, undefined}, State#state.table), 149 | OuterAcc = case Final of 150 | {FoldKey, VPKList} -> 151 | %% one last IFT to send off 152 | FoldFun(FoldKey, VPKList, OuterAcc0); 153 | undefined -> 154 | %% this partition was empty 155 | OuterAcc0 156 | end, 157 | {reply, OuterAcc, State}. 158 | 159 | is_empty(State) -> 160 | 0 == ets:info(State#state.table, size). 161 | 162 | drop(State) -> 163 | maybe_delete(State). 164 | 165 | maybe_delete(State) -> 166 | case lists:member(State#state.table, ets:all()) of 167 | true -> 168 | ets:delete(State#state.table), 169 | ok; 170 | false -> 171 | ok 172 | end. 173 | 174 | b(Binary) when is_binary(Binary) -> Binary; 175 | b(List) when is_list(List) -> iolist_to_binary(List). 176 | -------------------------------------------------------------------------------- /riak/multidict.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # (c) 2005 Ian Bicking and contributors; written for Paste 16 | # (http://pythonpaste.org) Licensed under the MIT license: 17 | # http://www.opensource.org/licenses/mit-license.php 18 | 19 | 20 | class MultiDict(dict): 21 | 22 | """ 23 | An ordered dictionary that can have multiple values for each key. 24 | Adds the methods getall, getone, mixed, and add to the normal 25 | dictionary interface. 26 | """ 27 | 28 | def __init__(self, *args, **kw): 29 | if len(args) > 1: 30 | raise TypeError( 31 | "MultiDict can only be called with one positional argument") 32 | if args: 33 | if hasattr(args[0], 'iteritems'): 34 | items = list(args[0].iteritems()) 35 | elif hasattr(args[0], 'items'): 36 | items = list(args[0].items()) 37 | else: 38 | items = list(args[0]) 39 | self._items = items 40 | else: 41 | self._items = [] 42 | self._items.extend(list(kw.items())) 43 | 44 | def __getitem__(self, key): 45 | for k, v in self._items: 46 | if k == key: 47 | return v 48 | raise KeyError(repr(key)) 49 | 50 | def __setitem__(self, key, value): 51 | try: 52 | del self[key] 53 | except KeyError: 54 | pass 55 | self._items.append((key, value)) 56 | 57 | def add(self, key, value): 58 | """ 59 | Add the key and value, not overwriting any previous value. 60 | """ 61 | self._items.append((key, value)) 62 | 63 | def getall(self, key): 64 | """ 65 | Return a list of all values matching the key (may be an empty list) 66 | """ 67 | result = [] 68 | for k, v in self._items: 69 | if key == k: 70 | result.append(v) 71 | return result 72 | 73 | def getone(self, key): 74 | """ 75 | Get one value matching the key, raising a KeyError if multiple 76 | values were found. 77 | """ 78 | v = self.getall(key) 79 | if not v: 80 | raise KeyError('Key not found: %r' % key) 81 | if len(v) > 1: 82 | raise KeyError('Multiple values match %r: %r' % (key, v)) 83 | return v[0] 84 | 85 | def mixed(self): 86 | """ 87 | Returns a dictionary where the values are either single 88 | values, or a list of values when a key/value appears more than 89 | once in this dictionary. This is similar to the kind of 90 | dictionary often used to represent the variables in a web 91 | request. 92 | """ 93 | result = {} 94 | multi = {} 95 | for key, value in self._items: 96 | if key in result: 97 | # We do this to not clobber any lists that are 98 | # *actual* values in this dictionary: 99 | if key in multi: 100 | result[key].append(value) 101 | else: 102 | result[key] = [result[key], value] 103 | multi[key] = None 104 | else: 105 | result[key] = value 106 | return result 107 | 108 | def dict_of_lists(self): 109 | """ 110 | Returns a dictionary where each key is associated with a 111 | list of values. 112 | """ 113 | result = {} 114 | for key, value in self._items: 115 | if key in result: 116 | result[key].append(value) 117 | else: 118 | result[key] = [value] 119 | return result 120 | 121 | def __delitem__(self, key): 122 | items = self._items 123 | found = False 124 | for i in range(len(items) - 1, -1, -1): 125 | if items[i][0] == key: 126 | del items[i] 127 | found = True 128 | if not found: 129 | raise KeyError(repr(key)) 130 | 131 | def __contains__(self, key): 132 | for k, v in self._items: 133 | if k == key: 134 | return True 135 | return False 136 | 137 | has_key = __contains__ 138 | 139 | def clear(self): 140 | self._items = [] 141 | 142 | def copy(self): 143 | return MultiDict(self) 144 | 145 | def setdefault(self, key, default=None): 146 | for k, v in self._items: 147 | if key == k: 148 | return v 149 | self._items.append((key, default)) 150 | return default 151 | 152 | def pop(self, key, *args): 153 | if len(args) > 1: 154 | raise TypeError("pop expected at most 2 arguments, got %s" % 155 | (1 + len(args))) 156 | for i in range(len(self._items)): 157 | if self._items[i][0] == key: 158 | v = self._items[i][1] 159 | del self._items[i] 160 | return v 161 | if args: 162 | return args[0] 163 | else: 164 | raise KeyError(repr(key)) 165 | 166 | def popitem(self): 167 | return self._items.pop() 168 | 169 | def update(self, other=None, **kwargs): 170 | if other is None: 171 | pass 172 | elif hasattr(other, 'items'): 173 | self._items.extend(other.items()) 174 | elif hasattr(other, 'keys'): 175 | for k in other.keys(): 176 | self._items.append((k, other[k])) 177 | else: 178 | for k, v in other: 179 | self._items.append((k, v)) 180 | if kwargs: 181 | self.update(kwargs) 182 | 183 | def __repr__(self): 184 | items = ', '.join(['(%r, %r)' % v for v in self._items]) 185 | return '%s([%s])' % (self.__class__.__name__, items) 186 | 187 | def __len__(self): 188 | return len(self._items) 189 | 190 | # 191 | # All the iteration: 192 | # 193 | 194 | def keys(self): 195 | return [k for k, v in self._items] 196 | 197 | def iterkeys(self): 198 | for k, v in self._items: 199 | yield k 200 | 201 | __iter__ = iterkeys 202 | 203 | def items(self): 204 | return self._items[:] 205 | 206 | def iteritems(self): 207 | return iter(self._items) 208 | 209 | def values(self): 210 | return [v for k, v in self._items] 211 | 212 | def itervalues(self): 213 | for k, v in self._items: 214 | yield v 215 | -------------------------------------------------------------------------------- /riak/node.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import math 16 | import time 17 | 18 | from threading import RLock 19 | 20 | 21 | class Decaying(object): 22 | """ 23 | A float value which decays exponentially toward 0 over time. This 24 | is used internally to select nodes for new connections that have 25 | had the least errors within the recent period. 26 | """ 27 | 28 | def __init__(self, p=0.0, e=math.e, r=None): 29 | """ 30 | Creates a new decaying error counter. 31 | 32 | :param p: the initial value (defaults to 0.0) 33 | :type p: float 34 | :param e: the exponent base (defaults to math.e) 35 | :type e: float 36 | :param r: timescale factor (defaults to decaying 50% over 10 37 | seconds, i.e. log(0.5) / 10) 38 | :type r: float 39 | """ 40 | self.p = p 41 | self.e = e 42 | self.r = r or (math.log(0.5) / 10) 43 | self.lock = RLock() 44 | self.t0 = time.time() 45 | 46 | def incr(self, d): 47 | """ 48 | Increases the value by the argument. 49 | 50 | :param d: the value to increase by 51 | :type d: float 52 | """ 53 | with self.lock: 54 | self.p = self.value() + d 55 | 56 | def value(self): 57 | """ 58 | Returns the current value (adjusted for the time decay) 59 | 60 | :rtype: float 61 | """ 62 | with self.lock: 63 | now = time.time() 64 | dt = now - self.t0 65 | self.t0 = now 66 | self.p = self.p * (math.pow(self.e, self.r * dt)) 67 | return self.p 68 | 69 | 70 | class RiakNode(object): 71 | """ 72 | The internal representation of a Riak node to which the client can 73 | connect. Encapsulates both the configuration for the node and 74 | error tracking used for node-selection. 75 | """ 76 | 77 | def __init__(self, host='127.0.0.1', http_port=8098, pb_port=8087, 78 | **unused_args): 79 | """ 80 | Creates a node. 81 | 82 | :param host: an IP address or hostname 83 | :type host: string 84 | :param http_port: the HTTP port of the node 85 | :type http_port: integer 86 | :param pb_port: the Protcol Buffers port of the node 87 | :type pb_port: integer 88 | """ 89 | self.host = host 90 | self.http_port = http_port 91 | self.pb_port = pb_port 92 | self.error_rate = Decaying() 93 | -------------------------------------------------------------------------------- /riak/pb/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | -------------------------------------------------------------------------------- /riak/resolver.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | 16 | def default_resolver(riak_object): 17 | """ 18 | The default conflict-resolution function, which does nothing. To 19 | implement a resolver, define a function that sets the 20 | :attr:`siblings ` property 21 | on the passed :class:`RiakObject ` 22 | instance to a list containing a single :class:`RiakContent 23 | ` object. 24 | 25 | :param riak_object: an object-in-conflict that will be resolved 26 | :type riak_object: :class:`RiakObject ` 27 | """ 28 | pass 29 | 30 | 31 | def last_written_resolver(riak_object): 32 | """ 33 | A conflict-resolution function that resolves by selecting the most 34 | recently-modified sibling by timestamp. 35 | 36 | :param riak_object: an object-in-conflict that will be resolved 37 | :type riak_object: :class:`RiakObject ` 38 | """ 39 | riak_object.siblings = [max(riak_object.siblings, 40 | key=lambda x: x.last_modified), ] 41 | -------------------------------------------------------------------------------- /riak/riak_error.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | 16 | class RiakError(Exception): 17 | """ 18 | Base class for exceptions generated in the Riak API. 19 | """ 20 | def __init__(self, *args, **kwargs): 21 | super(RiakError, self).__init__(*args, **kwargs) 22 | if len(args) > 0: 23 | self.value = args[0] 24 | else: 25 | self.value = 'unknown' 26 | 27 | def __str__(self): 28 | return repr(self.value) 29 | 30 | 31 | class ConflictError(RiakError): 32 | """ 33 | Raised when an operation is attempted on a 34 | :class:`~riak.riak_object.RiakObject` that has more than one 35 | sibling. 36 | """ 37 | def __init__(self, message='Object in conflict'): 38 | super(ConflictError, self).__init__(message) 39 | 40 | 41 | class ListError(RiakError): 42 | """ 43 | Raised when a list operation is attempted and 44 | riak.disable_list_exceptions is false. 45 | """ 46 | def __init__(self, message='Bucket and key list operations ' 47 | 'are expensive and should not be ' 48 | 'used in production.'): 49 | super(ListError, self).__init__(message) 50 | -------------------------------------------------------------------------------- /riak/table.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from six import string_types, PY2 16 | 17 | 18 | class Table(object): 19 | """ 20 | The ``Table`` object allows you to access properties on a Riak 21 | timeseries table and query timeseries data. 22 | """ 23 | def __init__(self, client, name): 24 | """ 25 | Returns a new ``Table`` instance. 26 | 27 | :param client: A :class:`RiakClient ` 28 | instance 29 | :type client: :class:`RiakClient ` 30 | :param name: The table's name 31 | :type name: string 32 | """ 33 | if not isinstance(name, string_types): 34 | raise TypeError('Table name must be a string') 35 | 36 | if PY2: 37 | try: 38 | name = name.encode('ascii') 39 | except UnicodeError: 40 | raise TypeError('Unicode table names are not supported.') 41 | 42 | self._client = client 43 | self.name = name 44 | 45 | def __str__(self): 46 | return self.name 47 | 48 | def __repr__(self): 49 | return self.name 50 | 51 | def new(self, rows, columns=None): 52 | """ 53 | A shortcut for manually instantiating a new 54 | :class:`~riak.ts_object.TsObject` 55 | 56 | :param rows: An list of lists with timeseries data 57 | :type rows: list 58 | :param columns: An list of Column names and types. Optional. 59 | :type columns: list 60 | :rtype: :class:`~riak.ts_object.TsObject` 61 | """ 62 | from riak.ts_object import TsObject 63 | 64 | return TsObject(self._client, self, rows, columns) 65 | 66 | def describe(self): 67 | """ 68 | Retrieves a timeseries table's description. 69 | 70 | :rtype: :class:`TsObject ` 71 | """ 72 | return self._client.ts_describe(self) 73 | 74 | def get(self, key): 75 | """ 76 | Gets a value from a timeseries table. 77 | 78 | :param key: The timeseries value's key. 79 | :type key: list 80 | :rtype: :class:`TsObject ` 81 | """ 82 | return self._client.ts_get(self, key) 83 | 84 | def delete(self, key): 85 | """ 86 | Deletes a value from a timeseries table. 87 | 88 | :param key: The timeseries value's key. 89 | :type key: list or dict 90 | :rtype: boolean 91 | """ 92 | return self._client.ts_delete(self, key) 93 | 94 | def query(self, query, interpolations=None): 95 | """ 96 | Queries a timeseries table. 97 | 98 | :param query: The timeseries query. 99 | :type query: string 100 | :rtype: :class:`TsObject ` 101 | """ 102 | return self._client.ts_query(self, query, interpolations) 103 | 104 | def stream_keys(self, timeout=None): 105 | """ 106 | Streams keys from a timeseries table. 107 | 108 | :rtype: list 109 | """ 110 | return self._client.ts_stream_keys(self, timeout) 111 | -------------------------------------------------------------------------------- /riak/tests/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import logging 16 | import os 17 | import socket 18 | import sys 19 | 20 | from riak.test_server import TestServer 21 | from riak.security import SecurityCreds 22 | 23 | USE_TEST_SERVER = int(os.environ.get('USE_TEST_SERVER', '0')) 24 | if USE_TEST_SERVER: 25 | HTTP_PORT = 9000 26 | PB_PORT = 9002 27 | test_server = TestServer() 28 | test_server.cleanup() 29 | test_server.prepare() 30 | test_server.start() 31 | 32 | try: 33 | __import__('riak.pb') 34 | HAVE_PROTO = True 35 | except ImportError: 36 | HAVE_PROTO = False 37 | 38 | 39 | def hostname_resolves(hostname): 40 | try: 41 | socket.gethostbyname(hostname) 42 | return 1 43 | except socket.error: 44 | return 0 45 | 46 | 47 | distutils_debug = os.environ.get('DISTUTILS_DEBUG', '0') 48 | if distutils_debug == '1': 49 | logger = logging.getLogger() 50 | logger.level = logging.DEBUG 51 | logger.addHandler(logging.StreamHandler(sys.stdout)) 52 | 53 | HOST = os.environ.get('RIAK_TEST_HOST', '127.0.0.1') 54 | 55 | PROTOCOL = os.environ.get('RIAK_TEST_PROTOCOL', 'pbc') 56 | 57 | PB_HOST = os.environ.get('RIAK_TEST_PB_HOST', HOST) 58 | PB_PORT = int(os.environ.get('RIAK_TEST_PB_PORT', '8087')) 59 | 60 | HTTP_HOST = os.environ.get('RIAK_TEST_HTTP_HOST', HOST) 61 | HTTP_PORT = int(os.environ.get('RIAK_TEST_HTTP_PORT', '8098')) 62 | 63 | # these ports are used to simulate errors, there shouldn't 64 | # be anything listening on either port. 65 | DUMMY_HTTP_PORT = int(os.environ.get('DUMMY_HTTP_PORT', '1023')) 66 | DUMMY_PB_PORT = int(os.environ.get('DUMMY_PB_PORT', '1022')) 67 | 68 | RUN_BTYPES = int(os.environ.get('RUN_BTYPES', '0')) 69 | RUN_DATATYPES = int(os.environ.get('RUN_DATATYPES', '0')) 70 | RUN_CLIENT = int(os.environ.get('RUN_CLIENT', '0')) 71 | RUN_INDEXES = int(os.environ.get('RUN_INDEXES', '0')) 72 | RUN_KV = int(os.environ.get('RUN_KV', '0')) 73 | RUN_MAPREDUCE = int(os.environ.get('RUN_MAPREDUCE', '0')) 74 | RUN_POOL = int(os.environ.get('RUN_POOL', '0')) 75 | RUN_RESOLVE = int(os.environ.get('RUN_RESOLVE', '0')) 76 | RUN_SEARCH = int(os.environ.get('RUN_SEARCH', '0')) 77 | RUN_TIMESERIES = int(os.environ.get('RUN_TIMESERIES', '0')) 78 | RUN_YZ = int(os.environ.get('RUN_YZ', '0')) 79 | 80 | if PROTOCOL != 'pbc': 81 | RUN_TIMESERIES = 0 82 | 83 | RUN_SECURITY = int(os.environ.get('RUN_SECURITY', '0')) 84 | if RUN_SECURITY: 85 | h = 'riak-test' 86 | if hostname_resolves(h): 87 | HOST = PB_HOST = HTTP_HOST = h 88 | else: 89 | raise AssertionError( 90 | 'RUN_SECURITY requires that the host name' + 91 | ' "riak-test" resolves to the IP address of a Riak node' + 92 | ' with security enabled.') 93 | 94 | SECURITY_USER = os.environ.get('RIAK_TEST_SECURITY_USER', 'riakpass') 95 | SECURITY_PASSWD = os.environ.get('RIAK_TEST_SECURITY_PASSWD', 'Test1234') 96 | 97 | SECURITY_CACERT = os.environ.get('RIAK_TEST_SECURITY_CACERT', 98 | 'tools/test-ca/certs/cacert.pem') 99 | SECURITY_REVOKED = os.environ.get('RIAK_TEST_SECURITY_REVOKED', 100 | 'tools/test-ca/crl/crl.pem') 101 | SECURITY_BAD_CERT = os.environ.get('RIAK_TEST_SECURITY_BAD_CERT', 102 | 'tools/test-ca/certs/badcert.pem') 103 | # Certificate-based Authentication only supported by PBC 104 | SECURITY_KEY = os.environ.get( 105 | 'RIAK_TEST_SECURITY_KEY', 106 | 'tools/test-ca/private/riakuser-client-cert-key.pem') 107 | SECURITY_CERT = os.environ.get('RIAK_TEST_SECURITY_CERT', 108 | 'tools/test-ca/certs/riakuser-client-cert.pem') 109 | SECURITY_CERT_USER = os.environ.get('RIAK_TEST_SECURITY_CERT_USER', 110 | 'riakuser') 111 | 112 | SECURITY_CIPHERS = 'DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:' + \ 113 | 'DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:AES128-SHA256:' + \ 114 | 'AES128-SHA:AES256-SHA256:AES256-SHA:RC4-SHA' 115 | 116 | SECURITY_CREDS = None 117 | if RUN_SECURITY: 118 | SECURITY_CREDS = SecurityCreds(username=SECURITY_USER, 119 | password=SECURITY_PASSWD, 120 | cacert_file=SECURITY_CACERT, 121 | ciphers=SECURITY_CIPHERS) 122 | -------------------------------------------------------------------------------- /riak/tests/base.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # -*- coding: utf-8 -*- 16 | import logging 17 | import random 18 | import riak 19 | 20 | from riak.client import RiakClient 21 | from riak.tests import HOST, PROTOCOL, PB_PORT, HTTP_PORT, SECURITY_CREDS 22 | 23 | 24 | class IntegrationTestBase(object): 25 | host = None 26 | pb_port = None 27 | http_port = None 28 | credentials = None 29 | 30 | @staticmethod 31 | def randint(): 32 | return random.randint(1, 999999) 33 | 34 | @staticmethod 35 | def randname(length=12): 36 | out = '' 37 | for i in range(length): 38 | out += chr(random.randint(ord('a'), ord('z'))) 39 | return out 40 | 41 | @classmethod 42 | def create_client(cls, host=None, http_port=None, pb_port=None, 43 | protocol=None, credentials=None, **kwargs): 44 | host = host or HOST 45 | http_port = http_port or HTTP_PORT 46 | pb_port = pb_port or PB_PORT 47 | 48 | if protocol is None: 49 | if hasattr(cls, 'protocol') and (cls.protocol is not None): 50 | protocol = cls.protocol 51 | else: 52 | protocol = PROTOCOL 53 | 54 | cls.protocol = protocol 55 | 56 | credentials = credentials or SECURITY_CREDS 57 | 58 | if hasattr(cls, 'client_options'): 59 | kwargs.update(cls.client_options) 60 | 61 | logger = logging.getLogger() 62 | logger.debug("RiakClient(protocol='%s', host='%s', pb_port='%d', " 63 | "http_port='%d', credentials='%s', kwargs='%s')", 64 | protocol, host, pb_port, http_port, credentials, kwargs) 65 | 66 | return RiakClient(protocol=protocol, 67 | host=host, 68 | http_port=http_port, 69 | credentials=credentials, 70 | pb_port=pb_port, 71 | **kwargs) 72 | 73 | def setUp(self): 74 | riak.disable_list_exceptions = True 75 | self.bucket_name = self.randname() 76 | self.key_name = self.randname() 77 | self.client = self.create_client() 78 | 79 | def tearDown(self): 80 | riak.disable_list_exceptions = False 81 | self.client.close() 82 | -------------------------------------------------------------------------------- /riak/tests/comparison.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # -*- coding: utf-8 -*- 16 | from six import PY2, PY3 17 | import collections 18 | import warnings 19 | 20 | 21 | class Comparison(object): 22 | ''' 23 | Provide a cross-version object comparison operator 24 | since its name changed between Python 2.x and Python 3.x 25 | ''' 26 | 27 | if PY3: 28 | # Stolen from Python 2.7.8's unittest 29 | _Mismatch = collections.namedtuple('Mismatch', 'actual expected value') 30 | 31 | def _count_diff_all_purpose(self, actual, expected): 32 | ''' 33 | Returns list of (cnt_act, cnt_exp, elem) 34 | triples where the counts differ 35 | ''' 36 | # elements need not be hashable 37 | s, t = list(actual), list(expected) 38 | m, n = len(s), len(t) 39 | NULL = object() 40 | result = [] 41 | for i, elem in enumerate(s): 42 | if elem is NULL: 43 | continue 44 | cnt_s = cnt_t = 0 45 | for j in range(i, m): 46 | if s[j] == elem: 47 | cnt_s += 1 48 | s[j] = NULL 49 | for j, other_elem in enumerate(t): 50 | if other_elem == elem: 51 | cnt_t += 1 52 | t[j] = NULL 53 | if cnt_s != cnt_t: 54 | diff = self._Mismatch(cnt_s, cnt_t, elem) 55 | result.append(diff) 56 | 57 | for i, elem in enumerate(t): 58 | if elem is NULL: 59 | continue 60 | cnt_t = 0 61 | for j in range(i, n): 62 | if t[j] == elem: 63 | cnt_t += 1 64 | t[j] = NULL 65 | diff = self._Mismatch(0, cnt_t, elem) 66 | result.append(diff) 67 | return result 68 | 69 | def _count_diff_hashable(self, actual, expected): 70 | ''' 71 | Returns list of (cnt_act, cnt_exp, elem) triples 72 | where the counts differ 73 | ''' 74 | # elements must be hashable 75 | s, t = self._ordered_count(actual), self._ordered_count(expected) 76 | result = [] 77 | for elem, cnt_s in s.items(): 78 | cnt_t = t.get(elem, 0) 79 | if cnt_s != cnt_t: 80 | diff = self._Mismatch(cnt_s, cnt_t, elem) 81 | result.append(diff) 82 | for elem, cnt_t in t.items(): 83 | if elem not in s: 84 | diff = self._Mismatch(0, cnt_t, elem) 85 | result.append(diff) 86 | return result 87 | 88 | def _ordered_count(self, iterable): 89 | 'Return dict of element counts, in the order they were first seen' 90 | c = collections.OrderedDict() 91 | for elem in iterable: 92 | c[elem] = c.get(elem, 0) + 1 93 | return c 94 | 95 | def assertItemsEqual(self, expected_seq, actual_seq, msg=None): 96 | """An unordered sequence specific comparison. It asserts that 97 | actual_seq and expected_seq have the same element counts. 98 | Equivalent to:: 99 | 100 | self.assertEqual(Counter(iter(actual_seq)), 101 | Counter(iter(expected_seq))) 102 | 103 | Asserts that each element has the same count in both sequences. 104 | Example: 105 | - [0, 1, 1] and [1, 0, 1] compare equal. 106 | - [0, 0, 1] and [0, 1] compare unequal. 107 | """ 108 | first_seq, second_seq = list(expected_seq), list(actual_seq) 109 | with warnings.catch_warnings(): 110 | try: 111 | first = collections.Counter(first_seq) 112 | second = collections.Counter(second_seq) 113 | except TypeError: 114 | # Handle case with unhashable elements 115 | differences = self._count_diff_all_purpose(first_seq, 116 | second_seq) 117 | else: 118 | if first == second: 119 | return 120 | differences = self._count_diff_hashable(first_seq, 121 | second_seq) 122 | 123 | if differences: 124 | standardMsg = 'Element counts were not equal:\n' 125 | lines = ['First has %d, Second has %d: %r' % 126 | diff for diff in differences] 127 | diffMsg = '\n'.join(lines) 128 | standardMsg = self._truncateMessage(standardMsg, diffMsg) 129 | 130 | def assert_raises_regex(self, exception, regexp): 131 | if PY2: 132 | return self.assertRaisesRegexp(exception, regexp) 133 | else: 134 | return self.assertRaisesRegex(exception, regexp) 135 | -------------------------------------------------------------------------------- /riak/tests/pool-grinder.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright 2010-present Basho Technologies, Inc. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | from __future__ import print_function 17 | from six import PY2 18 | from threading import Thread 19 | import sys 20 | from pool import Pool 21 | from random import SystemRandom 22 | from time import sleep 23 | if PY2: 24 | from Queue import Queue 25 | else: 26 | from queue import Queue 27 | sys.path.append("../transports/") 28 | 29 | 30 | class SimplePool(Pool): 31 | def __init__(self): 32 | self.count = 0 33 | Pool.__init__(self) 34 | 35 | def create_resource(self): 36 | self.count += 1 37 | return [self.count] 38 | 39 | def destroy_resource(self, resource): 40 | del resource[:] 41 | 42 | 43 | class EmptyListPool(Pool): 44 | def create_resource(self): 45 | return [] 46 | 47 | 48 | def test(): 49 | started = Queue() 50 | n = 1000 51 | threads = [] 52 | touched = [] 53 | pool = EmptyListPool() 54 | rand = SystemRandom() 55 | 56 | def _run(): 57 | psleep = rand.uniform(0.05, 0.1) 58 | with pool.transaction() as a: 59 | started.put(1) 60 | started.join() 61 | a.append(rand.uniform(0, 1)) 62 | if psleep > 1: 63 | print(psleep) 64 | sleep(psleep) 65 | 66 | for i in range(n): 67 | th = Thread(target=_run) 68 | threads.append(th) 69 | th.start() 70 | 71 | for i in range(n): 72 | started.get() 73 | started.task_done() 74 | 75 | for element in pool: 76 | touched.append(element) 77 | 78 | for thr in threads: 79 | thr.join() 80 | 81 | if set(pool.elements) != set(touched): 82 | print(set(pool.elements) - set(touched)) 83 | return False 84 | else: 85 | return True 86 | 87 | 88 | ret = True 89 | count = 0 90 | while ret: 91 | ret = test() 92 | count += 1 93 | print(count) 94 | 95 | 96 | # INSTRUMENTED FUNCTION 97 | 98 | # def __claim_elements(self): 99 | # #print('waiting for self lock') 100 | # with self.lock: 101 | # if self.__all_claimed(): # and self.unlocked: 102 | # #print('waiting on releaser lock') 103 | # with self.releaser: 104 | # print('waiting for release'') 105 | # print('targets', self.targets) 106 | # print('tomb', self.targets[0].tomb) 107 | # print('claimed', self.targets[0].claimed) 108 | # print(self.releaser) 109 | # print(self.lock) 110 | # print(self.unlocked) 111 | # self.releaser.wait(1) 112 | # for element in self.targets: 113 | # if element.tomb: 114 | # self.targets.remove(element) 115 | # #self.unlocked.remove(element) 116 | # continue 117 | # if not element.claimed: 118 | # self.targets.remove(element) 119 | # self.unlocked.append(element) 120 | # element.claimed = True 121 | -------------------------------------------------------------------------------- /riak/tests/suite.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import os.path 16 | import unittest 17 | 18 | 19 | def additional_tests(): 20 | top_level = os.path.join(os.path.dirname(__file__), "../../") 21 | start_dir = os.path.dirname(__file__) 22 | suite = unittest.TestSuite() 23 | suite.addTest(unittest.TestLoader().discover(start_dir, 24 | top_level_dir=top_level)) 25 | return suite 26 | -------------------------------------------------------------------------------- /riak/tests/test_datetime.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # -*- coding: utf-8 -*- 16 | import datetime 17 | import unittest 18 | 19 | from riak.util import epoch, epoch_tz, \ 20 | unix_time_millis 21 | 22 | # NB: without tzinfo, this is UTC 23 | ts0 = datetime.datetime(2015, 1, 1, 12, 1, 2, 987000) 24 | ts0_ts = 1420113662987 25 | ts0_ts_pst = 1420142462987 26 | 27 | 28 | class DatetimeUnitTests(unittest.TestCase): 29 | def test_get_unix_time_without_tzinfo(self): 30 | self.assertIsNone(epoch.tzinfo) 31 | self.assertIsNotNone(epoch_tz.tzinfo) 32 | self.assertIsNone(ts0.tzinfo) 33 | utm = unix_time_millis(ts0) 34 | self.assertEqual(utm, ts0_ts) 35 | 36 | def test_get_unix_time_with_tzinfo(self): 37 | try: 38 | import pytz 39 | tz = pytz.timezone('America/Los_Angeles') 40 | ts0_pst = tz.localize(ts0) 41 | utm = unix_time_millis(ts0_pst) 42 | self.assertEqual(utm, ts0_ts_pst) 43 | except ImportError: 44 | pass 45 | -------------------------------------------------------------------------------- /riak/tests/test_filters.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | # -*- coding: utf-8 -*- 16 | import unittest 17 | 18 | from riak.mapreduce import RiakKeyFilter 19 | from riak import key_filter 20 | 21 | 22 | class FilterTests(unittest.TestCase): 23 | def test_simple(self): 24 | f1 = RiakKeyFilter("tokenize", "-", 1) 25 | self.assertEqual(f1._filters, [["tokenize", "-", 1]]) 26 | 27 | def test_add(self): 28 | f1 = RiakKeyFilter("tokenize", "-", 1) 29 | f2 = RiakKeyFilter("eq", "2005") 30 | f3 = f1 + f2 31 | self.assertEqual(list(f3), [["tokenize", "-", 1], ["eq", "2005"]]) 32 | 33 | def test_and(self): 34 | f1 = RiakKeyFilter("starts_with", "2005-") 35 | f2 = RiakKeyFilter("ends_with", "-01") 36 | f3 = f1 & f2 37 | self.assertEqual(list(f3), 38 | [["and", 39 | [["starts_with", "2005-"]], 40 | [["ends_with", "-01"]]]]) 41 | 42 | def test_multi_and(self): 43 | f1 = RiakKeyFilter("starts_with", "2005-") 44 | f2 = RiakKeyFilter("ends_with", "-01") 45 | f3 = RiakKeyFilter("matches", "-11-") 46 | f4 = f1 & f2 & f3 47 | self.assertEqual(list(f4), [["and", 48 | [["starts_with", "2005-"]], 49 | [["ends_with", "-01"]], 50 | [["matches", "-11-"]], 51 | ]]) 52 | 53 | def test_or(self): 54 | f1 = RiakKeyFilter("starts_with", "2005-") 55 | f2 = RiakKeyFilter("ends_with", "-01") 56 | f3 = f1 | f2 57 | self.assertEqual(list(f3), [["or", [["starts_with", "2005-"]], 58 | [["ends_with", "-01"]]]]) 59 | 60 | def test_multi_or(self): 61 | f1 = RiakKeyFilter("starts_with", "2005-") 62 | f2 = RiakKeyFilter("ends_with", "-01") 63 | f3 = RiakKeyFilter("matches", "-11-") 64 | f4 = f1 | f2 | f3 65 | self.assertEqual(list(f4), [["or", 66 | [["starts_with", "2005-"]], 67 | [["ends_with", "-01"]], 68 | [["matches", "-11-"]], 69 | ]]) 70 | 71 | def test_chaining(self): 72 | f1 = key_filter.tokenize("-", 1).eq("2005") 73 | f2 = key_filter.tokenize("-", 2).eq("05") 74 | f3 = f1 & f2 75 | self.assertEqual(list(f3), [["and", 76 | [["tokenize", "-", 1], ["eq", "2005"]], 77 | [["tokenize", "-", 2], ["eq", "05"]] 78 | ]]) 79 | -------------------------------------------------------------------------------- /riak/tests/test_misc.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import unittest 16 | 17 | 18 | class MiscTests(unittest.TestCase): 19 | def test_timeout_validation(self): 20 | from riak.client.operations import _validate_timeout 21 | # valid cases 22 | try: 23 | _validate_timeout(None) 24 | _validate_timeout(None, infinity_ok=True) 25 | _validate_timeout('infinity', infinity_ok=True) 26 | _validate_timeout(1234) 27 | _validate_timeout(1234567898765432123456789) 28 | except ValueError: 29 | self.fail('_validate_timeout() unexpectedly raised ValueError') 30 | # invalid cases 31 | with self.assertRaises(ValueError): 32 | _validate_timeout('infinity') 33 | with self.assertRaises(ValueError): 34 | _validate_timeout('infinity-foo') 35 | with self.assertRaises(ValueError): 36 | _validate_timeout('foobarbaz') 37 | with self.assertRaises(ValueError): 38 | _validate_timeout('1234') 39 | with self.assertRaises(ValueError): 40 | _validate_timeout(0) 41 | with self.assertRaises(ValueError): 42 | _validate_timeout(12.34) 43 | -------------------------------------------------------------------------------- /riak/tests/test_server_test.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import sys 16 | import unittest 17 | 18 | from riak.test_server import TestServer 19 | 20 | 21 | @unittest.skipIf(sys.platform == 'win32', 'Windows is not supported') 22 | class TestServerTestCase(unittest.TestCase): 23 | def setUp(self): 24 | self.test_server = TestServer() 25 | 26 | def tearDown(self): 27 | pass 28 | 29 | def test_options_defaults(self): 30 | self.assertEqual( 31 | self.test_server.app_config["riak_core"]["handoff_port"], 9001) 32 | self.assertEqual( 33 | self.test_server.app_config["riak_kv"]["pb_ip"], "127.0.0.1") 34 | 35 | def test_merge_riak_core_options(self): 36 | self.test_server = TestServer(riak_core={"handoff_port": 10000}) 37 | self.assertEqual( 38 | self.test_server.app_config["riak_core"]["handoff_port"], 10000) 39 | 40 | def test_merge_riak_search_options(self): 41 | self.test_server = TestServer( 42 | riak_search={"search_backend": "riak_search_backend"}) 43 | self.assertEqual( 44 | self.test_server.app_config["riak_search"]["search_backend"], 45 | "riak_search_backend") 46 | 47 | def test_merge_riak_kv_options(self): 48 | self.test_server = TestServer(riak_kv={"pb_ip": "192.168.2.1"}) 49 | self.assertEqual(self.test_server.app_config["riak_kv"]["pb_ip"], 50 | "192.168.2.1") 51 | 52 | def test_merge_vmargs(self): 53 | self.test_server = TestServer(vm_args={"-P": 65000}) 54 | self.assertEqual(self.test_server.vm_args["-P"], 65000) 55 | 56 | def test_set_ring_state_dir(self): 57 | self.assertEqual( 58 | self.test_server.app_config["riak_core"]["ring_state_dir"], 59 | "/tmp/riak/test_server/data/ring") 60 | 61 | def test_set_default_tmp_dir(self): 62 | self.assertEqual(self.test_server.temp_dir, "/tmp/riak/test_server") 63 | 64 | def test_set_non_default_tmp_dir(self): 65 | tmp_dir = '/not/the/default/dir' 66 | server = TestServer(tmp_dir=tmp_dir) 67 | self.assertEqual(server.temp_dir, tmp_dir) 68 | 69 | 70 | def suite(): 71 | suite = unittest.TestSuite() 72 | suite.addTest(TestServerTestCase()) 73 | return suite 74 | -------------------------------------------------------------------------------- /riak/tests/test_util.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import datetime 16 | import unittest 17 | 18 | from riak.util import is_timeseries_supported, \ 19 | datetime_from_unix_time_millis, \ 20 | unix_time_millis 21 | 22 | 23 | class UtilUnitTests(unittest.TestCase): 24 | # NB: 25 | # 144379690 secs, 987 msecs past epoch 26 | # 144379690987 total msecs past epoch 27 | def test_conv_ms_timestamp_to_datetime_and_back(self): 28 | if is_timeseries_supported(): 29 | # this is what would be stored in Riak TS 30 | v = 144379690987 31 | dt = datetime_from_unix_time_millis(v) 32 | 33 | # This is how Python represents the above 34 | utp = 144379690.987000 35 | dtp = datetime.datetime.utcfromtimestamp(utp) 36 | self.assertEqual(dt, dtp) 37 | 38 | utm = unix_time_millis(dt) 39 | self.assertEqual(v, utm) 40 | else: 41 | pass 42 | 43 | def test_conv_datetime_to_unix_millis(self): 44 | # This is the "native" Python unix timestamp including 45 | # microseconds, as float. timedelta "total_seconds()" 46 | # returns a value like this 47 | if is_timeseries_supported(): 48 | v = 144379690.987000 49 | d = datetime.datetime.utcfromtimestamp(v) 50 | utm = unix_time_millis(d) 51 | self.assertEqual(utm, 144379690987) 52 | else: 53 | pass 54 | 55 | def test_unix_millis_validation(self): 56 | v = 144379690.987 57 | with self.assertRaises(ValueError): 58 | datetime_from_unix_time_millis(v) 59 | 60 | def test_unix_millis_small_value(self): 61 | if is_timeseries_supported(): 62 | # this is what would be stored in Riak TS 63 | v = 1001 64 | dt = datetime_from_unix_time_millis(v) 65 | 66 | # This is how Python represents the above 67 | utp = 1.001 68 | dtp = datetime.datetime.utcfromtimestamp(utp) 69 | self.assertEqual(dt, dtp) 70 | 71 | utm = unix_time_millis(dt) 72 | self.assertEqual(v, utm) 73 | else: 74 | pass 75 | 76 | def test_is_timeseries_supported(self): 77 | v = (2, 7, 10) 78 | self.assertEqual(True, is_timeseries_supported(v)) 79 | v = (2, 7, 11) 80 | self.assertEqual(True, is_timeseries_supported(v)) 81 | v = (2, 7, 12) 82 | self.assertEqual(True, is_timeseries_supported(v)) 83 | v = (3, 3, 6) 84 | self.assertEqual(False, is_timeseries_supported(v)) 85 | v = (3, 4, 3) 86 | self.assertEqual(False, is_timeseries_supported(v)) 87 | v = (3, 4, 4) 88 | self.assertEqual(True, is_timeseries_supported(v)) 89 | v = (3, 4, 5) 90 | self.assertEqual(True, is_timeseries_supported(v)) 91 | v = (3, 5, 0) 92 | self.assertEqual(False, is_timeseries_supported(v)) 93 | v = (3, 5, 1) 94 | self.assertEqual(True, is_timeseries_supported(v)) 95 | -------------------------------------------------------------------------------- /riak/tests/yz_setup.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import logging 16 | import riak 17 | 18 | from riak import RiakError 19 | from riak.tests import RUN_YZ 20 | from riak.tests.base import IntegrationTestBase 21 | 22 | 23 | def yzSetUp(*yzdata): 24 | if RUN_YZ: 25 | riak.disable_list_exceptions = True 26 | c = IntegrationTestBase.create_client() 27 | for yz in yzdata: 28 | logging.debug("yzSetUp: %s", yz) 29 | c.create_search_index(yz['index'], timeout=30000) 30 | if yz['btype'] is not None: 31 | t = c.bucket_type(yz['btype']) 32 | b = t.bucket(yz['bucket']) 33 | else: 34 | b = c.bucket(yz['bucket']) 35 | # Keep trying to set search bucket property until it succeeds 36 | index_set = False 37 | while not index_set: 38 | try: 39 | b.set_property('search_index', yz['index']) 40 | index_set = True 41 | except RiakError: 42 | pass 43 | c.close() 44 | 45 | 46 | def yzTearDown(c, *yzdata): 47 | if RUN_YZ: 48 | riak.disable_list_exceptions = True 49 | c = IntegrationTestBase.create_client() 50 | for yz in yzdata: 51 | logging.debug("yzTearDown: %s", yz) 52 | if yz['btype'] is not None: 53 | t = c.bucket_type(yz['btype']) 54 | b = t.bucket(yz['bucket']) 55 | else: 56 | b = c.bucket(yz['bucket']) 57 | b.set_property('search_index', '_dont_index_') 58 | c.delete_search_index(yz['index']) 59 | for keys in b.stream_keys(): 60 | for key in keys: 61 | b.delete(key) 62 | c.close() 63 | riak.disable_list_exceptions = False 64 | -------------------------------------------------------------------------------- /riak/transports/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | -------------------------------------------------------------------------------- /riak/transports/feature_detect.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from distutils.version import LooseVersion 16 | from riak.util import lazy_property 17 | 18 | 19 | versions = { 20 | 1: LooseVersion('1.0.0'), 21 | 1.1: LooseVersion('1.1.0'), 22 | 1.2: LooseVersion('1.2.0'), 23 | 1.4: LooseVersion('1.4.0'), 24 | 1.44: LooseVersion('1.4.4'), 25 | 2.0: LooseVersion('2.0.0'), 26 | 2.1: LooseVersion('2.1.0'), 27 | 2.12: LooseVersion('2.1.2') 28 | } 29 | 30 | 31 | class FeatureDetection(object): 32 | """ 33 | Implements boolean methods that can be checked for the presence of 34 | specific server-side features. Subclasses must implement the 35 | :meth:`_server_version` method to use this functionality, which 36 | should return the server's version as a string. 37 | 38 | :class:`FeatureDetection` is a parent class of 39 | :class:`Transport `. 40 | """ 41 | 42 | def _server_version(self): 43 | """ 44 | Gets the server version from the server. To be implemented by 45 | the individual transport class. 46 | 47 | :rtype: string 48 | """ 49 | raise NotImplementedError 50 | 51 | def phaseless_mapred(self): 52 | """ 53 | Whether MapReduce requests can be submitted without phases. 54 | 55 | :rtype: bool 56 | """ 57 | return self.server_version >= versions[1.1] 58 | 59 | def pb_indexes(self): 60 | """ 61 | Whether secondary index queries are supported over Protocol 62 | Buffers 63 | 64 | :rtype: bool 65 | """ 66 | return self.server_version >= versions[1.2] 67 | 68 | def pb_search_admin(self): 69 | """ 70 | Whether search administration is supported over Protocol Buffers 71 | 72 | :rtype: bool 73 | """ 74 | return self.server_version >= versions[2.0] 75 | 76 | def pb_search(self): 77 | """ 78 | Whether search queries are supported over Protocol Buffers 79 | 80 | :rtype: bool 81 | """ 82 | return self.server_version >= versions[1.2] 83 | 84 | def pb_conditionals(self): 85 | """ 86 | Whether conditional fetch/store semantics are supported over 87 | Protocol Buffers 88 | 89 | :rtype: bool 90 | """ 91 | return self.server_version >= versions[1] 92 | 93 | def quorum_controls(self): 94 | """ 95 | Whether additional quorums and FSM controls are available, 96 | e.g. primary quorums, basic_quorum, notfound_ok 97 | 98 | :rtype: bool 99 | """ 100 | return self.server_version >= versions[1] 101 | 102 | def tombstone_vclocks(self): 103 | """ 104 | Whether 'not found' responses might include vclocks 105 | 106 | :rtype: bool 107 | """ 108 | return self.server_version >= versions[1] 109 | 110 | def pb_head(self): 111 | """ 112 | Whether partial-fetches (vclock and metadata only) are 113 | supported over Protocol Buffers 114 | 115 | :rtype: bool 116 | """ 117 | return self.server_version >= versions[1] 118 | 119 | def pb_clear_bucket_props(self): 120 | """ 121 | Whether bucket properties can be cleared over Protocol 122 | Buffers. 123 | 124 | :rtype: bool 125 | """ 126 | return self.server_version >= versions[1.4] 127 | 128 | def pb_all_bucket_props(self): 129 | """ 130 | Whether all normal bucket properties are supported over 131 | Protocol Buffers. 132 | 133 | :rtype: bool 134 | """ 135 | return self.server_version >= versions[1.4] 136 | 137 | def counters(self): 138 | """ 139 | Whether CRDT counters are supported. 140 | 141 | :rtype: bool 142 | """ 143 | return self.server_version >= versions[1.4] 144 | 145 | def bucket_stream(self): 146 | """ 147 | Whether streaming bucket lists are supported. 148 | 149 | :rtype: bool 150 | """ 151 | return self.server_version >= versions[1.4] 152 | 153 | def client_timeouts(self): 154 | """ 155 | Whether client-supplied timeouts are supported. 156 | 157 | :rtype: bool 158 | """ 159 | return self.server_version >= versions[1.4] 160 | 161 | def stream_indexes(self): 162 | """ 163 | Whether secondary indexes support streaming responses. 164 | 165 | :rtype: bool 166 | """ 167 | return self.server_version >= versions[1.4] 168 | 169 | def index_term_regex(self): 170 | """ 171 | Whether secondary indexes supports a regexp term filter. 172 | 173 | :rtype: bool 174 | """ 175 | return self.server_version >= versions[1.44] 176 | 177 | def bucket_types(self): 178 | """ 179 | Whether bucket-types are supported. 180 | 181 | :rtype: bool 182 | """ 183 | return self.server_version >= versions[2.0] 184 | 185 | def datatypes(self): 186 | """ 187 | Whether datatypes are supported. 188 | 189 | :rtype: bool 190 | """ 191 | return self.server_version >= versions[2.0] 192 | 193 | def preflists(self): 194 | """ 195 | Whether bucket/key preflists are supported. 196 | 197 | :rtype: bool 198 | """ 199 | return self.server_version >= versions[2.1] 200 | 201 | def write_once(self): 202 | """ 203 | Whether write-once operations are supported. 204 | 205 | :rtype: bool 206 | """ 207 | return self.server_version >= versions[2.1] 208 | 209 | @lazy_property 210 | def server_version(self): 211 | return LooseVersion(self._server_version()) 212 | -------------------------------------------------------------------------------- /riak/transports/http/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import socket 16 | import select 17 | 18 | from six import PY2 19 | from riak.security import SecurityError, USE_STDLIB_SSL 20 | from riak.transports.pool import Pool 21 | from riak.transports.http.transport import HttpTransport 22 | 23 | if USE_STDLIB_SSL: 24 | import ssl 25 | from riak.transports.security import configure_ssl_context 26 | else: 27 | import OpenSSL.SSL 28 | from riak.transports.security import RiakWrappedSocket,\ 29 | configure_pyopenssl_context 30 | 31 | if PY2: 32 | from httplib import HTTPConnection, \ 33 | NotConnected, \ 34 | IncompleteRead, \ 35 | ImproperConnectionState, \ 36 | BadStatusLine, \ 37 | HTTPSConnection 38 | else: 39 | from http.client import HTTPConnection, \ 40 | HTTPSConnection, \ 41 | NotConnected, \ 42 | IncompleteRead, \ 43 | ImproperConnectionState, \ 44 | BadStatusLine 45 | 46 | 47 | class NoNagleHTTPConnection(HTTPConnection): 48 | """ 49 | Setup a connection class which does not use Nagle - deal with 50 | latency on PUT requests lower than MTU 51 | """ 52 | def connect(self): 53 | """ 54 | Set TCP_NODELAY on socket 55 | """ 56 | HTTPConnection.connect(self) 57 | self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) 58 | 59 | 60 | # Inspired by 61 | # http://code.activestate.com/recipes/577548-https-httplib-client-connection-with-certificate-v/ 62 | class RiakHTTPSConnection(HTTPSConnection): 63 | def __init__(self, 64 | host, 65 | port, 66 | credentials, 67 | pkey_file=None, 68 | cert_file=None, 69 | timeout=None): 70 | """ 71 | Class to make a HTTPS connection, 72 | with support for full client-based SSL Authentication 73 | 74 | :param host: Riak host name 75 | :type host: str 76 | :param port: Riak host port number 77 | :type port: int 78 | :param credentials: Security Credential settings 79 | :type credentials: SecurityCreds 80 | :param pkey_file: PEM formatted file that contains your private key 81 | :type pkey_file: str 82 | :param cert_file: PEM formatted certificate chain file 83 | :type cert_file: str 84 | :param timeout: Number of seconds before timing out 85 | :type timeout: int 86 | """ 87 | if PY2: 88 | # NB: it appears that pkey_file / cert_file are never set 89 | # in riak/transports/http/connection.py#_connect() method 90 | pkf = pkey_file 91 | if pkf is None and credentials is not None: 92 | pkf = credentials._pkey_file 93 | 94 | cf = cert_file 95 | if cf is None and credentials is not None: 96 | cf = credentials._cert_file 97 | 98 | HTTPSConnection.__init__(self, 99 | host, 100 | port, 101 | key_file=pkf, 102 | cert_file=cf) 103 | else: 104 | super(RiakHTTPSConnection, self). \ 105 | __init__(host=host, 106 | port=port, 107 | key_file=credentials._pkey_file, 108 | cert_file=credentials._cert_file) 109 | self.pkey_file = pkey_file 110 | self.cert_file = cert_file 111 | self.credentials = credentials 112 | self.timeout = timeout 113 | 114 | def connect(self): 115 | """ 116 | Connect to a host on a given (SSL) port using PyOpenSSL. 117 | """ 118 | sock = socket.create_connection((self.host, self.port), self.timeout) 119 | if not USE_STDLIB_SSL: 120 | ssl_ctx = configure_pyopenssl_context(self.credentials) 121 | 122 | # attempt to upgrade the socket to TLS 123 | cxn = OpenSSL.SSL.Connection(ssl_ctx, sock) 124 | cxn.set_connect_state() 125 | while True: 126 | try: 127 | cxn.do_handshake() 128 | except OpenSSL.SSL.WantReadError: 129 | select.select([sock], [], []) 130 | continue 131 | except OpenSSL.SSL.Error as e: 132 | raise SecurityError('bad handshake - ' + str(e)) 133 | break 134 | 135 | self.sock = RiakWrappedSocket(cxn, sock) 136 | self.credentials._check_revoked_cert(self.sock) 137 | else: 138 | ssl_ctx = configure_ssl_context(self.credentials) 139 | if self.timeout is not None: 140 | sock.settimeout(self.timeout) 141 | self.sock = ssl.SSLSocket(sock=sock, 142 | keyfile=self.credentials.pkey_file, 143 | certfile=self.credentials.cert_file, 144 | cert_reqs=ssl.CERT_REQUIRED, 145 | ca_certs=self.credentials.cacert_file, 146 | ciphers=self.credentials.ciphers, 147 | server_hostname=self.host) 148 | self.sock.context = ssl_ctx 149 | 150 | 151 | class HttpPool(Pool): 152 | """ 153 | A pool of HTTP(S) transport connections. 154 | """ 155 | def __init__(self, client, **options): 156 | self.client = client 157 | self.options = options 158 | self.connection_class = NoNagleHTTPConnection 159 | if self.client._credentials: 160 | self.connection_class = RiakHTTPSConnection 161 | 162 | super(HttpPool, self).__init__() 163 | 164 | def create_resource(self): 165 | node = self.client._choose_node() 166 | return HttpTransport(node=node, 167 | client=self.client, 168 | connection_class=self.connection_class, 169 | **self.options) 170 | 171 | def destroy_resource(self, transport): 172 | transport.close() 173 | 174 | 175 | CONN_CLOSED_ERRORS = ( 176 | NotConnected, 177 | IncompleteRead, 178 | ImproperConnectionState, 179 | BadStatusLine 180 | ) 181 | 182 | 183 | def is_retryable(err): 184 | """ 185 | Determines if the given exception is something that is 186 | network/socket-related and should thus cause the HTTP connection 187 | to close and the operation retried on another node. 188 | 189 | :rtype: boolean 190 | """ 191 | for errtype in CONN_CLOSED_ERRORS: 192 | if isinstance(err, errtype): 193 | return True 194 | return False 195 | -------------------------------------------------------------------------------- /riak/transports/http/connection.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import base64 16 | 17 | from six import PY2 18 | from riak.util import str_to_bytes 19 | 20 | if PY2: 21 | from httplib import NotConnected, HTTPConnection 22 | else: 23 | from http.client import NotConnected, HTTPConnection 24 | 25 | 26 | class HttpConnection(object): 27 | """ 28 | Connection and low-level request methods for HttpTransport. 29 | """ 30 | 31 | def _request(self, method, uri, headers={}, body='', stream=False): 32 | """ 33 | Given a Method, URL, Headers, and Body, perform and HTTP 34 | request, and return a 3-tuple containing the response status, 35 | response headers (as httplib.HTTPMessage), and response body. 36 | """ 37 | response = None 38 | headers.setdefault('Accept', 39 | 'multipart/mixed, application/json, */*;q=0.5') 40 | 41 | if self._client._credentials: 42 | self._security_auth_headers(self._client._credentials.username, 43 | self._client._credentials.password, 44 | headers) 45 | 46 | try: 47 | self._connection.request(method, uri, body, headers) 48 | try: 49 | response = self._connection.getresponse(buffering=True) 50 | except TypeError: 51 | response = self._connection.getresponse() 52 | 53 | if stream: 54 | # The caller is responsible for fully reading the 55 | # response and closing it when streaming. 56 | response_body = response 57 | else: 58 | response_body = response.read() 59 | finally: 60 | if response and not stream: 61 | response.close() 62 | 63 | return response.status, response.msg, response_body 64 | 65 | def _connect(self): 66 | """ 67 | Use the appropriate connection class; optionally with security. 68 | """ 69 | timeout = None 70 | if self._options is not None and 'timeout' in self._options: 71 | timeout = self._options['timeout'] 72 | 73 | if self._client._credentials: 74 | self._connection = self._connection_class( 75 | host=self._node.host, 76 | port=self._node.http_port, 77 | credentials=self._client._credentials, 78 | timeout=timeout) 79 | else: 80 | self._connection = self._connection_class( 81 | host=self._node.host, 82 | port=self._node.http_port, 83 | timeout=timeout) 84 | # Forces the population of stats and resources before any 85 | # other requests are made. 86 | self.server_version 87 | 88 | def close(self): 89 | """ 90 | Closes the underlying HTTP connection. 91 | """ 92 | try: 93 | self._connection.close() 94 | except NotConnected: 95 | pass 96 | 97 | # These are set by the HttpTransport initializer 98 | _connection_class = HTTPConnection 99 | _node = None 100 | 101 | def _security_auth_headers(self, username, password, headers): 102 | """ 103 | Add in the requisite HTTP Authentication Headers 104 | 105 | :param username: Riak Security Username 106 | :type str 107 | :param password: Riak Security Password 108 | :type str 109 | :param headers: Dictionary of headers 110 | :type dict 111 | """ 112 | userColonPassword = username + ":" + password 113 | b64UserColonPassword = base64. \ 114 | b64encode(str_to_bytes(userColonPassword)).decode("ascii") 115 | headers['Authorization'] = 'Basic %s' % b64UserColonPassword 116 | -------------------------------------------------------------------------------- /riak/transports/http/search.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | 16 | class XMLSearchResult(object): 17 | # Match tags that are document fields 18 | fieldtags = ['str', 'int', 'date'] 19 | 20 | def __init__(self): 21 | # Results 22 | self.num_found = 0 23 | self.max_score = 0.0 24 | self.docs = [] 25 | 26 | # Parser state 27 | self.currdoc = None 28 | self.currfield = None 29 | self.currvalue = None 30 | 31 | def start(self, tag, attrib): 32 | if tag == 'result': 33 | self.num_found = int(attrib['numFound']) 34 | self.max_score = float(attrib['maxScore']) 35 | elif tag == 'doc': 36 | self.currdoc = {} 37 | elif tag in self.fieldtags and self.currdoc is not None: 38 | self.currfield = attrib['name'] 39 | 40 | def end(self, tag): 41 | if tag == 'doc' and self.currdoc is not None: 42 | self.docs.append(self.currdoc) 43 | self.currdoc = None 44 | elif tag in self.fieldtags and self.currdoc is not None: 45 | if tag == 'int': 46 | self.currvalue = int(self.currvalue) 47 | self.currdoc[self.currfield] = self.currvalue 48 | self.currfield = None 49 | self.currvalue = None 50 | 51 | def data(self, data): 52 | if self.currfield: 53 | # riak_solr_output adds NL + 6 spaces 54 | data = data.rstrip() 55 | if self.currvalue: 56 | self.currvalue += data 57 | else: 58 | self.currvalue = data 59 | 60 | def close(self): 61 | return {'num_found': self.num_found, 62 | 'max_score': self.max_score, 63 | 'docs': self.docs} 64 | -------------------------------------------------------------------------------- /riak/transports/http/stream.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import json 16 | import re 17 | 18 | from cgi import parse_header 19 | from email import message_from_string 20 | from riak.util import decode_index_value 21 | from riak.client.index_page import CONTINUATION 22 | from riak import RiakError 23 | from six import PY2 24 | 25 | 26 | class HttpStream(object): 27 | """ 28 | Base class for HTTP streaming iterators. 29 | """ 30 | 31 | BLOCK_SIZE = 2048 32 | 33 | def __init__(self, response): 34 | self.response = response 35 | self.buffer = '' 36 | self.response_done = False 37 | self.resource = None 38 | 39 | def __iter__(self): 40 | return self 41 | 42 | def _read(self): 43 | chunk = self.response.read(self.BLOCK_SIZE) 44 | if PY2: 45 | if chunk == '': 46 | self.response_done = True 47 | self.buffer += chunk 48 | else: 49 | if chunk == b'': 50 | self.response_done = True 51 | self.buffer += chunk.decode('utf-8') 52 | 53 | def __next__(self): 54 | raise NotImplementedError 55 | 56 | def next(self): 57 | raise NotImplementedError 58 | 59 | def attach(self, resource): 60 | self.resource = resource 61 | 62 | def close(self): 63 | self.resource.release() 64 | 65 | 66 | class HttpJsonStream(HttpStream): 67 | _json_field = None 68 | 69 | def next(self): 70 | # Python 2.x Version 71 | while '}' not in self.buffer and not self.response_done: 72 | self._read() 73 | 74 | if '}' in self.buffer: 75 | idx = self.buffer.index('}') + 1 76 | chunk = self.buffer[:idx] 77 | self.buffer = self.buffer[idx:] 78 | jsdict = json.loads(chunk) 79 | if 'error' in jsdict: 80 | self.close() 81 | raise RiakError(jsdict['error']) 82 | field = jsdict[self._json_field] 83 | return field 84 | else: 85 | raise StopIteration 86 | 87 | def __next__(self): 88 | # Python 3.x Version 89 | return self.next() 90 | 91 | 92 | class HttpKeyStream(HttpJsonStream): 93 | """ 94 | Streaming iterator for list-keys over HTTP 95 | """ 96 | _json_field = u'keys' 97 | 98 | 99 | class HttpBucketStream(HttpJsonStream): 100 | """ 101 | Streaming iterator for list-buckets over HTTP 102 | """ 103 | _json_field = u'buckets' 104 | 105 | 106 | class HttpMultipartStream(HttpStream): 107 | """ 108 | Streaming iterator for multipart messages over HTTP 109 | """ 110 | def __init__(self, response): 111 | super(HttpMultipartStream, self).__init__(response) 112 | ctypehdr = response.getheader('content-type') 113 | _, params = parse_header(ctypehdr) 114 | self.boundary_re = re.compile('\r?\n--%s(?:--)?\r?\n' % 115 | re.escape(params['boundary'])) 116 | self.next_boundary = None 117 | self.seen_first = False 118 | 119 | def next(self): 120 | # multipart/mixed starts with a boundary, then the first part. 121 | if not self.seen_first: 122 | self.read_until_boundary() 123 | self.advance_buffer() 124 | self.seen_first = True 125 | 126 | self.read_until_boundary() 127 | 128 | if self.next_boundary: 129 | part = self.advance_buffer() 130 | message = message_from_string(part) 131 | return message 132 | else: 133 | raise StopIteration 134 | 135 | def __next__(self): 136 | # Python 3.x Version 137 | return self.next() 138 | 139 | def try_match(self): 140 | self.next_boundary = self.boundary_re.search(self.buffer) 141 | return self.next_boundary 142 | 143 | def advance_buffer(self): 144 | part = self.buffer[:self.next_boundary.start()] 145 | self.buffer = self.buffer[self.next_boundary.end():] 146 | self.next_boundary = None 147 | return part 148 | 149 | def read_until_boundary(self): 150 | while not self.try_match() and not self.response_done: 151 | self._read() 152 | 153 | 154 | class HttpMapReduceStream(HttpMultipartStream): 155 | """ 156 | Streaming iterator for MapReduce over HTTP 157 | """ 158 | 159 | def next(self): 160 | message = super(HttpMapReduceStream, self).next() 161 | payload = json.loads(message.get_payload()) 162 | return payload['phase'], payload['data'] 163 | 164 | def __next__(self): 165 | # Python 3.x Version 166 | return self.next() 167 | 168 | 169 | class HttpIndexStream(HttpMultipartStream): 170 | """ 171 | Streaming iterator for secondary indexes over HTTP 172 | """ 173 | 174 | def __init__(self, response, index, return_terms): 175 | super(HttpIndexStream, self).__init__(response) 176 | self.index = index 177 | self.return_terms = return_terms 178 | 179 | def next(self): 180 | message = super(HttpIndexStream, self).next() 181 | payload = json.loads(message.get_payload()) 182 | if u'error' in payload: 183 | raise RiakError(payload[u'error']) 184 | elif u'keys' in payload: 185 | return payload[u'keys'] 186 | elif u'results' in payload: 187 | structs = payload[u'results'] 188 | # Format is {"results":[{"2ikey":"primarykey"}, ...]} 189 | return [self._decode_pair(list(d.items())[0]) for d in structs] 190 | elif u'continuation' in payload: 191 | return CONTINUATION(payload[u'continuation']) 192 | 193 | def __next__(self): 194 | # Python 3.x Version 195 | return self.next() 196 | 197 | def _decode_pair(self, pair): 198 | return (decode_index_value(self.index, pair[0]), pair[1]) 199 | -------------------------------------------------------------------------------- /riak/transports/tcp/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import errno 16 | import socket 17 | 18 | from riak.transports.pool import Pool, ConnectionClosed 19 | from riak.transports.tcp.transport import TcpTransport 20 | 21 | 22 | class TcpPool(Pool): 23 | """ 24 | A resource pool of TCP transports. 25 | """ 26 | def __init__(self, client, **options): 27 | super(TcpPool, self).__init__() 28 | self._client = client 29 | self._options = options 30 | 31 | def create_resource(self): 32 | node = self._client._choose_node() 33 | return TcpTransport(node=node, 34 | client=self._client, 35 | **self._options) 36 | 37 | def destroy_resource(self, tcp): 38 | tcp.close() 39 | 40 | 41 | # These are a specific set of socket errors 42 | # that could be raised on send/recv that indicate 43 | # that the socket is closed or reset, and is not 44 | # usable. On seeing any of these errors, the socket 45 | # should be closed, and the connection re-established. 46 | CONN_CLOSED_ERRORS = ( 47 | errno.EHOSTUNREACH, 48 | errno.ECONNRESET, 49 | errno.ECONNREFUSED, 50 | errno.ECONNABORTED, 51 | errno.ETIMEDOUT, 52 | errno.EBADF, 53 | errno.EPIPE 54 | ) 55 | 56 | 57 | def is_retryable(err): 58 | """ 59 | Determines if the given exception is something that is 60 | network/socket-related and should thus cause the TCP connection to 61 | close and the operation retried on another node. 62 | 63 | :rtype: boolean 64 | """ 65 | if isinstance(err, ConnectionClosed): 66 | # NB: only retryable if we're not mid-streaming 67 | if err.mid_stream: 68 | return False 69 | else: 70 | return True 71 | elif isinstance(err, socket.error): 72 | code = err.args[0] 73 | return code in CONN_CLOSED_ERRORS 74 | else: 75 | return False 76 | -------------------------------------------------------------------------------- /riak/transports/tcp/stream.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import json 16 | 17 | import riak.pb.messages 18 | 19 | from riak.util import decode_index_value, bytes_to_str 20 | from riak.client.index_page import CONTINUATION 21 | from riak.codecs.ttb import TtbCodec 22 | from six import PY2 23 | 24 | 25 | class PbufStream(object): 26 | """ 27 | Used internally by TcpTransport to implement streaming 28 | operations. Implements the iterator interface. 29 | """ 30 | 31 | _expect = None 32 | 33 | def __init__(self, transport, codec): 34 | self.finished = False 35 | self.transport = transport 36 | self.codec = codec 37 | self.resource = None 38 | self._mid_stream = False 39 | 40 | def __iter__(self): 41 | return self 42 | 43 | def next(self): 44 | if self.finished: 45 | raise StopIteration 46 | 47 | try: 48 | resp_code, data = self.transport._recv_msg( 49 | mid_stream=self._mid_stream) 50 | self.codec.maybe_riak_error(resp_code, data) 51 | expect = self._expect 52 | self.codec.maybe_incorrect_code(resp_code, expect) 53 | resp = self.codec.parse_msg(expect, data) 54 | except: 55 | self.finished = True 56 | raise 57 | finally: 58 | self._mid_stream = True 59 | 60 | if self._is_done(resp): 61 | self.finished = True 62 | 63 | return resp 64 | 65 | def __next__(self): 66 | # Python 3.x Version 67 | return self.next() 68 | 69 | def _is_done(self, response): 70 | # This could break if new messages don't name the field the 71 | # same thing. 72 | return response.done 73 | 74 | def attach(self, resource): 75 | self.resource = resource 76 | 77 | def close(self): 78 | # We have to drain the socket to make sure that we don't get 79 | # weird responses when some other request comes after a 80 | # failed/prematurely-terminated one. 81 | try: 82 | while self.next(): 83 | pass 84 | except StopIteration: 85 | pass 86 | self.resource.release() 87 | 88 | 89 | class PbufKeyStream(PbufStream): 90 | """ 91 | Used internally by TcpTransport to implement key-list streams. 92 | """ 93 | 94 | _expect = riak.pb.messages.MSG_CODE_LIST_KEYS_RESP 95 | 96 | def next(self): 97 | response = super(PbufKeyStream, self).next() 98 | 99 | if response.done and len(response.keys) is 0: 100 | raise StopIteration 101 | 102 | return response.keys 103 | 104 | def __next__(self): 105 | # Python 3.x Version 106 | return self.next() 107 | 108 | 109 | class PbufMapredStream(PbufStream): 110 | """ 111 | Used internally by TcpTransport to implement MapReduce 112 | streams. 113 | """ 114 | 115 | _expect = riak.pb.messages.MSG_CODE_MAP_RED_RESP 116 | 117 | def next(self): 118 | response = super(PbufMapredStream, self).next() 119 | 120 | if response.done and not response.HasField('response'): 121 | raise StopIteration 122 | 123 | return response.phase, json.loads(bytes_to_str(response.response)) 124 | 125 | def __next__(self): 126 | # Python 3.x Version 127 | return self.next() 128 | 129 | 130 | class PbufBucketStream(PbufStream): 131 | """ 132 | Used internally by TcpTransport to implement key-list streams. 133 | """ 134 | 135 | _expect = riak.pb.messages.MSG_CODE_LIST_BUCKETS_RESP 136 | 137 | def next(self): 138 | response = super(PbufBucketStream, self).next() 139 | 140 | if response.done and len(response.buckets) is 0: 141 | raise StopIteration 142 | 143 | return response.buckets 144 | 145 | def __next__(self): 146 | # Python 3.x Version 147 | return self.next() 148 | 149 | 150 | class PbufIndexStream(PbufStream): 151 | """ 152 | Used internally by TcpTransport to implement Secondary Index 153 | streams. 154 | """ 155 | 156 | _expect = riak.pb.messages.MSG_CODE_INDEX_RESP 157 | 158 | def __init__(self, transport, codec, index, return_terms=False): 159 | super(PbufIndexStream, self).__init__(transport, codec) 160 | self.index = index 161 | self.return_terms = return_terms 162 | 163 | def next(self): 164 | response = super(PbufIndexStream, self).next() 165 | 166 | if response.done and not (response.keys or 167 | response.results or 168 | response.continuation): 169 | raise StopIteration 170 | 171 | if self.return_terms and response.results: 172 | return [(decode_index_value(self.index, r.key), 173 | bytes_to_str(r.value)) 174 | for r in response.results] 175 | elif response.keys: 176 | if PY2: 177 | return response.keys[:] 178 | else: 179 | return [bytes_to_str(key) for key in response.keys] 180 | elif response.continuation: 181 | return CONTINUATION(bytes_to_str(response.continuation)) 182 | 183 | def __next__(self): 184 | # Python 3.x Version 185 | return self.next() 186 | 187 | 188 | class PbufTsKeyStream(PbufStream, TtbCodec): 189 | """ 190 | Used internally by TcpTransport to implement TS key-list streams. 191 | """ 192 | 193 | _expect = riak.pb.messages.MSG_CODE_TS_LIST_KEYS_RESP 194 | 195 | def __init__(self, transport, codec, convert_timestamp=False): 196 | super(PbufTsKeyStream, self).__init__(transport, codec) 197 | self._convert_timestamp = convert_timestamp 198 | 199 | def next(self): 200 | response = super(PbufTsKeyStream, self).next() 201 | 202 | if response.done and len(response.keys) is 0: 203 | raise StopIteration 204 | 205 | keys = [] 206 | for tsrow in response.keys: 207 | keys.append(self.codec.decode_timeseries_row(tsrow, 208 | convert_timestamp=self._convert_timestamp)) 209 | 210 | return keys 211 | 212 | def __next__(self): 213 | # Python 3.x Version 214 | return self.next() 215 | -------------------------------------------------------------------------------- /riak/ts_object.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import collections 16 | 17 | from riak import RiakError 18 | from riak.table import Table 19 | 20 | TsColumns = collections.namedtuple('TsColumns', ['names', 'types']) 21 | 22 | 23 | class TsObject(object): 24 | """ 25 | The TsObject holds information about Timeseries data, plus the data 26 | itself. 27 | """ 28 | def __init__(self, client, table, rows=None, columns=None): 29 | """ 30 | Construct a new TsObject. 31 | 32 | :param client: A RiakClient object. 33 | :type client: :class:`RiakClient ` 34 | :param table: The table for the timeseries data as a Table object. 35 | :type table: :class:`Table` 36 | :param rows: An list of lists with timeseries data 37 | :type rows: list 38 | :param columns: A TsColumns tuple. Optional 39 | :type columns: :class:`TsColumns` 40 | """ 41 | 42 | if not isinstance(table, Table): 43 | raise ValueError('table must be an instance of Table.') 44 | 45 | self.client = client 46 | self.table = table 47 | 48 | if rows is not None and not isinstance(rows, list): 49 | raise RiakError("TsObject rows parameter must be a list.") 50 | else: 51 | self.rows = rows 52 | 53 | if columns is not None and \ 54 | not isinstance(columns, TsColumns): 55 | raise RiakError( 56 | "TsObject columns parameter must be a TsColumns instance") 57 | else: 58 | self.columns = columns 59 | 60 | def store(self): 61 | """ 62 | Store the timeseries data in Riak. 63 | :rtype: boolean 64 | """ 65 | return self.client.ts_put(self) 66 | -------------------------------------------------------------------------------- /riak/tz.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from datetime import tzinfo, timedelta 16 | 17 | ZERO = timedelta(0) 18 | 19 | 20 | class UTC(tzinfo): 21 | """UTC""" 22 | 23 | def utcoffset(self, dt): 24 | return ZERO 25 | 26 | def tzname(self, dt): 27 | return "UTC" 28 | 29 | def dst(self, dt): 30 | return ZERO 31 | 32 | 33 | utc = UTC() 34 | -------------------------------------------------------------------------------- /riak/util.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import print_function 16 | 17 | import datetime 18 | import sys 19 | import warnings 20 | 21 | from collections import Mapping 22 | from six import string_types, PY2 23 | 24 | epoch = datetime.datetime.utcfromtimestamp(0) 25 | try: 26 | import pytz 27 | epoch_tz = pytz.utc.localize(epoch) 28 | except ImportError: 29 | from riak.tz import utc 30 | epoch_tz = datetime.datetime.fromtimestamp(0, tz=utc) 31 | 32 | 33 | def unix_time_millis(dt): 34 | if dt.tzinfo: 35 | td = dt - epoch_tz 36 | else: 37 | td = dt - epoch 38 | tdms = ((td.days * 24 * 3600) + td.seconds) * 1000 39 | ms = td.microseconds // 1000 40 | return tdms + ms 41 | 42 | 43 | def datetime_from_unix_time_millis(ut): 44 | if isinstance(ut, float): 45 | raise ValueError('unix timestamp must not be a float, ' 46 | 'it must be total milliseconds since ' 47 | 'epoch as an integer') 48 | utms = ut / 1000.0 49 | return datetime.datetime.utcfromtimestamp(utms) 50 | 51 | 52 | def is_timeseries_supported(v=None): 53 | if v is None: 54 | v = sys.version_info 55 | return v < (3,) or (v[:3] >= (3, 4, 4) and v[:3] != (3, 5, 0)) 56 | 57 | 58 | def quacks_like_dict(object): 59 | """Check if object is dict-like""" 60 | return isinstance(object, Mapping) 61 | 62 | 63 | def deep_merge(a, b): 64 | """Merge two deep dicts non-destructively 65 | 66 | Uses a stack to avoid maximum recursion depth exceptions 67 | 68 | >>> a = {'a': 1, 'b': {1: 1, 2: 2}, 'd': 6} 69 | >>> b = {'c': 3, 'b': {2: 7}, 'd': {'z': [1, 2, 3]}} 70 | >>> c = deep_merge(a, b) 71 | >>> from pprint import pprint; pprint(c) 72 | {'a': 1, 'b': {1: 1, 2: 7}, 'c': 3, 'd': {'z': [1, 2, 3]}} 73 | """ 74 | assert quacks_like_dict(a), quacks_like_dict(b) 75 | dst = a.copy() 76 | 77 | stack = [(dst, b)] 78 | while stack: 79 | current_dst, current_src = stack.pop() 80 | for key in current_src: 81 | if key not in current_dst: 82 | current_dst[key] = current_src[key] 83 | else: 84 | if (quacks_like_dict(current_src[key]) and 85 | quacks_like_dict(current_dst[key])): 86 | stack.append((current_dst[key], current_src[key])) 87 | else: 88 | current_dst[key] = current_src[key] 89 | return dst 90 | 91 | 92 | def deprecated(message, stacklevel=3): 93 | """ 94 | Prints a deprecation warning to the console. 95 | """ 96 | warnings.warn(message, UserWarning, stacklevel=stacklevel) 97 | 98 | 99 | class lazy_property(object): 100 | ''' 101 | A method decorator meant to be used for lazy evaluation and 102 | memoization of an object attribute. The property should represent 103 | immutable data, as it replaces itself on first access. 104 | ''' 105 | def __init__(self, fget): 106 | self.fget = fget 107 | self.func_name = fget.__name__ 108 | 109 | def __get__(self, obj, cls): 110 | if obj is None: 111 | return None 112 | value = self.fget(obj) 113 | setattr(obj, self.func_name, value) 114 | return value 115 | 116 | 117 | def decode_index_value(index, value): 118 | if "_int" in bytes_to_str(index): 119 | return str_to_long(value) 120 | elif PY2: 121 | return str(value) 122 | else: 123 | return bytes_to_str(value) 124 | 125 | 126 | def bytes_to_str(value, encoding='utf-8'): 127 | if isinstance(value, string_types) or value is None: 128 | return value 129 | elif isinstance(value, list): 130 | return [bytes_to_str(elem) for elem in value] 131 | else: 132 | return value.decode(encoding) 133 | 134 | 135 | def str_to_bytes(value, encoding='utf-8'): 136 | if PY2 or value is None: 137 | return value 138 | elif isinstance(value, list): 139 | return [str_to_bytes(elem) for elem in value] 140 | else: 141 | return value.encode(encoding) 142 | 143 | 144 | def str_to_long(value, base=10): 145 | if value is None: 146 | return None 147 | elif PY2: 148 | return long(value, base) # noqa 149 | else: 150 | return int(value, base) 151 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [nosetests] 2 | verbosity=2 3 | with-coverage=1 4 | cover-package=riak 5 | cover-erase=1 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import codecs 4 | import sys 5 | 6 | from setuptools import setup, find_packages 7 | from version import get_version 8 | from commands import setup_timeseries, build_messages 9 | 10 | install_requires = ['six >= 1.8.0', 'basho_erlastic >= 2.1.1'] 11 | requires = ['six(>=1.8.0)', 'basho_erlastic(>= 2.1.1)'] 12 | 13 | if sys.version_info[:3] <= (2, 7, 9): 14 | install_requires.append("pyOpenSSL >= 0.14") 15 | requires.append("pyOpenSSL(>=0.14)") 16 | 17 | if sys.version_info[:3] <= (3, 0, 0): 18 | install_requires.append('protobuf >=2.4.1, <2.7.0') 19 | requires.append('protobuf(>=2.4.1, <2.7.0)') 20 | else: 21 | install_requires.append('python3_protobuf >=2.4.1, <2.6.0') 22 | requires.append('python3_protobuf(>=2.4.1, <2.6.0)') 23 | 24 | with codecs.open('README.md', 'r', 'utf-8') as f: 25 | readme_md = f.read() 26 | 27 | try: 28 | import pypandoc 29 | long_description = pypandoc.convert('README.md', 'rst') 30 | with codecs.open('README.rst', 'w', 'utf-8') as f: 31 | f.write(long_description) 32 | except(IOError, ImportError): 33 | long_description = readme_md 34 | 35 | setup( 36 | name='riak', 37 | version=get_version(), 38 | packages=find_packages(), 39 | requires=requires, 40 | install_requires=install_requires, 41 | package_data={'riak': ['erl_src/*']}, 42 | description='Python client for Riak', 43 | long_description=long_description, 44 | zip_safe=True, 45 | options={'easy_install': {'allow_hosts': 'pypi.python.org'}}, 46 | include_package_data=True, 47 | license='Apache 2', 48 | platforms='Platform Independent', 49 | author='Basho Technologies', 50 | author_email='clients@basho.com', 51 | test_suite='riak.tests.suite', 52 | url='https://github.com/basho/riak-python-client', 53 | cmdclass={ 54 | 'build_messages': build_messages, 55 | 'setup_timeseries': setup_timeseries 56 | }, 57 | classifiers=['License :: OSI Approved :: Apache Software License', 58 | 'Intended Audience :: Developers', 59 | 'Operating System :: OS Independent', 60 | 'Programming Language :: Python :: 2.7', 61 | 'Programming Language :: Python :: 3.3', 62 | 'Programming Language :: Python :: 3.4', 63 | 'Programming Language :: Python :: 3.5', 64 | 'Topic :: Database'] 65 | ) 66 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # Tox (http://tox.testrun.org/) is a tool for running tests 2 | # in multiple virtualenvs. This configuration file will run the 3 | # test suite on all supported python versions. 4 | 5 | [tox] 6 | envlist = py2, py3 7 | 8 | [testenv] 9 | install_command = pip install --upgrade {packages} 10 | commands = {envpython} setup.py test 11 | deps = 12 | pip 13 | pytz 14 | passenv = RUN_* SKIP_* RIAK_* 15 | -------------------------------------------------------------------------------- /version.py: -------------------------------------------------------------------------------- 1 | # Copyright 2010-present Basho Technologies, Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | """ 16 | Gets the current version number. 17 | If in a git repository, it is the current git tag. 18 | Otherwise it is the one contained in the PKG-INFO file. 19 | 20 | To use this script, simply import it in your setup.py file 21 | and use the results of get_version() as your package version:: 22 | 23 | from version import * 24 | 25 | setup( 26 | version=get_version() 27 | ) 28 | """ 29 | 30 | from __future__ import print_function 31 | from os.path import dirname, isdir, join 32 | import re 33 | from subprocess import CalledProcessError, Popen, PIPE 34 | 35 | try: 36 | from subprocess import check_output 37 | except ImportError: 38 | def check_output(*popenargs, **kwargs): 39 | """Run command with arguments and return its output as a byte string. 40 | 41 | If the exit code was non-zero it raises a CalledProcessError. The 42 | CalledProcessError object will have the return code in the returncode 43 | attribute and output in the output attribute. 44 | 45 | The arguments are the same as for the Popen constructor. Example: 46 | 47 | >>> check_output(["ls", "-l", "/dev/null"]) 48 | 'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n' 49 | 50 | The stdout argument is not allowed as it is used internally. 51 | To capture standard error in the result, use stderr=STDOUT. 52 | 53 | >>> import sys 54 | >>> check_output(["/bin/sh", "-c", 55 | ... "ls -l non_existent_file ; exit 0"], 56 | ... stderr=sys.stdout) 57 | 'ls: non_existent_file: No such file or directory\n' 58 | """ 59 | if 'stdout' in kwargs: 60 | raise ValueError('stdout argument not allowed, it will be ' 61 | 'overridden.') 62 | process = Popen(stdout=PIPE, *popenargs, **kwargs) 63 | output, unused_err = process.communicate() 64 | retcode = process.poll() 65 | if retcode: 66 | cmd = kwargs.get("args") 67 | if cmd is None: 68 | cmd = popenargs[0] 69 | raise CalledProcessError(retcode, cmd) 70 | return output 71 | 72 | version_re = re.compile('^Version: (.+)$', re.M) 73 | 74 | __all__ = ['get_version'] 75 | 76 | 77 | def get_version(): 78 | d = dirname(__file__) 79 | 80 | if isdir(join(d, '.git')): 81 | # Get the version using "git describe". 82 | cmd = 'git describe --tags --match [0-9]*'.split() 83 | try: 84 | version = check_output(cmd).decode().strip() 85 | except CalledProcessError: 86 | print('Unable to get version number from git tags') 87 | exit(1) 88 | 89 | # PEP 386 compatibility 90 | if '-' in version: 91 | version = '.post'.join(version.split('-')[:2]) 92 | 93 | else: 94 | # Extract the version from the PKG-INFO file. 95 | import codecs 96 | with codecs.open(join(d, 'PKG-INFO'), 'r', 'utf-8') as f: 97 | version = version_re.search(f.read()).group(1) 98 | 99 | return version 100 | 101 | 102 | if __name__ == '__main__': 103 | print(get_version()) 104 | --------------------------------------------------------------------------------