├── rebar ├── doc ├── man │ └── man1 │ │ ├── riak.1.gz │ │ └── riak-admin.1.gz ├── basho-doc-style.iorg ├── basic-mapreduce.txt ├── architecture.txt ├── basic-setup.txt ├── raw-http-howto.txt ├── doc-style.css ├── basic-client.txt └── js-mapreduce.org ├── releasenotes ├── riak-0.1.txt ├── riak-0.12.1.txt ├── riak-0.10.1.txt ├── riak-0.9.1.txt ├── riak-0.2.txt ├── riak-0.8.txt ├── riak-0.4.txt ├── riak-0.3.txt ├── riak-0.7.1.txt ├── riak-0.6.txt ├── riak-0.5.txt ├── riak-0.11.0.txt ├── riak-0.9.txt ├── riak-0.7.txt ├── riak-0.10.txt └── riak-0.12.0.txt ├── rebar.config ├── NOTICE ├── rel ├── files │ ├── vm.args │ ├── erl │ ├── nodetool │ ├── app.config │ ├── riak │ └── riak-admin ├── vars.config ├── vars │ ├── dev1_vars.config │ ├── dev2_vars.config │ └── dev3_vars.config └── reltool.config ├── THANKS ├── .hgignore ├── client_lib └── README ├── orgbatch.el ├── dialyzer.ignore-warnings ├── Makefile ├── .hgtags ├── README.org ├── README ├── LICENSE ├── TRANSITION.org └── TRANSITION /rebar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/b/riak/master/rebar -------------------------------------------------------------------------------- /doc/man/man1/riak.1.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/b/riak/master/doc/man/man1/riak.1.gz -------------------------------------------------------------------------------- /releasenotes/riak-0.1.txt: -------------------------------------------------------------------------------- 1 | riak-0.1 2 | 3 | The initial public release of Riak. 4 | 5 | -------------------------------------------------------------------------------- /doc/man/man1/riak-admin.1.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/b/riak/master/doc/man/man1/riak-admin.1.gz -------------------------------------------------------------------------------- /rebar.config: -------------------------------------------------------------------------------- 1 | {sub_dirs, ["rel"]}. 2 | 3 | {require_otp_vsn, "R13B04|R14"}. 4 | 5 | {cover_enabled, true}. 6 | 7 | {erl_opts, [debug_info, fail_on_warning]}. 8 | 9 | {deps, [ 10 | {riak_kv, "0.12.0", {hg, "http://bitbucket.org/basho/riak_kv", "tip"}} 11 | ]}. 12 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Riak 2 | Copyright 2007-2009 Basho Technologies 3 | 4 | This product contains code developed at Basho Technologies. 5 | (http://www.basho.com/) 6 | 7 | The gen_server2 and priority_queue modules are provided 8 | by LShift Ltd, and can also be found in the RabbitMQ 9 | sources at http://www.rabbitmq.com/ 10 | 11 | -------------------------------------------------------------------------------- /releasenotes/riak-0.12.1.txt: -------------------------------------------------------------------------------- 1 | ---------------------------------------------- 2 | Riak 0.12.1 Release Notes 3 | ---------------------------------------------- 4 | 5 | Bugs Fixed 6 | ------------------------------- 7 | 507 - Bitcask leaks file-handles when reading lock data for merge 8 | 523 - M/R job timeout not propagating in all cases 9 | 10 | 11 | All bug and issue numbers reference https://issues.basho.com. 12 | 13 | -------------------------------------------------------------------------------- /rel/files/vm.args: -------------------------------------------------------------------------------- 1 | 2 | ## Name of the riak node 3 | -name {{node}} 4 | 5 | ## Cookie for distributed erlang 6 | -setcookie riak 7 | 8 | ## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive 9 | ## (Disabled by default..use with caution!) 10 | ##-heart 11 | 12 | ## Enable kernel poll and a few async threads 13 | +K true 14 | +A 64 15 | 16 | ## Increase number of concurrent ports/sockets 17 | -env ERL_MAX_PORTS 4096 18 | 19 | ## Tweak GC to run more often 20 | -env ERL_FULLSWEEP_AFTER 10 21 | 22 | -------------------------------------------------------------------------------- /releasenotes/riak-0.10.1.txt: -------------------------------------------------------------------------------- 1 | ---------------------- 2 | Riak 0.10.1 Release Notes 3 | ---------------------- 4 | 5 | Riak 0.10.1 addresses two issues: 6 | 7 | - An issue has been fixed, which caused map/reduce queries to never 8 | be run on the node where the query originated. This both improves 9 | the efficiency of some queries, as well as fixes map/reduce on 10 | single-node clusters. 11 | 12 | - The "deps" make target was added as a dependency of the "rel" make 13 | target, such that "make rel" does the right thing the first time. 14 | -------------------------------------------------------------------------------- /THANKS: -------------------------------------------------------------------------------- 1 | The following people have contributed to Riak: 2 | 3 | Andy Gross 4 | Justin Sheehy 5 | Robert Ahrens 6 | Jeremy Latt 7 | Bryan Fink 8 | Ryan Tilder 9 | John Muellerleile 10 | Dave Smith 11 | Christopher Brown 12 | Tuncer Ayaz 13 | Rusty Klophaus 14 | Jay Doane 15 | Martin Scholl 16 | Jayson Baird 17 | Kirill A. Korinskiy 18 | Kevin Smith 19 | Jonathan Lee 20 | Sean Cribbs 21 | Matthew Curry 22 | Tony Falco 23 | Grant Schofield 24 | Jon Meredith 25 | Mark Phillips 26 | Ray Cote 27 | Adam Hunter 28 | Cliff Moon 29 | Rajiv M. Ranganath 30 | Francisco Treacy 31 | Bruce Lowekamp 32 | Dan Reverri 33 | -------------------------------------------------------------------------------- /doc/basho-doc-style.iorg: -------------------------------------------------------------------------------- 1 | #+COMMENT: -*- mode: org -*- 2 | #+COMMENT: Standard definitions for org-mode documents. 3 | #+COMMENT: Include this file with 4 | #+COMMENT: #+SETUPFILE: "basho-doc-style.iorg" 5 | #+COMMENT: at the top of your org-mode file 6 | 7 | #+COMMENT: the extension of this filename is ".iorg" instead of just 8 | #+COMMENT: ".org" to prevent the make orgs target from attempting to 9 | #+COMMENT: compile this file to basho-doc-style.html 10 | 11 | #+OPTIONS: author:nil creator:nil 12 | 13 | #+COMMENT: HTML style 14 | #+STYLE: 15 | -------------------------------------------------------------------------------- /.hgignore: -------------------------------------------------------------------------------- 1 | syntax regex 2 | (^|/)\.svn($|/) 3 | (^|/)\.hg($|/) 4 | (^|/)\.hgtags($|/) 5 | (^|/)\.DS_Store 6 | doc/.*(html|css|info|png)$ 7 | www/edoc/* 8 | www/README 9 | www/LICENSE 10 | www/TODO 11 | www/javadoc/* 12 | ebin/.*\.(beam)$ 13 | (^|/)erl_crash.dump$ 14 | .*~$ 15 | log/.*.log$ 16 | log/.*.log.*$ 17 | .*\.orig$ 18 | priv/log/.* 19 | priv/weblogs/.* 20 | ^priv/store 21 | ^priv/ringstate/* 22 | ^client_lib/java/build/* 23 | ^client_lib/java/dist/* 24 | ^client_lib/java/javadoc/* 25 | test\.log 26 | ^test/.* 27 | ^data/.* 28 | .*/.eunit/.* 29 | ^rel/riak/.* 30 | ^dev/.* 31 | include/.*_pb.hrl$ 32 | c_src/deps/(nsprpub|js|mozilla|nspr_release)/.* 33 | c_src/(nsprpub|js|mozilla|nspr_release|system)/.* 34 | c_src/include/js/.* 35 | c_src/.*(\.o|\.a)$ 36 | erlang_js/priv/.*.so$ 37 | client_lib/python/.*.pyc 38 | -------------------------------------------------------------------------------- /client_lib/README: -------------------------------------------------------------------------------- 1 | Riak client libraries are available for many languages. Rather than 2 | bundle them with the Riak server source code, we have given them each 3 | their own source repository. Currently, official Riak client language 4 | libraries include: 5 | 6 | + Javascript 7 | http://bitbucket.org/basho/riak-javascript-client 8 | 9 | + Python 10 | http://bitbucket.org/basho/riak-python-client 11 | 12 | + Ruby 13 | http://bitbucket.org/basho/riak-ruby-client 14 | http://github.com/seancribbs/ripple/ 15 | 16 | + Java 17 | http://bitbucket.org/basho/riak-java-client 18 | 19 | + PHP 20 | http://bitbucket.org/basho/riak-php-client 21 | 22 | + Erlang 23 | http://bitbucket.org/basho/riak-erlang-client 24 | (using protocol buffers instead of distributed Erlang) 25 | -------------------------------------------------------------------------------- /releasenotes/riak-0.9.1.txt: -------------------------------------------------------------------------------- 1 | ---------------------- 2 | Riak 0.9.1 Release Notes 3 | ---------------------- 4 | 5 | Riak 0.9.1 addresses a few minor issues: 6 | 7 | - raw_link_walker_resource no longer returns errors when linked 8 | objects are missing (the missing objects are just not included in 9 | the response) [http://issues.basho.com/show_bug.cgi?id=40] 10 | 11 | - if multiple map/reduce phases specify "keep":true, the results from 12 | each phase are grouped into their own list; streameing responses 13 | with multiple "keep":true phases include a "phase": 14 | field [http://issues.basho.com/show_bug.cgi?id=66] 15 | 16 | - the bucket name and key can now be extracted from objects in the 17 | Python and PHP clients 18 | 19 | - links to binary downloads in admin.org were updated 20 | -------------------------------------------------------------------------------- /rel/vars.config: -------------------------------------------------------------------------------- 1 | %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*- 2 | %% ex: ts=4 sw=4 et 3 | 4 | %% 5 | %% etc/app.config 6 | %% 7 | {ring_state_dir, "data/ring"}. 8 | {web_ip, "127.0.0.1"}. 9 | {web_port, 8098}. 10 | {handoff_port, 8099}. 11 | {pb_ip, "127.0.0.1"}. 12 | {pb_port, 8087}. 13 | {bitcask_data_root, "data/bitcask"}. 14 | {sasl_error_log, "log/sasl-error.log"}. 15 | {sasl_log_dir, "log/sasl"}. 16 | 17 | %% 18 | %% etc/vm.args 19 | %% 20 | {node, "riak@127.0.0.1"}. 21 | 22 | %% 23 | %% bin/riak 24 | %% 25 | {runner_script_dir, "$(cd ${0%/*} && pwd)"}. 26 | {runner_base_dir, "${RUNNER_SCRIPT_DIR%/*}"}. 27 | {runner_etc_dir, "$RUNNER_BASE_DIR/etc"}. 28 | {runner_log_dir, "$RUNNER_BASE_DIR/log"}. 29 | {pipe_dir, "/tmp/$RUNNER_BASE_DIR/"}. 30 | {runner_user, ""}. 31 | -------------------------------------------------------------------------------- /rel/vars/dev1_vars.config: -------------------------------------------------------------------------------- 1 | %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*- 2 | %% ex: ts=4 sw=4 et 3 | 4 | %% 5 | %% etc/app.config 6 | %% 7 | {ring_state_dir, "data/ring"}. 8 | {web_ip, "127.0.0.1"}. 9 | {web_port, 8091}. 10 | {handoff_port, 8101}. 11 | {pb_ip, "127.0.0.1"}. 12 | {pb_port, 8081}. 13 | {bitcask_data_root, "data/bitcask"}. 14 | {sasl_error_log, "log/sasl-error.log"}. 15 | {sasl_log_dir, "log/sasl"}. 16 | 17 | %% 18 | %% etc/vm.args 19 | %% 20 | {node, "dev1@127.0.0.1"}. 21 | 22 | %% 23 | %% bin/riak 24 | %% 25 | {runner_script_dir, "$(cd ${0%/*} && pwd)"}. 26 | {runner_base_dir, "${RUNNER_SCRIPT_DIR%/*}"}. 27 | {runner_etc_dir, "$RUNNER_BASE_DIR/etc"}. 28 | {runner_log_dir, "$RUNNER_BASE_DIR/log"}. 29 | {pipe_dir, "/tmp/$RUNNER_BASE_DIR/"}. 30 | {runner_user, ""}. 31 | -------------------------------------------------------------------------------- /rel/vars/dev2_vars.config: -------------------------------------------------------------------------------- 1 | %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*- 2 | %% ex: ts=4 sw=4 et 3 | 4 | %% 5 | %% etc/app.config 6 | %% 7 | {ring_state_dir, "data/ring"}. 8 | {web_ip, "127.0.0.1"}. 9 | {web_port, 8092}. 10 | {handoff_port, 8102}. 11 | {pb_ip, "127.0.0.1"}. 12 | {pb_port, 8082}. 13 | {bitcask_data_root, "data/bitcask"}. 14 | {sasl_error_log, "log/sasl-error.log"}. 15 | {sasl_log_dir, "log/sasl"}. 16 | 17 | %% 18 | %% etc/vm.args 19 | %% 20 | {node, "dev2@127.0.0.1"}. 21 | 22 | %% 23 | %% bin/riak 24 | %% 25 | {runner_script_dir, "$(cd ${0%/*} && pwd)"}. 26 | {runner_base_dir, "${RUNNER_SCRIPT_DIR%/*}"}. 27 | {runner_etc_dir, "$RUNNER_BASE_DIR/etc"}. 28 | {runner_log_dir, "$RUNNER_BASE_DIR/log"}. 29 | {pipe_dir, "/tmp/$RUNNER_BASE_DIR/"}. 30 | {runner_user, ""}. 31 | -------------------------------------------------------------------------------- /rel/vars/dev3_vars.config: -------------------------------------------------------------------------------- 1 | %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*- 2 | %% ex: ts=4 sw=4 et 3 | 4 | %% 5 | %% etc/app.config 6 | %% 7 | {ring_state_dir, "data/ring"}. 8 | {web_ip, "127.0.0.1"}. 9 | {web_port, 8093}. 10 | {handoff_port, 8103}. 11 | {pb_ip, "127.0.0.1"}. 12 | {pb_port, 8083}. 13 | {bitcask_data_root, "data/bitcask"}. 14 | {sasl_error_log, "log/sasl-error.log"}. 15 | {sasl_log_dir, "log/sasl"}. 16 | 17 | %% 18 | %% etc/vm.args 19 | %% 20 | {node, "dev3@127.0.0.1"}. 21 | 22 | %% 23 | %% bin/riak 24 | %% 25 | {runner_script_dir, "$(cd ${0%/*} && pwd)"}. 26 | {runner_base_dir, "${RUNNER_SCRIPT_DIR%/*}"}. 27 | {runner_etc_dir, "$RUNNER_BASE_DIR/etc"}. 28 | {runner_log_dir, "$RUNNER_BASE_DIR/log"}. 29 | {pipe_dir, "/tmp/$RUNNER_BASE_DIR/"}. 30 | {runner_user, ""}. 31 | -------------------------------------------------------------------------------- /releasenotes/riak-0.2.txt: -------------------------------------------------------------------------------- 1 | riak-0.2 2 | 3 | Eventer: 4 | - add partition to riak_vnode get/put/delete eventer notifications 5 | - export client-side send_event(EventName, EventDetail) for custom events 6 | - add Bucket, Key to riak_delete:delete_start event 7 | 8 | Documentation: 9 | - most pages validate cleanly now (except for edoc) 10 | - more detail on the HTTP interface 11 | 12 | Performance: 13 | - removed an unused async put operation 14 | - removed mnesia, locking, and spawn in riak_vnode 15 | - set garbage collection fullsweep_after to 20, globally 16 | 17 | HTTP Interface: 18 | - new PHP client 19 | - allow dynamic creation of buckets (and schema) via HTTP/JSON 20 | - fix for sibling merge with dynamic json buckets 21 | - useful default behavior for linkwalking JSON data 22 | 23 | Administration: 24 | - note in setup instructions that R13 or later is needed 25 | - added debug-restart.sh for interactive shell mode 26 | - explicitly put demo node on 127.0.0.1 27 | - allow_mult default value is now false; now "last write wins" by default 28 | -------------------------------------------------------------------------------- /releasenotes/riak-0.8.txt: -------------------------------------------------------------------------------- 1 | ---------------------- 2 | Riak 0.8 Release Notes 3 | ---------------------- 4 | 5 | Riak 0.8 is the first release of Riak with full support for map/reduce 6 | processing exposed entirely via an HTTP/JavaScript interface. In summary: 7 | 8 | - exposed javascript as a first class map/reduce query language 9 | - map/reduce queries can be submitted via HTTP/JSON (default endpoint 10 | is "/mapred") 11 | - numerous fixes and tweaks to map/reduce query performance and 12 | timeout behavior 13 | 14 | Riak 0.8 also includes a number of minor improvements such as: 15 | 16 | - improved handoff/rebalancing performance. 17 | - added configuration parameter 'handoff_concurrency', which controls 18 | how many vnodes per host can engage in handoff at once. 19 | - added client_lib/riak.py, a python client for the 'raw' riak http 20 | interface (based on jiak.py by Jay Baird). 21 | - statistics about node performance, VM state, and Riak cluster status 22 | are now exposed over HTTP/JSON (default endpoint is "/stats") 23 | - general stability improvements such as improved supervision for vnodes 24 | and map/reduce processes 25 | 26 | -------------------------------------------------------------------------------- /rel/files/erl: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | ## This script replaces the default "erl" in erts-VSN/bin. This is necessary 4 | ## as escript depends on erl and in turn, erl depends on having access to a 5 | ## bootscript (start.boot). Note that this script is ONLY invoked as a side-effect 6 | ## of running escript -- the embedded node bypasses erl and uses erlexec directly 7 | ## (as it should). 8 | ## 9 | ## Note that this script makes the assumption that there is a start_clean.boot 10 | ## file available in $ROOTDIR/release/VSN. 11 | 12 | # Determine the abspath of where this script is executing from. 13 | ERTS_BIN_DIR=$(cd ${0%/*} && pwd) 14 | 15 | # Now determine the root directory -- this script runs from erts-VSN/bin, 16 | # so we simply need to strip off two dirs from the end of the ERTS_BIN_DIR 17 | # path. 18 | ROOTDIR=${ERTS_BIN_DIR%/*/*} 19 | 20 | # Parse out release and erts info 21 | START_ERL=`cat $ROOTDIR/releases/start_erl.data` 22 | ERTS_VSN=${START_ERL% *} 23 | APP_VSN=${START_ERL#* } 24 | 25 | BINDIR=$ROOTDIR/erts-$ERTS_VSN/bin 26 | EMU=beam 27 | PROGNAME=`echo $0 | sed 's/.*\///'` 28 | CMD="$BINDIR/erlexec" 29 | export EMU 30 | export ROOTDIR 31 | export BINDIR 32 | export PROGNAME 33 | 34 | exec $CMD -boot $ROOTDIR/releases/$APP_VSN/start_clean ${1+"$@"} -------------------------------------------------------------------------------- /releasenotes/riak-0.4.txt: -------------------------------------------------------------------------------- 1 | riak-0.4 2 | 3 | This release changes the storage backend API in a backward- 4 | incompatible fashion, invalidating old stores. However, translation 5 | from the old form to the new is not difficult. Please feel free to 6 | contact the Riak developers if you need assistance. 7 | 8 | tests: 9 | - an eunit harness which is run via "make test" 10 | - unit tests cover over 40% of code in riak source 11 | 12 | riak_api: 13 | - removed this gen_server entirely, allowing shorter request paths 14 | 15 | riak_bucket: 16 | - default bucket properties are now configurable via riak.erlenv 17 | 18 | riak_client: 19 | - now directly starts remote FSMs 20 | 21 | riak_ring_manager: 22 | - the ring state is now in an ets table instead of gen_server state 23 | (for improved speed with many readers and occasional writes) 24 | 25 | riak_sup: 26 | - more OTP-friendly child specifications 27 | 28 | riak_keys_fsm: 29 | - a new and improved way of providing a list of a bucket's keys 30 | 31 | riak_eventer: 32 | - eventers can specify a pattern, and only matching events will be sent 33 | 34 | general: 35 | - storage backends are now aware of {Bucket,Key} instead of just a hash 36 | - many small improvements made in the course of expanding test coverage 37 | - initialization work in core FSMs deferred to reduce start() blocking 38 | -------------------------------------------------------------------------------- /releasenotes/riak-0.3.txt: -------------------------------------------------------------------------------- 1 | riak-0.3 2 | 3 | This release includes a backwards-incompatible change to riak_object, 4 | enforcing all object keys to be binaries. This only effects users of 5 | the Erlang native interface. Users of the HTTP/Jiak interface should 6 | not be affected. 7 | 8 | riak_object: 9 | - Enforcement of binary-only object keys. 10 | 11 | HTTP Interface: 12 | - new Java client. 13 | - HTTP results now include Link: headers (http://tools.ietf.org/html/draft-nottingham-http-link-header-06). 14 | - Upgraded to Webmachine-1.4 and mochiweb-trunk. 15 | - Fixed default web access log directory. 16 | 17 | Eventer: 18 | - Performance improvements (only calculate eventers on ring-state changes). 19 | - Add a local logfile facility via the riak_local_logfile configuration 20 | parameter. This should only be used for debugging as the profuse logging 21 | can compete with the I/O demands of Riak itself. 22 | 23 | Internals: 24 | - Fix for a ring-state gossip bug encountered when running large clusters. 25 | - Increased value of ERL_MAX_PORTS to 4096 to support larger ring sizes. 26 | 27 | Performance: 28 | - Turn riak_bucketkeys into a gen_server2 process with tuned GC parameters. 29 | - Enable kernel-poll in Erlang VM by default. 30 | 31 | Documentation: 32 | - Documented all configuration parameters in config/riak.erlenv 33 | 34 | -------------------------------------------------------------------------------- /orgbatch.el: -------------------------------------------------------------------------------- 1 | ;; setup erlang mode 2 | (let ((tp (shell-command-to-string "erl -noinput -eval \"io:format(\\\"~s\\\", [code:lib_dir(tools)])\" -run init stop"))) 3 | (setq load-path (cons (concat tp "/emacs") 4 | load-path)) 5 | (require 'erlang-start)) 6 | 7 | ;; Org-mode 8 | ;; specify ORG_PATH if you have org-mode installed somewhere 9 | ;; other than your site-lisp path 10 | (let ((orgpath (getenv "ORG_PATH"))) 11 | (if (stringp orgpath) 12 | (setq load-path 13 | (cons (concat orgpath "/lisp") 14 | (cons (concat orgpath "/contrib/lisp") 15 | load-path))))) 16 | (require 'org-install) 17 | (add-to-list 'auto-mode-alist '("\\.org\\'" . org-mode)) 18 | 19 | ;; allow css to set code highlighting style 20 | (setq org-export-htmlize-output-type 'css) 21 | 22 | ;; command-line --eval functions 23 | (defun riak-export-doc-file (filename format) 24 | (find-file filename) 25 | (cond ((eq format 'ascii) (org-export-as-ascii 3) 26 | (kill-this-buffer)) 27 | ((eq format 'html) (org-export-as-html 3) 28 | (kill-this-buffer)) 29 | (message "Unknown export format")) 30 | (kill-this-buffer)) 31 | 32 | (defun riak-export-doc-dir (directory format) 33 | (mapcar 34 | (lambda (filename) (riak-export-doc-file filename format)) 35 | (file-expand-wildcards (concat directory "/*.org")))) 36 | -------------------------------------------------------------------------------- /dialyzer.ignore-warnings: -------------------------------------------------------------------------------- 1 | ### 2 | ### This is not a comment, just a pattern that is 3 | ### very unlikely to match anything that we care about. 4 | ### Don't put blank lines in this file. 5 | ### 6 | ### Items that can be ignored because of EUnit macros doing odd 7 | ### things or Dialyzer not finding erlang:load_nif/2 (which definitely exists). 8 | ### 9 | Call to missing or unexported function erlang:load_nif/2 10 | The variable _ can never match since previous clauses completely covered the type 'false' 11 | Contract for function that does not exist: riak_client:for_dialyzer_only_ignore/2 12 | ### 13 | ### Items in third-party code that are harmless enough 14 | ### 15 | js_mochijson2.erl:530: The variable __V can never match since previous clauses completely covered the type 'true' 16 | mochijson2.erl:530: The variable __V can never match since previous clauses completely covered the type 'true' 17 | mochiweb_request.erl:246: The variable Length can never match since previous clauses completely covered the type 'chunked' | 'undefined' | integer() | {'unknown_transfer_encoding',_} 18 | mochiweb_request.erl:271: The call mochiweb:new_response({{_,port() | {'ssl',{'sslsocket',_,_}},_,_,_,_},binary() | maybe_improper_list() | integer(),gb_tree()}) contains an opaque term as 1st argument argument when a structured term of type {_,_,[any()] | tuple()} is expected 19 | mochiweb_socket_server.erl:128: The call erlang:integer_to_list(S::maybe_improper_list()) will never return since it differs in the 1st argument from the success typing arguments: (integer()) 20 | mochiweb_sup.erl:26: The pattern [Spec | _] can never match the type [] 21 | -------------------------------------------------------------------------------- /releasenotes/riak-0.7.1.txt: -------------------------------------------------------------------------------- 1 | ------------------------ 2 | Riak 0.7.1 Release Notes 3 | ------------------------ 4 | 5 | Core: 6 | 7 | - The handoff mechanism has been rewritten, yielding faster handoff with less CPU/Memory 8 | utilization. 9 | 10 | - riak_vnode's are now managed by an OTP supervisor. 11 | 12 | Backends: 13 | 14 | - riak_multi_backend is a "bucket-multiplexing" backend that allows different buckets 15 | to use different backend modules. 16 | 17 | - riak_cache_backend is a memory-only caching backend with configurable maximum size 18 | and object TTL. 19 | 20 | - Three new functions were added to the Storage Backend API: 21 | 22 | - drop/1 : Drops the underlying store. 23 | - is_empty/1: Returns true if the underlying store contains no data items 24 | - fold/3: Performs the equivalent of a lists:foldl/3 over the store. 25 | 26 | - The riak_osmos_backend has been removed. 27 | 28 | Administration: 29 | 30 | - The 'riak attach' command now works with multiple riak nodes running on the same 31 | physical machine. 32 | 33 | - The 'riak-admin backup' command now allows backups of either a single node or 34 | the entire cluster. 35 | 36 | HTTP Interfaces: 37 | 38 | - Upgraded to webmachine-1.5.2 39 | 40 | - The raw HTTP resource now accepts the "returnbody=true" query parameter. 41 | 42 | - Both the Jiak and Raw HTTP resources allow streaming of keys 43 | 44 | - Added a "ping" resource on "/ping", intended for use with HTTP load balancers. 45 | 46 | Miscellaneous: 47 | 48 | - Various bugfixes to the Ruby client. 49 | - Documentation improvements. 50 | -------------------------------------------------------------------------------- /rel/reltool.config: -------------------------------------------------------------------------------- 1 | %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*- 2 | %% ex: ts=4 sw=4 et 3 | {sys, [ 4 | {lib_dirs, ["../deps"]}, 5 | {rel, "riak", "0.12.1", 6 | [ 7 | kernel, 8 | stdlib, 9 | sasl, 10 | os_mon, 11 | crypto, 12 | runtime_tools, 13 | erlang_js, 14 | mochiweb, 15 | webmachine, 16 | luke, 17 | bitcask, 18 | riak_core, 19 | riak_kv 20 | ]}, 21 | {rel, "start_clean", "", 22 | [ 23 | kernel, 24 | stdlib 25 | ]}, 26 | {boot_rel, "riak"}, 27 | {profile, embedded}, 28 | {excl_sys_filters, ["^bin/.*", 29 | "^erts.*/bin/(dialyzer|typer)"]}, 30 | {excl_archive_filters, [".*"]}, 31 | {app, erlang_js, [{incl_cond, include}]}, 32 | {app, luke, [{incl_cond, include}]}, 33 | {app, ebloom, []}, 34 | {app, bitcask, [{incl_cond, include}]}, 35 | {app, riak_core, [{incl_cond, include}]}, 36 | {app, riak_kv, [{incl_cond, include}]}, 37 | {app, sasl, [{incl_cond, include}]} 38 | ]}. 39 | 40 | 41 | {target_dir, "riak"}. 42 | 43 | {overlay_vars, "vars.config"}. 44 | 45 | {overlay, [ 46 | {mkdir, "data/ring"}, 47 | {mkdir, "log/sasl"}, 48 | {copy, "files/erl", "{{erts_vsn}}/bin/erl"}, 49 | {copy, "files/nodetool", "{{erts_vsn}}/bin/nodetool"}, 50 | {template, "files/app.config", "etc/app.config"}, 51 | {template, "files/vm.args", "etc/vm.args"}, 52 | {template, "files/riak", "bin/riak"}, 53 | {template, "files/riak-admin", "bin/riak-admin"} 54 | ]}. 55 | 56 | 57 | -------------------------------------------------------------------------------- /releasenotes/riak-0.6.txt: -------------------------------------------------------------------------------- 1 | From Riak 0.5 to Riak 0.6 there were minor performance enhancements, a 2 | new in-memory storage backend, a streaming interface to map/reduce, 3 | and a few bug fixes. In a little more detail: 4 | 5 | - The R, W, and DW parameters for client requests are now exposed in 6 | all of the various jiak client libraries. 7 | 8 | - There is a new in-memory-only backend based on gb_trees which is 9 | faster than the ets_backend for many common uses. 10 | 11 | - Some bugs were fixed, such as proper handling of an empty ringstate 12 | directory when pruning. 13 | 14 | - The "bucket_mod" schema parameter can now be set over the HTTP interface. 15 | 16 | - Numerous space and time improvements around basic FSM operations, such as 17 | - Using nodes() instead of net_adm:ping() to find reachable nodes. 18 | - Doing less work making vtags 19 | - Using shorter, fixed-size client ids in vclocks 20 | 21 | - The map/reduce system now allows streaming to and from clients, 22 | allowing for indefinitely-sized input and output sets. This is 23 | only partially documented now, but those wishing to use it 24 | immediately should see that riak_client:mapred is now implemented 25 | in terms of riak_client:mapred_stream. 26 | 27 | - The riak_get_fsm now exits much more quickly in the common case, 28 | which causes cleanup activity such as read-repair and 29 | delete-finalization to happen faster. There is no longer a "delete 30 | window" of any noticable time between the success of a delete 31 | operation and when the document is actually removed from storage. 32 | 33 | - There is a new bucket parameter, "chash_keyfun", allowing 34 | per-bucket configuration of the consistent hashing function. 35 | (that function must still have the type f({Bucket,Key}) -> 160bit-binary) 36 | 37 | -------------------------------------------------------------------------------- /releasenotes/riak-0.5.txt: -------------------------------------------------------------------------------- 1 | riak-0.5 2 | 3 | By popular demand, this release changes the identifier type for 4 | buckets (Riak's namespaces) from atoms to binaries. Old code using 5 | the native interface must change accordingly; code using the jiak http 6 | interface will continue to work unchanged. To convert an old 7 | cluster's storage, simply do a backup and restore -- this will convert 8 | all of the data on the way back in. 9 | 10 | riak_client: 11 | - preferred use is now riak:client_connect(Nodename) instead of IP/Port 12 | - bugfix where timeouts could cause incorrect responses 13 | 14 | eventers: 15 | - no longer gen_event, now using simple erlang messaging 16 | - fully capable matchspecs now available for event matching 17 | - use monitor to watch for death of handlers 18 | - the example riak_event_logger is now a little friendlier 19 | 20 | riak_backup: 21 | - more robust due to explicit ring fetch and node connection 22 | 23 | testing: 24 | - more coverage, especially of backend interface, riak_object 25 | 26 | mapreduce: 27 | - less messaging via batched groups to each vnode 28 | - intermediate per-vnode reduce phase 29 | 30 | build process: 31 | - much more useful error message when building with erlang < R13 32 | 33 | ring management: 34 | - simplify metadata structure 35 | - fix filtered_preflist bug in case of small number of nodes 36 | - internal naming cleanup 37 | - prune old state files more aggressively 38 | 39 | configuration: 40 | - new 'start_apps' param for starting dependency applications 41 | (useful for some custom storage backends) 42 | 43 | native client interface: 44 | - added list_buckets and filter_keys 45 | 46 | client libraries: 47 | - url-style quoting of buckets and keys in http client content 48 | - initial version of ruby client 49 | - java client uses List instead of ArrayList 50 | 51 | 52 | -------------------------------------------------------------------------------- /releasenotes/riak-0.11.0.txt: -------------------------------------------------------------------------------- 1 | ------------------------- 2 | Riak 0.11.0 Release Notes 3 | ------------------------- 4 | 5 | Bitcask has arrived as the new default backend for Riak. Bitcask is Basho's 6 | new key/value store that should provide good balanced performance out 7 | of the box on most systems. Read more about it from the initial 8 | announcement here http://blog.basho.com/2010/04/27/hello,-bitcask/ 9 | 10 | Users that wish to upgrade from another backend will need to backup their 11 | cluster (using riak-admin) before changing the backend in app.config, restart 12 | riak and restore from the backup into bitcask. 13 | 14 | The protocol buffers client (PBC) interface has been enhanced to 15 | add map/reduce support and access to simple bucket properties. The 16 | erlang and python clients have been updated accordingly. 17 | 18 | Put operations using the local erlang client that request a returnbody 19 | have received a performance enhancement. Internally the put operation 20 | now returns the body directly, so an additional get is no longer 21 | required. The PBC and HTTP interfaces have been updated to use 22 | this new mechanism. 23 | 24 | Enhancements 25 | -------- 26 | 58 - provide default content-type of application/octet-stream if none 27 | present in the object. 28 | 74 - create ring directory if non-existant 29 | 142 - riak-admin leave restored 30 | 31 | Bugs Fixed 32 | ---------- 33 | 35 - {error, blah} messages are now passed on to javascript map/reduce jobs 34 | 104 - missing bucket/key into map/reduce job crashed it. Fixed. 35 | 193 - list keys sometimes uses downed nodes. Fixed. 36 | 208 - deleting a key with a post-commit hook set crashed riak. Fixed. 37 | 38 | Known Issues 39 | ------------ 40 | * During testing we discovered an issue where nodes do not leave the ring 41 | cleanly is the number of partitions is not a power of 2. Until this is 42 | resolved, only ring sizes that are powers of 2 should be chosen (e.g. 43 | 16, 32, 64, 256, 1024). The default ring size is 64 and is unaffected. 44 | -------------------------------------------------------------------------------- /releasenotes/riak-0.9.txt: -------------------------------------------------------------------------------- 1 | ---------------------- 2 | Riak 0.9 Release Notes 3 | ---------------------- 4 | 5 | Riak 0.9 deprecates the "Jiak" interface. The HTTP interface will now 6 | be managed fully through the raw_http_resource. In line with making 7 | this the official interface, the default base URL has moved from 8 | "/raw" to "/riak". 9 | 10 | New client libraries for several languages are included in the 11 | client_lib directory. These libraries include support for advanced 12 | features like sibling object versions and map/reduce. 13 | 14 | The build system has been greatly improved: 15 | - 32/64-bit, and other system properties, are now properly detected 16 | - developers: "rel/data" is no longer copied during "make dev" 17 | - developers: repeated runs of "make" should behave better 18 | 19 | Map/reduce handling has seen some reworking: 20 | - processing flows are now more cleanly described, thanks to the new 21 | 'luke' aplication 22 | - reduce phases can now be computed in parallel 23 | - better bubbling of errors encountered during phase execution 24 | - ability to specify timeout in HTTP request 25 | - ability to end an HTTP map/reduce query with a 'link' phase 26 | - ability to use a Javascript 'reduce' phase immediately after a 27 | 'link' phase 28 | - lists of lists returned from map/reduce phases are now preserved 29 | as such, instead of being flattened 30 | - better character-set protection (client-visible errors for 31 | non-Unicode data) 32 | 33 | Riak 0.9 also includes a number of minor improvements and bug fixes: 34 | - Webmachine 1.6 35 | - better handling of pipelined requests 36 | - proper 100 Continue format 37 | - man pages for the riak and riak-admin scripts are included in the 38 | doc/man directory 39 | - validity checking for some bucket-property HTTP requests 40 | - better handling of vnode errors during read requests 41 | - initial POST to /Bucket/Key now works just like initial PUT, 42 | instead of returning 404 43 | - better handling of whitespace in Link headers 44 | - the 'match anything' link URL "_,_,_" is now supported 45 | - add 'total' get/put stats to riak_stat 46 | - more unit test coverage 47 | - fix fold/3 in riak_ets_backend 48 | - case-insensitivity for X-Riak-Meta-* headers (Thanks, Jonathan.) 49 | - removal of bogus "unable to read *.ez" errors on startup 50 | - fixed a bug around using the "charset" parameter in the 51 | content-type header 52 | - fixed a bug around changing the content-type of an object 53 | -------------------------------------------------------------------------------- /releasenotes/riak-0.7.txt: -------------------------------------------------------------------------------- 1 | ---------------------- 2 | Riak 0.7 Release Notes 3 | ---------------------- 4 | 5 | - The Riak source tree has undergone a major restructuring as part of a 6 | transition to a new build tool -- rebar. This tool enforces OTP compliant 7 | directory structures for the riak application and its dependencies and 8 | provides an easy-to-use wrapper around the new reltool system available in 9 | Erlang R1303+. Using reltool makes it possible to construct a standalone 10 | server instance that can be used on machines without Erlang installed. This 11 | transition has also enabled us to introduce a standardized script for 12 | starting/stopping and other server management tasks. 13 | 14 | You can watch a screencast on Riak and Rebar here: 15 | 16 | http://blog.basho.com/2009/12/17/basho-screencast---dave-smith-introduces-riak-in-an-embedded-node/ 17 | 18 | - In addition to the existing Client:list_keys function, there is now also 19 | a Client:stream_list_keys function that provides the keys in a bucket over a 20 | series of messages instead of in a single list. This can be useful with 21 | larger buckets. The messages will be of the form {ReqId,{keys,Res}} followed 22 | by a single {ReqId, done} to notify the client that all keys have been sent. 23 | 24 | - riak_vnode has undergone a complete rewrite to fix some handoff-related bugs 25 | and to eliminate the need for a separate riak_vnode_sidekick process. 26 | 27 | - riak_stat : Riak can now track basic internal statistics about how many 28 | reads and writes happened, and with what latency, in the last minute. 29 | Add {riak_stat, true} to your Riak config file to enable stat tracking, then 30 | use riak_client:get_stats/1 to retrieve the latest stats. 31 | 32 | - Setting the bucket property "allow_mult" to 'false' enables Riak's 33 | automatic sibling resolution. Prior to this release, all resolution was 34 | done at read time. With 0.7, sibling resolution is done at write time for 35 | allow_mult=false buckets. 36 | 37 | - In fixing a bug in filtered_preflist, we also made Riak's default ring 38 | partition claiming function more explicitly evenly distributed in 0.7. The 39 | function will attempt to to produce an "optimal" claim for the N-value set 40 | by the "target_n_val" configuration parameter (3, by default). 41 | 42 | - The latest version of Webmachine (http://webmachine.basho.com) has been 43 | integrated into Riak, which increases HTTP/Jiak performance considerably. 44 | 45 | - Jonathan Lee contributed a patch that causes any object metadata keys 46 | beginning with "X-Riak-Meta" to be emitted as HTTP headers when using the 47 | "raw" HTTP interface. 48 | 49 | - Sean Cribbs refactored the Ruby client library to be more idiomatic and 50 | maintainable. 51 | 52 | - Jayson Baird contributed a new Python client that uses PyCURL when available. 53 | 54 | 55 | 56 | 57 | 58 | 59 | -------------------------------------------------------------------------------- /releasenotes/riak-0.10.txt: -------------------------------------------------------------------------------- 1 | ----------------------- 2 | Riak 0.10 Release Notes 3 | ----------------------- 4 | 5 | IMPORTANT: If you are upgrading an earlier version of Riak, please read 6 | the TRANSITION file in the directory above this 7 | (http://bitbucket.org/basho/riak/src/55a7f83c2ae2/TRANSITION) 8 | 9 | Riak 0.10 brings many new features and improvements. 10 | 11 | In addition to the existing HTTP interface, Riak 0.10 exposes a 12 | protocol buffers interface. This interface allows for basic put/get 13 | operations, and has client libraries in Python and Erlang. Support 14 | for map/reduce operations, and additional language libraries will be 15 | added soon. Read more about the interface on the wiki: 16 | https://wiki.basho.com/display/RIAK/PBC+API 17 | 18 | Riak 0.10 exposes the ability to tie into an object's update 19 | lifecycle, via pre-commit and post-commit hooks. Hook functions can 20 | be implemented in Erlang or Javascript. More detail on the wiki: 21 | https://wiki.basho.com/display/RIAK/Pre-+and+Post-Commit+Hooks 22 | 23 | Listing the keys stored in a bucket is faster an more reliable in 0.10. 24 | 25 | The "eventer" subsystem has been removed from Riak. It was found to 26 | be a bottleneck, as well as the main source of non-modularity in the 27 | code. This should not affect client code, unless it was explicitly 28 | calling 'riak_eventer:add_handler' function. 29 | 30 | Chunks of Riak have been rearranged for cleanliness and modularity. 31 | If you are upgrading an existing cluster to 0.10, please carefully 32 | read and follow the instructions in the TRANSITION file, referenced at 33 | the top of these release notes. Some filenames and app.config 34 | settings have changed, which you will want to account for in your 35 | installation. 36 | 37 | Erlang applications that Riak depends on (webmachine, erlang_js, etc.) 38 | have been moved out of the Riak repository, and into their own 39 | repositories. If you install a prebuilt Riak, this change will not 40 | affect you. If you build Riak from the source tarball, this change 41 | will not affect you. If you build Riak by cloning from the Bitbucket 42 | repository, your first "make" will download the additional source 43 | needed from Bitbucket. 44 | 45 | Client libraries have been moved into their own repositories. Instead 46 | of finding them under the client_lib subdirectory, please visit the 47 | appropriate repo: 48 | - http://bitbucket.org/basho/riak-javascript-client 49 | - http://bitbucket.org/basho/riak-python-client 50 | - http://bitbucket.org/basho/riak-ruby-client 51 | (same as) http://github.com/seancribbs/ripple/ 52 | - http://bitbucket.org/basho/riak-java-client 53 | - http://bitbucket.org/basho/riak-php-client 54 | - http://bitbucket.org/basho/riak-erlang-client 55 | (using protocol buffers instead of distributed Erlang) 56 | 57 | Riak 0.10 also includes a number of minor improvements and bug fixes: 58 | - multipart/mixed responses from the HTTP interface now use proper 59 | line termination (\r\n, instead of just \n) 60 | - fixed the "fold" functionality (used during handoff) for the 61 | multi-backend. 62 | - improvements to gossip corner cases 63 | - improvements to shell scripts for cross-platform compatibility 64 | - improvements for running map/reduce jobs on a degraded cluster 65 | -------------------------------------------------------------------------------- /rel/files/nodetool: -------------------------------------------------------------------------------- 1 | %% -*- erlang -*- 2 | %% ------------------------------------------------------------------- 3 | %% 4 | %% nodetool: Helper Script for interacting with live nodes 5 | %% 6 | %% ------------------------------------------------------------------- 7 | 8 | main(Args) -> 9 | %% Extract the args 10 | {RestArgs, TargetNode} = process_args(Args, [], undefined), 11 | 12 | %% See if the node is currently running -- if it's not, we'll bail 13 | case {net_kernel:hidden_connect_node(TargetNode), net_adm:ping(TargetNode)} of 14 | {true, pong} -> 15 | ok; 16 | {_, pang} -> 17 | io:format("Node ~p not responding to pings.\n", [TargetNode]), 18 | halt(1) 19 | end, 20 | 21 | case RestArgs of 22 | ["ping"] -> 23 | %% If we got this far, the node already responsed to a ping, so just dump 24 | %% a "pong" 25 | io:format("pong\n"); 26 | ["stop"] -> 27 | io:format("~p\n", [rpc:call(TargetNode, init, stop, [], 60000)]); 28 | ["restart"] -> 29 | io:format("~p\n", [rpc:call(TargetNode, init, restart, [], 60000)]); 30 | ["reboot"] -> 31 | io:format("~p\n", [rpc:call(TargetNode, init, reboot, [], 60000)]); 32 | ["rpc", Module, Function | RpcArgs] -> 33 | case rpc:call(TargetNode, list_to_atom(Module), list_to_atom(Function), [RpcArgs], 60000) of 34 | ok -> 35 | ok; 36 | {badrpc, Reason} -> 37 | io:format("RPC to ~p failed: ~p\n", [TargetNode, Reason]), 38 | halt(1); 39 | _ -> 40 | halt(1) 41 | end; 42 | Other -> 43 | io:format("Other: ~p\n", [Other]), 44 | io:format("Usage: nodetool {ping|stop|restart|reboot}\n") 45 | end, 46 | net_kernel:stop(). 47 | 48 | process_args([], Acc, TargetNode) -> 49 | {lists:reverse(Acc), TargetNode}; 50 | process_args(["-setcookie", Cookie | Rest], Acc, TargetNode) -> 51 | erlang:set_cookie(node(), list_to_atom(Cookie)), 52 | process_args(Rest, Acc, TargetNode); 53 | process_args(["-name", TargetName | Rest], Acc, _) -> 54 | ThisNode = append_node_suffix(TargetName, "_maint_"), 55 | {ok, _} = net_kernel:start([ThisNode, longnames]), 56 | process_args(Rest, Acc, nodename(TargetName)); 57 | process_args(["-sname", TargetName | Rest], Acc, _) -> 58 | ThisNode = append_node_suffix(TargetName, "_maint_"), 59 | {ok, _} = net_kernel:start([ThisNode, shortnames]), 60 | process_args(Rest, Acc, nodename(TargetName)); 61 | process_args([Arg | Rest], Acc, Opts) -> 62 | process_args(Rest, [Arg | Acc], Opts). 63 | 64 | 65 | nodename(Name) -> 66 | case string:tokens(Name, "@") of 67 | [_Node, _Host] -> 68 | list_to_atom(Name); 69 | [Node] -> 70 | [_, Host] = string:tokens(atom_to_list(node()), "@"), 71 | list_to_atom(lists:concat([Node, "@", Host])) 72 | end. 73 | 74 | append_node_suffix(Name, Suffix) -> 75 | case string:tokens(Name, "@") of 76 | [Node, Host] -> 77 | list_to_atom(lists:concat([Node, Suffix, os:getpid(), "@", Host])); 78 | [Node] -> 79 | list_to_atom(lists:concat([Node, Suffix, os:getpid()])) 80 | end. 81 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | RIAK_TAG = $(shell hg identify -t) 2 | 3 | .PHONY: rel stagedevrel deps 4 | 5 | all: deps compile 6 | 7 | compile: 8 | ./rebar compile 9 | 10 | deps: 11 | ./rebar get-deps 12 | 13 | clean: 14 | ./rebar clean 15 | 16 | distclean: clean devclean relclean ballclean 17 | ./rebar delete-deps 18 | 19 | test: 20 | ./rebar skip_deps=true eunit 21 | 22 | ## 23 | ## Release targets 24 | ## 25 | rel: deps 26 | ./rebar compile generate 27 | 28 | relclean: 29 | rm -rf rel/riak 30 | 31 | ## 32 | ## Developer targets 33 | ## 34 | stagedevrel: dev1 dev2 dev3 35 | $(foreach dev,$^,\ 36 | $(foreach dep,$(wildcard deps/*), rm -rf dev/$(dev)/lib/$(shell basename $(dep))-* && ln -sf $(abspath $(dep)) dev/$(dev)/lib;)) 37 | 38 | devrel: dev1 dev2 dev3 39 | 40 | dev1 dev2 dev3: 41 | mkdir -p dev 42 | (cd rel && ../rebar generate target_dir=../dev/$@ overlay_vars=vars/$@_vars.config) 43 | 44 | devclean: clean 45 | rm -rf dev 46 | 47 | stage : rel 48 | $(foreach dep,$(wildcard deps/*), rm -rf rel/riak/lib/$(shell basename $(dep))-* && ln -sf $(abspath $(dep)) rel/riak/lib;) 49 | 50 | ## 51 | ## Doc targets 52 | ## 53 | docs: 54 | ./rebar skip_deps=true doc 55 | @cp -R apps/luke/doc doc/luke 56 | @cp -R apps/riak_core/doc doc/riak_core 57 | @cp -R apps/riak_kv/doc doc/riak_kv 58 | 59 | orgs: orgs-doc orgs-README 60 | 61 | orgs-doc: 62 | @emacs -l orgbatch.el -batch --eval="(riak-export-doc-dir \"doc\" 'html)" 63 | 64 | orgs-README: 65 | @emacs -l orgbatch.el -batch --eval="(riak-export-doc-file \"README.org\" 'ascii)" 66 | @mv README.txt README 67 | 68 | APPS = kernel stdlib sasl erts ssl tools os_mon runtime_tools crypto inets \ 69 | xmerl webtool snmp public_key mnesia eunit syntax_tools compiler 70 | COMBO_PLT = $(HOME)/.riak_combo_dialyzer_plt 71 | 72 | check_plt: compile 73 | dialyzer --check_plt --plt $(COMBO_PLT) --apps $(APPS) \ 74 | deps/*/ebin 75 | 76 | build_plt: compile 77 | dialyzer --build_plt --output_plt $(COMBO_PLT) --apps $(APPS) \ 78 | deps/*/ebin 79 | 80 | dialyzer: compile 81 | @echo 82 | @echo Use "'make check_plt'" to check PLT prior to using this target. 83 | @echo Use "'make build_plt'" to build PLT prior to using this target. 84 | @echo 85 | @sleep 1 86 | dialyzer -Wno_return --plt $(COMBO_PLT) deps/*/ebin | \ 87 | fgrep -v -f ./dialyzer.ignore-warnings 88 | 89 | cleanplt: 90 | @echo 91 | @echo "Are you sure? It takes about 1/2 hour to re-build." 92 | @echo Deleting $(COMBO_PLT) in 5 seconds. 93 | @echo 94 | sleep 5 95 | rm $(COMBO_PLT) 96 | 97 | # Release tarball creation 98 | # Generates a tarball that includes all the deps sources so no checkouts are necessary 99 | 100 | distdir: 101 | $(if $(findstring tip,$(RIAK_TAG)),$(error "You can't generate a release tarball from tip")) 102 | mkdir distdir 103 | hg clone . distdir/riak-clone 104 | cd distdir/riak-clone; \ 105 | hg update -r $(RIAK_TAG) 106 | cd distdir/riak-clone; \ 107 | hg archive ../$(RIAK_TAG); \ 108 | mkdir ../$(RIAK_TAG)/deps; \ 109 | make deps; \ 110 | for dep in deps/*; do cd $${dep} && hg archive ../../../$(RIAK_TAG)/$${dep}; cd ../..; done 111 | 112 | dist $(RIAK_TAG).tar.gz: distdir 113 | cd distdir; \ 114 | tar czf ../$(RIAK_TAG).tar.gz $(RIAK_TAG) 115 | 116 | ballclean: 117 | rm -rf $(RIAK_TAG).tar.gz distdir 118 | 119 | -------------------------------------------------------------------------------- /.hgtags: -------------------------------------------------------------------------------- 1 | 383abb86f65dc78f47af1265997ce021875c7bb4 riak-0.2 2 | 383abb86f65dc78f47af1265997ce021875c7bb4 riak-0.2 3 | f98d13b442f6ebf14645afd878919edacd9f1a98 riak-0.2 4 | 661615e510dda3fdf251aac4bd5bd648671c2fa4 riak-0.3 5 | bc231db18e1c0413318ade1b097314d725fdca1c riak-0.3.1 6 | 47fa3a14cc63a37f6273ea537178eb70baaf72e0 riak-0.4 7 | 47490716f4c9b70cf5495968d6c2d7e84e24f47c riak-0.5 8 | 33a5ea0cbe7a015b23f8fbfe54d80682113527ea riak-0.5.1 9 | fdc2c2e4cebea159a622dbb0716c5430a096030b riak-0.5.2 10 | 5ffa6ae7e6999f9c765930c36a8b24d0e8b5874d riak-0.5.3 11 | 11895faae1cec4e14f76ec1fb55b87e739299e05 riak-0.6 12 | 93750f3fbbe2b4ca7cba3dbac16bad4039b84822 riak-0.6.1 13 | 46af62b77937ee39060d55011ffc74768b88a011 riak-0.6.2 14 | 46af62b77937ee39060d55011ffc74768b88a011 riak-0.6.2 15 | 07bb099b7b102e603edfea30a0f3b780168b11f6 riak-0.6.2 16 | e38077f4e4aa8c0d479848ce3c1a421dbf5ede1b riak-0.6.3 17 | 8790e3560defe0a12d31ffde5ef96c6946ac67a6 riak-0.7 18 | 8790e3560defe0a12d31ffde5ef96c6946ac67a6 riak-0.7 19 | 7b2af3b4c96854dab8132acc299d3f24148fc141 riak-0.7 20 | 82100e18113bf79f5c790e0f4858b0f33854fced riak-0.7.1 21 | 82100e18113bf79f5c790e0f4858b0f33854fced riak-0.7.1 22 | b6515e496da1cc5cb2203c345b6bb0ef407785b4 riak-0.7.1 23 | b6515e496da1cc5cb2203c345b6bb0ef407785b4 riak-0.7.1 24 | e1357f00129f7f99090698f190302df042e125b4 riak-0.7.1 25 | 54d5bca2e6c090c44404badd9380e238d5f8ee4e riak-0.7.1.1 26 | 54d5bca2e6c090c44404badd9380e238d5f8ee4e riak-0.7.1.1 27 | 0000000000000000000000000000000000000000 riak-0.7.1.1 28 | 65b66ac0fc83bee4e561791ea89f61ed866ac950 riak-0.8 29 | f93b7551002b9fc094894732912caa410ee04296 riak-0.9 30 | f93b7551002b9fc094894732912caa410ee04296 riak-0.9 31 | d46096e73be8093e2f01fb1533fe005899554994 riak-0.9 32 | d46096e73be8093e2f01fb1533fe005899554994 riak-0.9 33 | 6a957305862c37d3caf44ca695c2a8ea78384257 riak-0.9 34 | 6a957305862c37d3caf44ca695c2a8ea78384257 riak-0.9 35 | dbaa6f4ec58592da617692aff224c1bcd2825b31 riak-0.9 36 | 76ce7cca95d8bebbdc7db8f6b745a581a9414bc2 riak-0.9.1 37 | df553e35cc92bf54bee642f6f9a7b1de10db26c1 pre-0.10 38 | 30776b429ad6ecd78e1d989092a32f47a959426b riak-0.10rc1 39 | 0ea6fbd96cbb9aa079da32a42884551b110d1bab riak-0.10rc2 40 | c09d8f7659ce415d8917afeef6c79e6d0f6c7bbc riak-0.10rc3 41 | 11abcb65bc852b2c2612a4022599fbd103049999 riak-0.10rc4 42 | 897610194f2637f94e7478363dddaee307bf99e9 riak-0.10 43 | 897610194f2637f94e7478363dddaee307bf99e9 riak-0.10 44 | 45f284c5e5c030e0f6f2848e2292e45075fcae44 riak-0.10 45 | 83e4e6dffdb7d2651fd2d1867d01e1f0c33e135c riak-0.10.1 46 | 4acbe738e3801a6f87f331872f2b8b98adebd821 riak-0.11.0-rc1 47 | 0f41da416d362f9fa3e159078c5f15b4fdbc9333 riak-0.11.0rc1 48 | 76cdac5560b0cd744f35f02f96f9e7b63689c15d riak-0.11.0rc2 49 | 79879cbfbcc742e822d13db1d9586d9fd7c8f193 riak-0.11.0rc3 50 | 79879cbfbcc742e822d13db1d9586d9fd7c8f193 riak-0.11.0rc3 51 | 0000000000000000000000000000000000000000 riak-0.11.0rc3 52 | 0000000000000000000000000000000000000000 riak-0.11.0rc3 53 | 0ab8228298cf94d6ac8f266b522ededbc28768de riak-0.11.0rc3 54 | 1cf46333d426eb1d0b2c109f4ba153b9d9ea1ec1 riak-0.11.0rc4 55 | 2970633893feab4d9cd6b61a417f82366394bea8 riak-0.11.0rc6 56 | f3314680c6ca9dbb363018db001a7f55283a8989 riak-0.11.0rc7 57 | 6e87b4b76e3c2282cae99a1e20f5748f5fda33cf riak-0.11.0rc8 58 | fe57a454d21ef96208c1ce28807a6e0b0687bf42 riak-0.11.0rc9 59 | 6d17efd690ea79cbec1375da5a443c251e3ff71a riak-0.11.0 60 | 01b5275fd2ad22ad36ee837610fdaa16e14bdec0 riak-0.12.0rc1 61 | 96aee914c29f569869f7d9480f6c89f93c226bc3 riak-0.12.0rc2 62 | c9ad3232e093485988b5666d1dc2d58072bf65eb riak-0.12.1 63 | 0efa139ee9ca19805c7c48a9112b56f1555592ca riak-0.12.0 64 | -------------------------------------------------------------------------------- /releasenotes/riak-0.12.0.txt: -------------------------------------------------------------------------------- 1 | ------------------------- 2 | Riak 0.12.0 Release Notes 3 | ------------------------- 4 | 5 | Riak now uses a new and improved mechanism for determining whether a 6 | node is fully online and ready to participate in Riak operations. This 7 | is especially important in failure recovery situations, as it allows 8 | the storage backend to complete a full integrity check and repair 9 | process. (134) 10 | 11 | Applications can now use the keywords "one", "quorum" (or "default"), 12 | and "all" in place of number values to set R, W, and DW quorum settings. 13 | This allows developers to specify intended consistency levels more 14 | clearly. (211, 276, 277, 322) 15 | 16 | The multi backend has been fixed so bitcask can be used with the 17 | other backends (274). If innostore is installed it must be upgraded to 1.0.1 18 | if it will be used with the multi backend. 19 | 20 | Enhancements 21 | ------------ 22 | 82 - HTTP API now returns 400 when quorum parameters exceed N-val. 23 | 83 - Default quorum parameters are now configurable in HTTP and Protobuf APIs. 24 | 97 - Riak now calls a BackendModule:stop/1 function, allowing cleanup during shutdown. 25 | 190 - HTTP API now returns 503 when Riak operation times out. 26 | 192 - HTTP API no longer list keys on a bucket by default. 27 | 283 - HTTP API now returns 404 when an object is missing, regardless of accept header. (202) 28 | 216 - The "/stats" page now includes read-repair stats. 29 | 219 - A node now verifies that the ring_creation_size matches before joining a cluster. 30 | 230 - Upgrade to latest version of Mochiweb. 31 | 237 - Added a 'mapByFields' built-in Map/Reduce function. 32 | 246 - Improved error reporting in post-commit hooks. 33 | 251 - More descriptive error message on malformed link walking operations. 34 | 256 - The /stats endpoint now shows Riak version number. 35 | 259 - Improve python client packaging. Publish on PyPI. 36 | 267 - Updated bucket defaults to improve replica distribution across physical nodes. 37 | 274 - Improvements to storage backend interface layer. 38 | 365 - Use updated "rebar eunit" task for running tests. 39 | 40 | Bugs Fixed 41 | ---------- 42 | 26 - The 'devrel' target now builds on CentOS. 43 | 27 - Fixed 'riak-admin' problem on some architectures, including Solaris. 44 | 138 - Fixed platform specific problems in Riak 'init.d' script. 45 | 205 - Fixed Bitcask errors on 32-bit Erlang. (331, 344) 46 | 229 - Fixed 'riak stop' error on Mac OSX Snow Leopard 10.6.3. 47 | 240 - Python client now properly escapes "/" in Buckets and Keys. 48 | 253 - Correctly pass missing object (not_found) results between Map/Reduce phases. 49 | 274 - Correctly forward 'info' messages from multi_backend to child backends. 50 | 278 - Make Riak backup work correctly in all cases when objects are deleted while backup is in progress. 51 | 280 - Fixed corner cases causing timestamp collision in Bitcask. 52 | 281 - Fixed premature tombstone collection in Bitcask. 53 | 301 - Fixed chunked mapreduce results to use correct line breaks (\r\n). 54 | 305 - Fixed possible race condition between get and Bitcask merge. 55 | 382 - Update Map/Reduce to honor timeout setting. 56 | 361 - Cleaned up Dialyzer warnings. (373, 374, 376, 381, 389) 57 | 382 - Update Map/Reduce to honor timeout setting. 58 | 402 - Make Bitcask data and hint files more resistant to corruption. 59 | 60 | Riak has been updated with the necessary changes to compile 61 | on Erlang R14A, but has not been thoroughly tested on R14A. 62 | Please continue to run Riak on R13B04 in production. (263, 264, 269) 63 | 64 | All bug and issue numbers reference https://issues.basho.com. 65 | -------------------------------------------------------------------------------- /rel/files/app.config: -------------------------------------------------------------------------------- 1 | %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*- 2 | %% ex: ts=4 sw=4 et 3 | [ 4 | %% Riak Core config 5 | {riak_core, [ 6 | %% Default location of ringstate 7 | {ring_state_dir, "{{ring_state_dir}}"}, 8 | 9 | %% riak_web_ip is the IP address that the Riak HTTP interface will 10 | %% bind to. If this is undefined, the HTTP interface will not run. 11 | {web_ip, "{{web_ip}}" }, 12 | 13 | %% riak_web_port is the TCP port that the Riak HTTP interface will 14 | %% bind to. 15 | {web_port, {{web_port}} }, 16 | 17 | %% riak_handoff_port is the TCP port that Riak uses for 18 | %% intra-cluster data handoff. 19 | {handoff_port, {{handoff_port}} } 20 | ]}, 21 | 22 | %% Riak KV config 23 | {riak_kv, [ 24 | %% Storage_backend specifies the Erlang module defining the storage 25 | %% mechanism that will be used on this node. 26 | {storage_backend, riak_kv_bitcask_backend}, 27 | 28 | %% pb_ip is the IP address that the Riak Protocol Buffers interface 29 | %% will bid to. If this is undefined, the interface will not run. 30 | {pb_ip, "{{pb_ip}}" }, 31 | 32 | %% pb_port is the TCP port that the Riak Protocol Buffers interface 33 | %% will bind to 34 | {pb_port, {{pb_port}} }, 35 | 36 | %% raw_name is the first part of all URLS used by the Riak raw HTTP 37 | %% interface. See riak_web.erl and raw_http_resource.erl for 38 | %% details. 39 | %{raw_name, "riak"}, 40 | 41 | %% mapred_name is URL used to submit map/reduce requests to Riak. 42 | {mapred_name, "mapred"}, 43 | 44 | %% js_vm_count is the number of Javascript VMs to start per Riak 45 | %% node. 8 is a good default for smaller installations. A larger 46 | %% number like 12 or 16 is appropriate for installations handling 47 | %% lots of map/reduce processing. 48 | {js_vm_count, 8}, 49 | 50 | %% js_max_vm_mem is the maximum amount of memory, in megabytes, 51 | %% allocated to the Javascript VMs. If unset, the default is 52 | %% 8MB. 53 | {js_max_vm_mem, 8}, 54 | 55 | %% js_thread_stack is the maximum amount of thread stack, in megabyes, 56 | %% allocate to the Javascript VMs. If unset, the default is 16MB. 57 | %% NOTE: This is not the same as the C thread stack. 58 | {js_thread_stack, 16}, 59 | 60 | %% js_source_dir should point to a directory containing Javascript 61 | %% source files which will be loaded by Riak when it initializes 62 | %% Javascript VMs. 63 | %{js_source_dir, "/tmp/js_source"}, 64 | 65 | %% vnode_mr_timeout controls how much time Riak will permit a map 66 | %% function to execute on a given vnode before trying another vnode. 67 | %% Value is in milliseconds and defaults to 1000ms if not set. 68 | %{vnode_mr_timeout, 1000}, 69 | 70 | %% riak_stat enables the use of the "riak-admin status" command to 71 | %% retrieve information the Riak node for performance and debugging needs 72 | {riak_kv_stat, true} 73 | ]}, 74 | 75 | %% Bitcask Config 76 | {bitcask, [ 77 | {data_root, "{{bitcask_data_root}}"} 78 | ]}, 79 | 80 | %% SASL config 81 | {sasl, [ 82 | {sasl_error_logger, {file, "{{sasl_error_log}}"}}, 83 | {errlog_type, error}, 84 | {error_logger_mf_dir, "{{sasl_log_dir}}"}, % Log directory 85 | {error_logger_mf_maxbytes, 10485760}, % 10 MB max file size 86 | {error_logger_mf_maxfiles, 5} % 5 files max 87 | ]} 88 | ]. 89 | -------------------------------------------------------------------------------- /rel/files/riak: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | RUNNER_SCRIPT_DIR={{runner_script_dir}} 4 | RUNNER_SCRIPT=${0##*/} 5 | 6 | RUNNER_BASE_DIR={{runner_base_dir}} 7 | RUNNER_ETC_DIR={{runner_etc_dir}} 8 | RUNNER_LOG_DIR={{runner_log_dir}} 9 | PIPE_DIR={{pipe_dir}} 10 | RUNNER_USER={{runner_user}} 11 | 12 | # Make sure this script is running as the appropriate user 13 | if [ "$RUNNER_USER" -a "x$LOGNAME" != "x$RUNNER_USER" ]; then 14 | type -p sudo > /dev/null 2>&1 15 | if [ $? -ne 0 ]; then 16 | echo "sudo doesn't appear to be installed and your EUID isn't $RUNNER_USER" 1>&2 17 | exit 1 18 | fi 19 | echo "Attempting to restart script through sudo -u $RUNNER_USER" 20 | exec sudo -u $RUNNER_USER -i $RUNNER_SCRIPT_DIR/$RUNNER_SCRIPT $@ 21 | fi 22 | 23 | # Make sure CWD is set to runner base dir 24 | cd $RUNNER_BASE_DIR 25 | 26 | # Make sure log directory exists 27 | mkdir -p $RUNNER_LOG_DIR 28 | 29 | # Extract the target node name from node.args 30 | NAME_ARG=`grep '\-[s]*name' $RUNNER_ETC_DIR/vm.args` 31 | if [ -z "$NAME_ARG" ]; then 32 | echo "vm.args needs to have either -name or -sname parameter." 33 | exit 1 34 | fi 35 | 36 | # Extract the target cookie 37 | COOKIE_ARG=`grep '\-setcookie' $RUNNER_ETC_DIR/vm.args` 38 | if [ -z "$COOKIE_ARG" ]; then 39 | echo "vm.args needs to have a -setcookie parameter." 40 | exit 1 41 | fi 42 | 43 | # Identify the script name 44 | SCRIPT=`basename $0` 45 | 46 | # Parse out release and erts info 47 | START_ERL=`cat $RUNNER_BASE_DIR/releases/start_erl.data` 48 | ERTS_VSN=${START_ERL% *} 49 | APP_VSN=${START_ERL#* } 50 | 51 | # Add ERTS bin dir to our path 52 | ERTS_PATH=$RUNNER_BASE_DIR/erts-$ERTS_VSN/bin 53 | 54 | # Setup command to control the node 55 | NODETOOL="$ERTS_PATH/escript $ERTS_PATH/nodetool $NAME_ARG $COOKIE_ARG" 56 | 57 | # Check the first argument for instructions 58 | case "$1" in 59 | start) 60 | # Make sure there is not already a node running 61 | RES=`$NODETOOL ping` 62 | if [ "$RES" == "pong" ]; then 63 | echo "Node is already running!" 64 | exit 1 65 | fi 66 | export HEART_COMMAND="$RUNNER_SCRIPT_DIR/$SCRIPT start" 67 | mkdir -p $PIPE_DIR 68 | $ERTS_PATH/run_erl -daemon $PIPE_DIR/ $RUNNER_LOG_DIR \ 69 | "exec $RUNNER_SCRIPT_DIR/$SCRIPT console" 2>&1 70 | ;; 71 | 72 | stop) 73 | case `uname` in 74 | Darwin) 75 | # Make sure we explicitly set this because iTerm.app doesn't for 76 | # some reason. 77 | COMMAND_MODE=unix2003 78 | esac 79 | # Wait for the node to completely stop... 80 | PID=`ps -ef|grep "$RUNNER_BASE_DIR/.*/[b]eam.smp"|awk '{print $2}'` 81 | $NODETOOL stop 82 | while `kill -0 $PID 2>/dev/null`; 83 | do 84 | sleep 1 85 | done 86 | ;; 87 | 88 | restart) 89 | ## Restart the VM without exiting the process 90 | $NODETOOL restart 91 | ;; 92 | 93 | reboot) 94 | ## Restart the VM completely (uses heart to restart it) 95 | $NODETOOL reboot 96 | ;; 97 | 98 | ping) 99 | ## See if the VM is alive 100 | $NODETOOL ping 101 | ;; 102 | 103 | attach) 104 | # Make sure a node IS running 105 | RES=`$NODETOOL ping` 106 | if [ "$RES" != "pong" ]; then 107 | echo "Node is not running!" 108 | exit 1 109 | fi 110 | 111 | shift 112 | $ERTS_PATH/to_erl $PIPE_DIR 113 | ;; 114 | 115 | console) 116 | RES=`$NODETOOL ping` 117 | if [ "$RES" == "pong" ]; then 118 | echo "Node is already running - use '$SCRIPT attach' instead" 119 | exit 1 120 | fi 121 | # Setup beam-required vars 122 | ROOTDIR=$RUNNER_BASE_DIR 123 | BINDIR=$ROOTDIR/erts-$ERTS_VSN/bin 124 | EMU=beam 125 | PROGNAME=`echo $0 | sed 's/.*\///'` 126 | CMD="$BINDIR/erlexec -boot $RUNNER_BASE_DIR/releases/$APP_VSN/$SCRIPT \ 127 | -embedded -config $RUNNER_ETC_DIR/app.config \ 128 | -args_file $RUNNER_ETC_DIR/vm.args -- ${1+"$@"}" 129 | export EMU 130 | export ROOTDIR 131 | export BINDIR 132 | export PROGNAME 133 | 134 | # Dump environment info for logging purposes 135 | echo "Exec: $CMD" 136 | echo "Root: $ROOTDIR" 137 | 138 | # Log the startup 139 | logger -t "$SCRIPT[$$]" "Starting up" 140 | 141 | # Start the VM 142 | exec $CMD 143 | ;; 144 | 145 | *) 146 | echo "Usage: $SCRIPT {start|stop|restart|reboot|ping|console|attach}" 147 | exit 1 148 | ;; 149 | esac 150 | 151 | exit 0 152 | -------------------------------------------------------------------------------- /rel/files/riak-admin: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | RUNNER_SCRIPT_DIR={{runner_script_dir}} 4 | RUNNER_SCRIPT=${0##*/} 5 | 6 | RUNNER_BASE_DIR={{runner_base_dir}} 7 | RUNNER_ETC_DIR={{runner_etc_dir}} 8 | RUNNER_LOG_DIR={{runner_log_dir}} 9 | RUNNER_USER={{runner_user}} 10 | 11 | # Make sure this script is running as the appropriate user 12 | if [ "$RUNNER_USER" -a "x$LOGNAME" != "x$RUNNER_USER" ]; then 13 | type -p sudo > /dev/null 2>&1 14 | if [ $? -ne 0 ]; then 15 | echo "sudo doesn't appear to be installed and your EUID isn't $RUNNER_USER" 1>&2 16 | exit 1 17 | fi 18 | echo "Attempting to restart script through sudo -u $RUNNER_USER" 19 | exec sudo -u $RUNNER_USER -i $RUNNER_SCRIPT_DIR/$RUNNER_SCRIPT $@ 20 | fi 21 | 22 | # Make sure CWD is set to runner base dir 23 | cd $RUNNER_BASE_DIR 24 | 25 | # Extract the target node name from node.args 26 | NAME_ARG=`grep '\-[s]*name' $RUNNER_ETC_DIR/vm.args` 27 | if [ -z "$NAME_ARG" ]; then 28 | echo "vm.args needs to have either -name or -sname parameter." 29 | exit 1 30 | fi 31 | 32 | # Learn how to specify node name for connection from remote nodes 33 | echo "$NAME_ARG" | grep '^-sname' > /dev/null 2>&1 34 | if [ "X$?" = "X0" ]; then 35 | NAME_PARAM="-sname" 36 | NAME_HOST="" 37 | else 38 | NAME_PARAM="-name" 39 | echo "$NAME_ARG" | grep '@.*' > /dev/null 2>&1 40 | if [ "X$?" = "X0" ]; then 41 | NAME_HOST=`echo "${NAME_ARG}" | sed -e 's/.*\(@.*\)$/\1/'` 42 | else 43 | NAME_HOST="" 44 | fi 45 | fi 46 | 47 | # Extract the target cookie 48 | COOKIE_ARG=`grep '\-setcookie' $RUNNER_ETC_DIR/vm.args` 49 | if [ -z "$COOKIE_ARG" ]; then 50 | echo "vm.args needs to have a -setcookie parameter." 51 | exit 1 52 | fi 53 | 54 | # Identify the script name 55 | SCRIPT=`basename $0` 56 | 57 | # Parse out release and erts info 58 | START_ERL=`cat $RUNNER_BASE_DIR/releases/start_erl.data` 59 | ERTS_VSN=${START_ERL% *} 60 | APP_VSN=${START_ERL#* } 61 | 62 | # Add ERTS bin dir to our path 63 | ERTS_PATH=$RUNNER_BASE_DIR/erts-$ERTS_VSN/bin 64 | 65 | # Setup command to control the node 66 | NODETOOL="$ERTS_PATH/escript $ERTS_PATH/nodetool $NAME_ARG $COOKIE_ARG" 67 | 68 | # Check the first argument for instructions 69 | case "$1" in 70 | join) 71 | # Make sure the local node IS running 72 | RES=`$NODETOOL ping` 73 | if [ "$RES" != "pong" ]; then 74 | echo "Node is not running!" 75 | exit 1 76 | fi 77 | 78 | shift 79 | 80 | $NODETOOL rpc riak_kv_console join $@ 81 | ;; 82 | 83 | leave) 84 | # Make sure the local node is running 85 | RES=`$NODETOOL ping` 86 | if [ "$RES" != "pong" ]; then 87 | echo "Node is not running!" 88 | exit 1 89 | fi 90 | 91 | shift 92 | $NODETOOL rpc riak_kv_console leave $@ 93 | ;; 94 | 95 | status) 96 | # Make sure the local node IS running 97 | RES=`$NODETOOL ping` 98 | if [ "$RES" != "pong" ]; then 99 | echo "Node is not running!" 100 | exit 1 101 | fi 102 | shift 103 | 104 | $NODETOOL rpc riak_kv_console status $@ 105 | ;; 106 | 107 | js_reload) 108 | # Reload all Javascript VMs 109 | RES=`$NODETOOL ping` 110 | if [ "$RES" != "pong" ]; then 111 | echo "Node is not running!" 112 | exit 1 113 | fi 114 | 115 | $NODETOOL rpc riak_kv_js_manager reload 116 | ;; 117 | 118 | reip) 119 | ACTION=$1 120 | shift 121 | if [ $# -lt 2 ]; then 122 | echo "Usage $SCRIPT $ACTION " 123 | exit 1 124 | fi 125 | # Make sure the local node IS not running 126 | RES=`$NODETOOL ping` 127 | if [ "$RES" == "pong" ]; then 128 | echo "Node must be down to re-ip." 129 | exit 1 130 | fi 131 | OLDNODE=$1 132 | NEWNODE=$2 133 | $ERTS_PATH/erl -noshell \ 134 | -config $RUNNER_ETC_DIR/app.config \ 135 | -eval "riak_kv_console:$ACTION(['$OLDNODE', '$NEWNODE'])" \ 136 | -s init stop 137 | ;; 138 | 139 | restore) 140 | ACTION=$1 141 | shift 142 | 143 | if [ $# -lt 3 ]; then 144 | echo "Usage: $SCRIPT $ACTION " 145 | exit 1 146 | fi 147 | 148 | NODE=$1 149 | COOKIE=$2 150 | FILENAME=$3 151 | 152 | $ERTS_PATH/erl -noshell $NAME_PARAM riak_kv_backup$NAME_HOST -setcookie $COOKIE \ 153 | -eval "riak_kv_backup:$ACTION('$NODE', \"$FILENAME\")" -s init stop 154 | ;; 155 | 156 | backup) 157 | ACTION=$1 158 | shift 159 | 160 | if [ $# -lt 4 ]; then 161 | echo "Usage: $SCRIPT $ACTION [node|all]" 162 | exit 1 163 | fi 164 | 165 | NODE=$1 166 | COOKIE=$2 167 | FILENAME=$3 168 | TYPE=$4 169 | 170 | $ERTS_PATH/erl -noshell $NAME_PARAM riak_kv_backup$NAME_HOST -setcookie $COOKIE \ 171 | -eval "riak_kv_backup:$ACTION('$NODE', \"$FILENAME\", \"$TYPE\")" -s init stop 172 | ;; 173 | 174 | test) 175 | # Make sure the local node IS running 176 | RES=`$NODETOOL ping` 177 | if [ "$RES" != "pong" ]; then 178 | echo "Node is not running!" 179 | exit 1 180 | fi 181 | 182 | shift 183 | 184 | # Parse out the node name to pass to the client 185 | NODE_NAME=${NAME_ARG#* } 186 | 187 | $ERTS_PATH/erl -noshell $NAME_PARAM riak_test$NAME_HOST $COOKIE_ARG \ 188 | -eval "riak:client_test(\"$NODE_NAME\")" -s init stop 189 | 190 | ;; 191 | *) 192 | echo "Usage: $SCRIPT { join | leave | backup | restore | test | status | reip | js_reload }" 193 | exit 1 194 | ;; 195 | esac 196 | -------------------------------------------------------------------------------- /doc/basic-mapreduce.txt: -------------------------------------------------------------------------------- 1 | Introduction to Map/Reduce on Riak 2 | ------ 3 | 4 | This document describes Riak's implementation of a data processing 5 | system based on the MapReduce[1] programming paradigm popularized by 6 | Google. It assumes that you have already set up Riak, and know the 7 | basics about dealing with Riak clients. For more information on these 8 | prerequisites, see riak/doc/basic-setup.txt and 9 | riak/doc/basic-client.txt. 10 | 11 | Quick and Dirty Example 12 | --- 13 | 14 | If you have a Riak client hanging around, you can execute Map/Reduce 15 | queries on it like this: 16 | 17 | 1> Count = fun(G, undefined, none) -> 18 | [dict:from_list([{I, 1} || I <- riak_object:get_value(G)])] 19 | end. 20 | 2> Merge = fun(Gcounts, none) -> 21 | [lists:foldl(fun(G, Acc) -> 22 | dict:merge(fun(_, X, Y) -> X+Y end, 23 | G, Acc) 24 | end, 25 | dict:new(), 26 | Gcounts)] 27 | end. 28 | 3> {ok, [R]} = Client:mapred([{<<"groceries">>, <<"mine">>}, 29 | {<<"groceries">>, <<"yours">>}], 30 | [{map, {qfun, Count}, none, false}, 31 | {reduce, {qfun, Merge}, none, true}]). 32 | 4> L = dict:to_list(R). 33 | 34 | If the "mine" and "yours" objects in the groceries bucket had values 35 | of ["bread", "cheese"], ["bread", "butter"], the sequence of commands 36 | above would result in L being bound to 37 | [{"bread", 2},{"cheese",1},{"butter",1}]. 38 | 39 | 40 | Details 41 | --- 42 | 43 | More importantly, riak_client:mapred takes two lists as arguments. 44 | The first list contains bucket key pairs, which are the keys to the 45 | "starting values" of the Map/Reduce query. The second list are the 46 | steps of the query. 47 | 48 | 49 | Map Steps 50 | --- 51 | 52 | Map steps expect as input a list of bucket/key pairs, just like the 53 | first argument to the riak_client:mapred function. Riak executes a 54 | map step by looking up values for keys in the input list and executing 55 | the map function referenced in the step. 56 | 57 | Map steps take the form: 58 | 59 | {map, FunTerm, Arg, Accumulate} 60 | 61 | Where: 62 | 63 | FunTerm is a reference to the function that will compute the map of 64 | each value. A function referenced by a FunTerm must be arity-3, 65 | accepting the arguments: 66 | 67 | Value: the value found at a key. This will be a Riak object 68 | (defined by the riak_object module) if a value was found, or the 69 | tuple {error, notfound} if a bucket/key was put in the input 70 | list, but not found in the Riak cluster. 71 | 72 | Data: An optional piece of data attached to the bucket/key tuple. 73 | If instead of {Bucket, Key}, {{Bucket, Key}, Data} is passed as 74 | input to a map step, that Data will be passed to the map 75 | function in this argument. Data will be the atom 'undefined' if 76 | the former form is used. 77 | 78 | Arg: The Arg from the map step definition. The same Arg is passed 79 | to every execution of the map function in this step. 80 | 81 | Functions may be referenced in two ways: 82 | 83 | {modfun, Module, Function} where Module and Function are atoms 84 | that name an Erlang function in a specific module 85 | 86 | {qfun, Function} where Function is a callable fun term 87 | 88 | The function must return a *list* of values. The lists returned 89 | by all executions of the map function for this step will be 90 | appended and passed to the next step. 91 | 92 | Arg: The third argument passed to the function referenced in FunTerm. 93 | 94 | Accumulate: If true, the output of this map step will be included in 95 | the final return of the mapred function. If false, the output will 96 | be discarded after the next step. 97 | 98 | 99 | Reduce Steps 100 | --- 101 | 102 | A reduce step takes the form: 103 | 104 | {reduce, FunTerm, Arg, Acc} 105 | 106 | Where FunTerm, Arg, and Acc are mostly the same as their definition 107 | for a map step, but the function referenced in FunTerm is instead of 108 | arity 2. Its parameters are: 109 | 110 | ValueList: The list of values produce by the preceeding step of the 111 | Map/Reduce. 112 | 113 | Arg: The Arg from the step definition. 114 | 115 | The function should again produce a list of values, but it must also 116 | be true that the function is commutative, associative, and 117 | idempotent. That is, if the input list [a,b,c,d] is valid for a given 118 | F, then all of the following must produce the same result: 119 | 120 | F([a,b,c,d]) 121 | F([a,d] ++ F([c,b])) 122 | F([F([a]),F([c]),F([b]),F([d])]) 123 | 124 | 125 | Where does the code run? 126 | --- 127 | 128 | So, all well and good, but you could code the same abstraction in a 129 | couple of hours, right? Just fetch each object and run your function. 130 | 131 | Well, not so fast. This map/reduce isn't just an abstraction, it 132 | fully exploits data locality. That is to say, both map and reduce 133 | functions run on Riak nodes. Map nodes are even run on the node where 134 | the data is already located. 135 | 136 | This means a few things to you: 137 | 138 | - If you use the {modfun, Module, Function} form of the FunTerm in the 139 | map/reduce step definition, that Module must be in the code path of 140 | the Riak node. This isn't a huge concern for libraries that ship 141 | with Erlang, but for any of your custom code, you'll need to make 142 | sure it's loadable. 143 | 144 | - If you use the {modfun, Module, Function} form of the FunTerm in the 145 | map/reduce step definition, you'll need to force the Riak nodes to 146 | reload the Module if you make a change to it. 147 | 148 | The easiest way to reload a module on a Riak node is to get a Riak 149 | client, then call Client:reload_all(Module). 150 | 151 | - If you need to do a Riak 'get' inside of a map or reduce function, 152 | you can use riak:local_client/0 to get a Riak client instead of 153 | riak:client_connect/1. 154 | 155 | - Your map and reduce functions are running on a Riak node, which 156 | means that that Riak node is spending CPU time doing something other 157 | than responding to 'get' and 'put' requests. 158 | 159 | - If you use the {qfun, Fun} form, your callable function, and its 160 | environment will be shipped to the Riak cluster, and each node on 161 | which it runs. This is both a benefit (in that you have the full 162 | power of closures) and a danger (in that you must be mindful of 163 | closing over very large data structures). 164 | 165 | [1] http://labs.google.com/papers/mapreduce.html 166 | -------------------------------------------------------------------------------- /doc/architecture.txt: -------------------------------------------------------------------------------- 1 | Riak Architecture Overview 2 | ------ 3 | 4 | This document describes the design of Riak. It is meant to give both 5 | a description of the shape of the system and also some background in 6 | the terminology used throughout. 7 | 8 | Summary and High-level View 9 | --- 10 | 11 | Riak is a distributed key-value store, strongly influenced by the 12 | Dynamo Paper[1] and the CAP Theorem[2]. It supports high availability 13 | by allowing tunable levels of guarantees for durability and eventual 14 | consistency. 15 | 16 | A Riak cluster is generally run on a set of well-connected physical 17 | hosts. Each host in the cluster runs one Riak node. Each Riak node 18 | runs a set of virtual nodes, or "vnodes", that are each responsible 19 | for storing a separate portion of the key space. 20 | 21 | Nodes are not clones of each other, nor do they all participate in 22 | fulfilling every request. The extent to which data is replicated, and 23 | when, and with what merge strategy and failure model, is configurable 24 | at runtime. 25 | 26 | 27 | The Ring 28 | --- 29 | 30 | (Much of this section is discussed in the Dynamo paper, but it's a 31 | good summary of how Riak implements the necessities.) 32 | 33 | Riak's client interface speaks of "buckets" and "keys". Internally, 34 | Riak computes a 160-bit binary hash of the bucket/key pair, and maps 35 | this value to a position on an ordered "ring" of all such values. 36 | This ring is divided into partitions. Each Riak vnode is responsible 37 | for a partition (we say that it "claims" that partition). 38 | 39 | The nodes of a Riak cluster each attempt to run roughly an equal 40 | number of vnodes. In the general case, this means that each node in 41 | the cluster is responsible for 1/(number of nodes) of the ring, or 42 | (number of partitions)/(number of nodes) vnodes. For example, if two 43 | nodes define a 16-partition cluster, then each node will run 8 vnodes. 44 | Nodes claim their partitions at random intervals around the ring, in 45 | an attempt at even distribution. 46 | 47 | When a value is being stored in the cluster, any node may participate 48 | as the coordinator for the request. The coordinating node consults 49 | the ring state to determine which vnode owns the partition in which 50 | the value's key belongs, then sends the "put" request to that vnode, 51 | as well as the vnodes responsible for the next N-1 partitions in the 52 | ring, where N is a bucket-configurable parameter that describes how 53 | many copies of the value to store. The put request may also specify 54 | that at least W (=< N) of those vnodes reply with success, and that DW 55 | (=< W) reply with success only after durably storing the value. 56 | 57 | A fetch, or "get", request operates similarly, sending requests to the 58 | vnode that "claims" the partition in which the key resides, as well as 59 | to the next N-1 partitions. The request also specifies R (=< N), the 60 | number of vnodes that must reply before a response is returned. 61 | 62 | The riak node startup are defined by the riak_core_sup and riak_kv_sup 63 | modules. 64 | 65 | The riak ring is defined by the riak_core_ring module. 66 | 67 | Vnodes are defined by the riak_kv_vnode module. 68 | 69 | Gets and Puts are driven by the riak_kv_get_fsm and riak_kv_put_fsm 70 | modules, respectively. 71 | 72 | 73 | Gossiping 74 | --- 75 | 76 | The ring state is shared around the cluster by means of a "gossip 77 | protocol". Whenever a node changes its claim on the ring, it 78 | announces its change via this protocol. It also periodically 79 | re-announces what it knows about the ring, in case any nodes missed 80 | previous updates. 81 | 82 | This gossip protocol is defined in the riak_core_gossip and 83 | riak_core_ring_manager modules. 84 | 85 | 86 | Vclocks 87 | --- 88 | 89 | With any node able to drive any request, and not all nodes needing to 90 | participate in each request, it is necessary to have a method for 91 | keeping track of which version of a value is current. This is where 92 | vclocks come in. The vclocks used in Riak are based on the work 93 | of Leslie Lamport.[3] 94 | 95 | When a value is stored in Riak, it is tagged with a vclock, 96 | establishing its initial version. For each update, the vclock is 97 | extended in such a way that Riak can later compare to versions of the 98 | object and determine: 99 | 100 | 1. Whether one object is a direct descendant of the other. 101 | 2. Whether the objects are direct descendants of a common parent. 102 | 3. Whether the objects are unrelated in recent heritage. 103 | 104 | Using this knowledge, Riak can possibly auto-repair out-of-sync data, 105 | or at least provide a client with an opportunity to reconcile 106 | divergent changesets in an application specific manner. 107 | 108 | Riak's vclock usage is defined by the vclock module in the riak source 109 | directory. 110 | 111 | Riak attempts to move data toward a consistent state across nodes, 112 | but it doesn't do so by comparing each and every object on each node. 113 | Instead, nodes gossip a "merkle tree"[4], which allows them to quickly 114 | decide which values need comparing. 115 | 116 | Riak's merkle trees are defined by the merkerl module in the riak source 117 | directory. 118 | 119 | Backends 120 | --- 121 | 122 | Sharing data among nodes, on rings, etc. is all well and good, but at 123 | some point, it has to actually be stored somewhere - like on disk! 124 | Because Riak is relevant to a wide variety of applications, its 125 | "backend" storage system is a pluggable one. 126 | 127 | Each node may be configured with a different Erlang module for doing 128 | the simple storage, at the vnode level, below all of the 129 | interconnected cluster details. At the backend level, a module only 130 | needs to define "get", "put", "delete", and "keys list" functions that 131 | receive a key and value which are both Erlang binaries. The backend 132 | can consider these binaries completely opaque data, or examine them to 133 | make decisions about how best to store them. 134 | 135 | Several backends are packaged with Riak: 136 | 137 | 1. riak_kv_bitcask_backend, which stores data in the Bitcask key/value 138 | store. 139 | 2. riak_kv_cache_backend, which behaves as an LRU-with-timed-expiry cache 140 | 3. riak_kv_dets_backend, which stores data on-disk in DETS tables 141 | 4. riak_kv_ets_backend, which stores data in ETS tables (which makes it 142 | volatile storage, but great for debugging) 143 | 5. riak_kv_fs_backend, which stores data directly to files in a nested 144 | directory structure on disk 145 | 6. riak_kv_gb_trees_backend, which stores data in Erlang gb_trees. 146 | 147 | [1] http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf 148 | [2] http://portal.acm.org/citation.cfm?doid=564585.564601 149 | [3] http://portal.acm.org/citation.cfm?id=359563 150 | [4] http://portal.acm.org/citation.cfm?id=704751 151 | -------------------------------------------------------------------------------- /doc/basic-setup.txt: -------------------------------------------------------------------------------- 1 | Riak Setup Instructions 2 | ------ 3 | 4 | This document explains how to set up a Riak cluster. It assumes that 5 | you have already downloaded and successfully built Riak. For help with 6 | those steps, please refer to riak/README. 7 | 8 | 9 | Overview 10 | --- 11 | 12 | Riak has many knobs to tweak, affecting everything from distribution 13 | to disk storage. This document will attempt to give a description of 14 | the common configuration parameters, then describe two typical setups, 15 | one small, one large. 16 | 17 | 18 | Configuration 19 | --- 20 | 21 | Configurations are stored in the simple text files vm.args and 22 | app.config. Initial versions of these files are stored in the 23 | rel/files/ subdirectory of the riak source tree. When a release 24 | is generated, these files are copied to rel/riak/etc/. 25 | 26 | vm.args 27 | --- 28 | 29 | The vm.args configuration file sets the parameters passed on the 30 | command line to the Erlang VM. Lines starting with a '#' are 31 | comments, and are ignored. The other lines are concatenated and 32 | passed on the command line verbatim. 33 | 34 | Two important parameters to configure here are "-name", the name to 35 | give the Erlang node running Riak, and "-setcookie", the cookie that 36 | all Riak nodes need to share in order to communicate. 37 | 38 | app.config 39 | --- 40 | 41 | The app.config configuration file is formatted as an Erlang VM config 42 | file. The syntax is simply: 43 | 44 | [ 45 | {AppName, [ 46 | {Option1, Value1}, 47 | {Option2, Value2}, 48 | ... 49 | ]}, 50 | ... 51 | ]. 52 | 53 | Normally, this will look something like: 54 | 55 | [ 56 | {riak_kv, [ 57 | {storage_backend, riak_kv_dets_backend}, 58 | {riak_kv_dets_backend_root, "data/dets"} 59 | ]}, 60 | {sasl, [ 61 | {sasl_error_logger, {file, "log/sasl-error.log"}} 62 | ]} 63 | ]. 64 | 65 | This would set the 'storage_backend' and 'riak_kv_dets_backend_root' 66 | options for the 'riak_kv' application, and the 'sasl_error_logger' option 67 | for the 'sasl' application. 68 | 69 | The following parameters can be used in app.config to configure Riak 70 | behavior. Some of the terminology used below is better explained in 71 | riak/doc/architecture.txt. 72 | 73 | cluster_name: string 74 | The name of the cluster. Can be anything. Used mainly in saving 75 | ring configuration. All nodes should have the same cluster name. 76 | 77 | gossip_interval: integer 78 | The period, in milliseconds, at which ring state gossiping will 79 | happen. A good default is 60000 (sixty seconds). Best not to 80 | change it unless you know what you're doing. 81 | 82 | ring_creation_size: integer 83 | The number of partitions to divide the keyspace into. This can be 84 | any number, but you probably don't want to go lower than 16, and 85 | production deployments will probably want something like 1024 or 86 | greater. This is a very difficult parameter to change after your 87 | ring has been created, so choose a number that allows for growth, if 88 | you expect to add nodes to this cluster in the future. 89 | 90 | ring_state_dir: string 91 | Directory in which the ring state should be stored. Ring state is 92 | stored to allow an entire cluster to be restarted. 93 | 94 | storage_backend: atom 95 | 96 | Name of the module that implements the storage for a vnode. The 97 | backends that ship with Riak are riak_kv_bitcask_backend, 98 | riak_kv_cache_backend, riak_dets_backend, riak_ets_backend, 99 | riak_fs_backend, and riak_kv_gb_trees_backend. Some backends have 100 | their own set of configuration parameters. 101 | 102 | riak_kv_bitcask_backend: 103 | Stores data in the Bitcask key/value store. 104 | 105 | riak_kv_cache_backend: 106 | Behaves as an LRU-with-timed-expiry cache. 107 | 108 | riak_kv_dets_backend: 109 | A backend that uses DETS to store its data. 110 | 111 | riak_kv_dets_backend_root: string 112 | The directory under which this backend will store its files. 113 | 114 | riak_kv_ets_backend: 115 | A backend that uses ETS to store its data. 116 | 117 | riak_kv_fs_backend: 118 | A backend that uses the filesystem directly to store data. Data 119 | are stored in Erlang binary format in files in a directory 120 | structure on disk. 121 | 122 | riak_fs_backend_root: string 123 | The directory under which this backend will store its files. 124 | 125 | riak_kv_gb_trees_backend: 126 | A backend that uses Erlang gb_trees to store data. 127 | 128 | 129 | Single-node Configuration 130 | --- 131 | 132 | If you're running a single Riak node, you likely don't need to change 133 | any configuration at all. After compiling and generating the release 134 | ("./rebar compile generate"), simply start Riak from the rel/ 135 | directory. (Details about the "riak" control script can be found in 136 | the README.) 137 | 138 | 139 | Large (Production) Configuration 140 | --- 141 | 142 | If you're running any sort of cluster that could be labeled 143 | "production", "deployment", "scalable", "enterprise", or any other 144 | word implying that the cluster will be running interminably with 145 | on-going maintenance, then you will want to change configurations a 146 | bit. Some recommended changes: 147 | 148 | * Uncomment the "-heart" line in vm.args. This will cause the "heart" 149 | utility to monitor the Riak node, and restart it if it stops. 150 | 151 | * Change the name of the Riak node in vm.args from riak@127.0.0.1 to 152 | riak@VISIBLE.HOSTNAME. This will allow Riak nodes on separate machines 153 | to communicate. 154 | 155 | * Change 'web_ip' in the 'riak_core' section of app.config if you'll be 156 | accessing that interface from a non-host-local client. 157 | 158 | * Consider adding a 'ring_creation_size' entry to app.config, and 159 | setting it to a number higher than the default of 64. More 160 | partitions will allow you to add more Riak nodes later, if you need 161 | to. 162 | 163 | To get the cluster, up and running, first start Riak on each node with 164 | the usual "riak start" command. Next, tell each node to join the 165 | cluster with the riak-admin script: 166 | 167 | box2$ bin/riak-admin join riak@box1.example.com 168 | Sent join request to riak@box1.example.com 169 | 170 | To check that all nodes have joined the cluster, attach a console to 171 | any node, and request the ring from the ring manager, then check that 172 | all nodes are represented: 173 | 174 | $ bin/riak attach 175 | Attaching to /tmp/erlang.pipe.1 (^D to exit) 176 | (riak@box1.example.com)1> {ok, R} = riak_core_ring_manager:get_my_ring(). 177 | {ok,{chstate,'riak@box1.example.com', 178 | ...snip... 179 | (riak@box1.example.com)2> riak_core_ring:all_members(R). 180 | ['riak@box1.example.com','riak@box2.example.com'] 181 | 182 | 183 | Your cluster should now be ready to accept requests. See 184 | riak/doc/basic-client.txt for simple instructions on connecting, 185 | storing and fetching data. 186 | 187 | Starting more nodes in production is just as easy: 188 | 189 | 1. Install Riak on another host, modifying hostnames in configuration 190 | files, if necessary. 191 | 2. Start the node with "riak start" 192 | 3. Add the node to the cluster with 193 | "riak-admin join ExistingClusterNode" 194 | 195 | 196 | Developer Configuration 197 | --- 198 | 199 | If you're hacking on Riak, and you need to run multiple nodes on a 200 | single physical machine, use the "devrel" make command: 201 | 202 | $ make devrel 203 | mkdir -p dev 204 | (cd rel && ../rebar generate target_dir=../dev/dev1 overlay_vars=vars/dev1_vars.config) 205 | ==> rel (generate) 206 | mkdir -p dev 207 | (cd rel && ../rebar generate target_dir=../dev/dev2 overlay_vars=vars/dev2_vars.config) 208 | ==> rel (generate) 209 | mkdir -p dev 210 | (cd rel && ../rebar generate target_dir=../dev/dev3 overlay_vars=vars/dev3_vars.config) 211 | ==> rel (generate) 212 | 213 | This make target creates a release, and then modifies configuration 214 | files such that each Riak node uses a different Erlang node name 215 | (dev1-3), web port (8091-3), data directory (dev/dev1-3/data/), etc. 216 | 217 | Start each developer node just as you would a regular node, but use 218 | the 'riak' script in dev/devN/bin/. 219 | 220 | 221 | -------------------------------------------------------------------------------- /README.org: -------------------------------------------------------------------------------- 1 | #+SETUPFILE: "doc/basho-doc-style.iorg" 2 | 3 | Welcome to Riak. 4 | 5 | * Overview 6 | Riak is a distributed, decentralized data storage system. 7 | 8 | Below, you will find the "quick start" directions for setting up and 9 | using Riak. For more information, browse the following files: 10 | 11 | * README: this file 12 | * TODO: a list of improvements planned for Riak 13 | * LICENSE: the license under which Riak is released 14 | * apps/ the source tree for Riak and all its dependencies 15 | * doc/ 16 | - admin.org: Riak Administration Guide 17 | - architecture.txt: details about the underlying design of Riak 18 | - basic-client.txt: slightly more detail on using Riak 19 | - basic-setup.txt: slightly more detail on setting up Riak 20 | - basic-mapreduce.txt: introduction to map/reduce on Riak 21 | - js-mapreduce.org: using Javascript with Riak map/reduce 22 | - man/riak.1.gz: manual page for the riak(1) command 23 | - man/riak-admin.1.gz manual page for the riak-admin(1) command 24 | - raw-http-howto.txt: using the Riak HTTP interface 25 | 26 | 27 | 28 | * Quick Start 29 | 30 | This section assumes that you have copy of the Riak source tree. To get 31 | started, you need to: 32 | 1. Build Riak 33 | 2. Start the Riak server 34 | 3. Connect a client and store/fetch data 35 | 36 | ** Building Riak 37 | 38 | Assuming you have a working Erlang (R13B04 or later) installation, 39 | building Riak should be as simple as: 40 | 41 | #+BEGIN_EXAMPLE 42 | $ cd $RIAK 43 | $ make rel 44 | #+END_EXAMPLE 45 | 46 | ** Starting Riak 47 | 48 | Once you have successfully built Riak, you can start the server with the 49 | following commands: 50 | 51 | #+BEGIN_EXAMPLE 52 | $ cd $RIAK/rel/riak 53 | $ bin/riak start 54 | #+END_EXAMPLE 55 | 56 | Now, verify that the server started up cleanly and is working: 57 | 58 | : $ bin/riak-admin test 59 | 60 | Note that the $RIAK/rel/riak directory is a complete, self-contained instance 61 | of Riak and Erlang. It is strongly suggested that you move this directory 62 | outside the source tree if you plan to run a production instance. 63 | 64 | ** Connecting a client to Riak 65 | 66 | Now that you have a functional server, let's try storing some data in 67 | it. First, start up a erlang node using our embedded version of erlang: 68 | 69 | #+BEGIN_EXAMPLE 70 | $ erts-/bin/erl -name riaktest@127.0.0.1 -setcookie riak 71 | 72 | Eshell V5.7.4 (abort with ^G) 73 | (riaktest@127.0.0.1)1> 74 | #+END_EXAMPLE 75 | 76 | Now construct the node name of Riak server and make sure we can talk to it: 77 | 78 | #+BEGIN_EXAMPLE 79 | (riaktest@127.0.0.1)4> RiakNode = 'riak@127.0.0.1'. 80 | 81 | (riaktest@127.0.0.1)2> net_adm:ping(RiakNode). 82 | pong 83 | (riaktest@127.0.0.1)2> 84 | #+END_EXAMPLE 85 | 86 | We are now ready to start the Riak client: 87 | 88 | #+BEGIN_EXAMPLE 89 | (riaktest@127.0.0.1)2> {ok, C} = riak:client_connect(RiakNode). 90 | {ok,{riak_client,'riak@127.0.0.1',<<4,136,81,151>>}} 91 | #+END_EXAMPLE 92 | 93 | Let's create a shopping list for bread at /groceries/mine: 94 | 95 | #+BEGIN_EXAMPLE 96 | (riaktest@127.0.0.1)6> O0 = riak_object:new(<<"groceries">>, <<"mine">>, ["bread"]). 97 | O0 = riak_object:new(<<"groceries">>, <<"mine">>, ["bread"]). 98 | {r_object,<<"groceries">>,<<"mine">>, 99 | [{r_content,{dict,0,16,16,8,80,48, 100 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 101 | {{[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 102 | ["bread"]}], 103 | [], 104 | {dict,1,16,16,8,80,48, 105 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 106 | {{[],[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 107 | undefined} 108 | 109 | (riaktest@127.0.0.1)3> C:put(O0, 1). 110 | #+END_EXAMPLE 111 | 112 | Now, read the list back from the Riak server and extract the value 113 | 114 | #+BEGIN_EXAMPLE 115 | (riaktest@127.0.0.1)4> {ok, O1} = C:get(<<"groceries">>, <<"mine">>, 1). 116 | {ok,{r_object,<<"groceries">>,<<"mine">>, 117 | [{r_content,{dict,2,16,16,8,80,48, 118 | {[],[],[],[],[],[],[],[],[],[],[],[],...}, 119 | {{[],[],[],[],[],[], 120 | [["X-Riak-Last-Modified",87|...]], 121 | [],[],[],...}}}, 122 | ["bread"]}], 123 | [{"20090722191020-riaktest@127.0.0.1-riakdemo@127.0.0.1-266664", 124 | {1,63415509105}}], 125 | {dict,0,16,16,8,80,48, 126 | {[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 127 | {{[],[],[],[],[],[],[],[],[],[],[],...}}}, 128 | undefined}} 129 | 130 | (riaktest@127.0.0.1)5> %% extract the value 131 | (riaktest@127.0.0.1)5> V = riak_object:get_value(O1). 132 | ["bread"] 133 | #+END_EXAMPLE 134 | 135 | Add milk to our list of groceries and write the new value to Riak: 136 | 137 | #+BEGIN_EXAMPLE 138 | (riaktest@127.0.0.1)6> %% add milk to the list 139 | (riaktest@127.0.0.1)6> O2 = riak_object:update_value(O1, ["milk" | V]). 140 | {r_object,<<"groceries">>,<<"mine">>, 141 | [{r_content,{dict,2,16,16,8,80,48, 142 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 143 | {{[],[],[],[],[],[], 144 | [["X-Riak-Last-Modified",87,101,100|...]], 145 | [],[],[],[],[],...}}}, 146 | ["bread"]}], 147 | [{"20090722191020-riaktest@127.0.0.1-riakdemo@127.0.0.1-266664", 148 | {1,63415509105}}], 149 | {dict,0,16,16,8,80,48, 150 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 151 | {{[],[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 152 | ["milk","bread"]} 153 | 154 | (riaktest@127.0.0.1)7> %% store the new list 155 | (riaktest@127.0.0.1)7> C:put(O2, 1). 156 | ok 157 | #+END_EXAMPLE 158 | 159 | Finally, see what other keys are available in groceries bucket: 160 | 161 | #+BEGIN_EXAMPLE 162 | (riaktest@127.0.0.1)8> C:list_keys(<<"groceries">>). 163 | {ok,[<<"mine">>]} 164 | #+END_EXAMPLE 165 | 166 | ** Clients for Other Languages 167 | 168 | Client libraries are available for many languages. Rather than 169 | bundle them with the Riak server source code, we have given them 170 | each their own source repository. Currently, official Riak 171 | client language libraries include: 172 | 173 | + Javascript 174 | http://bitbucket.org/basho/riak-javascript-client 175 | 176 | + Python 177 | http://bitbucket.org/basho/riak-python-client 178 | 179 | + Ruby 180 | http://bitbucket.org/basho/riak-ruby-client 181 | http://github.com/seancribbs/ripple/ 182 | 183 | + Java 184 | http://bitbucket.org/basho/riak-java-client 185 | 186 | + PHP 187 | http://bitbucket.org/basho/riak-php-client 188 | 189 | + Erlang 190 | http://bitbucket.org/basho/riak-erlang-client 191 | (using protocol buffers instead of distributed Erlang) 192 | 193 | * Server Management 194 | 195 | ** Configuration 196 | Configuration for the Riak server is stored in $RIAK/rel/riak/etc 197 | directory. There are two files: 198 | - vm.args 199 | This file contains the arguments that are passed to the Erlang VM 200 | in which Riak runs. The default settings in this file shouldn't need to be 201 | changed for most environments. 202 | 203 | - app.config 204 | This file contains the configuration for the Erlang applications 205 | that run on the Riak server. 206 | 207 | More information about this files is available in doc/basic-setup.txt. 208 | 209 | ** Server Control 210 | *** bin/riak 211 | This script is the primary interface for starting and stopping the Riak 212 | server. 213 | 214 | To start a daemonized (background) instance of Riak: 215 | 216 | : $ bin/riak start 217 | 218 | Once a server is running in the background you can attach to the Erlang 219 | console via: 220 | 221 | : $ bin/riak attach 222 | 223 | Alternatively, if you want to run a foreground instance of Riak, start it 224 | with: 225 | 226 | : $ bin/riak console 227 | 228 | Stopping a foreground or background instance of Riak can be done from a 229 | shell prompt via: 230 | 231 | : $ bin/riak stop 232 | 233 | Or if you are attached/on the Erlang console: 234 | 235 | : (riak@127.0.0.1)1> q(). 236 | 237 | You can determine if the server is running by: 238 | 239 | : $ bin/riak ping 240 | 241 | *** bin/riak-admin 242 | This script provides access to general administration of the Riak server. 243 | The below commands assume you are running a default configuration for 244 | parameters such as cookie. 245 | 246 | To join a new Riak node to an existing cluster: 247 | 248 | #+BEGIN_EXAMPLE 249 | $ bin/riak start # If a local server is not already running 250 | $ bin/riak-admin join 251 | #+END_EXAMPLE 252 | 253 | (Note that you must have a local node already running for this to work) 254 | 255 | To verify that the local Riak node is able to read/write data: 256 | : $ bin/riak-admin test 257 | 258 | To backup a node or cluster run the following: 259 | : $ bin/riak-admin backup riak@X.X.X.X riak node 260 | : $ bin/riak-admin backup riak@X.X.X.X riak all 261 | 262 | Restores can function in two ways, if the backup file was of a node the 263 | node will be restored and if the backup file contains the data for a 264 | cluster all nodes in the cluster will be restored. 265 | 266 | To restore from a backup file: 267 | : $ riak-admin restore riak@X.X.X.X riak 268 | 269 | To view the status of a node: 270 | : $ bin/riak-admin status 271 | 272 | If you change the IP or node name you will need to use the reip command: 273 | : $ bin/riak-admin reip 274 | 275 | 276 | 277 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | Welcome to Riak. 2 | ================ 3 | 4 | Date: 2010-06-09 10:06:44 CDT 5 | 6 | 7 | 8 | Table of Contents 9 | ================= 10 | 1 Overview 11 | 2 Quick Start 12 | 2.1 Building Riak 13 | 2.2 Starting Riak 14 | 2.3 Connecting a client to Riak 15 | 2.4 Clients for Other Languages 16 | 3 Server Management 17 | 3.1 Configuration 18 | 3.2 Server Control 19 | 3.2.1 bin/riak 20 | 3.2.2 bin/riak-admin 21 | 22 | 23 | 1 Overview 24 | ~~~~~~~~~~~ 25 | Riak is a distributed, decentralized data storage system. 26 | 27 | Below, you will find the "quick start" directions for setting up and 28 | using Riak. For more information, browse the following files: 29 | 30 | * README: this file 31 | * TODO: a list of improvements planned for Riak 32 | * LICENSE: the license under which Riak is released 33 | * apps/ the source tree for Riak and all its dependencies 34 | * doc/ 35 | - admin.org: Riak Administration Guide 36 | - architecture.txt: details about the underlying design of Riak 37 | - basic-client.txt: slightly more detail on using Riak 38 | - basic-setup.txt: slightly more detail on setting up Riak 39 | - basic-mapreduce.txt: introduction to map/reduce on Riak 40 | - js-mapreduce.org: using Javascript with Riak map/reduce 41 | - man/riak.1.gz: manual page for the riak(1) command 42 | - man/riak-admin.1.gz manual page for the riak-admin(1) command 43 | - raw-http-howto.txt: using the Riak HTTP interface 44 | 45 | 46 | 47 | 2 Quick Start 48 | ~~~~~~~~~~~~~~ 49 | 50 | This section assumes that you have copy of the Riak source tree. To get 51 | started, you need to: 52 | 1. Build Riak 53 | 2. Start the Riak server 54 | 3. Connect a client and store/fetch data 55 | 56 | 2.1 Building Riak 57 | ================== 58 | 59 | Assuming you have a working Erlang (R13B04 or later) installation, 60 | building Riak should be as simple as: 61 | 62 | 63 | $ cd $RIAK 64 | $ make rel 65 | 66 | 2.2 Starting Riak 67 | ================== 68 | 69 | Once you have successfully built Riak, you can start the server with the 70 | following commands: 71 | 72 | 73 | $ cd $RIAK/rel/riak 74 | $ bin/riak start 75 | 76 | Now, verify that the server started up cleanly and is working: 77 | 78 | $ bin/riak-admin test 79 | 80 | Note that the $RIAK/rel/riak directory is a complete, self-contained instance 81 | of Riak and Erlang. It is strongly suggested that you move this directory 82 | outside the source tree if you plan to run a production instance. 83 | 84 | 2.3 Connecting a client to Riak 85 | ================================ 86 | 87 | Now that you have a functional server, let's try storing some data in 88 | it. First, start up a erlang node using our embedded version of erlang: 89 | 90 | 91 | $ erts-/bin/erl -name riaktest@127.0.0.1 -setcookie riak 92 | 93 | Eshell V5.7.4 (abort with ^G) 94 | (riaktest@127.0.0.1)1> 95 | 96 | Now construct the node name of Riak server and make sure we can talk to it: 97 | 98 | 99 | (riaktest@127.0.0.1)4> RiakNode = 'riak@127.0.0.1'. 100 | 101 | (riaktest@127.0.0.1)2> net_adm:ping(RiakNode). 102 | pong 103 | (riaktest@127.0.0.1)2> 104 | 105 | We are now ready to start the Riak client: 106 | 107 | 108 | (riaktest@127.0.0.1)2> {ok, C} = riak:client_connect(RiakNode). 109 | {ok,{riak_client,'riak@127.0.0.1',<<4,136,81,151>>}} 110 | 111 | Let's create a shopping list for bread at /groceries/mine: 112 | 113 | 114 | (riaktest@127.0.0.1)6> O0 = riak_object:new(<<"groceries">>, <<"mine">>, ["bread"]). 115 | O0 = riak_object:new(<<"groceries">>, <<"mine">>, ["bread"]). 116 | {r_object,<<"groceries">>,<<"mine">>, 117 | [{r_content,{dict,0,16,16,8,80,48, 118 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 119 | {{[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 120 | ["bread"]}], 121 | [], 122 | {dict,1,16,16,8,80,48, 123 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 124 | {{[],[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 125 | undefined} 126 | 127 | (riaktest@127.0.0.1)3> C:put(O0, 1). 128 | 129 | Now, read the list back from the Riak server and extract the value 130 | 131 | 132 | (riaktest@127.0.0.1)4> {ok, O1} = C:get(<<"groceries">>, <<"mine">>, 1). 133 | {ok,{r_object,<<"groceries">>,<<"mine">>, 134 | [{r_content,{dict,2,16,16,8,80,48, 135 | {[],[],[],[],[],[],[],[],[],[],[],[],...}, 136 | {{[],[],[],[],[],[], 137 | ["X-Riak-Last-Modified",87|...], 138 | [],[],[],...}}}, 139 | ["bread"]}], 140 | [{"20090722191020-riaktest@127.0.0.1-riakdemo@127.0.0.1-266664", 141 | {1,63415509105}}], 142 | {dict,0,16,16,8,80,48, 143 | {[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 144 | {{[],[],[],[],[],[],[],[],[],[],[],...}}}, 145 | undefined}} 146 | 147 | (riaktest@127.0.0.1)5> %% extract the value 148 | (riaktest@127.0.0.1)5> V = riak_object:get_value(O1). 149 | ["bread"] 150 | 151 | Add milk to our list of groceries and write the new value to Riak: 152 | 153 | 154 | (riaktest@127.0.0.1)6> %% add milk to the list 155 | (riaktest@127.0.0.1)6> O2 = riak_object:update_value(O1, ["milk" | V]). 156 | {r_object,<<"groceries">>,<<"mine">>, 157 | [{r_content,{dict,2,16,16,8,80,48, 158 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 159 | {{[],[],[],[],[],[], 160 | ["X-Riak-Last-Modified",87,101,100|...], 161 | [],[],[],[],[],...}}}, 162 | ["bread"]}], 163 | [{"20090722191020-riaktest@127.0.0.1-riakdemo@127.0.0.1-266664", 164 | {1,63415509105}}], 165 | {dict,0,16,16,8,80,48, 166 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 167 | {{[],[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 168 | ["milk","bread"]} 169 | 170 | (riaktest@127.0.0.1)7> %% store the new list 171 | (riaktest@127.0.0.1)7> C:put(O2, 1). 172 | ok 173 | 174 | Finally, see what other keys are available in groceries bucket: 175 | 176 | 177 | (riaktest@127.0.0.1)8> C:list_keys(<<"groceries">>). 178 | {ok,[<<"mine">>]} 179 | 180 | 2.4 Clients for Other Languages 181 | ================================ 182 | 183 | Client libraries are available for many languages. Rather than 184 | bundle them with the Riak server source code, we have given them 185 | each their own source repository. Currently, official Riak 186 | client language libraries include: 187 | 188 | + Javascript 189 | [http://bitbucket.org/basho/riak-javascript-client] 190 | 191 | + Python 192 | [http://bitbucket.org/basho/riak-python-client] 193 | 194 | + Ruby 195 | [http://bitbucket.org/basho/riak-ruby-client] 196 | [http://github.com/seancribbs/ripple/] 197 | 198 | + Java 199 | [http://bitbucket.org/basho/riak-java-client] 200 | 201 | + PHP 202 | [http://bitbucket.org/basho/riak-php-client] 203 | 204 | + Erlang 205 | [http://bitbucket.org/basho/riak-erlang-client] 206 | (using protocol buffers instead of distributed Erlang) 207 | 208 | 3 Server Management 209 | ~~~~~~~~~~~~~~~~~~~~ 210 | 211 | 3.1 Configuration 212 | ================== 213 | Configuration for the Riak server is stored in $RIAK/rel/riak/etc 214 | directory. There are two files: 215 | - vm.args 216 | This file contains the arguments that are passed to the Erlang VM 217 | in which Riak runs. The default settings in this file shouldn't need to be 218 | changed for most environments. 219 | 220 | - app.config 221 | This file contains the configuration for the Erlang applications 222 | that run on the Riak server. 223 | 224 | More information about this files is available in doc/basic-setup.txt. 225 | 226 | 3.2 Server Control 227 | =================== 228 | 229 | 3.2.1 bin/riak 230 | --------------- 231 | This script is the primary interface for starting and stopping the Riak 232 | server. 233 | 234 | To start a daemonized (background) instance of Riak: 235 | 236 | $ bin/riak start 237 | 238 | Once a server is running in the background you can attach to the Erlang 239 | console via: 240 | 241 | $ bin/riak attach 242 | 243 | Alternatively, if you want to run a foreground instance of Riak, start it 244 | with: 245 | 246 | $ bin/riak console 247 | 248 | Stopping a foreground or background instance of Riak can be done from a 249 | shell prompt via: 250 | 251 | $ bin/riak stop 252 | 253 | Or if you are attached/on the Erlang console: 254 | 255 | (riak@127.0.0.1)1> q(). 256 | 257 | You can determine if the server is running by: 258 | 259 | $ bin/riak ping 260 | 261 | 3.2.2 bin/riak-admin 262 | --------------------- 263 | This script provides access to general administration of the Riak server. 264 | The below commands assume you are running a default configuration for 265 | parameters such as cookie. 266 | 267 | To join a new Riak node to an existing cluster: 268 | 269 | 270 | $ bin/riak start # If a local server is not already running 271 | $ bin/riak-admin join 272 | 273 | (Note that you must have a local node already running for this to work) 274 | 275 | To verify that the local Riak node is able to read/write data: 276 | $ bin/riak-admin test 277 | 278 | To backup a node or cluster run the following: 279 | $ bin/riak-admin backup riak@X.X.X.X riak node 280 | $ bin/riak-admin backup riak@X.X.X.X riak all 281 | 282 | Restores can function in two ways, if the backup file was of a node the 283 | node will be restored and if the backup file contains the data for a 284 | cluster all nodes in the cluster will be restored. 285 | 286 | To restore from a backup file: 287 | $ riak-admin restore riak@X.X.X.X riak 288 | 289 | To view the status of a node: 290 | $ bin/riak-admin status 291 | 292 | If you change the IP or node name you will need to use the reip command: 293 | $ bin/riak-admin reip 294 | 295 | 296 | 297 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | -------------------------------------------------------------------------------- /TRANSITION.org: -------------------------------------------------------------------------------- 1 | #+OPTIONS: author:nil timestamp:nil 2 | 3 | Migrating to Riak 0.11 from earlier versions of Riak 4 | 5 | * Riak 0.10 to 0.11 6 | 7 | ** Customized configuration, not switching backends 8 | 9 | If you have customized your configuration file (etc/app.config), 10 | and you will be keeping your customized configuration file, your 11 | transition should be as easy as stopping the 0.10 node, installing 12 | the 0.11 binaries, and starting the node back up. The node should 13 | find its ring and data in the same location as before, and will 14 | continue working as if nothing changed. 15 | 16 | ** Default configuration, or switching backends 17 | 18 | If you're using the default Riak configuration file, you'll need to 19 | be aware that the default storage backend has changed from DETS to 20 | Basho's new Bitcask. You'll need to choose one of two upgrade 21 | paths in order to migrate your data from your old storage backend 22 | to the new one (these steps will also work, in general, from any 23 | backend to any other backend). Your options are: backup/restore, 24 | and online rolling upgrade. 25 | 26 | *** Backup/Restore 27 | 28 | The easiest way to migrate from one storage backend to another is 29 | to backup the cluster as it stands, shut it down, switch the 30 | backend (by upgrading or editing the config file), start the 31 | cluster back up, and then restore the data to the cluster. Follow 32 | these steps to do this: 33 | 34 | 1. With a running cluster, use the "riak-admin backup" command to 35 | backup the cluster's data: 36 | 37 | $ riak-admin backup riak@127.0.0.1 riak riak-backup.dat all 38 | 39 | This script will exit when backup is complete. If your cluster 40 | has a lot of data, you can backup each node individually by 41 | switching "all" to "node" in the command line above, and then 42 | running the command once for each node, changing the node name 43 | in the command line. 44 | 45 | 2. Shut down the cluster. 46 | 47 | $ riak stop 48 | 49 | Run this command on each node. 50 | 51 | 3. Upgrade each node by installing the new 0.11 package. If 52 | you're just switching backends (not upgrading), modify the 53 | configuration file in this step. 54 | 55 | 4. Start the cluster 56 | 57 | $ riak start 58 | 59 | Run this command on each node. The nodes will automatically 60 | reconnect to each other. 61 | 62 | 5. Reload the cluster's data 63 | 64 | $ riak-admin restore riak@127.0.0.1 riak riak-backup.dat 65 | 66 | If you used the whole-cluster ("all") backup mode, you're now 67 | finished. If you backed up each node individually, you'll need 68 | to run this command for each of those backup files (note that 69 | they have node names appended to what you provided on the 70 | backup command line). 71 | 72 | Once the restore script exits, all data should be restored to the 73 | cluster. After checking that this is true, you may remove the 74 | data directories for the old storage backend. 75 | 76 | *** Online Rolling Upgrade 77 | 78 | If you need your Riak cluster to remain running during the backend 79 | transition, and you have spare network capacity, you can use 80 | Riak's easy node addition/removal process to switch backends. 81 | 82 | 1. Configure a new node, with the new backend you want to use, and 83 | add it to the cluster. 84 | 85 | newnode$ riak start 86 | newnode$ riak-admin join oldclusternode@10.0.0.100 87 | 88 | 2. Remove one of the old-backend nodes from the cluster. 89 | 90 | oldnode0$ riak-admin leave 91 | 92 | 3. When the old-backend node completes handoff, it will exit (use 93 | 'ps' or similar to watch for it to shut down). Once it has 94 | exited, upgrade to 0.11 (or switch the backend) and restart the 95 | node. 96 | 97 | 4. Repeat steps 2 and 3 for each other old-backend node in the 98 | cluster. 99 | 100 | 5. When all nodes are running on the new backend, shutdown the new 101 | node you set up in step 1 (if you wish, or leave it up if you like). 102 | 103 | Note: You can skip step 1 if you are running a cluster of more 104 | than one node, and your cluster can tolerate (capacity- and 105 | throughput-wise) being temporarily one node smaller. Step 1 is 106 | merely an attempt to add extra capacity to the system before 107 | beging an expensive operation. 108 | 109 | * Riak 0.9 and earlier to 0.11 110 | 111 | If you are upgrading from a Riak version earlier than 0.10, your 112 | only option for upgrade is backup and restore. Please use the 113 | backup/restore method described in the 0.10-to-0.11-transition 114 | section above. 115 | 116 | You will also be interested in the following section, which 117 | discusses migration from 0.9 to 0.10. 118 | 119 | 120 | * Migrating from Riak 0.9.x to 0.10 121 | 122 | * Overview 123 | Riak has undergone significant restructuring in the transition from 124 | version 0.9.x to version 0.10. If you are using the binary builds, 125 | please skip directly to the Configuration and Clients sections. If 126 | you are building from source yourself, please review the whole of 127 | this document. 128 | 129 | NOTE: If the only files you have changed in your Riak source clone 130 | are those underneath the "rel" directory 131 | ("rel/overlay/etc/app.config", for example), the safest way to 132 | update is to make a fresh clone of the Riak repository, and then 133 | skip to the Configuration and Clients sections of this document for 134 | details about migrating your configurations and data. 135 | 136 | * Requirements 137 | 138 | ** Erlang/OTP R13B04 139 | 140 | Riak 0.10 uses new features ("NIFs") provided by the latest 141 | Erlang/OTP release, R13B04. If you are building from source, you 142 | will need this release or a newer one. 143 | 144 | ** Mercurial 145 | 146 | Riak 0.10 has moved several of its components into external 147 | repositories. If you are building from source, you will need 148 | Mercurial installed to allow the Rebar build system to retrieve 149 | code from these external repositories. 150 | 151 | * Dependencies 152 | 153 | ** Mochiweb, Webmachine, Erlang_js 154 | mochiweb, webmachine, and erlang_js are now pulled into the "deps" 155 | subdirectory, instead of being included in the "apps" subdirectory. 156 | If you are pulling 0.10 code into a repository that formerly had 157 | 0.9.x in it, please remove the apps/mochiweb, apps/webmachine, and 158 | apps/erlang_js directories from your source tree. 159 | 160 | There is a chance that your update will also leave an "apps/riak" 161 | directory hanging around. If it does, please remove this directory 162 | (Riak code has moved into the "apps/riak_core" and "apps/riak_kv" 163 | directories). 164 | 165 | ** make deps 166 | The "all" make target (and, by extension, the "rel" target as 167 | well), depend on a new "deps" target, which handles the fetching 168 | the dependencies (mochiweb, webmachine, erlang_js). 169 | 170 | * Source 171 | 172 | ** Core/KV Split 173 | We've drawn a line through 0.9.x Riak, and divided it into two 174 | things, one called "riak_core", and the other called "riak_kv". 175 | 176 | The things that live in riak_core are those that deal with cluster 177 | membership. Ring-claiming and the like. 178 | 179 | The things that live in riak_kv are those that deal with storing 180 | data. Get and Put FSMs, backends, etc. 181 | 182 | ** Clients 183 | We've also moved the clients out of the client_lib subdirectory, 184 | and into their own language-specific repositories on BitBucket. At 185 | http://bitbucket.org/basho/, you should find: 186 | 187 | + riak-python-client 188 | + riak-php-client 189 | + riak-erlang-client 190 | + riak-java-client 191 | + riak-javascript-client 192 | + riak-ruby-client 193 | 194 | * Configuration 195 | 196 | ** app.config 197 | 198 | Splitting the "riak" Erlang application into the "riak_core" and 199 | "riak_kv" Erlang applications means that configuration options for 200 | each component need to move around in etc/app.config. 201 | 202 | Where before etc/app.config would have contained a section like: 203 | 204 | {riak, [ 205 | %% many settings here 206 | ]}, 207 | 208 | Now, etc/app.config should contain two sections like: 209 | 210 | {riak_core, [ 211 | %% core-specific settings 212 | ]}, 213 | {riak_kv, [ 214 | %% kv-specific settings 215 | ]}, 216 | 217 | The list of settings that moved to the riak_core section are: 218 | 219 | + choose_claim_fun 220 | + cluster_name - string, defaults to "default" 221 | + default_bucket_props 222 | + gossip_interval - integer, defaults to 60k msec 223 | + ring_creation_size - integer, defaults to 64 224 | + ring_state_dir - string 225 | + target_n_val - integer, defaults to 3 226 | + wants_claim_fun 227 | + web_ip - string. Used to be "riak_web_ip" 228 | + web_logdir - string. 229 | + web_port - integer. Used to be "riak_web_port" 230 | 231 | IMPORTANT: Note the rename of "riak_web_*" to just "web_*" 232 | 233 | The list of settings that moved to the riak_kv section are: 234 | 235 | + add_paths - list, defaults to [] 236 | + handoff_concurrency - integer, defaults to 4 237 | + js_source_dir - string 238 | + js_vm_count - integer 239 | + mapred_name - string 240 | + raw_name - string 241 | + riak_kv_stat - boolean. 242 | + stats_urlpath - string 243 | + storage_backend - atom. Backend names are now prefixed as "riak_kv_" instead of just "riak_". 244 | + pb_ip - string 245 | + pb_port - integer 246 | 247 | IMPORTANT: The default backend has changed names from 248 | riak_dets_backend to riak_kv_dets_bakend. Other backends have 249 | changed names as well. This rename does not affect you if you are 250 | using the Innostore backend. 251 | 252 | If you did not have any of these settings defined in etc/app.config, 253 | you still do not need to define them in your new etc/app.config. 254 | 255 | ** Ring Storage 256 | Periodically, Riak nodes save the state of their ring to disk. In 257 | 0.9, these files were named "data/ring/riak_ring.*", but in 0.10, 258 | they're named "data/ring/riak_core_ring.*". Renaming the old files 259 | to the new scheme is all you need to do to make the switch. 260 | 261 | If you referenced any Riak modules in your bucket properties, you 262 | will also need to change those references to point to the new 263 | module names after your cluster is running. 264 | 265 | ** Your Data 266 | The rest of your cluster's data, stored in the "data" directory 267 | ("data/dets" or "data/innodb", for example) 268 | should be safe to either leave in place, or copy to your new 269 | install location, depending on how you upgraded. 270 | -------------------------------------------------------------------------------- /doc/raw-http-howto.txt: -------------------------------------------------------------------------------- 1 | How to Use the Riak HTTP Interface 2 | 3 | Step 1 in using the Riak HTTP interface is to enable the Riak web 4 | interface. Add these two lines to your riak config, in the 'riak_core' section: 5 | 6 | {web_ip, "127.0.0.1"}. 7 | {web_port, 8098}. 8 | 9 | Now start Riak with your config file, and you'll find the HTTP 10 | interface at http://127.0.0.1:8098/riak/... If you'd rather have some 11 | prefix other than "riak", add another line to your config: 12 | 13 | {raw_name, "myprefix"}. 14 | 15 | You'll find that all buckets exist, and are ready to give you details 16 | about themselves at /riak/BucketName: 17 | 18 | $ curl -i http://127.0.0.1:8098/riak/example 19 | HTTP/1.1 200 OK 20 | Vary: Accept-Encoding 21 | Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic) 22 | Date: Wed, 14 Jul 2010 03:28:46 GMT 23 | Content-Type: application/json 24 | Content-Length: 370 25 | 26 | {"props":{"name":"bucket","n_val":3,"allow_mult":false,"last_write_wins":false,"precommit":[],"postcommit":[],"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"old_vclock":86400,"young_vclock":20,"big_vclock":50,"small_vclock":10,"r":"quorum","w":"quorum","dw":"quorum","rw":"quorum"}} 27 | 28 | It is not necessary to "create" or otherwise "touch" a bucket before storing documents into it, but if you want to change a property at runtime, it's as simple as PUTing to the bucket: 29 | 30 | $ curl -X PUT -H "content-type: application/json" \ 31 | http://127.0.0.1:8098/riak/example --data "{\"props\":{\"n_val\":4}}" 32 | 33 | This would change the n_val of the bucket to 4. 34 | 35 | Storing data is just as easy - just PUT to the bucket and key: 36 | 37 | $ curl -X PUT -H "content-type: text/plain" \ 38 | http://127.0.0.1:8098/riak/example/foo --data "I have a document." 39 | 40 | The Riak HTTP interface requires only that you include a content-type, 41 | but no attempt to validate the content is made. You could use 42 | application/json just as easily as image/gif - it's up to you to 43 | provide the correct body. Whatever content type you include here will 44 | be the content type that Riak serves whenever a client attempts to GET 45 | this document. 46 | 47 | To get the document back, just GET the bucket and key: 48 | 49 | $ curl -i http://127.0.0.1:8098/riak/example/foo 50 | HTTP/1.1 200 OK 51 | X-Riak-Vclock: a85hYGBgzGDKBVIsLGtY2zOYEhnzWBkE12w8ypcFAA== 52 | Vary: Accept-Encoding 53 | Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic) 54 | Link: ; rel="up" 55 | Last-Modified: Wed, 14 Jul 2010 03:33:37 GMT 56 | Etag: 1wSftQ956ViTpGGs8dKQ68 57 | Date: Wed, 14 Jul 2010 03:33:45 GMT 58 | Content-Type: text/plain 59 | Content-Length: 18 60 | 61 | I have a document. 62 | 63 | You'll notice one odd-looking header in that response: X-Riak-Vclock. 64 | This is the vclock you want to provide with your next write to that 65 | object, in order to indicate the causality of your modification. For 66 | example: 67 | 68 | $ curl -X PUT -H "content-type: text/plain" \ 69 | -H "X-Riak-Vclock: a85hYGBgzGDKBVIsLGtY2zOYEhnzWBkE12w8ypcFAA==" \ 70 | http://127.0.0.1:8098/riak/example/foo \ 71 | --data "I have a modified document." 72 | 73 | This command will modify the document, which we can verify with a 74 | second GET: 75 | 76 | $ curl -i http://127.0.0.1:8098/riak/example/foo 77 | HTTP/1.1 200 OK 78 | X-Riak-Vclock: a85hYGBgymDKBVIsLGtY2zOYEhnzWBkE12w8ygcVZm8+GwwVPggSzgIA 79 | Vary: Accept-Encoding 80 | Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic) 81 | Link: ; rel="up" 82 | Last-Modified: Wed, 14 Jul 2010 03:36:33 GMT 83 | Etag: 59nvzTQTQ6ixb9jxASLDKS 84 | Date: Wed, 14 Jul 2010 03:36:36 GMT 85 | Content-Type: text/plain 86 | Content-Length: 27 87 | 88 | I have a modified document. 89 | 90 | To delete a document, simply issue a DELETE request: 91 | 92 | $ curl -X DELETE http://127.0.0.1:8098/riak/example/foo 93 | 94 | You'll find that further GETs of that URL return status code 404. 95 | 96 | For each of the key-level, document requests, you may also specify the 97 | query parameters 'r', 'w', 'dw', and 'rw', to tune the R (read), W 98 | (write), DW (durable write), and RW (read-write, for delete) value for 99 | that request. For instance: 100 | 101 | $ curl http://127.0.0.1:8098/riak/example/foo?r=1 102 | 103 | Would get the "foo" document in the "example" bucket using an R-value of 1. 104 | 105 | 106 | == Advanced Topic 1: Siblings (multiple values) == 107 | 108 | Documents in Riak can have multiple, conflicting values if the 109 | 'allow_mult' property has been set to 'true' for a bucket. For 110 | example, if you issued the following: 111 | 112 | $ curl -X PUT -H "content-type: application/json" \ 113 | http://127.0.0.1:8098/riak/example \ 114 | --data "{\"props\":{\"allow_mult\":true}}" 115 | $ curl -X PUT -H "content-type: text/plain" \ 116 | http://127.0.0.1:8098/riak/example/sib --data "one thing" 117 | $ curl -X PUT -H "content-type: text/plain" \ 118 | http://127.0.0.1:8098/riak/example/sib --data "another" 119 | 120 | You will have created two siblings for the "sib" document in the 121 | "example" bucket. Riak won't know what to do with these siblings if 122 | you ask for the "sib" document, so instead it will just tell you that 123 | they're both there: 124 | 125 | $ curl -i http://127.0.0.1:8098/riak/example/sib 126 | HTTP/1.1 300 Multiple Choices 127 | X-Riak-Vclock: a85hYGBgzmDKBVIszBJiPRlMiYx5rAyzNmw8ygcVZoyuU4YKz0cIszUnMZ0/pIAskQUA 128 | Vary: Accept, Accept-Encoding 129 | Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic) 130 | Date: Wed, 14 Jul 2010 03:53:08 GMT 131 | Content-Type: text/plain 132 | Content-Length: 55 133 | 134 | Siblings: 135 | xKntdvwv9b1sIj1LArbox 136 | 3QApcpvLPS1FyKu8IrN1sV 137 | 138 | The strings listed in the body are the vtags of each sibling. To 139 | examine each sibling, perform the same GET, but add a "vtag" query 140 | parameter to the URL: 141 | 142 | $ curl http://127.0.0.1:8098/riak/example/sib?vtag=xKntdvwv9b1sIj1LArbox 143 | one thing 144 | $ curl http://127.0.0.1:8098/riak/example/sib?vtag=3QApcpvLPS1FyKu8IrN1sV 145 | another 146 | 147 | If you'd rather see all of the siblings at once, set your Accept 148 | header to multipart/mixed. Riak will hand back each of the versions 149 | as a separate part of a multipart/mixed document: 150 | 151 | $ curl -i -H "accept: multipart/mixed" http://127.0.0.1:8098/riak/example/sib 152 | HTTP/1.1 300 Multiple Choices 153 | X-Riak-Vclock: a85hYGBgzmDKBVIszBJiPRlMiYx5rAyzNmw8ygcVZoyuU4YKz0cIszUnMZ0/pIAskQUA 154 | Vary: Accept, Accept-Encoding 155 | Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic) 156 | Date: Wed, 14 Jul 2010 03:55:24 GMT 157 | Content-Type: multipart/mixed; boundary=FOs2aRNqxNs5pyKd7eHBia2Pg7x 158 | Content-Length: 390 159 | 160 | 161 | --FOs2aRNqxNs5pyKd7eHBia2Pg7x 162 | Content-Type: text/plain 163 | Link: ; rel="up" 164 | Etag: xKntdvwv9b1sIj1LArbox 165 | Last-Modified: Wed, 14 Jul 2010 03:53:03 GMT 166 | 167 | another 168 | --FOs2aRNqxNs5pyKd7eHBia2Pg7x 169 | Content-Type: text/plain 170 | Link: ; rel="up" 171 | Etag: 3QApcpvLPS1FyKu8IrN1sV 172 | Last-Modified: Wed, 14 Jul 2010 03:52:58 GMT 173 | 174 | one thing 175 | --FOs2aRNqxNs5pyKd7eHBia2Pg7x-- 176 | 177 | To resolve the conflict, just issue another PUT, with the body you 178 | want, and the vclock from this version: 179 | 180 | $ curl -X PUT -H "content-type: text/plain" \ 181 | -H "X-Riak-Vclock: a85hYGBgzmDKBVIszBJiPRlMiYx5rAyzNmw8ygcVZoyuU4YKz0cIszUnMZ0/pIAskQUA" \ 182 | http://127.0.0.1:8098/riak/example/sib --data "resolved" 183 | 184 | And you'll see that things are back to normal: 185 | 186 | $ curl http://127.0.0.1:8098/riak/example/sib 187 | resolved 188 | 189 | 190 | == Advanced Topic 2: Link walking == 191 | 192 | As with other Riak documents, you are free to specify links in your 193 | documents in any fashion you wish, as long as you also write a 194 | function for extracting them at map/reduce time. However, the HTTP 195 | interface provides a function that will handle link parsing and 196 | extraction for you, if you are able to describe your links in a Link 197 | HTTP header. 198 | 199 | Riak's Link header syntax is based on Mark Nottingham's work. You can 200 | read more about it at: 201 | http://www.mnot.net/drafts/draft-nottingham-http-link-header-00.txt 202 | 203 | For Riak, the goal is to provide a link from one document to another. 204 | For instance, you may want to link from the "jane" document in the 205 | "person" bucket to the "xyz" document in the "memo" bucket. To do 206 | this, you'd add a header of the following format to your PUT to 207 | /riak/person/jane: 208 | 209 | Link: ; riaktag="author" 210 | 211 | Multiple links should be separated by commas: 212 | 213 | Link: ; riaktag="author", ; riaktag="reader" 214 | 215 | Performing a GET on a resource with links will return a Link header 216 | of the same format. 217 | 218 | To walk these links, use the URL-walking syntax: 219 | 220 | http://127.0.0.1:8098/riak/person/jane/memo,_,_ 221 | 222 | This request would return all of the documents in the "memo" bucket 223 | that the "jane" document links to. You could get just the "memo" 224 | documents with links tagged "author" by asking for: 225 | 226 | http://127.0.0.1:8098/riak/person/jane/memo,author,_ 227 | 228 | The response of a link walk request is always multipart/mixed content. 229 | Each part of the multipart response body is a representation of the 230 | result of the corresponding link step. That representation is also a 231 | multipart/mixed document. Each part of this inner multipart document 232 | is a representation of the Riak object that was walked to at that 233 | step. For documents with siblings, one of the siblings is chosen 234 | arbitrarily, and an X-Riak-Sibling-VTags header is added to its 235 | representation to alert the user that this is the case. 236 | 237 | A few examples are approriate: 238 | 239 | $ curl -X PUT -H "content-type: text/plain" \ 240 | http://127.0.0.1:8098/riak/memo/xyz \ 241 | --data "my reading list: New York Times, Wired" 242 | $ curl -X PUT -H "content-type: text/plain" \ 243 | http://127.0.0.1:8098/riak/memo/abc \ 244 | --data "todos: have meeting, make phone call" 245 | $ curl -X PUT -H "content-type: text/plain" \ 246 | -H "link: ; riaktag=\"author\", ; riaktag=\"reader\"" \ 247 | http://127.0.0.1:8098/riak/person/jane --data "Name: Jane Doe" 248 | 249 | $ curl -i http://127.0.0.1:8098/riak/person/jane/memo,_,_ 250 | HTTP/1.1 200 OK 251 | Server: MochiWeb/1.1 WebMachine/1.7 (participate in the frantic) 252 | Expires: Wed, 14 Jul 2010 04:48:32 GMT 253 | Date: Wed, 14 Jul 2010 04:38:32 GMT 254 | Content-Type: multipart/mixed; boundary=W6v94jEJ22hI5cfwbqinnGBmwpV 255 | Content-Length: 753 256 | 257 | 258 | --W6v94jEJ22hI5cfwbqinnGBmwpV 259 | Content-Type: multipart/mixed; boundary=SRzlI2jh68aWdVONf9n1yfDBEVx 260 | 261 | --SRzlI2jh68aWdVONf9n1yfDBEVx 262 | X-Riak-Vclock: a85hYGBgzGDKBVIsbLfe7clgSmTMY2Uw3r3xKF8WAA== 263 | Location: /riak/memo/xyz 264 | Content-Type: text/plain 265 | Link: ; rel="up" 266 | Etag: MY6SrlVU7iWVVCndk47NB 267 | Last-Modified: Wed, 14 Jul 2010 04:38:11 GMT 268 | 269 | my reading list: New York Times, Wired 270 | --SRzlI2jh68aWdVONf9n1yfDBEVx 271 | X-Riak-Vclock: a85hYGBgzGDKBVIsDByHMjOYEhnzWBksd288ypcFAA== 272 | Location: /riak/memo/abc 273 | Content-Type: text/plain 274 | Link: ; rel="up" 275 | Etag: 5sTdNwHNK2rykq4kqRSMtW 276 | Last-Modified: Wed, 14 Jul 2010 04:38:17 GMT 277 | 278 | todos: have meeting, make phone call 279 | --SRzlI2jh68aWdVONf9n1yfDBEVx-- 280 | 281 | --W6v94jEJ22hI5cfwbqinnGBmwpV-- 282 | 283 | -------------------------------------------------------------------------------- /TRANSITION: -------------------------------------------------------------------------------- 1 | Migrating to Riak 0.11 from earlier versions of Riak 2 | ==================================================== 3 | 4 | 5 | 6 | 7 | Table of Contents 8 | ================= 9 | 1 Riak 0.10 to 0.11 10 | 1.1 Customized configuration, not switching backends 11 | 1.2 Default configuration, or switching backends 12 | 1.2.1 Backup/Restore 13 | 1.2.2 Online Rolling Upgrade 14 | 2 Riak 0.9 and earlier to 0.11 15 | 3 Migrating from Riak 0.9.x to 0.10 16 | 4 Overview 17 | 5 Requirements 18 | 5.1 Erlang/OTP R13B04 19 | 5.2 Mercurial 20 | 6 Dependencies 21 | 6.1 Mochiweb, Webmachine, Erlang_js 22 | 6.2 make deps 23 | 7 Source 24 | 7.1 Core/KV Split 25 | 7.2 Clients 26 | 8 Configuration 27 | 8.1 app.config 28 | 8.2 Ring Storage 29 | 8.3 Your Data 30 | 31 | 32 | 1 Riak 0.10 to 0.11 33 | ~~~~~~~~~~~~~~~~~~~~ 34 | 35 | 1.1 Customized configuration, not switching backends 36 | ===================================================== 37 | 38 | If you have customized your configuration file (etc/app.config), 39 | and you will be keeping your customized configuration file, your 40 | transition should be as easy as stopping the 0.10 node, installing 41 | the 0.11 binaries, and starting the node back up. The node should 42 | find its ring and data in the same location as before, and will 43 | continue working as if nothing changed. 44 | 45 | 1.2 Default configuration, or switching backends 46 | ================================================= 47 | 48 | If you're using the default Riak configuration file, you'll need to 49 | be aware that the default storage backend has changed from DETS to 50 | Basho's new Bitcask. You'll need to choose one of two upgrade 51 | paths in order to migrate your data from your old storage backend 52 | to the new one (these steps will also work, in general, from any 53 | backend to any other backend). Your options are: backup/restore, 54 | and online rolling upgrade. 55 | 56 | 1.2.1 Backup/Restore 57 | --------------------- 58 | 59 | The easiest way to migrate from one storage backend to another is 60 | to backup the cluster as it stands, shut it down, switch the 61 | backend (by upgrading or editing the config file), start the 62 | cluster back up, and then restore the data to the cluster. Follow 63 | these steps to do this: 64 | 65 | 1. With a running cluster, use the "riak-admin backup" command to 66 | backup the cluster's data: 67 | 68 | $ riak-admin backup riak@127.0.0.1 riak riak-backup.dat all 69 | 70 | This script will exit when backup is complete. If your cluster 71 | has a lot of data, you can backup each node individually by 72 | switching "all" to "node" in the command line above, and then 73 | running the command once for each node, changing the node name 74 | in the command line. 75 | 76 | 2. Shut down the cluster. 77 | 78 | $ riak stop 79 | 80 | Run this command on each node. 81 | 82 | 3. Upgrade each node by installing the new 0.11 package. If 83 | you're just switching backends (not upgrading), modify the 84 | configuration file in this step. 85 | 86 | 4. Start the cluster 87 | 88 | $ riak start 89 | 90 | Run this command on each node. The nodes will automatically 91 | reconnect to each other. 92 | 93 | 5. Reload the cluster's data 94 | 95 | $ riak-admin restore riak@127.0.0.1 riak riak-backup.dat 96 | 97 | If you used the whole-cluster ("all") backup mode, you're now 98 | finished. If you backed up each node individually, you'll need 99 | to run this command for each of those backup files (note that 100 | they have node names appended to what you provided on the 101 | backup command line). 102 | 103 | Once the restore script exits, all data should be restored to the 104 | cluster. After checking that this is true, you may remove the 105 | data directories for the old storage backend. 106 | 107 | 1.2.2 Online Rolling Upgrade 108 | ----------------------------- 109 | 110 | If you need your Riak cluster to remain running during the backend 111 | transition, and you have spare network capacity, you can use 112 | Riak's easy node addition/removal process to switch backends. 113 | 114 | 1. Configure a new node, with the new backend you want to use, and 115 | add it to the cluster. 116 | 117 | newnode$ riak start 118 | newnode$ riak-admin join oldclusternode@10.0.0.100 119 | 120 | 2. Remove one of the old-backend nodes from the cluster. 121 | 122 | oldnode0$ riak-admin leave 123 | 124 | 3. When the old-backend node completes handoff, it will exit (use 125 | 'ps' or similar to watch for it to shut down). Once it has 126 | exited, upgrade to 0.11 (or switch the backend) and restart the 127 | node. 128 | 129 | 4. Repeat steps 2 and 3 for each other old-backend node in the 130 | cluster. 131 | 132 | 5. When all nodes are running on the new backend, shutdown the new 133 | node you set up in step 1 (if you wish, or leave it up if you like). 134 | 135 | Note: You can skip step 1 if you are running a cluster of more 136 | than one node, and your cluster can tolerate (capacity- and 137 | throughput-wise) being temporarily one node smaller. Step 1 is 138 | merely an attempt to add extra capacity to the system before 139 | beging an expensive operation. 140 | 141 | 2 Riak 0.9 and earlier to 0.11 142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 143 | 144 | If you are upgrading from a Riak version earlier than 0.10, your 145 | only option for upgrade is backup and restore. Please use the 146 | backup/restore method described in the 0.10-to-0.11-transition 147 | section above. 148 | 149 | You will also be interested in the following section, which 150 | discusses migration from 0.9 to 0.10. 151 | 152 | 153 | 3 Migrating from Riak 0.9.x to 0.10 154 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 155 | 156 | 4 Overview 157 | ~~~~~~~~~~~ 158 | Riak has undergone significant restructuring in the transition from 159 | version 0.9.x to version 0.10. If you are using the binary builds, 160 | please skip directly to the Configuration and Clients sections. If 161 | you are building from source yourself, please review the whole of 162 | this document. 163 | 164 | NOTE: If the only files you have changed in your Riak source clone 165 | are those underneath the "rel" directory 166 | ("rel/overlay/etc/app.config", for example), the safest way to 167 | update is to make a fresh clone of the Riak repository, and then 168 | skip to the Configuration and Clients sections of this document for 169 | details about migrating your configurations and data. 170 | 171 | 5 Requirements 172 | ~~~~~~~~~~~~~~~ 173 | 174 | 5.1 Erlang/OTP R13B04 175 | ====================== 176 | 177 | Riak 0.10 uses new features ("NIFs") provided by the latest 178 | Erlang/OTP release, R13B04. If you are building from source, you 179 | will need this release or a newer one. 180 | 181 | 5.2 Mercurial 182 | ============== 183 | 184 | Riak 0.10 has moved several of its components into external 185 | repositories. If you are building from source, you will need 186 | Mercurial installed to allow the Rebar build system to retrieve 187 | code from these external repositories. 188 | 189 | 6 Dependencies 190 | ~~~~~~~~~~~~~~~ 191 | 192 | 6.1 Mochiweb, Webmachine, Erlang_js 193 | ==================================== 194 | mochiweb, webmachine, and erlang_js are now pulled into the "deps" 195 | subdirectory, instead of being included in the "apps" subdirectory. 196 | If you are pulling 0.10 code into a repository that formerly had 197 | 0.9.x in it, please remove the apps/mochiweb, apps/webmachine, and 198 | apps/erlang_js directories from your source tree. 199 | 200 | There is a chance that your update will also leave an "apps/riak" 201 | directory hanging around. If it does, please remove this directory 202 | (Riak code has moved into the "apps/riak_core" and "apps/riak_kv" 203 | directories). 204 | 205 | 6.2 make deps 206 | ============== 207 | The "all" make target (and, by extension, the "rel" target as 208 | well), depend on a new "deps" target, which handles the fetching 209 | the dependencies (mochiweb, webmachine, erlang_js). 210 | 211 | 7 Source 212 | ~~~~~~~~~ 213 | 214 | 7.1 Core/KV Split 215 | ================== 216 | We've drawn a line through 0.9.x Riak, and divided it into two 217 | things, one called "riak_core", and the other called "riak_kv". 218 | 219 | The things that live in riak_core are those that deal with cluster 220 | membership. Ring-claiming and the like. 221 | 222 | The things that live in riak_kv are those that deal with storing 223 | data. Get and Put FSMs, backends, etc. 224 | 225 | 7.2 Clients 226 | ============ 227 | We've also moved the clients out of the client_lib subdirectory, 228 | and into their own language-specific repositories on BitBucket. At 229 | [http://bitbucket.org/basho/], you should find: 230 | 231 | + riak-python-client 232 | + riak-php-client 233 | + riak-erlang-client 234 | + riak-java-client 235 | + riak-javascript-client 236 | + riak-ruby-client 237 | 238 | 8 Configuration 239 | ~~~~~~~~~~~~~~~~ 240 | 241 | 8.1 app.config 242 | =============== 243 | 244 | Splitting the "riak" Erlang application into the "riak_core" and 245 | "riak_kv" Erlang applications means that configuration options for 246 | each component need to move around in etc/app.config. 247 | 248 | Where before etc/app.config would have contained a section like: 249 | 250 | {riak, [ 251 | %% many settings here 252 | ]}, 253 | 254 | Now, etc/app.config should contain two sections like: 255 | 256 | {riak_core, [ 257 | %% core-specific settings 258 | ]}, 259 | {riak_kv, [ 260 | %% kv-specific settings 261 | ]}, 262 | 263 | The list of settings that moved to the riak_core section are: 264 | 265 | + choose_claim_fun 266 | + cluster_name - string, defaults to "default" 267 | + default_bucket_props 268 | + gossip_interval - integer, defaults to 60k msec 269 | + ring_creation_size - integer, defaults to 64 270 | + ring_state_dir - string 271 | + target_n_val - integer, defaults to 3 272 | + wants_claim_fun 273 | + web_ip - string. Used to be "riak_web_ip" 274 | + web_logdir - string. 275 | + web_port - integer. Used to be "riak_web_port" 276 | 277 | IMPORTANT: Note the rename of "riak_web_*" to just "web_*" 278 | 279 | The list of settings that moved to the riak_kv section are: 280 | 281 | + add_paths - list, defaults to [] 282 | + handoff_concurrency - integer, defaults to 4 283 | + js_source_dir - string 284 | + js_vm_count - integer 285 | + mapred_name - string 286 | + raw_name - string 287 | + riak_kv_stat - boolean. 288 | + stats_urlpath - string 289 | + storage_backend - atom. Backend names are now prefixed as "riak_kv_" instead of just "riak_". 290 | + pb_ip - string 291 | + pb_port - integer 292 | 293 | IMPORTANT: The default backend has changed names from 294 | riak_dets_backend to riak_kv_dets_bakend. Other backends have 295 | changed names as well. This rename does not affect you if you are 296 | using the Innostore backend. 297 | 298 | If you did not have any of these settings defined in etc/app.config, 299 | you still do not need to define them in your new etc/app.config. 300 | 301 | 8.2 Ring Storage 302 | ================= 303 | Periodically, Riak nodes save the state of their ring to disk. In 304 | 0.9, these files were named "data/ring/riak_ring.*", but in 0.10, 305 | they're named "data/ring/riak_core_ring.*". Renaming the old files 306 | to the new scheme is all you need to do to make the switch. 307 | 308 | If you referenced any Riak modules in your bucket properties, you 309 | will also need to change those references to point to the new 310 | module names after your cluster is running. 311 | 312 | 8.3 Your Data 313 | ============== 314 | The rest of your cluster's data, stored in the "data" directory 315 | ("data/dets" or "data/innodb", for example) 316 | should be safe to either leave in place, or copy to your new 317 | install location, depending on how you upgraded. 318 | -------------------------------------------------------------------------------- /doc/doc-style.css: -------------------------------------------------------------------------------- 1 | body { 2 | color: #000000; 3 | background-color: #ffffff; 4 | } 5 | .org-bold { 6 | /* bold */ 7 | font-weight: bold; 8 | } 9 | .org-bold-italic { 10 | /* bold-italic */ 11 | font-weight: bold; 12 | font-style: italic; 13 | } 14 | .org-border { 15 | /* border */ 16 | background-color: #000000; 17 | } 18 | .org-buffer-menu-buffer { 19 | /* buffer-menu-buffer */ 20 | font-weight: bold; 21 | } 22 | .org-builtin { 23 | /* font-lock-builtin-face */ 24 | color: #da70d6; 25 | } 26 | .org-button { 27 | /* button */ 28 | text-decoration: underline; 29 | } 30 | .org-comint-highlight-input { 31 | /* comint-highlight-input */ 32 | font-weight: bold; 33 | } 34 | .org-comint-highlight-prompt { 35 | /* comint-highlight-prompt */ 36 | color: #00008b; 37 | } 38 | .org-comment { 39 | /* font-lock-comment-face */ 40 | color: #a8a8a8; 41 | } 42 | .org-comment-delimiter { 43 | /* font-lock-comment-delimiter-face */ 44 | color: #a8a8a8; 45 | } 46 | .org-constant { 47 | /* font-lock-constant-face */ 48 | color: #4682b4; 49 | } 50 | .org-cursor { 51 | /* cursor */ 52 | background-color: #000000; 53 | } 54 | .org-default { 55 | /* default */ 56 | color: #000000; 57 | background-color: #ffffff; 58 | } 59 | .org-diff-added { 60 | } 61 | .org-diff-changed { 62 | } 63 | .org-diff-context { 64 | /* diff-context */ 65 | color: #7f7f7f; 66 | } 67 | .org-diff-file-header { 68 | /* diff-file-header */ 69 | background-color: #b3b3b3; 70 | font-weight: bold; 71 | } 72 | .org-diff-function { 73 | /* diff-function */ 74 | background-color: #d9d9d9; 75 | } 76 | .org-diff-header { 77 | /* diff-header */ 78 | background-color: #d9d9d9; 79 | } 80 | .org-diff-hunk-header { 81 | /* diff-hunk-header */ 82 | background-color: #d9d9d9; 83 | } 84 | .org-diff-index { 85 | /* diff-index */ 86 | background-color: #b3b3b3; 87 | font-weight: bold; 88 | } 89 | .org-diff-indicator-added { 90 | } 91 | .org-diff-indicator-changed { 92 | } 93 | .org-diff-indicator-removed { 94 | } 95 | .org-diff-nonexistent { 96 | /* diff-nonexistent */ 97 | background-color: #b3b3b3; 98 | font-weight: bold; 99 | } 100 | .org-diff-removed { 101 | } 102 | .org-doc { 103 | /* font-lock-doc-face */ 104 | color: #a0522d; 105 | } 106 | .org-escape-glyph { 107 | /* escape-glyph */ 108 | color: #a52a2a; 109 | } 110 | .org-file-name-shadow { 111 | /* file-name-shadow */ 112 | color: #7f7f7f; 113 | } 114 | .org-fixed-pitch { 115 | } 116 | .org-fringe { 117 | /* fringe */ 118 | background-color: #f2f2f2; 119 | } 120 | .org-function-name { 121 | /* font-lock-function-name-face */ 122 | color: #0000ff; 123 | } 124 | .org-header-line { 125 | /* header-line */ 126 | color: #333333; 127 | background-color: #e5e5e5; 128 | } 129 | .org-help-argument-name { 130 | /* help-argument-name */ 131 | font-style: italic; 132 | } 133 | .org-highlight { 134 | /* highlight */ 135 | background-color: #b4eeb4; 136 | } 137 | .org-isearch { 138 | /* isearch */ 139 | color: #b0e2ff; 140 | background-color: #cd00cd; 141 | } 142 | .org-italic { 143 | /* italic */ 144 | font-style: italic; 145 | } 146 | .org-keyword { 147 | /* font-lock-keyword-face */ 148 | color: #a020f0; 149 | } 150 | .org-lazy-highlight { 151 | /* lazy-highlight */ 152 | background-color: #afeeee; 153 | } 154 | .org-link { 155 | /* link */ 156 | color: #0000ff; 157 | text-decoration: underline; 158 | } 159 | .org-link-visited { 160 | /* link-visited */ 161 | color: #8b008b; 162 | text-decoration: underline; 163 | } 164 | .org-makefile-makepp-perl { 165 | /* makefile-makepp-perl */ 166 | background-color: #bfefff; 167 | } 168 | .org-makefile-shell { 169 | } 170 | .org-makefile-space { 171 | /* makefile-space */ 172 | background-color: #ff69b4; 173 | } 174 | .org-makefile-targets { 175 | /* makefile-targets */ 176 | color: #0000ff; 177 | } 178 | .org-match { 179 | /* match */ 180 | background-color: #ffff00; 181 | } 182 | .org-menu { 183 | } 184 | .org-negation-char { 185 | } 186 | .org-next-error { 187 | /* next-error */ 188 | background-color: #eedc82; 189 | } 190 | .org-nobreak-space { 191 | /* nobreak-space */ 192 | color: #a52a2a; 193 | text-decoration: underline; 194 | } 195 | .org-org-agenda-clocking { 196 | /* org-agenda-clocking */ 197 | background-color: #ffff00; 198 | } 199 | .org-org-agenda-column-dateline { 200 | /* org-agenda-column-dateline */ 201 | background-color: #e5e5e5; 202 | } 203 | .org-org-agenda-date { 204 | /* org-agenda-date */ 205 | color: #0000ff; 206 | } 207 | .org-org-agenda-date-today { 208 | /* org-agenda-date-today */ 209 | color: #0000ff; 210 | font-weight: bold; 211 | font-style: italic; 212 | } 213 | .org-org-agenda-date-weekend { 214 | /* org-agenda-date-weekend */ 215 | color: #0000ff; 216 | font-weight: bold; 217 | } 218 | .org-org-agenda-dimmed-todo { 219 | /* org-agenda-dimmed-todo-face */ 220 | color: #7f7f7f; 221 | } 222 | .org-org-agenda-done { 223 | /* org-agenda-done */ 224 | color: #228b22; 225 | } 226 | .org-org-agenda-restriction-lock { 227 | /* org-agenda-restriction-lock */ 228 | background-color: #ffff00; 229 | } 230 | .org-org-agenda-structure { 231 | /* org-agenda-structure */ 232 | color: #0000ff; 233 | } 234 | .org-org-archived { 235 | /* org-archived */ 236 | color: #7f7f7f; 237 | } 238 | .org-org-block { 239 | /* org-block */ 240 | color: #7f7f7f; 241 | } 242 | .org-org-checkbox { 243 | /* org-checkbox */ 244 | font-weight: bold; 245 | } 246 | .org-org-checkbox-statistics-done { 247 | /* org-checkbox-statistics-done */ 248 | color: #228b22; 249 | font-weight: bold; 250 | } 251 | .org-org-checkbox-statistics-todo { 252 | /* org-checkbox-statistics-todo */ 253 | color: #ff0000; 254 | font-weight: bold; 255 | } 256 | .org-org-clock-overlay { 257 | /* org-clock-overlay */ 258 | background-color: #ffff00; 259 | } 260 | .org-org-code { 261 | /* org-code */ 262 | color: #7f7f7f; 263 | } 264 | .org-org-column { 265 | /* org-column */ 266 | background-color: #e5e5e5; 267 | } 268 | .org-org-column-title { 269 | /* org-column-title */ 270 | background-color: #e5e5e5; 271 | font-weight: bold; 272 | text-decoration: underline; 273 | } 274 | .org-org-date { 275 | /* org-date */ 276 | color: #a020f0; 277 | text-decoration: underline; 278 | } 279 | .org-org-done { 280 | /* org-done */ 281 | color: #228b22; 282 | font-weight: bold; 283 | } 284 | .org-org-drawer { 285 | /* org-drawer */ 286 | color: #0000ff; 287 | } 288 | .org-org-ellipsis { 289 | /* org-ellipsis */ 290 | color: #b8860b; 291 | text-decoration: underline; 292 | } 293 | .org-org-footnote { 294 | /* org-footnote */ 295 | color: #a020f0; 296 | text-decoration: underline; 297 | } 298 | .org-org-formula { 299 | /* org-formula */ 300 | color: #b22222; 301 | } 302 | .org-org-headline-done { 303 | /* org-headline-done */ 304 | color: #bc8f8f; 305 | } 306 | .org-org-hide { 307 | /* org-hide */ 308 | color: #ffffff; 309 | } 310 | .org-org-latex-and-export-specials { 311 | /* org-latex-and-export-specials */ 312 | color: #8b4513; 313 | } 314 | .org-org-level-1 { 315 | /* org-level-1 */ 316 | color: #0000ff; 317 | } 318 | .org-org-level-2 { 319 | /* org-level-2 */ 320 | color: #b8860b; 321 | } 322 | .org-org-level-3 { 323 | /* org-level-3 */ 324 | color: #a020f0; 325 | } 326 | .org-org-level-4 { 327 | /* org-level-4 */ 328 | color: #b22222; 329 | } 330 | .org-org-level-5 { 331 | /* org-level-5 */ 332 | color: #228b22; 333 | } 334 | .org-org-level-6 { 335 | /* org-level-6 */ 336 | color: #5f9ea0; 337 | } 338 | .org-org-level-7 { 339 | /* org-level-7 */ 340 | color: #da70d6; 341 | } 342 | .org-org-level-8 { 343 | /* org-level-8 */ 344 | color: #bc8f8f; 345 | } 346 | .org-org-link { 347 | /* org-link */ 348 | color: #a020f0; 349 | text-decoration: underline; 350 | } 351 | .org-org-meta-line { 352 | /* org-meta-line */ 353 | color: #a8a8a8; 354 | } 355 | .org-org-mode-line-clock { 356 | /* org-mode-line-clock */ 357 | color: #000000; 358 | background-color: #bfbfbf; 359 | } 360 | .org-org-property-value { 361 | } 362 | .org-org-quote { 363 | /* org-quote */ 364 | color: #7f7f7f; 365 | } 366 | .org-org-scheduled { 367 | /* org-scheduled */ 368 | color: #006400; 369 | } 370 | .org-org-scheduled-previously { 371 | /* org-scheduled-previously */ 372 | color: #b22222; 373 | } 374 | .org-org-scheduled-today { 375 | /* org-scheduled-today */ 376 | color: #006400; 377 | } 378 | .org-org-sexp-date { 379 | /* org-sexp-date */ 380 | color: #a020f0; 381 | } 382 | .org-org-special-keyword { 383 | /* org-special-keyword */ 384 | color: #bc8f8f; 385 | } 386 | .org-org-table { 387 | /* org-table */ 388 | color: #0000ff; 389 | } 390 | .org-org-tag { 391 | /* org-tag */ 392 | font-weight: bold; 393 | } 394 | .org-org-target { 395 | /* org-target */ 396 | text-decoration: underline; 397 | } 398 | .org-org-time-grid { 399 | /* org-time-grid */ 400 | color: #b8860b; 401 | } 402 | .org-org-todo { 403 | /* org-todo */ 404 | color: #ff0000; 405 | font-weight: bold; 406 | } 407 | .org-org-upcoming-deadline { 408 | /* org-upcoming-deadline */ 409 | color: #b22222; 410 | } 411 | .org-org-verbatim { 412 | /* org-verbatim */ 413 | color: #7f7f7f; 414 | } 415 | .org-org-verse { 416 | /* org-verse */ 417 | color: #7f7f7f; 418 | } 419 | .org-org-warning { 420 | /* org-warning */ 421 | color: #ff0000; 422 | font-weight: bold; 423 | } 424 | .org-outline-1 { 425 | /* outline-1 */ 426 | color: #0000ff; 427 | } 428 | .org-outline-2 { 429 | /* outline-2 */ 430 | color: #b8860b; 431 | } 432 | .org-outline-3 { 433 | /* outline-3 */ 434 | color: #a020f0; 435 | } 436 | .org-outline-4 { 437 | /* outline-4 */ 438 | color: #da70d6; 439 | } 440 | .org-outline-5 { 441 | /* outline-5 */ 442 | color: #a8a8a8; 443 | } 444 | .org-outline-6 { 445 | /* outline-6 */ 446 | color: #4682b4; 447 | } 448 | .org-outline-7 { 449 | /* outline-7 */ 450 | color: #228b22; 451 | } 452 | .org-outline-8 { 453 | /* outline-8 */ 454 | color: #a0522d; 455 | } 456 | .org-preprocessor { 457 | /* font-lock-preprocessor-face */ 458 | color: #da70d6; 459 | } 460 | .org-preview { 461 | /* preview-face */ 462 | background-color: #f5f5dc; 463 | } 464 | .org-preview-reference { 465 | } 466 | .org-query-replace { 467 | /* query-replace */ 468 | color: #b0e2ff; 469 | background-color: #cd00cd; 470 | } 471 | .org-regexp-grouping-backslash { 472 | /* font-lock-regexp-grouping-backslash */ 473 | font-weight: bold; 474 | } 475 | .org-regexp-grouping-construct { 476 | /* font-lock-regexp-grouping-construct */ 477 | font-weight: bold; 478 | } 479 | .org-region { 480 | /* region */ 481 | background-color: #eedc82; 482 | } 483 | .org-scroll-bar { 484 | } 485 | .org-secondary-selection { 486 | /* secondary-selection */ 487 | background-color: #ffff00; 488 | } 489 | .org-sgml-namespace { 490 | /* sgml-namespace */ 491 | color: #da70d6; 492 | } 493 | .org-sh-escaped-newline { 494 | /* sh-escaped-newline */ 495 | color: #a0522d; 496 | } 497 | .org-sh-heredoc { 498 | /* sh-heredoc */ 499 | color: #d2b48c; 500 | } 501 | .org-sh-quoted-exec { 502 | /* sh-quoted-exec */ 503 | color: #ff00ff; 504 | } 505 | .org-shadow { 506 | /* shadow */ 507 | color: #7f7f7f; 508 | } 509 | .org-string { 510 | /* font-lock-string-face */ 511 | color: #a0522d; 512 | } 513 | .org-tool-bar { 514 | /* tool-bar */ 515 | color: #000000; 516 | background-color: #bfbfbf; 517 | } 518 | .org-tooltip { 519 | /* tooltip */ 520 | color: #000000; 521 | background-color: #ffffe0; 522 | } 523 | .org-trailing-whitespace { 524 | /* trailing-whitespace */ 525 | background-color: #ff0000; 526 | } 527 | .org-type { 528 | /* font-lock-type-face */ 529 | color: #228b22; 530 | } 531 | .org-underline { 532 | /* underline */ 533 | text-decoration: underline; 534 | } 535 | .org-variable-name { 536 | /* font-lock-variable-name-face */ 537 | color: #b8860b; 538 | } 539 | .org-variable-pitch { 540 | } 541 | .org-vertical-border { 542 | } 543 | .org-warning { 544 | /* font-lock-warning-face */ 545 | color: #ff0000; 546 | font-weight: bold; 547 | } 548 | .org-widget-button { 549 | /* widget-button */ 550 | font-weight: bold; 551 | } 552 | .org-widget-button-pressed { 553 | /* widget-button-pressed */ 554 | color: #ff0000; 555 | } 556 | .org-widget-documentation { 557 | /* widget-documentation */ 558 | color: #006400; 559 | } 560 | .org-widget-field { 561 | /* widget-field */ 562 | background-color: #d9d9d9; 563 | } 564 | .org-widget-inactive { 565 | /* widget-inactive */ 566 | color: #7f7f7f; 567 | } 568 | .org-widget-single-line-field { 569 | /* widget-single-line-field */ 570 | background-color: #d9d9d9; 571 | } 572 | 573 | a { 574 | background-color: inherit; 575 | font: inherit; 576 | text-decoration: inherit; 577 | } 578 | a:hover { 579 | text-decoration: underline; 580 | } 581 | -------------------------------------------------------------------------------- /doc/basic-client.txt: -------------------------------------------------------------------------------- 1 | Riak Client Usage Introduction 2 | ------ 3 | 4 | This document assumes that you have already started your Riak cluster. 5 | For instructions on that prerequisite, refer to 6 | riak/doc/basic-setup.txt. 7 | 8 | Overview 9 | --- 10 | 11 | To talk to riak, all you need is an Erlang node with the Riak ebin 12 | directories in its code path. Once this shell is up, use 13 | riak:client_connect/1 to get connected. The client returned from 14 | client_connect is defined by the riak_client module, and supports the 15 | simple functions get, put, delete, and others. 16 | 17 | 18 | Starting Your Client Erlang Node 19 | --- 20 | 21 | Riak client nodes must use "long names" and have the Riak ebin 22 | directoriess in their code path. The easiest way to start a node of 23 | this nature is: 24 | 25 | $ export ERL_LIBS=$PATH_TO_RIAK/apps 26 | $ erl -name myclient@127.0.0.1 -setcookie cookie 27 | 28 | Note: If you are using a precompiled version of Riak your ERL_LIBS 29 | would be /usr/lib/riak/lib 30 | 31 | You'll know you've done this correctly if you can execute the 32 | following commands and get a path to a beam file, instead of the atom 33 | 'non_existing': 34 | 35 | (myclient@127.0.0.1)1> code:which(riak). 36 | "../riak_kv/ebin/riak.beam" 37 | 38 | Connecting 39 | --- 40 | 41 | Once you have your node running, pass your Riak server nodename 42 | to riak:client_connect/1 to connect and get a client. This can 43 | be as simple as: 44 | 45 | 3> {ok, Client} = riak:client_connect('riak@127.0.0.1'). 46 | {ok,{riak_client,'riak@127.0.0.1', <<1,112,224,226>>}} 47 | 48 | 49 | Storing New Data 50 | --- 51 | 52 | Each bit of data in Riak is stored in a "bucket" at a "key" that is 53 | unique to that bucket. The bucket is intended as an organizational 54 | aid, for example to help segregate data by type, but Riak doesn't care 55 | what values it stores, so choose whatever scheme suits you. Buckets 56 | and keys must both be binaries. 57 | 58 | Before storing your data, you must wrap it in a riak_object: 59 | 60 | 4> Object = riak_object:new(<<"groceries">>, <<"mine">>, ["eggs", "bacon"]). 61 | {r_object,<<"groceries">>,<<"mine">>, 62 | [{r_content,{dict,0,16,16,8,80,48, 63 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 64 | {{[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 65 | ["eggs","bacon"]}], 66 | [], 67 | {dict,0,16,16,8,80,48, 68 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 69 | {{[],[],[],[],[],[],[],[],[],[],[],[],[],...}}}, 70 | undefined} 71 | 72 | Then, using the client you opened earlier, store the object: 73 | 74 | 5> Client:put(Object, 1). 75 | ok 76 | 77 | If the return value of the last command was anything but the atom 78 | 'ok', then the store failed. The return value may give you a clue as 79 | to why the store failed, but check the Troubleshooting section below if 80 | not. 81 | 82 | The object is now stored in Riak, but you may be wondering about the 83 | additional parameter to Client:put. There are five different 'put' 84 | functions: put/1, put/2, put/3, put/4 and put/5. The lower-arity functions pass 85 | defaults for the parameters they leave out of the higher-arity 86 | functions. The available parameters, in order are: 87 | 88 | Object: the riak_object to store 89 | W: the minimum number of nodes that must respond 90 | with success for the write to be considered successful 91 | DW: the minimum number of nodes that must respond 92 | with success *after durably storing* the object for the 93 | write to be considered successful 94 | Timeout: the number of milliseconds to wait for W and DW responses 95 | before exiting with a timeout 96 | Options: List of options that control certain behavior. 97 | 98 | returnbody: Return the stored object. 99 | 100 | 101 | The default timeout is currently 60 seconds, and put/2 passes its W 102 | value as the DW value. So, the example above asks the client to store 103 | Object, waiting for 1 successful durable write response, waiting a 104 | maximum of 60 seconds for success. 105 | 106 | See riak/doc/architecture.txt for more information about W and DW 107 | values. 108 | 109 | 110 | Fetching Data 111 | --- 112 | 113 | At some point you'll want that data back. Using the same bucket and 114 | key you used before: 115 | 116 | 6> {ok, O} = Client:get(<<"groceries">>, <<"mine">>, 1). 117 | {ok,{r_object,<<"groceries">>,<<"mine">>, 118 | [{r_content,{dict,2,16,16,8,80,48, 119 | {[],[],[],[],[],[],[],[],[],[],[],[],...}, 120 | {{[],[],[],[],[],[], 121 | [["X-Riak-Last-Modified",87|...]], 122 | [],[],[],...}}}, 123 | ["eggs","bacon"]}], 124 | [{"20090722142711-myclient@127.0.0.1-riak@127.0.0.1-916345", 125 | {1,63415492187}}], 126 | {dict,0,16,16,8,80,48, 127 | {[],[],[],[],[],[],[],[],[],[],[],[],[],...}, 128 | {{[],[],[],[],[],[],[],[],[],[],[],...}}}, 129 | undefined}} 130 | 7> riak_object:get_value(O). 131 | ["eggs","bacon"] 132 | 133 | Like 'put', there are multiple 'get' functions: get/2, get/3, and get/4. Their 134 | parameters are: 135 | 136 | Bucket: the bucket in which the object is stored 137 | Key: the key under which the object is stored 138 | R: the minimum number of nodes that must respond 139 | with success for the read to be considered successful 140 | Timeout: the number of milliseconds to wait for R responses 141 | before exiting with a timeout 142 | 143 | So, the example 'get' above requested the "mine" object in the 144 | "groceries" bucket, demanding at least one successful response in 60 145 | seconds. 146 | 147 | 148 | Modifying Data 149 | --- 150 | 151 | Say you had the "grocery list" from the examples above, reminding you 152 | to get ["eggs","bacon"], and you want to add "milk" to it. The 153 | easiest way is: 154 | 155 | 8> {ok, Oa} = Client:get(<<"groceries">>, <<"mine">>, 1). 156 | ... 157 | 9> Ob = riak_object:update_value(Oa, ["milk"|riak_object:get_value(Oa)]). 158 | ... 159 | 10> Client:put(Ob). 160 | ok 161 | 162 | That is, fetch the object from Riak, modify its value with 163 | riak_object:update_value/2, then store the modified object back in 164 | Riak. You can get your updated object to convince yourself that your 165 | list is updated: 166 | 167 | 11> {ok, Oc} = Client:get(<<"groceries">>, <<"mine">>, 1). 168 | ... 169 | 12> riak_object:get_value(Oc). 170 | ["milk","eggs","bacon"]. 171 | 172 | 173 | Siblings 174 | --- 175 | 176 | By default, Riak does not expose siblings to clients. It is, however, 177 | important to be aware of their presence. 178 | 179 | Riak is able to provide high availability, in part, due to its ability 180 | to accept put requests, even when it can't tell if that put would 181 | overwrite data. Through the use of vector clocks Riak is able to track 182 | and resolve conflicting versions of an object. For more information 183 | about how Riak uses vclocks, see riak/doc/architecture.txt. 184 | 185 | Siblings occur when Riak accepts conflicting values for an object that 186 | cannot be resolved with vector clocks. 187 | 188 | Let's continue the example from above but this time we will tell Riak to 189 | return siblings to the client. Siblings can be allowed per bucket by 190 | setting 'allow_mult' to 'true': 191 | 192 | 13> Client:set_bucket(<<"groceries">>, [{allow_mult, true}]). 193 | ok 194 | 195 | To create a sibling, we'll fire up a new client and define our grocery 196 | list with different items: 197 | 198 | 14> {ok, Client2} = riak:client_connect('riak@127.0.0.1'). 199 | ... 200 | 15> Client2:put(riak_object:new(<<"groceries">>, <<"mine">>, ["bread","cheese"]), 1). 201 | ... 202 | 16> {ok, O2} = Client2:get(<<"groceries">>, <<"mine">>, 1). 203 | ... 204 | 17> riak_object:get_value(O2). 205 | ** exception error: no match of right hand side value 206 | [{{dict,2,16,16,8,80,48, 207 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, 208 | {{[],[],[],[],[],[],[],[],[],[], 209 | [[<<"X-Riak-VTag">>,49,54,106,80|...]], 210 | [],[], 211 | [[<<"X-Ri"...>>|{...}]], 212 | [],[]}}}, 213 | ["bread","cheese"]}, 214 | {{dict,2,16,16,8,80,48, 215 | {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, 216 | {{[],[],[],[],[],[],[],[],[],[], 217 | [[<<"X-Riak-VTag">>,50,97,53|...]], 218 | [],[], 219 | [[<<...>>|...]], 220 | [],[]}}}, 221 | ["milk","eggs","bacon"]}] 222 | in function riak_object:get_value/1 223 | 224 | Now that Riak is returning siblings it seems our get_value/1 function 225 | is broken. So, what happened? The function get_value/1 does not know 226 | how to handle siblings; to see both sets of conflicting data, use 227 | riak_object:get_values/1: 228 | 229 | 18> riak_object:get_values(O2). 230 | [["bread","cheese"],["milk","eggs","bacon"]] 231 | 232 | It is up to the client to "merge" the siblings in whatever way suits 233 | your application. In this case, we really did just want bread and 234 | cheese in the list, so: 235 | 236 | 19> O3 = riak_object:update_value(O2, ["bread","cheese"]). 237 | ... 238 | 20> Client2:put(O3, 1). 239 | ok 240 | 241 | Now, we can fetch the grocery list and check its value: 242 | 243 | 21> {ok, O4} = Client2:get(<<"groceries">>, <<"mine">>, 1). 244 | ... 245 | 22> riak_object:get_value(O4). 246 | ["bread","cheese"] 247 | 248 | To return to the default behavior we can set 'allow_mult' back to false: 249 | 250 | 23> Client2:set_bucket(<<"groceries">>, [{allow_mult, false}]). 251 | 252 | 253 | Listing Keys 254 | --- 255 | 256 | Most uses of key-value stores are structured in such a way that 257 | requests know which keys they want in a bucket. Sometimes, though, 258 | it's necessary to find out what keys are available (when debugging, 259 | for example). For that, there is list_keys: 260 | 261 | 1> Client:list_keys(<<"groceries">>). 262 | {ok, ["mine"]}. 263 | 264 | Note that keylist updates are asynchronous to the object storage 265 | primitives, and may not be updated immediately after a put or delete. 266 | This function is primarily intended as a debugging aid. 267 | 268 | Deleting Data 269 | --- 270 | 271 | Throwing away data is quick and simple: just use the delete function 272 | in the riak_client module: 273 | 274 | 1> Client:delete(<<"groceries">>, <<"mine">>, 1). 275 | ok 276 | 277 | As with get, delete has arity-2, arity-3 and arity-4 functions, with 278 | parameters: 279 | 280 | Bucket: the bucket the object is in 281 | Key: the key to delete 282 | RW: the number of nodes to wait for responses from 283 | Timeout: the number of milliseconds to wait for responses 284 | 285 | So, the command demonstrated above tries to delete the object with the 286 | key "mine" in the bucket "groceries", waiting up to 60 seconds for at 287 | least one node to respond. 288 | 289 | Issuing a delete for an object that does not exist returns an error 290 | tuple. For example, calling the same delete as above a second time: 291 | 292 | 2> Client:delete(<<"groceries">>, <<"mine">>, 1). 293 | {error,notfound} 294 | 295 | 296 | Bucket Properties 297 | --- 298 | 299 | As seen in the examples above, simply storing a key/value in a bucket 300 | causes the bucket to come into existence with some default parameters. 301 | To view the settings for a bucket, use the get_bucket function in the 302 | riak_client module: 303 | 304 | 1> Client:get_bucket(<<"groceries">>). 305 | [{name,<<"groceries">>}, 306 | {n_val,3}, 307 | {allow_mult,false}, 308 | {last_write_wins,false}, 309 | {precommit,[]}, 310 | {postcommit,[]}, 311 | {chash_keyfun,{riak_core_util,chash_std_keyfun}}, 312 | {linkfun,{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}}, 313 | {old_vclock,86400}, 314 | {young_vclock,20}, 315 | {big_vclock,50}, 316 | {small_vclock,10}, 317 | {r,quorum}, 318 | {w,quorum}, 319 | {dw,quorum}, 320 | {rw,quorum}] 321 | 322 | If the default parameters do not suit your application, you can alter 323 | them on a per-bucket basis with set_bucket. You should do this before 324 | storing any data in the bucket, but many of the settings will "just 325 | work" if they are modified after the bucket contains data. 326 | 327 | An interesting bucket setting to take note of is 'n_val'. n_val tells 328 | Riak how many copies of an object to make. More properly, for a bucket 329 | with an n_val of N, Riak will store each object in the bucket in N 330 | different vnodes (see riak/doc/architecture.txt for a description of 331 | the term "vnode"). 332 | 333 | Most of the time, the default n_val of 3 works perfectly. Three 334 | provides some durability and availability over 1 or 2. 335 | 336 | If you are attempting to enhance either durability or speed, however, 337 | increasing n_val may make sense *if you also modify the R, W, and DW* 338 | values in your get and put calls. Changing n_val on its own will have 339 | little effect without also modifying the put- and get-time parameters. 340 | 341 | 342 | Troubleshooting 343 | --- 344 | 345 | {nodedown, ...} 346 | - 347 | 348 | If all of your riak_client calls are exiting with exceptions that 349 | describe themselves as, roughly: 350 | 351 | ** exception exit: {{nodedown,'riak@127.0.0.1'}, 352 | {gen_server,call, 353 | [{riak_api,'riak@127.0.0.1'}, 354 | {get,groceries,"mine",1,15000}, 355 | 15000]}} 356 | in function gen_server:call/3 357 | 358 | The node that your client was connected to is down. Try connecting a 359 | new client. 360 | 361 | 362 | riak:client_connect/1 returns {error,timeout} 363 | - 364 | 365 | The Riak node you are connecting to is down. 366 | 367 | 368 | {error,notfound} 369 | - 370 | 371 | The bucket/key combination you requested was not found. 372 | -------------------------------------------------------------------------------- /doc/js-mapreduce.org: -------------------------------------------------------------------------------- 1 | #+SETUPFILE: "basho-doc-style.iorg" 2 | #+TITLE: Using Javascript with Riak Map/Reduce 3 | 4 | Riak supports writing map/reduce query functions in Javascript, as 5 | well as specifying query execution over HTTP. This document will 6 | teach you how to use these features. 7 | 8 | * Simple Example 9 | 10 | This section hits the ground running with a quick example to 11 | demonstrate what HTTP/Javascript map/reduce looks like in Riak. 12 | This example will store several chunks of text in Riak, and then 13 | compute a word counts on the set of documents. 14 | 15 | ** Load data 16 | 17 | We will use the Riak HTTP interface to store the texts we want to 18 | process: 19 | 20 | #+BEGIN_EXAMPLE 21 | $ curl -X PUT -H "content-type: text/plain" \ 22 | http://localhost:8098/riak/alice/p1 --data-binary @- 23 | Alice was beginning to get very tired of sitting by her sister on the 24 | bank, and of having nothing to do: once or twice she had peeped into the 25 | book her sister was reading, but it had no pictures or conversations in 26 | it, 'and what is the use of a book,' thought Alice 'without pictures or 27 | conversation?' 28 | ^D 29 | $ curl -X PUT -H "content-type: text/plain" \ 30 | http://localhost:8098/riak/alice/p2 --data-binary @- 31 | So she was considering in her own mind (as well as she could, for the 32 | hot day made her feel very sleepy and stupid), whether the pleasure 33 | of making a daisy-chain would be worth the trouble of getting up and 34 | picking the daisies, when suddenly a White Rabbit with pink eyes ran 35 | close by her. 36 | ^D 37 | $ curl -X PUT -H "content-type: text/plain" \ 38 | http://localhost:8098/riak/alice/p5 --data-binary @- 39 | The rabbit-hole went straight on like a tunnel for some way, and then 40 | dipped suddenly down, so suddenly that Alice had not a moment to think 41 | about stopping herself before she found herself falling down a very deep 42 | well. 43 | #+END_EXAMPLE 44 | 45 | ** Run query 46 | 47 | With data loaded, we can now run a query: 48 | 49 | #+BEGIN_EXAMPLE 50 | $ curl -X POST -H "content-type: application/json" http://localhost:8098/mapred --data @- 51 | {"inputs":[["alice","p1"],["alice","p2"],["alice","p5"]],"query":[{"map":{"language":"javascript","source":"function(v) { var m = v.values[0].data.toLowerCase().match('\\\\w*','g'); var r = []; for(var i in m) if (m[i] != '') { var o = {}; o[m[i]] = 1; r.push(o); } return r; }"}},{"reduce":{"language":"javascript","source":"function(v) { var r = {}; for (var i in v) { for(var w in v[i]) { if (w in r) r[w] += v[i][w]; else r[w] = v[i][w]; } } return [r]; }"}}]} 52 | ^D 53 | #+END_EXAMPLE 54 | 55 | And we end up with the word counts for the three documents. 56 | 57 | #+BEGIN_EXAMPLE 58 | [{"the":8,"rabbit":2,"hole":1,"went":1,"straight":1,"on":2,"like":1,"a":6,"tunnel":1,"for":2,"some":1,"way":1,"and":5,"then":1,"dipped":1,"suddenly":3,"down":2,"so":2,"that":1,"alice":3,"had":3,"not":1,"moment":1,"to":3,"think":1,"about":1,"stopping":1,"herself":2,"before":1,"she":4,"found":1,"falling":1,"very":3,"deep":1,"well":2,"was":3,"considering":1,"in":2,"her":5,"own":1,"mind":1,"as":2,"could":1,"hot":1,"day":1,"made":1,"feel":1,"sleepy":1,"stupid":1,"whether":1,"pleasure":1,"of":5,"making":1,"daisy":1,"chain":1,"would":1,"be":1,"worth":1,"trouble":1,"getting":1,"up":1,"picking":1,"daisies":1,"when":1,"white":1,"with":1,"pink":1,"eyes":1,"ran":1,"close":1,"by":2,"beginning":1,"get":1,"tired":1,"sitting":1,"sister":2,"bank":1,"having":1,"nothing":1,"do":1,"once":1,"or":3,"twice":1,"peeped":1,"into":1,"book":2,"reading":1,"but":1,"it":2,"no":1,"pictures":2,"conversations":1,"what":1,"is":1,"use":1,"thought":1,"without":1,"conversation":1}] 59 | #+END_EXAMPLE 60 | 61 | ** Explanation 62 | 63 | For more details about what each bit of syntax means, and other 64 | syntax options, read the following sections. As a quick 65 | explanation of how this example map/reduce query worked, though: 66 | 67 | 1. The objects named =p1=, =p2=, and =p5= from the =alice= bucket 68 | were given as inputs to the query. 69 | 70 | 2. The map function from the phase was run on each object. The 71 | function: 72 | 73 | #+BEGIN_SRC javascript 74 | function(v) { 75 | var m = v.values[0].data.match('\\w*','g'); 76 | var r = []; 77 | for(var i in m) 78 | if (m[i] != '') { 79 | var o = {}; 80 | o[m[i]] = 1; 81 | r.push(o); 82 | } 83 | return r; 84 | } 85 | #+END_SRC 86 | 87 | creates a list of JSON objects, one for each word (non-unique) 88 | in the text. The object has as a key, the word, and as the 89 | value for that key, the integer 1. 90 | 91 | 3. The reduce function from the phase was run on the outputs of the 92 | map functions. The function: 93 | 94 | #+BEGIN_SRC javascript 95 | function(v) { 96 | var r = {}; 97 | for (var i in v) { 98 | for(var w in v[i]) { 99 | if (w in r) 100 | r[w] += v[i][w]; 101 | else 102 | r[w] = v[i][w]; 103 | } 104 | } 105 | return [r]; 106 | } 107 | #+END_SRC 108 | 109 | looks at each JSON object in the input list. It steps through 110 | each key in each object, and produces a new object. That new 111 | object has a key for each key in every other object, the value 112 | of that key being the sum of the values of that key in the other 113 | objects. It returns this new object in a list, because it may 114 | be run a second time on a list including that object and more 115 | inputs from the map phase. 116 | 117 | 4. The final output is a list with one element: a JSON object with 118 | a key for each word in all of the documents (unique), with the 119 | value of that key being the number of times the word appeared in 120 | the documents. 121 | 122 | * Query Syntax 123 | 124 | Map/Reduce queries are issued over HTTP via a POST to the /mapred 125 | resource. The body should be =application/json= of the form 126 | ={"inputs":[...inputs...],"query":[...query...]}=. 127 | 128 | Map/Reduce queries have a default timeout of 60000 milliseconds 129 | (60 seconds). The default timeout can be overridden by supplying 130 | a different value, in milliseconds, in the JSON document 131 | ={"inputs":[...inputs...],"query":[...query...],"timeout": 90000}= 132 | 133 | ** Inputs 134 | 135 | The list of input objects is given as a list of 2-element lists of 136 | the form =[Bucket,Key]= or 3-element lists of the form 137 | =[Bucket,Key,KeyData]=. 138 | 139 | You may also pass just the name of a bucket 140 | (={"inputs":"mybucket",...}=), which is equivalent to passing all 141 | of the keys in that bucket as inputs (i.e. "a map/reduce across the 142 | whole bucket"). You should be aware that this triggers the 143 | somewhat expensive "list keys" operation, so you should use it 144 | sparingly. 145 | 146 | ** Query 147 | 148 | The query is given as a list of phases, each phase being of the 149 | form ={PhaseType:{...spec...}}=. Valid =PhaseType= values are 150 | "map", "reduce", and "link". 151 | 152 | Every phase spec may include a =keep= field, which must have a 153 | boolean value: =true= means that the results of this phase should 154 | be included in the final result of the map/reduce, =false= means 155 | the results of this phase should be used only by the next phase. 156 | Omitting the =keep= field accepts its default value, which is 157 | =false= for all phases except the final phase (Riak assumes that 158 | you were most interested in the results of the last phase of your 159 | map/reduce query). 160 | 161 | *** Map 162 | 163 | Map phases must be told where to find the code for the function to 164 | execute, and what language that function is in. 165 | 166 | Function source can be specified directly in the query by using 167 | the "source" spec field. Function source can also be loaded from 168 | a pre-stored riak object by providing "bucket" and "key" fields in 169 | the spec. 170 | 171 | For example: 172 | 173 | :{"map":{"language":"javascript","source":"function(v) { return [v]; }","keep":true}} 174 | 175 | would run the Javascript function given in the spec, and include 176 | the results in the final output of the m/r query. 177 | 178 | :{"map":{"language":"javascript","bucket":"myjs","key":"mymap","keep":false}} 179 | 180 | would run the Javascript function declared in the content of the 181 | Riak object under =mymap= in the =myjs= bucket, and the results of 182 | the funciton would not be included in the final output of the m/r 183 | query. 184 | 185 | Map phases may also be passed static arguments by using the "arg" 186 | spec field. 187 | 188 | *** Reduce 189 | 190 | Reduce phases look exactly like map phases, but are labeled "reduce". 191 | 192 | *** Link 193 | 194 | Link phases accept =bucket= and =tag= fields that specify which 195 | links match the link query. The string "_" (underscore) in each 196 | field means "match all", while any other string means "match 197 | exactly this string". If either field is left out, it is 198 | considered to be set to "_" (match all). 199 | 200 | For example: 201 | 202 | :{"link":{"bucket":"foo","keep":false}} 203 | 204 | Would follow all links pointing to objects in the =foo= bucket, 205 | regardless of their tag. 206 | 207 | * Javascript Functions 208 | ** Function Parameters 209 | *** Map functions 210 | 211 | Map functions are passed three parameters: the object that the map 212 | is being applied to, the "keydata" for that object, and the static 213 | argument for the phase. 214 | 215 | The object will be a JSON object of the form: 216 | 217 | #+BEGIN_EXAMPLE 218 | { 219 | "bucket":BucketAsString, 220 | "key":KeyAsString, 221 | "vclock":VclockAsString, 222 | "values":[ 223 | { 224 | "metadata":{ 225 | "X-Riak-VTag":VtagAsString, 226 | "X-riak-Last-Modified":LastModAsString, 227 | ...other metadata... 228 | }, 229 | "data":ObjectData 230 | }, 231 | ...other metadata/data values (siblings)... 232 | ] 233 | } 234 | #+END_EXAMPLE 235 | 236 | =object.values[0].data= is probably what you will be interested in 237 | most of the time, but the rest of the details of the object are 238 | provided for your use. 239 | 240 | The "keydata" is the third element of the item from the input 241 | bucket/key list (called =KeyData= in the [[Inputs]] section above), or 242 | "undefined" if none was provided. 243 | 244 | The static argument for the phase is the value of the =arg= field 245 | from the map spec in the query list. 246 | 247 | A map phase should produce a list of results. You will see errors 248 | if the output of your map function is not a list. Return the 249 | empty list if your map function chooses not to produce output. 250 | *** Reduce functions 251 | 252 | Reduce functions are passed two parameters: a list of inputs to 253 | reduce, and the static argument for the phase. 254 | 255 | The list of inputs to reduce may contain values from previous 256 | executions of the reduce function. It will also contain results 257 | produced by the preceding map or reduce phase. 258 | 259 | The static argument for the phase is the value of the =arg= field 260 | from the reduce spec in the query list. 261 | 262 | A reduce phase should produce a list of results. You will see 263 | errors if the output of your reduce function is not a list. The 264 | function should return an empty list, if it has no other output to 265 | produce. 266 | 267 | *** Link functions 268 | 269 | If you are storing data through the HTTP interface, and using the 270 | =Link= HTTP header, you do not need to worry about writing a 271 | link-extraction function. Just use the predefined 272 | =raw_link_walker_resource:mapreduce_linkfun/3=. 273 | 274 | But, if you need to extract links from your data in some other 275 | manner, there are many ways to specify Javascript functions to do 276 | that. They all start with setting the =linkfun= bucket property. 277 | Through the HTTP interface: 278 | 279 | :$ curl -X PUT -H "application/json" http://localhost:8098/riak/bucket \ 280 | :> --data "{\"props\":{\"linkfun\":{...function...}}}" 281 | 282 | The three ways to fill in the value of the =linkfun= key are: 283 | 284 | + Quoted source code, as the value of the =jsanon= key: 285 | 286 | :{"jsanon":"function(v,kd,bt) { return []; }"} 287 | 288 | + The bucket and key of an object containing the function source: 289 | 290 | :{"jsanon":{"bucket":Bucket,"key":Key}} 291 | 292 | + The name of a predefined Javascript function: 293 | 294 | :{"jsfun":FunctionName} 295 | 296 | The function has basically the same contract as a map function. 297 | The first argument is the object from which links should be 298 | extracted. The second argument is the =KeyData= for the object. 299 | 300 | The third argument is a Javascript object representing the links 301 | to match at return. The two fields in the object, =bucket= and 302 | =tag=, will have the values given in the link phase spec from the 303 | query. 304 | 305 | The link fun should return a list of the same form as the =inputs= 306 | list: 2-item bucket/key lists, or 3-item bucket/key/keydata lists. 307 | 308 | * How Map/Reduce Queries Work 309 | 310 | ** Map/Reduce Intro 311 | 312 | The main goal of Map/Reduce is to spread the processing of a query 313 | across many systems to take advantage of parallel processing power. 314 | This is generally done by dividing the query into several steps, 315 | dividing the dataset into several chunks, and then running those 316 | step/chunk pairs in separate physical hosts. 317 | 318 | One step type is called "map". Map functions take one piece of 319 | data as input, and produce zero or more results as output. If 320 | you're familiar with "mapping over a list" in functional 321 | programming style, you're already familiar with "map" steps in a 322 | map/reduce query. 323 | 324 | Another step type is called "reduce". The purpose of a "reduce" 325 | step is to combine the output of many "map" step evaluations, into 326 | one result. 327 | 328 | The common example of a map/reduce query involves a "map" step that 329 | takes a body of text as input, and produces a word count for that 330 | body of text. A reduce step then takes the word counts produced 331 | from many bodies of text and either sums them to provide a word 332 | count for the corpus, or filters them to produce a list of 333 | documents containing only certain counts. 334 | 335 | ** Riak-specific Map/Reduce 336 | 337 | *** How Riak Spreads Processing 338 | 339 | Riak's map/reduce has an additional goal: increasing data-locality. 340 | When processing a large dataset, it's often much more efficient to 341 | take the computation to the data than it is to bring the data to 342 | the computation. 343 | 344 | It is Riak's solution to the data-locality problem that determines 345 | how Riak spreads the processing across the cluster. In the same 346 | way that any Riak node can coordinate a read or write by sending 347 | requests directly to the other nodes responsible for maintaining 348 | that data, any Riak node can also coordinate a map/reduce query by 349 | sending a map-step evaluation request directly to the node 350 | responsible for maintaining the input data. Map-step results are 351 | sent back to the coordinating node, where reduce-step processing 352 | can produce a unified result. 353 | 354 | Put more simply: Riak runs map-step functions right on the node 355 | holding the input data for those functions, and it runs reduce-step 356 | functions on the node coordinating the map/reduce query. 357 | 358 | *** How Riak's Map/Reduce Queries Are Specified 359 | 360 | Map/Reduce queries in Riak have two components: a list of inputs 361 | and a list of "steps", or "phases". 362 | 363 | Each element of the input list is a bucket-key pair. This 364 | bucket-key pair may also be annotated with "key-data", which will 365 | be passed as an argument to a map function, when evaluated on the 366 | object stored under that bucket-key pair. 367 | 368 | Each element of the phases list is a description of a map 369 | function, a reduce function, or a link function. The description 370 | includes where to find the code for the phase function (for map 371 | and reduce phases), static data passed to the function every time 372 | it is executed during that phase, and a flag indicating whether or 373 | not to include the results of that phase in the final output of 374 | the query. 375 | 376 | The phase list describes the chain of operations each input will 377 | flow through. That is, the initial inputs will be fed to the 378 | first phase in the list, and the output of that phase will be fed 379 | as input to the next phase in the list. This stream will continue 380 | through the final phase. 381 | 382 | *** How a Map Phase Works in Riak 383 | 384 | The input list to a map phase must be a list of (possibly 385 | annotated) bucket-key pairs. For each pair, Riak will send the 386 | request to evaluate the map function to the partition that is 387 | responsible for storing the data for that bucket-key. The vnode 388 | hosting that partition will lookup the object stored under that 389 | bucket-key, and evaluate the map function with the object as an 390 | argument. The other arguments to the function will be the 391 | annotation, if any is included, with the bucket-key, and the 392 | static data for the phase, as specified in the query. 393 | 394 | *** How a Reduce Phase Works in Riak 395 | 396 | Reduce phases accept any list of data as input, and produce any 397 | list of data as output. They also receive a phase-static value, 398 | specified in the query definition. 399 | 400 | The important thing to understand is that the function defining 401 | the reduce phase may be evaluated multiple times, and the input of 402 | later evaluations will include the input of earlier evaluations. 403 | 404 | For example, a reduce phase may implement the "set-union" 405 | function. In that case, the first set of inputs might be 406 | =[1,2,2,3]=, and the output would be =[1,2,3]=. When the phase 407 | receives more inputs, say =[3,4,5]=, the function will be called 408 | with the concatentation of the two lists: =[1,2,3,3,4,5]=. 409 | 410 | Other systems refer to the second application of the reduce 411 | function as a "re-reduce". There are at least a couple of 412 | reduce-query implementation strategies that work with Riak's model. 413 | 414 | One strategy is to implement the phase preceeding the reduce 415 | phase, such that its output is "the same shape" as the output of 416 | the reduce phase. This is how the examples in this document are 417 | written, and the way that we have found produces cleaner code. 418 | 419 | An alternate strategy is to make the output of a reduce phase 420 | recognizable, such that it can be extracted from the input list on 421 | subsequent applications. For example, if inputs from the 422 | preceeding phase are numbers, outputs from the reduce phase could 423 | be objects or strings. This would allow the function to find the 424 | previous result, and apply new inputs to it. 425 | 426 | *** How a Link Phase Works in Riak 427 | 428 | Link phases find links matching patterns specified in the query 429 | definition. The patterns specify which buckets and tags links 430 | must have. 431 | 432 | "Following a link" means adding it to the output list of this 433 | phase. The output of this phase is often most useful as input to 434 | a map phase, or another reduce phase. 435 | 436 | *** Using Named Functions 437 | 438 | Riak can also use pre-defined named functions for map and reduce 439 | phase processing. Named functions are invoked with the following 440 | form: 441 | 442 | #+BEGIN_EXAMPLE 443 | {"map": {"language": "javascript", "name": "Riak.mapValues", "keep": true}} 444 | 445 | {"reduce": {"language": "javascript", "name": "Riak.reduceSort", "keep": true}} 446 | #+END_EXAMPLE 447 | 448 | The key =name= in both examples points to the name of the function 449 | to be used. Riak expects the function to be defined prior to the 450 | execution of the phase using it. 451 | 452 | **** Defining Named Functions 453 | 454 | Defining a named function for Riak is a simple process. 455 | 456 | 1. Create a Javascript source file containing the definitions for 457 | all the functions you would like Riak to pre-define. 458 | 2. Edit the =app.config= of your Riak nodes and add the line 459 | ={js_source_dir, }= to the =riak= 460 | configuration block. == should point to 461 | the directory where the file created in step #1 was saved. 462 | 3. Start using the functions in your map/reduce jobs. 463 | 464 | When =js_source_dir= is enabled, Riak scans the directory for 465 | files ending in =.js=. These files are then loaded into each 466 | Javascript VM when it is created. 467 | 468 | NOTE: Named functions must be available on all nodes in a cluster 469 | for proper map/reduce results. 470 | 471 | **** Why Use Named Functions? 472 | 473 | Named functions can be better than anonymous functions in certain 474 | situations. Since named functions live in a file they can be 475 | managed using source code control and deployed automatically 476 | using tools such as Chef or Puppet. This can be a significant 477 | advantage when administrating large Riak clusters. 478 | 479 | More important, though, is the fact that named functions execute 480 | much faster than the equivalent anonymous functions. Invoking 481 | anonymous functions requires Riak to ensure the anonymous 482 | function is defined before invoking it. Named functions allow 483 | Riak to skip the definition check and execute the function call 484 | immediately. 485 | 486 | Also, since named functions do not change between invocations, 487 | Riak is able to cache named function call results and short 488 | circuit the call entirely. Currently, Riak performs this 489 | optimization on named functions executed during map phases only. 490 | 491 | In general, anonymous functions should be used during development 492 | and named functions should be used for production deployments 493 | where possible. This combination provides the maximum flexibility 494 | and performance. 495 | 496 | **** Riak-Supplied Functions 497 | 498 | Riak supplies several named functions out of the box. These 499 | functions are defined on a global Javascript object named =Riak= 500 | and should not be modified or overridden. These functions, along 501 | with descriptions and notes on their use are described in the 502 | next two sections. 503 | 504 | ***** Named Map Functions 505 | 506 | + =Riak.mapValues(values, keyData, arg)= 507 | Extracts and returns only the values contained in a bucket and key. 508 | 509 | + =Riak.mapValuesJson(values, keyData, arg)= 510 | Same as =mapValues= except the values are passed through a JSON 511 | decoder first. 512 | 513 | ***** Named Reduce Functions 514 | 515 | + =Riak.reduceSum(values, arg)= 516 | Returns the sum of =values= 517 | 518 | + =Riak.reduceMin(values, arg)= 519 | Returns the minimum value from =values= 520 | 521 | + =Riak.reduceMax(values, arg)= 522 | Returns the maximum value from =values= 523 | 524 | + =Riak.reduceSort(values, arg)= 525 | Returns the sorted version of =values=. If =arg= is the source 526 | to a Javascript function, it will be eval'd and used to 527 | control the sort via =Array.sort=. 528 | 529 | + =Riak.reduceLimit(values, arg)= 530 | Returns the leftmost n members of values where =arg= is used as n. 531 | 532 | + =Riak.reduceSlice(values, arg)= 533 | Returns a slice of the values array. =arg= must be a two 534 | element array containing the starting and ending positions for 535 | the slice. 536 | --------------------------------------------------------------------------------