├── .gitignore ├── .travis.yml ├── CHANGELOG.md ├── LICENSE ├── README.md ├── config ├── config.exs ├── dev.exs └── test.exs ├── lib ├── kvx.ex └── kvx │ ├── adapters │ └── ex_shards │ │ └── bucket_shards.ex │ ├── bucket.ex │ └── exceptions.ex ├── mix.exs ├── mix.lock └── test ├── bucket_shards_test.exs └── test_helper.exs /.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc 12 | 13 | # If the VM crashes, it generates a dump, let's ignore it too. 14 | erl_crash.dump 15 | 16 | # Also ignore archive artifacts (built via "mix archive.build"). 17 | *.ez 18 | 19 | # Others 20 | *.o 21 | *.beam 22 | *.plt 23 | erl_crash.dump 24 | .DS_Store 25 | ._* 26 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: elixir 2 | elixir: 3 | - 1.4.0 4 | - 1.3.2 5 | otp_release: 6 | - 19.2 7 | - 18.3 8 | sudo: false 9 | before_script: 10 | - mix deps.get --only test 11 | script: 12 | - mix test 13 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Change Log 2 | 3 | ## [v0.1.3](https://github.com/cabol/kvx/tree/v0.1.3) (2017-02-14) 4 | [Full Changelog](https://github.com/cabol/kvx/compare/v0.1.2...v0.1.3) 5 | 6 | **Closed issues:** 7 | 8 | - Add travis-ci support [\#6](https://github.com/cabol/kvx/issues/6) 9 | - Migrate from `shards` to `ex\_shards` [\#5](https://github.com/cabol/kvx/issues/5) 10 | - Add specs to public/API functions [\#2](https://github.com/cabol/kvx/issues/2) 11 | 12 | ## [v0.1.2](https://github.com/cabol/kvx/tree/v0.1.2) (2016-12-04) 13 | [Full Changelog](https://github.com/cabol/kvx/compare/v0.1.1...v0.1.2) 14 | 15 | ## [v0.1.1](https://github.com/cabol/kvx/tree/v0.1.1) (2016-09-09) 16 | [Full Changelog](https://github.com/cabol/kvx/compare/v0.1.0...v0.1.1) 17 | 18 | ## [v0.1.0](https://github.com/cabol/kvx/tree/v0.1.0) (2016-09-06) 19 | 20 | 21 | \* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)* -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2016 Carlos Andres Bolaños R.A. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # KVX [![Build Status](https://travis-ci.org/cabol/kvx.svg?branch=master)](https://travis-ci.org/cabol/kvx) 2 | 3 | This is a simple/basic in-memory Key/Value Store written in [**Elixir**](http://elixir-lang.org/) 4 | and using [**ExShards**](https://github.com/cabol/ex_shards) as default adapter. 5 | 6 | Again, **KVX** is a simple library, most of the work is done by **ExShards**, and 7 | its typical use case might be as a **Cache**. 8 | 9 | ## Usage 10 | 11 | Add `kvx` to your Mix dependencies: 12 | 13 | ```elixir 14 | defp deps do 15 | [{:kvx, "~> 0.1"}] 16 | end 17 | ``` 18 | 19 | In an existing or new module: 20 | 21 | ```elixir 22 | defmodule MyTestMod do 23 | use KVX.Bucket 24 | end 25 | ``` 26 | 27 | ## Getting Started! 28 | 29 | Let's try it out, compile your project and start an interactive console: 30 | 31 | ``` 32 | $ mix deps.get 33 | $ mix compile 34 | $ iex -S mix 35 | ``` 36 | 37 | Now let's play with `kvx`: 38 | 39 | ```elixir 40 | > MyTestMod.new(:mybucket) 41 | :mybucket 42 | 43 | > MyTestMod.set(:mybucket, :fruit, "banana") 44 | :mybucket 45 | 46 | > MyTestMod.mset(:mybucket, male_users: 200, female_users: 150) 47 | :mybucket 48 | 49 | > MyTestMod.get(:mybucket, :female_users) 50 | 150 51 | 52 | > MyTestMod.mget(:mybucket, [:male_users, :female_users]) 53 | [200, 150] 54 | 55 | > MyTestMod.find_all(:mybucket) 56 | [fruit: "banana", male_users: 200, female_users: 150] 57 | 58 | > MyTestMod.delete(:mybucket, :male_users) 59 | :mybucket 60 | 61 | > MyTestMod.get(:mybucket, :male_users) 62 | nil 63 | 64 | > MyTestMod.flush!(:mybucket) 65 | :mybucket 66 | 67 | > MyTestMod.find_all(:mybucket) 68 | [] 69 | ``` 70 | 71 | ## Configuration 72 | 73 | Most of the configuration that goes into the `config` is specific to the adapter. 74 | But there are some common/shared options such as: `:adapter` and `:ttl`. E.g.: 75 | 76 | ```elixir 77 | config :kvx, 78 | adapter: KVX.Bucket.ExShards, 79 | ttl: 1 # the ttl in seconds 80 | ``` 81 | 82 | Now, in case of the adapter `KVX.Bucket.ExShards`, it has some extra options 83 | like `module`. E.g.: 84 | 85 | ```elixir 86 | config :kvx, 87 | adapter: KVX.Bucket.ExShards, 88 | ttl: 1, # the ttl in seconds 89 | module: ExShards.Local 90 | ``` 91 | 92 | Besides, you can define bucket options in the config: 93 | 94 | ```elixir 95 | config :kvx, 96 | adapter: KVX.Bucket.ExShards, 97 | ttl: 43200, # the ttl in seconds 98 | module: ExShards, 99 | buckets: [ 100 | mybucket1: [ 101 | n_shards: 4 102 | ], 103 | mybucket2: [ 104 | n_shards: 8 105 | ] 106 | ] 107 | ``` 108 | 109 | In case of **ExShards** adapter, run-time options when calling `new/2` function, are 110 | the same as `ExShards.new/2`. E.g.: 111 | 112 | ```elixir 113 | MyModule.new(:mybucket, [n_shards: 4]) 114 | ``` 115 | 116 | > **NOTE:** For more information check [KVX.Bucket.ExShards](./lib/kvx/adapters/ex_shards/bucket_shards.ex). 117 | 118 | ## Running Tests 119 | 120 | ``` 121 | $ mix test 122 | ``` 123 | 124 | ### Coverage 125 | 126 | ``` 127 | $ mix coveralls 128 | ``` 129 | 130 | > **NOTE:** For more coverage options check [**excoveralls**](https://github.com/parroty/excoveralls). 131 | 132 | ## Example 133 | 134 | As we mentioned before, one of the most typical use case might be 135 | use **KVX** as a **Cache**. Now, let's suppose you're working with 136 | [**Ecto**](https://github.com/elixir-ecto/ecto), and you want to be 137 | able to cache data when you call `Ecto.Repo.get/3`, and on other hand, 138 | be able to handle eviction, remove/update cached data when they 139 | change or mutate – typically when you call `Ecto.Repo.insert/2`, 140 | `Ecto.Repo.update/2`, etc. 141 | 142 | To do so, let's implement our own `CacheableRepo` to encapsulate 143 | data access and caching logic. First let's create our bucket and 144 | the `Ecto.Repo` in two separated modules: 145 | 146 | ```elixir 147 | defmodule MyApp.Bucket do 148 | use KVX.Bucket 149 | end 150 | 151 | defmodule MyApp.Repo do 152 | use Ecto.Repo, otp_app: :myapp 153 | end 154 | ``` 155 | 156 | Now, let's code our `CacheableRepo`, re-implementing some `Ecto.Repo` 157 | functions but adding caching. It is as simple as this: 158 | 159 | ```elixir 160 | defmodule MyApp.CacheableRepo do 161 | alias MyApp.Repo 162 | alias MyApp.Bucket 163 | 164 | require Logger 165 | 166 | def get(queryable, id, opts \\ []) do 167 | get(&Repo.get/3, queryable, id, opts) 168 | end 169 | 170 | def get!(queryable, id, opts \\ []) do 171 | get(&Repo.get!/3, queryable, id, opts) 172 | end 173 | 174 | def get_by(queryable, clauses, opts \\ []) do 175 | get(&Repo.get_by/3, queryable, clauses, opts) 176 | end 177 | 178 | def get_by!(queryable, clauses, opts \\ []) do 179 | get(&Repo.get_by!/3, queryable, clauses, opts) 180 | end 181 | 182 | defp get(fun, queryable, key, opts) do 183 | b = bucket(queryable) 184 | case Bucket.get(b, key) do 185 | nil -> 186 | value = fun.(queryable, key, opts) 187 | if value != nil do 188 | Logger.debug "CACHING : #{inspect key} => #{inspect value}" 189 | Bucket.set(b, key, value) 190 | end 191 | value 192 | value -> 193 | Logger.debug "CACHED : #{inspect key} => #{inspect value}" 194 | value 195 | end 196 | end 197 | 198 | def insert(struct, opts \\ []) do 199 | case Repo.insert(struct, opts) do 200 | {:ok, schema} = rs -> 201 | schema 202 | |> bucket 203 | |> Bucket.delete(schema.id) 204 | rs 205 | error -> 206 | error 207 | end 208 | end 209 | 210 | def insert!(struct, opts \\ []) do 211 | rs = Repo.insert!(struct, opts) 212 | rs 213 | |> bucket 214 | |> Bucket.delete(rs.id) 215 | rs 216 | end 217 | 218 | def update(struct, opts \\ []) do 219 | case Repo.update(struct, opts) do 220 | {:ok, schema} = rs -> 221 | schema 222 | |> bucket 223 | |> Bucket.set(schema.id, schema) 224 | rs 225 | error -> 226 | error 227 | end 228 | end 229 | 230 | def update!(struct, opts \\ []) do 231 | rs = Repo.update!(struct, opts) 232 | rs 233 | |> bucket 234 | |> Bucket.set(rs.id, rs) 235 | rs 236 | end 237 | 238 | def delete(struct, opts \\ []) do 239 | case Repo.delete(struct, opts) do 240 | {:ok, schema} = rs -> 241 | schema 242 | |> bucket 243 | |> Bucket.delete(schema.id) 244 | rs 245 | error -> 246 | error 247 | end 248 | end 249 | 250 | def delete!(struct, opts \\ []) do 251 | rs = Repo.delete!(struct, opts) 252 | rs 253 | |> bucket 254 | |> Bucket.delete(rs.id) 255 | rs 256 | end 257 | 258 | # function to resolve what bucket depending on the given schema 259 | defp bucket(%{__struct__: struct}), do: Bucket.new(struct) 260 | defp bucket(struct) when is_atom(struct), do: Bucket.new(struct) 261 | defp bucket(_), do: Bucket.new(:default) 262 | end 263 | ``` 264 | 265 | Now that we have our `CacheableRepo`, it can be used instead of `Ecto.Repo` 266 | (since it is a wrapper on top of it, but it adds caching) for data you 267 | consider can be cached, for example, you can use it from your 268 | **Phoenix Controllers** – in case you're using [Phoenix](http://www.phoenixframework.org/). 269 | 270 | ## Copyright and License 271 | 272 | Copyright (c) 2016 Carlos Andres Bolaños R.A. 273 | 274 | **KVX** source code is licensed under the [**MIT License**](LICENSE.md). 275 | -------------------------------------------------------------------------------- /config/config.exs: -------------------------------------------------------------------------------- 1 | # This file is responsible for configuring your application 2 | # and its dependencies with the aid of the Mix.Config module. 3 | use Mix.Config 4 | 5 | # KVX config 6 | config :kvx, 7 | adapter: KVX.Bucket.ExShards 8 | 9 | # Import environment specific config. 10 | import_config "#{Mix.env}.exs" 11 | -------------------------------------------------------------------------------- /config/dev.exs: -------------------------------------------------------------------------------- 1 | use Mix.Config 2 | 3 | # KVX config 4 | config :kvx, 5 | adapter: KVX.Bucket.ExShards, 6 | ttl: 300 7 | -------------------------------------------------------------------------------- /config/test.exs: -------------------------------------------------------------------------------- 1 | use Mix.Config 2 | 3 | # KVX config 4 | config :kvx, 5 | ttl: 1, 6 | buckets: [ 7 | mybucket: [ 8 | n_shards: 2 9 | ] 10 | ] 11 | -------------------------------------------------------------------------------- /lib/kvx.ex: -------------------------------------------------------------------------------- 1 | defmodule KVX do 2 | @moduledoc """ 3 | This is a simple/basic in-memory Key/Value Store written in 4 | [**Elixir**](http://elixir-lang.org/) and using 5 | [**ExShards**](https://github.com/cabol/ex_shards) 6 | as default adapter. 7 | 8 | Again, **KVX** is a simple library, most of the work 9 | is done by **ExShards**, and its typical use case might 10 | be as a **Cache**. 11 | 12 | ## Adapters 13 | 14 | **KVX** was designed to be flexible and support multiple 15 | backends. We currently ship with one backend: 16 | 17 | * `KVX.Bucket.ExShards` - uses [ExShards](https://github.com/cabol/ex_shards), 18 | to implement the `KVX.Bucket` interface. 19 | 20 | **KVX** adapters config might looks like: 21 | 22 | config :kvx, 23 | adapter: KVX.Bucket.ExShards, 24 | ttl: 43200, 25 | module: ExShards, 26 | buckets: [ 27 | mybucket1: [ 28 | n_shards: 4 29 | ], 30 | mybucket2: [ 31 | n_shards: 8 32 | ] 33 | ] 34 | 35 | In case of `ExShards` adapter, run-time options when calling `new/2` 36 | function are the same as `ExShards.new/2`. E.g.: 37 | 38 | MyModule.new(:mybucket, [n_shards: 4]) 39 | 40 | ## Example 41 | 42 | Check the example [**HERE**](https://github.com/cabol/kvx#example). 43 | """ 44 | end 45 | -------------------------------------------------------------------------------- /lib/kvx/adapters/ex_shards/bucket_shards.ex: -------------------------------------------------------------------------------- 1 | defmodule KVX.Bucket.ExShards do 2 | @moduledoc """ 3 | ExShards adapter. This is the default adapter supported by `KVX`. 4 | ExShards adapter only works with `set` and `ordered_set` table types. 5 | 6 | ExShards extra config options: 7 | 8 | * `:module` - internal ExShards module to use. By default, `ExShards` 9 | module is used, which is a wrapper on top of `ExShards.Local` and 10 | `ExShards.Dist`. 11 | * `:buckets` - this can be used to set bucket options in config, 12 | so it can be loaded when the bucket is created. See example below. 13 | 14 | Run-time options when calling `new/2` function, are the same as 15 | `ExShards.new/2`. For example: 16 | 17 | MyModule.new(:mybucket, [n_shards: 4]) 18 | 19 | ## Example: 20 | 21 | config :kvx, 22 | adapter: KVX.Bucket.ExShards, 23 | ttl: 43200, 24 | module: ExShards, 25 | buckets: [ 26 | mybucket1: [ 27 | n_shards: 4 28 | ], 29 | mybucket2: [ 30 | n_shards: 8 31 | ] 32 | ] 33 | 34 | For more information about `ExShards`: 35 | 36 | * [GitHub](https://github.com/cabol/ex_shards) 37 | * [GitHub](https://github.com/cabol/shards) 38 | * [Blog Post](http://cabol.github.io/posts/2016/04/14/sharding-support-for-ets.html) 39 | """ 40 | 41 | @behaviour KVX.Bucket 42 | 43 | @mod (Application.get_env(:kvx, :module, ExShards)) 44 | @default_ttl (Application.get_env(:kvx, :ttl, :infinity)) 45 | 46 | require Ex2ms 47 | 48 | ## Setup Commands 49 | 50 | def new(bucket, opts \\ []) when is_atom(bucket) do 51 | case Process.whereis(bucket) do 52 | nil -> new_bucket(bucket, opts) 53 | _ -> bucket 54 | end 55 | end 56 | 57 | defp new_bucket(bucket, opts) do 58 | opts = maybe_get_bucket_opts(bucket, opts) 59 | @mod.new(bucket, opts) 60 | end 61 | 62 | defp maybe_get_bucket_opts(bucket, []) do 63 | :kvx 64 | |> Application.get_env(:buckets, []) 65 | |> Keyword.get(bucket, []) 66 | end 67 | defp maybe_get_bucket_opts(_, opts), do: opts 68 | 69 | ## Storage Commands 70 | 71 | def add(bucket, key, value, ttl \\ @default_ttl) do 72 | case get(bucket, key) do 73 | nil -> set(bucket, key, value, ttl) 74 | _ -> raise KVX.ConflictError, key: key, value: value 75 | end 76 | end 77 | 78 | def set(bucket, key, value, ttl \\ @default_ttl) do 79 | @mod.set(bucket, {key, value, seconds_since_epoch(ttl)}) 80 | end 81 | 82 | def mset(bucket, entries, ttl \\ @default_ttl) when is_list(entries) do 83 | entries |> Enum.each(fn({key, value}) -> 84 | ^bucket = set(bucket, key, value, ttl) 85 | end) 86 | bucket 87 | end 88 | 89 | ## Retrieval Commands 90 | 91 | def get(bucket, key) do 92 | case @mod.lookup(bucket, key) do 93 | [{^key, value, ttl}] -> 94 | if ttl > seconds_since_epoch(0) do 95 | value 96 | else 97 | true = @mod.delete(bucket, key) 98 | nil 99 | end 100 | _ -> 101 | nil 102 | end 103 | end 104 | 105 | def mget(bucket, keys) when is_list(keys) do 106 | for key <- keys do 107 | get(bucket, key) 108 | end 109 | end 110 | 111 | def find_all(bucket, query \\ nil) do 112 | do_find_all(bucket, query) 113 | end 114 | 115 | defp do_find_all(bucket, nil) do 116 | do_find_all(bucket, Ex2ms.fun do object -> object end) 117 | end 118 | defp do_find_all(bucket, query) do 119 | bucket 120 | |> @mod.select(query) 121 | |> Enum.reduce([], fn({k, v, ttl}, acc) -> 122 | case ttl > seconds_since_epoch(0) do 123 | true -> 124 | [{k, v} | acc] 125 | _ -> 126 | true = @mod.delete(bucket, k) 127 | acc 128 | end 129 | end) 130 | end 131 | 132 | ## Cleanup functions 133 | 134 | def delete(bucket, key) do 135 | true = @mod.delete(bucket, key) 136 | bucket 137 | end 138 | 139 | def delete(bucket) do 140 | true = @mod.delete(bucket) 141 | bucket 142 | end 143 | 144 | def flush(bucket) do 145 | true = @mod.delete_all_objects(bucket) 146 | bucket 147 | end 148 | 149 | ## Extended functions 150 | 151 | def __ex_shards_mod__, do: @mod 152 | 153 | def __default_ttl__, do: @default_ttl 154 | 155 | ## Private functions 156 | 157 | defp seconds_since_epoch(diff) when is_integer(diff) do 158 | {mega, secs, _} = :os.timestamp() 159 | mega * 1000000 + secs + diff 160 | end 161 | defp seconds_since_epoch(:infinity), do: :infinity 162 | defp seconds_since_epoch(diff), do: raise ArgumentError, "ttl #{inspect diff} is invalid." 163 | end 164 | -------------------------------------------------------------------------------- /lib/kvx/bucket.ex: -------------------------------------------------------------------------------- 1 | defmodule KVX.Bucket do 2 | @moduledoc """ 3 | Defines a Bucket. 4 | 5 | A bucket maps to an underlying data store, controlled by the 6 | adapter. For example, `KVX` ships with a `KVX.Bucket.ExShards` 7 | adapter that stores data into `ExShards` distributed memory 8 | storage – [ExShards](https://github.com/cabol/ex_shards). 9 | 10 | For example, the bucket: 11 | 12 | defmodule MyModule do 13 | use use KVX.Bucket 14 | end 15 | 16 | Could be configured with: 17 | 18 | config :kvx, 19 | adapter: KVX.Bucket.ExShards, 20 | ttl: 10 21 | 22 | Most of the configuration that goes into the `config` is specific to 23 | the adapter, so check `KVX.Bucket.ExShards` documentation for more 24 | information. However, some configuration is shared across 25 | all adapters, they are: 26 | 27 | * `:ttl` - The time in seconds to wait until the `key` expires. 28 | Value `:infinity` will wait indefinitely (default: 3600) 29 | 30 | Check adapters documentation for more information. 31 | """ 32 | 33 | @type bucket :: atom 34 | @type key :: term 35 | @type value :: term 36 | @type ttl :: integer | :infinity 37 | 38 | @doc false 39 | defmacro __using__(_opts) do 40 | quote do 41 | @behaviour KVX.Bucket 42 | 43 | @adapter (Application.get_env(:kvx, :adapter, KVX.Bucket.ExShards)) 44 | @default_ttl (Application.get_env(:kvx, :ttl, :infinity)) 45 | 46 | def __adapter__ do 47 | @adapter 48 | end 49 | 50 | def __ttl__ do 51 | @default_ttl 52 | end 53 | 54 | def new(bucket, opts \\ []) do 55 | @adapter.new(bucket, opts) 56 | end 57 | 58 | def add(bucket, key, value, ttl \\ @default_ttl) do 59 | @adapter.add(bucket, key, value, ttl) 60 | end 61 | 62 | def set(bucket, key, value, ttl \\ @default_ttl) do 63 | @adapter.set(bucket, key, value, ttl) 64 | end 65 | 66 | def mset(bucket, kv_pairs, ttl \\ @default_ttl) when is_list(kv_pairs) do 67 | @adapter.mset(bucket, kv_pairs, ttl) 68 | end 69 | 70 | def get(bucket, key) do 71 | @adapter.get(bucket, key) 72 | end 73 | 74 | def mget(bucket, keys) when is_list(keys) do 75 | @adapter.mget(bucket, keys) 76 | end 77 | 78 | def find_all(bucket, query \\ nil) do 79 | @adapter.find_all(bucket, query) 80 | end 81 | 82 | def delete(bucket, key) do 83 | @adapter.delete(bucket, key) 84 | end 85 | 86 | def delete(bucket) do 87 | @adapter.delete(bucket) 88 | end 89 | 90 | def flush(bucket) do 91 | @adapter.flush(bucket) 92 | end 93 | end 94 | end 95 | 96 | ## Setup Commands 97 | 98 | @doc """ 99 | Creates a new bucket if it doesn't exist. If the bucket already exist, 100 | nothing happens – it works as an idempotent operation. 101 | 102 | ## Example 103 | 104 | MyBucket.new(:mybucket) 105 | """ 106 | @callback new(bucket, [term]) :: bucket 107 | 108 | ## Storage Commands 109 | 110 | @doc """ 111 | Store this data, only if it does not already exist. If an item already 112 | exists and an add fails with a `KVX.ConflictError` exception. 113 | 114 | If `bucket` doesn't exist, it will raise an argument error. 115 | 116 | ## Example 117 | 118 | MyBucket.add(:mybucket, "hello", "world") 119 | """ 120 | @callback add(bucket, key, value, ttl) :: bucket | KVX.ConflictError 121 | 122 | @doc """ 123 | Most common command. Store this data, possibly overwriting any existing data. 124 | 125 | If `bucket` doesn't exist, it will raise an argument error. 126 | 127 | ## Example 128 | 129 | MyBucket.set(:mybucket, "hello", "world") 130 | """ 131 | @callback set(bucket, key, value, ttl) :: bucket 132 | 133 | @doc """ 134 | Store this bulk data, possibly overwriting any existing data. 135 | 136 | If `bucket` doesn't exist, it will raise an argument error. 137 | 138 | ## Example 139 | 140 | MyBucket.mset(:mybucket, [{"a": 1}, {"b", "2"}]) 141 | """ 142 | @callback mset(bucket, [{key, value}], ttl) :: bucket 143 | 144 | ## Retrieval Commands 145 | 146 | @doc """ 147 | Get the value of `key`. If the key does not exist the special value `nil` 148 | is returned. 149 | 150 | If `bucket` doesn't exist, it will raise an argument error. 151 | 152 | ## Example 153 | 154 | MyBucket.get(:mybucket, "hello") 155 | """ 156 | @callback get(bucket, key) :: value | nil 157 | 158 | @doc """ 159 | Returns the values of all specified keys. For every key that does not hold 160 | a string value or does not exist, the special value `nil` is returned. 161 | Because of this, the operation never fails. 162 | 163 | If `bucket` doesn't exist, it will raise an argument error. 164 | 165 | ## Example 166 | 167 | MyBucket.mget(:mybucket, ["hello", "world"]) 168 | """ 169 | @callback mget(bucket, [key]) :: [value | nil] 170 | 171 | @doc """ 172 | Returns all objects/tuples `{key, value}` that matches with the specified 173 | `query`. The `query` type/spec depends on each adapter implementation – 174 | `:ets.match_spec` in case of `KVX.Bucket.ExShards`. 175 | 176 | If `bucket` doesn't exist, it will raise an argument error. 177 | 178 | ## Example 179 | 180 | MyBucket.find_all(bucket, Ex2ms.fun do object -> object end) 181 | """ 182 | @callback find_all(bucket, query :: term) :: [{key, value}] 183 | 184 | ## Cleanup functions 185 | 186 | @doc """ 187 | Removes an item from the bucket, if it exists. 188 | 189 | If `bucket` doesn't exist, it will raise an argument error. 190 | 191 | ## Example 192 | 193 | MyBucket.delete(:mybucket, "hello") 194 | """ 195 | @callback delete(bucket, key) :: bucket 196 | 197 | @doc """ 198 | Deletes an entire bucket, if it exists. 199 | 200 | If `bucket` doesn't exist, it will raise an argument error. 201 | 202 | ## Example 203 | 204 | MyBucket.delete(:mybucket) 205 | """ 206 | @callback delete(bucket) :: bucket 207 | 208 | @doc """ 209 | Invalidate all existing cache items. 210 | 211 | If `bucket` doesn't exist, it will raise an argument error. 212 | 213 | ## Example 214 | 215 | MyBucket.flush(:mybucket) 216 | """ 217 | @callback flush(bucket) :: bucket 218 | end 219 | -------------------------------------------------------------------------------- /lib/kvx/exceptions.ex: -------------------------------------------------------------------------------- 1 | defmodule KVX.ConflictError do 2 | defexception [:message] 3 | 4 | def exception(opts) do 5 | key = Keyword.fetch!(opts, :key) 6 | val = Keyword.fetch!(opts, :value) 7 | 8 | msg = """ 9 | Expected non-existing slot but got #{key} => #{val}. 10 | """ 11 | 12 | %__MODULE__{message: msg} 13 | end 14 | end 15 | -------------------------------------------------------------------------------- /mix.exs: -------------------------------------------------------------------------------- 1 | defmodule KVX.Mixfile do 2 | use Mix.Project 3 | 4 | def project do 5 | [app: :kvx, 6 | version: "0.1.3", 7 | elixir: "~> 1.3", 8 | build_embedded: Mix.env == :prod, 9 | start_permanent: Mix.env == :prod, 10 | deps: deps(), 11 | test_coverage: [tool: ExCoveralls], 12 | preferred_cli_env: ["coveralls": :test, "coveralls.detail": :test, "coveralls.post": :test, "coveralls.html": :test], 13 | package: package(), 14 | description: """ 15 | Simple Elixir in-memory Key/Value Store using `cabol/ex_shards`. 16 | """] 17 | end 18 | 19 | def application do 20 | [applications: [:logger, :shards]] 21 | end 22 | 23 | defp deps do 24 | [{:ex_shards, "~> 0.2"}, 25 | {:ex2ms, "~> 1.4"}, 26 | {:ex_doc, ">= 0.0.0", only: :dev}, 27 | {:excoveralls, "~> 0.5.6", only: :test}] 28 | end 29 | 30 | defp package do 31 | [name: :kvx, 32 | maintainers: ["Carlos A Bolanos"], 33 | licenses: ["MIT"], 34 | links: %{"GitHub" => "https://github.com/cabol/kvx"}] 35 | end 36 | end 37 | -------------------------------------------------------------------------------- /mix.lock: -------------------------------------------------------------------------------- 1 | %{"certifi": {:hex, :certifi, "0.7.0", "861a57f3808f7eb0c2d1802afeaae0fa5de813b0df0979153cbafcd853ababaf", [:rebar3], []}, 2 | "earmark": {:hex, :earmark, "1.1.1", "433136b7f2e99cde88b745b3a0cfc3fbc81fe58b918a09b40fce7f00db4d8187", [:mix], []}, 3 | "ex2ms": {:hex, :ex2ms, "1.4.0", "e43b410888b45ba363ea6650db3736db3e455a0a412ec244ac633fede857bcb2", [:mix], []}, 4 | "ex_doc": {:hex, :ex_doc, "0.14.5", "c0433c8117e948404d93ca69411dd575ec6be39b47802e81ca8d91017a0cf83c", [:mix], [{:earmark, "~> 1.0", [hex: :earmark, optional: false]}]}, 5 | "ex_shards": {:hex, :ex_shards, "0.2.0", "ded14e46d76797246a4d574eb3c67b583f9cfbfa747828d3e52a85c8d2ef98d2", [:mix], [{:ex2ms, "~> 1.4", [hex: :ex2ms, optional: false]}, {:shards, "~> 0.4", [hex: :shards, optional: false]}]}, 6 | "excoveralls": {:hex, :excoveralls, "0.5.7", "5d26e4a7cdf08294217594a1b0643636accc2ad30e984d62f1d166f70629ff50", [:mix], [{:exjsx, "~> 3.0", [hex: :exjsx, optional: false]}, {:hackney, ">= 0.12.0", [hex: :hackney, optional: false]}]}, 7 | "exjsx": {:hex, :exjsx, "3.2.1", "1bc5bf1e4fd249104178f0885030bcd75a4526f4d2a1e976f4b428d347614f0f", [:mix], [{:jsx, "~> 2.8.0", [hex: :jsx, optional: false]}]}, 8 | "hackney": {:hex, :hackney, "1.6.5", "8c025ee397ac94a184b0743c73b33b96465e85f90a02e210e86df6cbafaa5065", [:rebar3], [{:certifi, "0.7.0", [hex: :certifi, optional: false]}, {:idna, "1.2.0", [hex: :idna, optional: false]}, {:metrics, "1.0.1", [hex: :metrics, optional: false]}, {:mimerl, "1.0.2", [hex: :mimerl, optional: false]}, {:ssl_verify_fun, "1.1.1", [hex: :ssl_verify_fun, optional: false]}]}, 9 | "idna": {:hex, :idna, "1.2.0", "ac62ee99da068f43c50dc69acf700e03a62a348360126260e87f2b54eced86b2", [:rebar3], []}, 10 | "jsx": {:hex, :jsx, "2.8.2", "7acc7d785b5abe8a6e9adbde926a24e481f29956dd8b4df49e3e4e7bcc92a018", [:mix, :rebar3], []}, 11 | "metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], []}, 12 | "mimerl": {:hex, :mimerl, "1.0.2", "993f9b0e084083405ed8252b99460c4f0563e41729ab42d9074fd5e52439be88", [:rebar3], []}, 13 | "shards": {:hex, :shards, "0.4.1", "51c2f52e10dcfa02ca81b7eeb74f21ac59f9140bc2c0e57d7a3abb111e94fe32", [:make, :rebar3], []}, 14 | "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.1", "28a4d65b7f59893bc2c7de786dec1e1555bd742d336043fe644ae956c3497fbe", [:make, :rebar], []}} 15 | -------------------------------------------------------------------------------- /test/bucket_shards_test.exs: -------------------------------------------------------------------------------- 1 | defmodule KVX.Bucket.ShardsTest do 2 | use ExUnit.Case 3 | use KVX.Bucket 4 | 5 | doctest KVX 6 | 7 | @bucket __MODULE__ 8 | 9 | require Ex2ms 10 | 11 | setup do 12 | @bucket = @bucket 13 | |> new([n_shards: 4]) 14 | |> flush 15 | 16 | on_exit fn -> 17 | assert_raise ArgumentError, fn -> 18 | @bucket 19 | |> delete 20 | |> find_all 21 | end 22 | end 23 | end 24 | 25 | test "default config" do 26 | assert KVX.Bucket.ExShards === __adapter__() 27 | assert 1 === __ttl__() 28 | end 29 | 30 | test "invalid bucket error" do 31 | assert_raise ArgumentError, fn -> 32 | set(:invalid, :k1, 1) 33 | end 34 | 35 | assert_raise ArgumentError, fn -> 36 | get(:invalid, :k1) 37 | end 38 | 39 | assert_raise ArgumentError, fn -> 40 | delete(:invalid, :k1) 41 | end 42 | end 43 | 44 | test "flush bucket" do 45 | @bucket 46 | |> new 47 | |> flush 48 | |> mset([k1: 1, k2: 2, k3: 3, k4: 4, k1: 1]) 49 | 50 | assert [k1: 1, k2: 2, k3: 3, k4: 4] === find_all(@bucket) |> Enum.sort 51 | 52 | rs = @bucket 53 | |> flush 54 | |> find_all 55 | |> Enum.sort 56 | 57 | assert [] === rs 58 | end 59 | 60 | test "storage and retrieval commands test" do 61 | @bucket 62 | |> mset([k1: 1, k2: 2, k3: 3, k4: 4, k1: 1]) 63 | 64 | assert 1 === get(@bucket, :k1) 65 | assert [1, 2] === mget(@bucket, [:k1, :k2]) 66 | assert nil === get(@bucket, :k11) 67 | 68 | assert_raise KVX.ConflictError, fn -> 69 | add(@bucket, :k1, 11) 70 | end 71 | add(@bucket, :kx, 123) 72 | assert 123 === get(@bucket, :kx) 73 | 74 | assert [k1: 1, k2: 2, k3: 3, k4: 4, kx: 123] === find_all(@bucket) |> Enum.sort 75 | 76 | ms1 = Ex2ms.fun do {_, v, _} = obj when rem(v, 2) == 0 -> obj end 77 | assert [k2: 2, k4: 4] === find_all(@bucket, ms1) |> Enum.sort 78 | 79 | nil = @bucket 80 | |> delete(:k1) 81 | |> get(:k1) 82 | end 83 | 84 | test "ttl test" do 85 | assert_raise ArgumentError, fn -> 86 | set(@bucket, :k1, 1, :foo) 87 | end 88 | 89 | @bucket 90 | |> mset([k1: 1, k2: 2, k3: 3], 2) 91 | |> set(:k4, 4, 3) 92 | |> set(:k5, 5, :infinity) 93 | 94 | assert [k1: 1, k2: 2, k3: 3, k4: 4, k5: 5] === find_all(@bucket) |> Enum.sort 95 | 96 | :timer.sleep(2000) 97 | assert [k4: 4, k5: 5] === find_all(@bucket) |> Enum.sort 98 | 99 | :timer.sleep(1000) 100 | assert nil === get(@bucket, :k4) 101 | assert [k5: 5] === find_all(@bucket) |> Enum.sort 102 | assert 5 === get(@bucket, :k5) 103 | end 104 | 105 | test "cleanup commands test" do 106 | @bucket 107 | |> mset([k1: 1, k2: 2, k3: 3]) 108 | 109 | assert [k1: 1, k2: 2, k3: 3] === find_all(@bucket) |> Enum.sort 110 | 111 | nil = @bucket 112 | |> delete(:k1) 113 | |> get(:k1) 114 | 115 | assert [k2: 2, k3: 3] === find_all(@bucket) |> Enum.sort 116 | end 117 | 118 | test "load bucket opts from config test" do 119 | assert :mybucket === new(:mybucket) 120 | assert 2 === :shards_state.n_shards(:mybucket) 121 | assert ExShards === KVX.Bucket.ExShards.__ex_shards_mod__ 122 | end 123 | end 124 | -------------------------------------------------------------------------------- /test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start() 2 | --------------------------------------------------------------------------------