├── .formatter.exs ├── .gitignore ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── LCR_LICENSE.md ├── LICENCED_USERS.md ├── MIT_LICENSE.md ├── README.md ├── config ├── config.exs ├── dev.exs ├── prod.exs └── test.exs ├── lib ├── event_bus_postgres.ex └── event_bus_postgres │ ├── application.ex │ ├── bucket.ex │ ├── config.ex │ ├── event_mapper.ex │ ├── models │ └── event.ex │ ├── queue.ex │ ├── repo.ex │ ├── store.ex │ ├── supervisors │ └── ttl.ex │ └── workers │ └── ttl.ex ├── mix.exs ├── mix.lock ├── priv ├── repo │ └── migrations │ │ └── 20180107070712_create_events.exs └── seeds.exs └── test ├── event_bus_postgres_test.exs ├── support └── data_case.ex └── test_helper.exs /.formatter.exs: -------------------------------------------------------------------------------- 1 | # Used by "mix format" 2 | [ 3 | inputs: ["mix.exs", "{config,lib,test}/**/*.{ex,exs}"], 4 | line_length: 80 5 | ] 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build/ 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover/ 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps/ 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc/ 12 | 13 | # Ignore .fetch files in case you like to edit your project deps locally. 14 | /.fetch 15 | 16 | # If the VM crashes, it generates a dump, let's ignore it too. 17 | erl_crash.dump 18 | 19 | # Also ignore archive artifacts (built via "mix archive.build"). 20 | *.ez 21 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) 5 | and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 6 | 7 | ## [0.4.x] 8 | - Added db_url and db_pool_size configurations to support of releases 9 | - Fixed dialyxir warnings 10 | - Added LCR license 11 | - Added Code of Conduct 12 | 13 | ## Previous versions 14 | - Opensourced as a sample addon using GenStage 15 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. 4 | 5 | We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality. 6 | 7 | Examples of unacceptable behavior by participants include: 8 | 9 | * The use of sexualized language or imagery 10 | * Unwelcome sexual advances 11 | * Discrimination based on age, disability, gender, nationality, race, religion, sexuality, or similar personal characteristic 12 | * Personal attacks 13 | * Trolling or insulting/derogatory comments 14 | * Public or private harassment 15 | * Bullying or systematic harassment 16 | * Publishing other's private information, such as physical or electronic addresses, without explicit permission 17 | * Other unethical or unprofessional conduct 18 | * Insulting, demeaning, hateful, or threatening remarks 19 | * Incitement to any of these 20 | 21 | Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team. 22 | 23 | This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. 24 | 25 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers. 26 | 27 | This Code of Conduct is adapted from the [Contributor Covenant][1], version 1.2.0, available at [http://contributor-covenant.org/version/1/2/0/][2]. 28 | 29 | [1]: http://contributor-covenant.org 30 | [2]: http://contributor-covenant.org/version/1/2/0/ 31 | -------------------------------------------------------------------------------- /LCR_LICENSE.md: -------------------------------------------------------------------------------- 1 | LCR License since 0.4.x 2 | 3 | If you use the library in a live project then you have to create a pull request 4 | on [Github](https://github.com/otobus/event_bus_postgres) and add yourself to 5 | the list on [LICENCED_USERS.md](https://github.com/otobus/event_bus_postgres/blob/master/LICENCED_USERS.md) 6 | file. 7 | -------------------------------------------------------------------------------- /LICENCED_USERS.md: -------------------------------------------------------------------------------- 1 | # Library Users 2 | 3 | | Github handler | 4 | |------------------------------------------------------------------------------| 5 | | @mustafaturan | 6 | 7 | As a license requirement; you have to add your self to the list if you are using 8 | the library in a live project. Please make sure you apply the markup convension. 9 | -------------------------------------------------------------------------------- /MIT_LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Mustafa Turan 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EventBus.Postgres 2 | 3 | Listen and save `event_bus` events to Postgres. 4 | 5 | ## Installation 6 | 7 | The package can be installed by adding `event_bus_postgres` to your list of dependencies in `mix.exs`: 8 | 9 | ```elixir 10 | def deps do 11 | [ 12 | {:event_bus_postgres, "~> 0.4.2"}, 13 | {:event_bus, "~> 1.6"} 14 | ] 15 | end 16 | ``` 17 | 18 | On the command line: 19 | 20 | ```shell 21 | mix deps.get 22 | cp _deps/event_bus_postgres/priv/repo/migrations/* priv/repo/migrations/ 23 | mix ecto.create -r EventBus.Postgres.Repo 24 | mix ecto.migrate -r EventBus.Postgres.Repo 25 | ``` 26 | 27 | ## Configuration 28 | 29 | In your config.exs (or dev.exs, test.exs, prod.exs); 30 | 31 | ```elixir 32 | config :event_bus_postgres, 33 | # Enable/disable PG consumer 34 | enabled: {:system, "EB_PG_ENABLED", "true"}, 35 | 36 | # If you want to disable auto delete set auto_delete_with_ttl to "false" 37 | auto_delete_with_ttl: {:system, "EB_PG_AUTO_DELETE_WITH_TTL", "true"}, 38 | 39 | # Default TTL for deletion, this value will be set when not given in Event struct 40 | default_ttl_in_ms: {:system, "EB_PG_DEFAULT_TTL_IN_MS", "900000"}, 41 | 42 | # Execute delete in given period 43 | deletion_period_in_ms: {:system, "EB_PG_DELETION_PERIOD_IN_MS", "600000"}, 44 | 45 | # GenStage config 46 | min_demand: {:system, "EB_PG_MIN_DEMAND", "75"}, # GenStage consumer 47 | max_demand: {:system, "EB_PG_MAX_DEMAND", "100"}, # GenStage consumer 48 | pool_size: {:system, "EB_PG_POOL_SIZE", "1"}, # GenStage consumer + DB Connection pool 49 | buffer_size: {:system, "EB_PG_BUFFER_SIZE", "200"}, # GenStage producer_consumer 50 | 51 | # Topic subscriptions seperated by semicolons ';' 52 | topics: {:system, "EB_PG_TOPICS", ".*"}, 53 | 54 | # Set DB url and pool size in here if you use Elixir releases 55 | db_url: {:system, "EB_PG_DATABASE_URL", nil}, 56 | db_pool_size: {:system, "EB_PG_DATABASE_POOL_SIZE", 1} 57 | 58 | # Regular ecto DB config 59 | config :event_bus_postgres, ecto_repos: [EventBus.Postgres.Repo] 60 | 61 | config :event_bus_postgres, EventBus.Postgres.Repo, 62 | # ..., 63 | # Set other configs depending on your DB needs 64 | adapter: Ecto.Adapters.Postgres, 65 | ssl: true # depending on your need 66 | ``` 67 | 68 | ## How does it work? 69 | 70 | ```markdown 71 | +-----+ 72 | | | GEN STAGE 73 | | | EVENTBUS +------------------------------------------+ 74 | | | CONSUMER | +---+ | 75 | | | +-----+ | | | | 76 | | | | | | | | +---+ | 77 | | | | E | | | | | | | 78 | | | | v | | | | | | | 79 | | | | e | | | | | | | 80 | | E | | n | | | E | | | | 81 | | l | | t | | +-------+ | v | | | | 82 | | i | topic | B | topic | | | e | | | 83 | | x | + | u | + | Q | | n | | B | +--+ 84 | | i |event_id| s | event_id | u | ask | t | ask | u | | | 85 | | r |------->| . |--------->| e |<-------| | <--------| c | BATCH | | 86 | | | | P | | u |------->| M | -------->| k |------>|DB| 87 | | E | | o | | e | pull | a | pull | e | INSERT| | 88 | | v | | s | | | | p | | t | | | 89 | | e | | t | | +-------+ | p | | | | +--+ 90 | | n | | g | | GENSTAGE | e | | | | 91 | | t | | r | | PRODUCER | r | | | | 92 | | B | | e | | | | | | | 93 | | u | | s | | | | | | | 94 | | s | +-----+ | | | | | | 95 | | |<-----------------------------------------| | +---+ | 96 | | | | fetch_event/1 | | CONSUMER | 97 | | | | | | | 98 | +-----+ | +---+ | 99 | | CONSUMER | 100 | | PRODUCER | 101 | +------------------------------------------+ 102 | ``` 103 | 104 | ## Documentation 105 | 106 | Module docs can be found at [https://hexdocs.pm/event_bus_postgres](https://hexdocs.pm/event_bus_postgres). 107 | 108 | ## Contributing 109 | 110 | ### Issues, Bugs, Documentation, Enhancements 111 | 112 | 1. Fork the project 113 | 114 | 2. Make your improvements and write your tests(make sure you covered all the cases). 115 | 116 | 3. Make a pull request. 117 | 118 | ### Contributors 119 | 120 | Special thanks to all [contributors](https://github.com/otobus/event_bus_postgres/graphs/contributors). 121 | 122 | ## Demo 123 | 124 | Let's create 100k rows of events in Postgres with 1 worker 125 | 126 | ```shell 127 | # Export ENV vars for event_bus_postgres configurations 128 | 129 | export EB_PG_ENABLED=true; 130 | export EB_PG_MIN_DEMAND=75; 131 | export EB_PG_MAX_DEMAND=100; 132 | export EB_PG_POOL_SIZE=1; # 1 worker 133 | export EB_PG_BUFFER_SIZE=200; 134 | export EB_PG_TOPICS=".*"; 135 | export EB_PG_DEFAULT_TTL="900000"; 136 | export EB_PG_DELETION_PERIOD="600000"; 137 | 138 | # Export ENV vars for Postgres db 139 | export EB_PG_DATABASE_URL=postgres://admin:123456@localhost:5432/event_bus_postgres_dev 140 | export EB_PG_DATABASE_POOL_SIZE=1 # 1 DB connection 141 | 142 | iex -S mix; 143 | ``` 144 | 145 | In the app console: 146 | 147 | ```elixir 148 | defmodule FakeSource do 149 | @moduledoc """ 150 | Fake event generator 151 | """ 152 | 153 | use EventBus.EventSource 154 | 155 | @doc """ 156 | Generates 100_000 events 157 | """ 158 | def generate_events do 159 | topic = :fake_event_initialized 160 | error_topic = :fake_event_erred 161 | EventBus.register_topic(topic) 162 | EventBus.register_topic(error_topic) 163 | source = "console" 164 | ttl = 600_000_000 165 | 166 | transaction_id = UUID.uuid4() 167 | 168 | :timer.tc(fn -> 169 | Enum.each(1..100_000, fn _ -> 170 | params = %{id: UUID.uuid4(), topic: topic, transaction_id: transaction_id, ttl: ttl, source: source, error_topic: error_topic} 171 | EventSource.notify(params) do 172 | "this is a fake event with id #{params[:id]}" 173 | end 174 | end) 175 | end) 176 | end 177 | end 178 | 179 | # All generated events will be saved to postgres automatically 180 | {time_spent, :ok} = FakeSource.generate_events() 181 | ``` 182 | 183 | ## Code of Conduct 184 | 185 | Please refer to [code of conduct](CODE_OF_CONDUCT.md) file. 186 | 187 | ## License 188 | 189 | [MIT](MIT_LICENSE.md), [LCR](LCR_LICENSE.md) 190 | 191 | Copyright (c) 2018 Mustafa Turan 192 | 193 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 194 | 195 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 196 | 197 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 198 | -------------------------------------------------------------------------------- /config/config.exs: -------------------------------------------------------------------------------- 1 | use Mix.Config 2 | 3 | config :event_bus_postgres, ecto_repos: [EventBus.Postgres.Repo] 4 | 5 | config :event_bus_postgres, 6 | enabled: {:system, "EB_PG_ENABLED", "true"}, 7 | auto_delete_with_ttl: {:system, "EB_PG_AUTO_DELETE_WITH_TTL", "true"}, 8 | min_demand: {:system, "EB_PG_MIN_DEMAND", "75"}, 9 | max_demand: {:system, "EB_PG_MAX_DEMAND", "100"}, 10 | pool_size: {:system, "EB_PG_POOL_SIZE", "1"}, 11 | buffer_size: {:system, "EB_PG_BUFFER_SIZE", "200"}, 12 | topics: {:system, "EB_PG_TOPICS", ".*"}, 13 | default_ttl_in_ms: {:system, "EB_PG_DEFAULT_TTL_IN_MS", "900000"}, 14 | deletion_period_in_ms: {:system, "EB_PG_DELETION_PERIOD_IN_MS", "600000"}, 15 | db_url: {:system, "EB_PG_DATABASE_URL", nil}, 16 | db_pool_size: {:system, "EB_PG_DATABASE_POOL_SIZE", 1} 17 | 18 | import_config "#{Mix.env()}.exs" 19 | -------------------------------------------------------------------------------- /config/dev.exs: -------------------------------------------------------------------------------- 1 | use Mix.Config 2 | 3 | # Configure your database 4 | config :event_bus_postgres, EventBus.Postgres.Repo, 5 | adapter: Ecto.Adapters.Postgres 6 | -------------------------------------------------------------------------------- /config/prod.exs: -------------------------------------------------------------------------------- 1 | use Mix.Config 2 | 3 | config :event_bus_postgres, EventBus.Postgres.Repo, 4 | adapter: Ecto.Adapters.Postgres, 5 | ssl: true 6 | -------------------------------------------------------------------------------- /config/test.exs: -------------------------------------------------------------------------------- 1 | use Mix.Config 2 | 3 | # Configure your database 4 | config :event_bus_postgres, EventBus.Postgres.Repo, 5 | adapter: Ecto.Adapters.Postgres, 6 | username: System.get_env()["DATABASE_POSTGRESQL_USERNAME"] || "postgres", 7 | password: System.get_env()["DATABASE_POSTGRESQL_PASSWORD"] || "", 8 | database: "event_bus_postgres_test", 9 | hostname: "localhost", 10 | pool: Ecto.Adapters.SQL.Sandbox 11 | -------------------------------------------------------------------------------- /lib/event_bus_postgres.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres do 2 | @moduledoc """ 3 | Postgres plugin for EventBus 4 | """ 5 | 6 | alias EventBus.Postgres.Queue 7 | 8 | @doc """ 9 | Deliver EventBus events to Postgres queue 10 | """ 11 | def process({_topic, _id} = event_shadow) do 12 | Queue.push(event_shadow) 13 | end 14 | end 15 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/application.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Application do 2 | @moduledoc false 3 | 4 | use Application 5 | 6 | alias EventBus.Postgres 7 | alias EventBus.Postgres.{Bucket, Config, EventMapper, Queue, Repo} 8 | alias EventBus.Postgres.Supervisor.TTL, as: TTLSupervisor 9 | 10 | def start(_type, _args) do 11 | link = 12 | Supervisor.start_link( 13 | workers(), 14 | strategy: :one_for_one, 15 | name: EventBus.Postgres.Supervisor 16 | ) 17 | 18 | if Config.enabled?() do 19 | EventBus.subscribe({Postgres, Config.topics()}) 20 | end 21 | 22 | link 23 | end 24 | 25 | defp workers do 26 | import Supervisor.Spec, warn: false 27 | 28 | [ 29 | supervisor(Repo, [], id: make_ref(), restart: :permanent), 30 | worker(Queue, [], id: make_ref(), restart: :permanent), 31 | worker(EventMapper, [], id: make_ref(), restart: :permanent) 32 | ] ++ bucket_workers() ++ auto_deletion_workers() 33 | end 34 | 35 | defp auto_deletion_workers do 36 | if Config.auto_delete_with_ttl?() do 37 | import Supervisor.Spec, warn: false 38 | 39 | [supervisor(TTLSupervisor, [], id: make_ref(), restart: :permanent)] 40 | else 41 | [] 42 | end 43 | end 44 | 45 | defp bucket_workers do 46 | import Supervisor.Spec, warn: false 47 | 48 | Enum.map(1..Config.pool_size(), fn _ -> 49 | worker(Bucket, [], id: make_ref(), restart: :permanent) 50 | end) 51 | end 52 | end 53 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/bucket.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Bucket do 2 | @moduledoc """ 3 | Event persistence worker 4 | """ 5 | 6 | use GenStage 7 | 8 | alias EventBus.Postgres.{Config, EventMapper, Store} 9 | 10 | def init(:ok) do 11 | { 12 | :consumer, 13 | :ok, 14 | subscribe_to: [ 15 | { 16 | EventMapper, 17 | min_demand: Config.min_demand(), max_demand: Config.max_demand() 18 | } 19 | ] 20 | } 21 | end 22 | 23 | def start_link do 24 | GenStage.start_link(__MODULE__, :ok) 25 | end 26 | 27 | @doc false 28 | def handle_events(events, _from, state) do 29 | Store.batch_insert(events) 30 | {:noreply, [], state} 31 | end 32 | end 33 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/config.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Config do 2 | @moduledoc """ 3 | Config reader 4 | """ 5 | 6 | @app :event_bus_postgres 7 | 8 | ############################################################################## 9 | # POSTGRES DB CONFIG START 10 | ############################################################################## 11 | 12 | @spec db_url() :: String.t() | nil 13 | def db_url do 14 | @app 15 | |> Application.get_env(:db_url) 16 | |> get_env_var() 17 | end 18 | 19 | @spec db_pool_size() :: integer() | nil 20 | def db_pool_size do 21 | @app 22 | |> Application.get_env(:db_pool_size) 23 | |> get_env_var() 24 | |> to_int() 25 | end 26 | 27 | ############################################################################## 28 | # POSTGRES DB CONFIG END 29 | ############################################################################## 30 | 31 | @spec enabled?() :: boolean() 32 | def enabled? do 33 | @app 34 | |> Application.get_env(:enabled) 35 | |> get_env_var() 36 | |> to_bool() 37 | end 38 | 39 | @spec auto_delete_with_ttl?() :: boolean() 40 | def auto_delete_with_ttl? do 41 | @app 42 | |> Application.get_env(:auto_delete_with_ttl, true) 43 | |> get_env_var() 44 | |> to_bool() 45 | end 46 | 47 | @spec pool_size() :: integer() 48 | def pool_size do 49 | @app 50 | |> Application.get_env(:pool_size, 1) 51 | |> get_env_var() 52 | |> to_int() 53 | end 54 | 55 | @spec buffer_size() :: integer() 56 | def buffer_size do 57 | @app 58 | |> Application.get_env(:buffer_size, 200) 59 | |> get_env_var() 60 | |> to_int() 61 | end 62 | 63 | @spec min_demand() :: integer() 64 | def min_demand do 65 | @app 66 | |> Application.get_env(:min_demand, 75) 67 | |> get_env_var() 68 | |> to_int() 69 | end 70 | 71 | @spec max_demand() :: integer() 72 | def max_demand do 73 | @app 74 | |> Application.get_env(:max_demand, 100) 75 | |> get_env_var() 76 | |> to_int() 77 | end 78 | 79 | @spec topics() :: list(String.t()) 80 | def topics do 81 | @app 82 | |> Application.get_env(:topics, "") 83 | |> get_env_var() 84 | |> to_list() 85 | end 86 | 87 | @spec default_ttl() :: integer() 88 | def default_ttl do 89 | @app 90 | |> Application.get_env(:default_ttl_in_ms, 900_000) 91 | |> get_env_var() 92 | |> to_int() 93 | |> to_microseconds() 94 | end 95 | 96 | @spec deletion_period() :: integer() 97 | def deletion_period do 98 | @app 99 | |> Application.get_env(:deletion_period_in_ms, default_ttl()) 100 | |> get_env_var() 101 | |> to_int() 102 | end 103 | 104 | defp get_env_var({:system, name, default}) do 105 | System.get_env(name) || default 106 | end 107 | 108 | defp get_env_var(item) do 109 | item 110 | end 111 | 112 | defp to_list(val) when is_list(val) do 113 | val 114 | end 115 | 116 | defp to_list(val) do 117 | String.split(val, ";") 118 | end 119 | 120 | defp to_int(val) when is_integer(val) do 121 | val 122 | end 123 | 124 | defp to_int(val) do 125 | String.to_integer(val) 126 | end 127 | 128 | defp to_microseconds(val) do 129 | val * 1_000 130 | end 131 | 132 | defp to_bool(val) do 133 | case "#{val}" do 134 | "1" -> 135 | true 136 | 137 | "true" -> 138 | true 139 | 140 | _ -> 141 | false 142 | end 143 | end 144 | end 145 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/event_mapper.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.EventMapper do 2 | @moduledoc """ 3 | Mapper for EventBus events to Postgres scheme 4 | """ 5 | 6 | use GenStage 7 | 8 | alias EventBus.Postgres 9 | alias EventBus.Postgres.{Config, Model.Event, Queue} 10 | 11 | def init(state) do 12 | { 13 | :producer_consumer, 14 | state, 15 | subscribe_to: [ 16 | { 17 | Queue, 18 | min_demand: Config.min_demand(), max_demand: Config.max_demand() 19 | } 20 | ], 21 | buffer_size: Config.buffer_size() 22 | } 23 | end 24 | 25 | def start_link(state \\ []) do 26 | GenStage.start_link(__MODULE__, state, name: __MODULE__) 27 | end 28 | 29 | def handle_events(event_shadows, _from, state) do 30 | events = 31 | Enum.map(event_shadows, fn {topic, id} -> 32 | event = EventBus.fetch_event({topic, id}) 33 | EventBus.mark_as_completed({Postgres, {topic, id}}) 34 | Event.from_eb_event(event) 35 | end) 36 | 37 | {:noreply, events, state} 38 | end 39 | 40 | def handle_demand(_demand, state) do 41 | {:noreply, [], state} 42 | end 43 | end 44 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/models/event.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Model.Event do 2 | @moduledoc """ 3 | Event model/struct 4 | """ 5 | 6 | use Ecto.Schema 7 | 8 | alias EventBus.Postgres.{Config, Model.Event} 9 | 10 | @primary_key {:id, :binary_id, autogenerate: true} 11 | @foreign_key_type :binary_id 12 | schema "events" do 13 | field(:data, :binary) 14 | field(:initialized_at, :integer) 15 | field(:occurred_at, :integer) 16 | field(:source, :string) 17 | field(:topic, :string) 18 | field(:transaction_id, Ecto.UUID) 19 | field(:ttl, :integer) 20 | end 21 | 22 | @doc false 23 | def from_eb_event(%EventBus.Model.Event{} = event) do 24 | %{ 25 | id: event.id, 26 | transaction_id: event.transaction_id, 27 | topic: "#{event.topic}", 28 | data: :erlang.term_to_binary(event.data), 29 | initialized_at: event.initialized_at, 30 | occurred_at: event.occurred_at || now(), 31 | source: event.source, 32 | ttl: event.ttl || Config.default_ttl() 33 | } 34 | end 35 | 36 | @doc false 37 | def to_eb_event(%Event{} = event) do 38 | %EventBus.Model.Event{ 39 | id: "#{event.id}", 40 | transaction_id: "#{event.transaction_id}", 41 | topic: :"#{event.topic}", 42 | data: :erlang.binary_to_term(event.data), 43 | initialized_at: event.initialized_at, 44 | occurred_at: event.occurred_at, 45 | source: event.source, 46 | ttl: event.ttl 47 | } 48 | end 49 | 50 | defp now do 51 | System.os_time(time_unit()) 52 | end 53 | 54 | defp time_unit do 55 | Application.get_env(:event_bus, :time_unit, :microsecond) 56 | end 57 | end 58 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/queue.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Queue do 2 | @moduledoc """ 3 | Postgres queue (producer) 4 | """ 5 | 6 | use GenStage 7 | 8 | def init(state) do 9 | {:producer, state, buffer_size: :infinity} 10 | end 11 | 12 | def start_link(state \\ []) do 13 | GenStage.start_link(__MODULE__, state, name: __MODULE__) 14 | end 15 | 16 | @doc """ 17 | Push event shadows to queue 18 | """ 19 | @spec push({atom(), String.t() | integer()}) :: :ok 20 | def push({_topic, _id} = event_shadow) do 21 | GenServer.cast(__MODULE__, {:push, event_shadow}) 22 | end 23 | 24 | @doc false 25 | def handle_cast({:push, event_shadow}, state) do 26 | {:noreply, [event_shadow], state} 27 | end 28 | 29 | @doc false 30 | def handle_demand(_demand, state) do 31 | {:noreply, [], state} 32 | end 33 | end 34 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/repo.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Repo do 2 | @moduledoc false 3 | 4 | use Ecto.Repo, 5 | otp_app: :event_bus_postgres, 6 | adapter: Ecto.Adapters.Postgres 7 | 8 | alias EventBus.Postgres.Config 9 | 10 | @doc false 11 | def init(_, opts) do 12 | opts = 13 | opts 14 | |> merge_db_url() 15 | |> merge_pool_size() 16 | 17 | {:ok, opts} 18 | end 19 | 20 | defp merge_db_url(opts) do 21 | case Keyword.has_key?(opts, :url) do 22 | false -> merge_val(opts, :url, Config.db_url()) 23 | 24 | true -> opts 25 | end 26 | end 27 | 28 | defp merge_pool_size(opts) do 29 | case Keyword.has_key?(opts, :pool_size) do 30 | false -> merge_val(opts, :pool_size, Config.db_pool_size()) 31 | 32 | true -> opts 33 | end 34 | end 35 | 36 | defp merge_val(opts, _key, nil) do 37 | opts 38 | end 39 | 40 | defp merge_val(opts, key, val) do 41 | Keyword.put(opts, key, val) 42 | end 43 | end 44 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/store.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Store do 2 | @moduledoc """ 3 | Basic db actions 4 | """ 5 | 6 | import Ecto.Query 7 | 8 | alias EventBus.Postgres.{Model.Event, Repo} 9 | 10 | @pagination_vars %{page: 1, per_page: 20, since: 0} 11 | @pagination_vars_with_transaction_id Map.put(@pagination_vars, :transaction_id, nil) 12 | 13 | @doc """ 14 | Fetch all events with pagination 15 | """ 16 | def all(%{page: page, per_page: per_page, since: since} \\ @pagination_vars) do 17 | offset = (page - 1) * per_page 18 | 19 | query = 20 | from( 21 | e in Event, 22 | where: e.occurred_at >= ^since, 23 | offset: ^offset, 24 | limit: ^per_page 25 | ) 26 | 27 | query 28 | |> Repo.all() 29 | |> Enum.map(fn event -> Event.to_eb_event(event) end) 30 | end 31 | 32 | @doc """ 33 | Total events per topic since given time 34 | """ 35 | def count_per_topic(%{since: since} \\ %{since: 0}) do 36 | query = 37 | from( 38 | e in Event, 39 | where: e.occurred_at >= ^since, 40 | group_by: e.topic, 41 | select: %{topic: e.topic, count: count(e.id)} 42 | ) 43 | 44 | Repo.all(query) 45 | end 46 | 47 | @doc """ 48 | Total events since given time 49 | """ 50 | def count(%{since: since} \\ %{since: 0}) do 51 | query = 52 | from( 53 | e in Event, 54 | where: e.occurred_at >= ^since 55 | ) 56 | 57 | Repo.aggregate(query, :count, :id) 58 | end 59 | 60 | @doc """ 61 | Fetch all events with pagination 62 | """ 63 | def find_all_by_transaction_id(%{page: page, per_page: per_page, since: since, transaction_id: transaction_id} \\ @pagination_vars_with_transaction_id) do 64 | query = 65 | from( 66 | e in Event, 67 | where: 68 | e.transaction_id == ^transaction_id and 69 | e.occurred_at >= ^since, 70 | offset: ^((page - 1) * per_page), 71 | limit: ^per_page 72 | ) 73 | 74 | query 75 | |> Repo.all() 76 | |> Enum.map(fn event -> Event.to_eb_event(event) end) 77 | end 78 | 79 | @doc """ 80 | Find an event 81 | """ 82 | def find(id) do 83 | case Repo.get(Event, id) do 84 | nil -> nil 85 | event -> Event.to_eb_event(event) 86 | end 87 | end 88 | 89 | @doc """ 90 | Delete an event 91 | """ 92 | def delete(id) do 93 | Repo.delete(%Event{id: id}) 94 | end 95 | 96 | @doc """ 97 | Batch insert 98 | """ 99 | def batch_insert([]) do 100 | :ok 101 | end 102 | 103 | def batch_insert(events) do 104 | Repo.insert_all(Event, events, on_conflict: :nothing) 105 | end 106 | 107 | @doc """ 108 | Delete expired events 109 | """ 110 | def delete_expired do 111 | now = System.os_time(:microsecond) 112 | 113 | query = 114 | from( 115 | e in Event, 116 | where: fragment("? + ? <= ?", e.occurred_at, e.ttl, ^now) 117 | ) 118 | 119 | Repo.delete_all(query) 120 | end 121 | end 122 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/supervisors/ttl.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Supervisor.TTL do 2 | @moduledoc false 3 | 4 | use Supervisor 5 | 6 | alias EventBus.Postgres.Worker.TTL, as: TTLWorker 7 | 8 | @doc false 9 | def start_link do 10 | Supervisor.start_link(__MODULE__, [], name: __MODULE__) 11 | end 12 | 13 | @doc false 14 | @spec init(list()) :: no_return() 15 | def init([]) do 16 | children = [ 17 | worker(TTLWorker, [], id: make_ref(), restart: :permanent) 18 | ] 19 | 20 | opts = [strategy: :one_for_one, name: __MODULE__] 21 | 22 | supervise(children, opts) 23 | end 24 | end 25 | -------------------------------------------------------------------------------- /lib/event_bus_postgres/workers/ttl.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Worker.TTL do 2 | @moduledoc false 3 | 4 | use GenServer 5 | 6 | alias EventBus.Postgres.{Config, Store} 7 | 8 | ## Callbacks 9 | 10 | @doc false 11 | def start_link do 12 | GenServer.start_link(__MODULE__, [], name: __MODULE__) 13 | end 14 | 15 | @doc false 16 | def init(_opts) do 17 | delete_later() 18 | {:ok, nil} 19 | end 20 | 21 | @doc false 22 | def handle_info(:delete_expired, state) do 23 | Store.delete_expired() 24 | delete_later() 25 | {:noreply, state} 26 | end 27 | 28 | defp delete_later do 29 | Process.send_after(self(), :delete_expired, Config.deletion_period()) 30 | end 31 | end 32 | -------------------------------------------------------------------------------- /mix.exs: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Mixfile do 2 | use Mix.Project 3 | 4 | def project do 5 | [ 6 | app: :event_bus_postgres, 7 | version: "0.4.2", 8 | elixir: "~> 1.5", 9 | elixirc_paths: elixirc_paths(Mix.env()), 10 | start_permanent: Mix.env() == :prod, 11 | aliases: aliases(), 12 | deps: deps(), 13 | description: description(), 14 | package: package() 15 | ] 16 | end 17 | 18 | # Run "mix help compile.app" to learn about applications. 19 | def application do 20 | [ 21 | extra_applications: [:logger], 22 | mod: {EventBus.Postgres.Application, []} 23 | ] 24 | end 25 | 26 | # Run "mix help deps" to learn about dependencies. 27 | defp deps do 28 | [ 29 | {:event_bus, ">= 1.6.0"}, 30 | {:ecto, "~> 3.0"}, 31 | {:ecto_sql, "~> 3.0"}, 32 | {:postgrex, ">= 0.0.0"}, 33 | {:gen_stage, "~> 1.0"}, 34 | {:uuid, "~> 1.1", only: [:dev, :test]}, 35 | {:ex_doc, ">= 0.0.0", only: :dev}, 36 | {:credo, "~> 1.4.0", only: :dev}, 37 | {:dialyxir, "~> 1.0.0-rc.3", only: [:dev], build: false} 38 | ] 39 | end 40 | 41 | # Specifies which paths to compile per environment. 42 | defp elixirc_paths(:test) do 43 | ["lib", "test/support"] 44 | end 45 | 46 | defp elixirc_paths(_) do 47 | ["lib"] 48 | end 49 | 50 | # Aliases are shortcuts or tasks specific to the current project. 51 | # For example, to create, migrate and run the seeds file at once: 52 | # 53 | # $ mix ecto.setup 54 | # 55 | # See the documentation for `Mix` for more info on aliases. 56 | defp aliases do 57 | [ 58 | "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"], 59 | "ecto.reset": ["ecto.drop", "ecto.setup"], 60 | test: ["ecto.create --quiet", "ecto.migrate", "test"] 61 | ] 62 | end 63 | 64 | defp description do 65 | """ 66 | Postgres event store for event_bus 67 | """ 68 | end 69 | 70 | defp package do 71 | [ 72 | name: :event_bus_postgres, 73 | files: [ 74 | "lib", 75 | "mix.exs", 76 | "priv", 77 | "README.md", 78 | "MIT_LICENSE.md", 79 | "LCR_LICENSE.md" 80 | ], 81 | maintainers: ["Mustafa Turan"], 82 | licenses: ["MIT", "LCR"], 83 | links: %{"GitHub" => "https://github.com/mustafaturan/event_bus_postgres"} 84 | ] 85 | end 86 | end 87 | -------------------------------------------------------------------------------- /mix.lock: -------------------------------------------------------------------------------- 1 | %{ 2 | "bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm", "7af5c7e09fe1d40f76c8e4f9dd2be7cebd83909f31fee7cd0e9eadc567da8353"}, 3 | "connection": {:hex, :connection, "1.0.4", "a1cae72211f0eef17705aaededacac3eb30e6625b04a6117c1b2db6ace7d5976", [:mix], [], "hexpm", "4a0850c9be22a43af9920a71ab17c051f5f7d45c209e40269a1938832510e4d9"}, 4 | "credo": {:hex, :credo, "1.4.0", "92339d4cbadd1e88b5ee43d427b639b68a11071b6f73854e33638e30a0ea11f5", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "1fd3b70dce216574ce3c18bdf510b57e7c4c85c2ec9cad4bff854abaf7e58658"}, 5 | "db_connection": {:hex, :db_connection, "1.1.3", "89b30ca1ef0a3b469b1c779579590688561d586694a3ce8792985d4d7e575a61", [:mix], [{:connection, "~> 1.0.2", [hex: :connection, repo: "hexpm", optional: false]}, {:poolboy, "~> 1.5", [hex: :poolboy, repo: "hexpm", optional: true]}, {:sbroker, "~> 1.0", [hex: :sbroker, repo: "hexpm", optional: true]}], "hexpm", "5f0a16a58312a610d5eb0b07506280c65f5137868ad479045f2a2dc4ced80550"}, 6 | "decimal": {:hex, :decimal, "1.8.1", "a4ef3f5f3428bdbc0d35374029ffcf4ede8533536fa79896dd450168d9acdf3c", [:mix], [], "hexpm", "3cb154b00225ac687f6cbd4acc4b7960027c757a5152b369923ead9ddbca7aec"}, 7 | "dialyxir": {:hex, :dialyxir, "1.0.0", "6a1fa629f7881a9f5aaf3a78f094b2a51a0357c843871b8bc98824e7342d00a5", [:mix], [{:erlex, ">= 0.2.6", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "aeb06588145fac14ca08d8061a142d52753dbc2cf7f0d00fc1013f53f8654654"}, 8 | "earmark": {:hex, :earmark, "1.4.4", "4821b8d05cda507189d51f2caeef370cf1e18ca5d7dfb7d31e9cafe6688106a4", [:mix], [], "hexpm", "1f93aba7340574847c0f609da787f0d79efcab51b044bb6e242cae5aca9d264d"}, 9 | "earmark_parser": {:hex, :earmark_parser, "1.4.12", "b245e875ec0a311a342320da0551da407d9d2b65d98f7a9597ae078615af3449", [:mix], [], "hexpm", "711e2cc4d64abb7d566d43f54b78f7dc129308a63bc103fbd88550d2174b3160"}, 10 | "ecto": {:hex, :ecto, "2.2.10", "e7366dc82f48f8dd78fcbf3ab50985ceeb11cb3dc93435147c6e13f2cda0992e", [:mix], [{:db_connection, "~> 1.1", [hex: :db_connection, repo: "hexpm", optional: true]}, {:decimal, "~> 1.2", [hex: :decimal, repo: "hexpm", optional: false]}, {:mariaex, "~> 0.8.0", [hex: :mariaex, repo: "hexpm", optional: true]}, {:poison, "~> 2.2 or ~> 3.0", [hex: :poison, repo: "hexpm", optional: true]}, {:poolboy, "~> 1.5", [hex: :poolboy, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.13.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:sbroker, "~> 1.0", [hex: :sbroker, repo: "hexpm", optional: true]}], "hexpm", "dbd3da2931dc3f82595143640a950d96faee6b887f84ed61a9c03e4070cba1df"}, 11 | "ecto_sql": {:hex, :ecto_sql, "3.7.0", "2fcaad4ab0c8d76a5afbef078162806adbe709c04160aca58400d5cbbe8eeac6", [:mix], [{:db_connection, "~> 2.2", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.7.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.4.0 or ~> 0.5.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.15.0 or ~> 1.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:tds, "~> 2.1.1", [hex: :tds, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "a26135dfa1d99bf87a928c464cfa25bba6535a4fe761eefa56077a4febc60f70"}, 12 | "erlex": {:hex, :erlex, "0.2.6", "c7987d15e899c7a2f34f5420d2a2ea0d659682c06ac607572df55a43753aa12e", [:mix], [], "hexpm", "2ed2e25711feb44d52b17d2780eabf998452f6efda104877a3881c2f8c0c0c75"}, 13 | "event_bus": {:hex, :event_bus, "1.6.1", "07331328b67ccc76d14a12872013464106390abaa47ea0d6a7755e3524899964", [:mix], [], "hexpm", "450b73213a8056c14710d8f0047aefc70f3bbef5a1a7847bb9ceda6fbcdbd42a"}, 14 | "ex_doc": {:hex, :ex_doc, "0.24.2", "e4c26603830c1a2286dae45f4412a4d1980e1e89dc779fcd0181ed1d5a05c8d9", [:mix], [{:earmark_parser, "~> 1.4.0", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_elixir, "~> 0.14", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1", [hex: :makeup_erlang, repo: "hexpm", optional: false]}], "hexpm", "e134e1d9e821b8d9e4244687fb2ace58d479b67b282de5158333b0d57c6fb7da"}, 15 | "gen_stage": {:hex, :gen_stage, "1.1.2", "b1656cd4ba431ed02c5656fe10cb5423820847113a07218da68eae5d6a260c23", [:mix], [], "hexpm", "9e39af23140f704e2b07a3e29d8f05fd21c2aaf4088ff43cb82be4b9e3148d02"}, 16 | "jason": {:hex, :jason, "1.2.0", "10043418c42d2493d0ee212d3fddd25d7ffe484380afad769a0a38795938e448", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "116747dbe057794c3a3e4e143b7c8390b29f634e16c78a7f59ba75bfa6852e7f"}, 17 | "makeup": {:hex, :makeup, "1.0.5", "d5a830bc42c9800ce07dd97fa94669dfb93d3bf5fcf6ea7a0c67b2e0e4a7f26c", [:mix], [{:nimble_parsec, "~> 0.5 or ~> 1.0", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "cfa158c02d3f5c0c665d0af11512fed3fba0144cf1aadee0f2ce17747fba2ca9"}, 18 | "makeup_elixir": {:hex, :makeup_elixir, "0.15.1", "b5888c880d17d1cc3e598f05cdb5b5a91b7b17ac4eaf5f297cb697663a1094dd", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}, {:nimble_parsec, "~> 1.1", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "db68c173234b07ab2a07f645a5acdc117b9f99d69ebf521821d89690ae6c6ec8"}, 19 | "makeup_erlang": {:hex, :makeup_erlang, "0.1.1", "3fcb7f09eb9d98dc4d208f49cc955a34218fc41ff6b84df7c75b3e6e533cc65f", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "174d0809e98a4ef0b3309256cbf97101c6ec01c4ab0b23e926a9e17df2077cbb"}, 20 | "nimble_parsec": {:hex, :nimble_parsec, "1.1.0", "3a6fca1550363552e54c216debb6a9e95bd8d32348938e13de5eda962c0d7f89", [:mix], [], "hexpm", "08eb32d66b706e913ff748f11694b17981c0b04a33ef470e33e11b3d3ac8f54b"}, 21 | "poolboy": {:hex, :poolboy, "1.5.1", "6b46163901cfd0a1b43d692657ed9d7e599853b3b21b95ae5ae0a777cf9b6ca8", [:rebar], [], "hexpm", "8f7168911120e13419e086e78d20e4d1a6776f1eee2411ac9f790af10813389f"}, 22 | "postgrex": {:hex, :postgrex, "0.13.5", "3d931aba29363e1443da167a4b12f06dcd171103c424de15e5f3fc2ba3e6d9c5", [:mix], [{:connection, "~> 1.0", [hex: :connection, repo: "hexpm", optional: false]}, {:db_connection, "~> 1.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: false]}], "hexpm", "a19b61193379cdee04b5b2361bf93d1eb170cd2eec0b18042617b07e1e15fbfb"}, 23 | "uuid": {:hex, :uuid, "1.1.8", "e22fc04499de0de3ed1116b770c7737779f226ceefa0badb3592e64d5cfb4eb9", [:mix], [], "hexpm", "c790593b4c3b601f5dc2378baae7efaf5b3d73c4c6456ba85759905be792f2ac"}, 24 | } 25 | -------------------------------------------------------------------------------- /priv/repo/migrations/20180107070712_create_events.exs: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.Repo.Migrations.CreateEvents do 2 | use Ecto.Migration 3 | 4 | def change do 5 | create table(:events, primary_key: false) do 6 | add :id, :uuid, primary_key: true 7 | add :transaction_id, :uuid 8 | add :topic, :string 9 | add :data, :bytea 10 | add :initialized_at, :bigint 11 | add :occurred_at, :bigint 12 | add :source, :string 13 | add :ttl, :integer 14 | end 15 | 16 | create index(:events, ["occurred_at DESC"]) 17 | create index(:events, [:topic]) 18 | create index(:events, [:transaction_id]) 19 | end 20 | end 21 | -------------------------------------------------------------------------------- /priv/seeds.exs: -------------------------------------------------------------------------------- 1 | # Script for populating the database. You can run it as: 2 | # 3 | # mix run priv/repo/seeds.exs 4 | # 5 | # Inside the script, you can read and write to any of your 6 | # repositories directly: 7 | # 8 | # EventBus.Postgres.Repo.insert!(%EventBus.Postgres.SomeSchema{}) 9 | # 10 | # We recommend using the bang functions (`insert!`, `update!` 11 | # and so on) as they will fail if something goes wrong. 12 | -------------------------------------------------------------------------------- /test/event_bus_postgres_test.exs: -------------------------------------------------------------------------------- 1 | defmodule EventBus.PostgresTest do 2 | use ExUnit.Case 3 | 4 | doctest EventBus.Postgres 5 | end 6 | -------------------------------------------------------------------------------- /test/support/data_case.ex: -------------------------------------------------------------------------------- 1 | defmodule EventBus.Postgres.DataCase do 2 | @moduledoc """ 3 | This module defines the setup for tests requiring 4 | access to the application's data layer. 5 | 6 | You may define functions here to be used as helpers in 7 | your tests. 8 | 9 | Finally, if the test case interacts with the database, 10 | it cannot be async. For this reason, every test runs 11 | inside a transaction which is reset at the beginning 12 | of the test unless the test case is marked as async. 13 | """ 14 | 15 | use ExUnit.CaseTemplate 16 | 17 | alias Ecto.{Adapters.SQL.Sandbox, Changeset} 18 | alias EventBus.Postgres.Repo 19 | 20 | using do 21 | quote do 22 | alias EventBus.Postgres.Repo 23 | 24 | import Ecto 25 | import Ecto.Changeset 26 | import Ecto.Query 27 | import EventBus.Postgres.DataCase 28 | end 29 | end 30 | 31 | setup tags do 32 | :ok = Sandbox.checkout(Repo) 33 | 34 | unless tags[:async] do 35 | Sandbox.mode(Repo, {:shared, self()}) 36 | end 37 | 38 | :ok 39 | end 40 | 41 | @doc """ 42 | A helper that transform changeset errors to a map of messages. 43 | 44 | assert {:error, changeset} = Accounts.create_user(%{password: "short"}) 45 | assert "password is too short" in errors_on(changeset).password 46 | assert %{password: ["password is too short"]} = errors_on(changeset) 47 | 48 | """ 49 | def errors_on(changeset) do 50 | Changeset.traverse_errors(changeset, fn {message, opts} -> 51 | Enum.reduce(opts, message, fn {key, value}, acc -> 52 | String.replace(acc, "%{#{key}}", to_string(value)) 53 | end) 54 | end) 55 | end 56 | end 57 | -------------------------------------------------------------------------------- /test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start() 2 | 3 | Ecto.Adapters.SQL.Sandbox.mode(EventBus.Postgres.Repo, :manual) 4 | --------------------------------------------------------------------------------