├── rebar ├── img ├── ping-rtt.png └── packets-in-rate.png ├── rtb.sh ├── rtb.yml.mqtt.example ├── rtb.yml.xmpp.example ├── src ├── rtb.app.src ├── rtb_sup.erl ├── rtb_plot.erl ├── rtb_sm.erl ├── rtb_http.erl ├── rtb_pool.erl ├── mod_xmpp_http.erl ├── rtb.erl ├── mod_xmpp_proxy65.erl ├── mqtt_socket.erl ├── rtb_stats.erl ├── rtb_watchdog.erl └── rtb_config.erl ├── include ├── rtb.hrl └── mqtt.hrl ├── plugins └── override_opts.erl ├── Makefile ├── rebar.config ├── CODE_OF_CONDUCT.md ├── cert.pem ├── CONTRIBUTING.md ├── LICENSE ├── c_src └── rtb_db.c └── README.md /rebar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/processone/rtb/HEAD/rebar -------------------------------------------------------------------------------- /img/ping-rtt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/processone/rtb/HEAD/img/ping-rtt.png -------------------------------------------------------------------------------- /img/packets-in-rate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/processone/rtb/HEAD/img/packets-in-rate.png -------------------------------------------------------------------------------- /rtb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | export XMPPB_LIMIT=65536 4 | 5 | erl +K true -pa ebin -pa deps/*/ebin +P $XMPPB_LIMIT +Q $XMPPB_LIMIT \ 6 | -sname rtb@localhost \ 7 | -s rtb -rtb config "\"rtb.yml\"" "$@" 8 | -------------------------------------------------------------------------------- /rtb.yml.mqtt.example: -------------------------------------------------------------------------------- 1 | ### 2 | ### RTB configuration file 3 | ### 4 | ### The parameters used in this configuration file are explained at 5 | ### 6 | ### https://github.com/processone/rtb/blob/master/README.md 7 | ### 8 | ### The configuration file is written in YAML 9 | ### 10 | ### ******************************************************* 11 | ### ******* !!! WARNING !!! ******* 12 | ### ******* YAML IS INDENTATION SENSITIVE ******* 13 | ### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY ******* 14 | ### ******************************************************* 15 | ### 16 | 17 | ### Mandatory options: common for all scenarios 18 | scenario: mqtt 19 | interval: 100 20 | capacity: 10 21 | 22 | ### Mandatory options for MQTT scenario 23 | client_id: rtb% 24 | servers: 25 | - tcp://127.0.0.1:1883 26 | 27 | ### Optional authentication 28 | ### username: user% 29 | ### password: pass% 30 | 31 | ### An HTTP port for the statistics web interface 32 | ### www_port: 8080 33 | 34 | ### Local Variables: 35 | ### mode: yaml 36 | ### End: 37 | ### vim: set filetype=yaml tabstop=8 38 | -------------------------------------------------------------------------------- /rtb.yml.xmpp.example: -------------------------------------------------------------------------------- 1 | ### 2 | ### RTB configuration file 3 | ### 4 | ### The parameters used in this configuration file are explained at 5 | ### 6 | ### https://github.com/processone/rtb/blob/master/README.md 7 | ### 8 | ### The configuration file is written in YAML 9 | ### 10 | ### ******************************************************* 11 | ### ******* !!! WARNING !!! ******* 12 | ### ******* YAML IS INDENTATION SENSITIVE ******* 13 | ### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY ******* 14 | ### ******************************************************* 15 | ### 16 | 17 | ### Mandatory options: common for all scenarios 18 | scenario: xmpp 19 | interval: 100 20 | capacity: 10 21 | certfile: cert.pem 22 | 23 | ### Mandatory options for XMPP scenario 24 | jid: user%@localhost 25 | password: pass% 26 | 27 | ### Server addresses 28 | ### Optional, but highly recommended to set 29 | ### servers: 30 | ### - tcp://127.0.0.1:5222 31 | ### - tcp://192.168.1.1:5222 32 | 33 | ### An HTTP port for the statistics web interface 34 | ### www_port: 8080 35 | 36 | ### Local Variables: 37 | ### mode: yaml 38 | ### End: 39 | ### vim: set filetype=yaml tabstop=8 40 | -------------------------------------------------------------------------------- /src/rtb.app.src: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2017 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | {application, rtb, 19 | [{description, "XMPP Benchmarking tool"}, 20 | {vsn, "1.0.0"}, 21 | {modules, []}, 22 | {registered, []}, 23 | {applications, [kernel, stdlib]}, 24 | {mod, {rtb, []}}, 25 | {env, []}]}. 26 | 27 | %% Local Variables: 28 | %% mode: erlang 29 | %% End: 30 | %% vim: set filetype=erlang tabstop=8: 31 | -------------------------------------------------------------------------------- /include/rtb.hrl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2018 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -record(metric, {name :: atom(), 19 | call :: fun(() -> integer()), 20 | type = gauge :: metric_type(), 21 | rate = true :: boolean()}). 22 | 23 | -type metric_type() :: gauge | counter | hist | bar. 24 | -type metric() :: #metric{}. 25 | 26 | -record(endpoint, {transport :: transport(), 27 | address :: inet:ip_address() | inet:hostname(), 28 | port :: inet:port_number(), 29 | host :: inet:hostname(), 30 | path :: string()}). 31 | 32 | -type transport() :: tcp | tls | ws | wss. 33 | -type endpoint() :: #endpoint{}. 34 | -------------------------------------------------------------------------------- /plugins/override_opts.erl: -------------------------------------------------------------------------------- 1 | -module(override_opts). 2 | -export([preprocess/2]). 3 | 4 | override_opts(override, Config, Opts) -> 5 | lists:foldl(fun({Opt, Value}, Conf) -> 6 | rebar_config:set(Conf, Opt, Value) 7 | end, Config, Opts); 8 | override_opts(add, Config, Opts) -> 9 | lists:foldl(fun({Opt, Value}, Conf) -> 10 | V = rebar_config:get_local(Conf, Opt, []), 11 | rebar_config:set(Conf, Opt, V ++ Value) 12 | end, Config, Opts); 13 | override_opts(del, Config, Opts) -> 14 | lists:foldl(fun({Opt, Value}, Conf) -> 15 | V = rebar_config:get_local(Conf, Opt, []), 16 | rebar_config:set(Conf, Opt, V -- Value) 17 | end, Config, Opts). 18 | 19 | preprocess(Config, _Dirs) -> 20 | Overrides = rebar_config:get_local(Config, overrides, []), 21 | TopOverrides = case rebar_config:get_xconf(Config, top_overrides, []) of 22 | [] -> Overrides; 23 | Val -> Val 24 | end, 25 | Config2 = rebar_config:set_xconf(Config, top_overrides, TopOverrides), 26 | try 27 | Config3 = case rebar_app_utils:load_app_file(Config2, _Dirs) of 28 | {ok, C, AppName, _AppData} -> 29 | lists:foldl(fun({Type, AppName2, Opts}, Conf1) when 30 | AppName2 == AppName -> 31 | override_opts(Type, Conf1, Opts); 32 | ({Type, Opts}, Conf1a) -> 33 | override_opts(Type, Conf1a, Opts); 34 | (_, Conf2) -> 35 | Conf2 36 | end, C, TopOverrides); 37 | _ -> 38 | Config2 39 | end, 40 | {ok, Config3, []} 41 | catch 42 | error:badarg -> {ok, Config2, []} 43 | end. 44 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | REBAR=./rebar 2 | 3 | all: src 4 | 5 | src: 6 | $(REBAR) get-deps compile 7 | 8 | clean: 9 | $(REBAR) clean 10 | 11 | distclean: clean 12 | rm -f config.status 13 | rm -f config.log 14 | rm -rf autom4te.cache 15 | rm -rf deps 16 | rm -rf ebin 17 | rm -rf priv 18 | rm -f vars.config 19 | rm -f compile_commands.json 20 | rm -rf dialyzer 21 | 22 | test: all 23 | $(REBAR) -v skip_deps=true eunit 24 | 25 | xref: all 26 | $(REBAR) skip_deps=true xref 27 | 28 | deps := $(wildcard deps/*/ebin) 29 | 30 | dialyzer/erlang.plt: 31 | @mkdir -p dialyzer 32 | @dialyzer --build_plt --output_plt dialyzer/erlang.plt \ 33 | -o dialyzer/erlang.log --apps kernel stdlib erts; \ 34 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 35 | 36 | dialyzer/deps.plt: 37 | @mkdir -p dialyzer 38 | @dialyzer --build_plt --output_plt dialyzer/deps.plt \ 39 | -o dialyzer/deps.log $(deps); \ 40 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 41 | 42 | dialyzer/rtb.plt: 43 | @mkdir -p dialyzer 44 | @dialyzer --build_plt --output_plt dialyzer/rtb.plt \ 45 | -o dialyzer/ejabberd.log ebin; \ 46 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 47 | 48 | erlang_plt: dialyzer/erlang.plt 49 | @dialyzer --plt dialyzer/erlang.plt --check_plt -o dialyzer/erlang.log; \ 50 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 51 | 52 | deps_plt: dialyzer/deps.plt 53 | @dialyzer --plt dialyzer/deps.plt --check_plt -o dialyzer/deps.log; \ 54 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 55 | 56 | rtb_plt: dialyzer/rtb.plt 57 | @dialyzer --plt dialyzer/rtb.plt --check_plt -o dialyzer/ejabberd.log; \ 58 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 59 | 60 | dialyzer: erlang_plt deps_plt rtb_plt 61 | @dialyzer --plts dialyzer/*.plt --no_check_plt \ 62 | --get_warnings -o dialyzer/error.log ebin; \ 63 | status=$$? ; if [ $$status -ne 2 ]; then exit $$status; else exit 0; fi 64 | 65 | check-syntax: 66 | gcc -o nul -S ${CHK_SOURCES} 67 | 68 | .PHONY: clean src test all dialyzer erlang_plt deps_plt rtb_plt 69 | -------------------------------------------------------------------------------- /rebar.config: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | 19 | {erl_opts, [debug_info, 20 | {src_dirs, ["src"]}, 21 | {i, "include"}, 22 | {i, "deps/fast_xml/include"}, 23 | {i, "deps/xmpp/include"}]}. 24 | 25 | {deps, [{lager, ".*", {git, "https://github.com/erlang-lager/lager", {tag, "3.9.1"}}}, 26 | {fast_yaml, ".*", {git, "https://github.com/processone/fast_yaml", {tag, "1.0.15"}}}, 27 | {p1_utils, ".*", {git, "https://github.com/processone/p1_utils", "6ff85e8"}}, 28 | {oneup, ".*", {git, "https://github.com/processone/oneup.git"}}, 29 | {gun, ".*", {git, "https://github.com/ninenines/gun.git", {tag, "1.3.0"}}}, 30 | {xmpp, ".*", {git, "https://github.com/processone/xmpp", "b6ea78e"}}]}. 31 | 32 | {port_env, [{"CFLAGS", "$CFLAGS -std=c99"}, 33 | {"CPPFLAGS", "$CPPFLAGS -std=c++11"}, 34 | {"LDFLAGS", "$LDFLAGS -lssl -lcrypto"}]}. 35 | 36 | {plugins, [override_opts]}. 37 | 38 | {overrides, [{add, [{port_env, [{"ERL_LDFLAGS", " -L$ERL_EI_LIBDIR -lei"}]}]}]}. 39 | 40 | {port_specs, [{"priv/bin/rtb_db", ["c_src/rtb_db.c"]}]}. 41 | 42 | {cover_enabled, true}. 43 | {cover_export_enabled, true}. 44 | 45 | {xref_checks, [undefined_function_calls, undefined_functions, deprecated_function_calls, deprecated_functions]}. 46 | 47 | {profiles, [{test, [{erl_opts, [{src_dirs, ["src", "test"]}]}]}]}. 48 | 49 | %% Local Variables: 50 | %% mode: erlang 51 | %% End: 52 | %% vim: set filetype=erlang tabstop=8: 53 | -------------------------------------------------------------------------------- /src/rtb_sup.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_sup). 19 | 20 | -behaviour(supervisor). 21 | 22 | %% API 23 | -export([start_link/0]). 24 | %% Supervisor callbacks 25 | -export([init/1]). 26 | 27 | -define(SHUTDOWN_TIMEOUT, timer:seconds(1)). 28 | 29 | %%=================================================================== 30 | %% API functions 31 | %%=================================================================== 32 | start_link() -> 33 | supervisor:start_link({local, ?MODULE}, ?MODULE, []). 34 | 35 | %%=================================================================== 36 | %% Supervisor callbacks 37 | %%=================================================================== 38 | init([]) -> 39 | Pool = lists:map( 40 | fun(I) -> 41 | Name = pool_name(I), 42 | worker(Name, rtb_pool, [Name, I]) 43 | end, lists:seq(1, cores())), 44 | {ok, {{one_for_all, 10, 1}, 45 | [worker(rtb_config), 46 | worker(rtb_sm), 47 | worker(rtb_stats), 48 | worker(rtb_http)|Pool]}}. 49 | 50 | %%%=================================================================== 51 | %%% Internal functions 52 | %%%=================================================================== 53 | pool_name(I) -> 54 | list_to_atom("rtb_pool_" ++ integer_to_list(I)). 55 | 56 | worker(Mod) -> 57 | worker(Mod, Mod). 58 | 59 | worker(Name, Mod) -> 60 | worker(Name, Mod, []). 61 | 62 | worker(Name, Mod, Args) -> 63 | {Name, {Mod, start_link, Args}, permanent, ?SHUTDOWN_TIMEOUT, worker, [Mod]}. 64 | 65 | cores() -> 66 | erlang:system_info(logical_processors). 67 | -------------------------------------------------------------------------------- /src/rtb_plot.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_plot). 19 | -compile([{parse_transform, lager_transform}]). 20 | 21 | %% API 22 | -export([render/2]). 23 | 24 | %%%=================================================================== 25 | %%% API 26 | %%%=================================================================== 27 | -spec render(atom(), atom()) -> ok. 28 | render(Name, Type) -> 29 | OutDir = rtb_http:docroot(), 30 | StatsDir = rtb_config:get_option(stats_dir), 31 | DataFile = filename:join(StatsDir, Name) ++ ".dat", 32 | do_render(atom_to_list(Name), DataFile, OutDir, Type). 33 | 34 | %%%=================================================================== 35 | %%% Internal functions 36 | %%%=================================================================== 37 | do_render(Name, DataFile, Dir, Type) -> 38 | OutFile = filename:join(Dir, Name ++ ".png"), 39 | Gnuplot = rtb_config:get_option(gnuplot), 40 | Cmds = ["set title '" ++ Name ++ "'"] ++ 41 | xlabel(Type) ++ ylabel(Type) ++ 42 | ["set grid", 43 | "set boxwidth 0.9", 44 | "set style " ++ style(Type), 45 | "set terminal png small size 400,300", 46 | "set output '" ++ OutFile ++ "'", 47 | "plot '" ++ DataFile ++ "' using 1:2 with " ++ with(Type) ++ " notitle"], 48 | Ret = os:cmd(Gnuplot ++ " -e \"" ++ string:join(Cmds, "; ") ++ "\""), 49 | check_ret(Name, Ret). 50 | 51 | style(hist) -> "fill solid 0.5"; 52 | style(_) -> "line 1 linecolor rgb '#0060ad' linetype 1". 53 | 54 | with(hist) -> "boxes"; 55 | with(_) -> "lines linestyle 1". 56 | 57 | ylabel(hist) -> ["set ylabel '%' rotate by 0"]; 58 | ylabel(rate) -> ["set ylabel 'number/sec'"]; 59 | ylabel(_) -> ["set ylabel 'number'"]. 60 | 61 | xlabel(hist) -> ["set xlabel 'round trip time (milliseconds)'"]; 62 | xlabel(_) -> ["set xlabel 'benchmark duration (seconds)'"]. 63 | 64 | check_ret(_, "") -> ok; 65 | check_ret(_, "Warning:" ++ _) -> ok; 66 | check_ret(Name, Err) -> 67 | lager:error("Failed to render '~s' graph:~n~s", [Name, Err]). 68 | -------------------------------------------------------------------------------- /src/rtb_sm.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_sm). 19 | -compile([{parse_transform, lager_transform}, 20 | {no_auto_import, [register/2, unregister/1]}]). 21 | -behaviour(p1_server). 22 | 23 | %% API 24 | -export([start_link/0, register/3, unregister/1, 25 | lookup/1, random/0, size/0]). 26 | 27 | %% gen_server callbacks 28 | -export([init/1, handle_call/3, handle_cast/2, handle_info/2, 29 | terminate/2, code_change/3]). 30 | 31 | -record(state, {}). 32 | 33 | %%%=================================================================== 34 | %%% API 35 | %%%=================================================================== 36 | start_link() -> 37 | gen_server:start_link({local, ?MODULE}, ?MODULE, [], []). 38 | 39 | register(I, Pid, Val) -> 40 | ets:insert(?MODULE, {I, {Pid, Val}}). 41 | 42 | unregister(I) -> 43 | try ets:delete(?MODULE, I) 44 | catch _:badarg -> true 45 | end. 46 | 47 | lookup(I) -> 48 | try ets:lookup_element(?MODULE, I, 2) of 49 | {Pid, Val} -> {ok, {Pid, I, Val}} 50 | catch _:badarg -> 51 | {error, notfound} 52 | end. 53 | 54 | random() -> 55 | case {ets:first(?MODULE), ets:last(?MODULE)} of 56 | {From, To} when is_integer(From), is_integer(To) -> 57 | Rnd = p1_rand:uniform(From, To), 58 | Next = case ets:next(?MODULE, Rnd) of 59 | '$end_of_table' -> From; 60 | K -> K 61 | end, 62 | case lookup(Next) of 63 | {ok, Val} -> {ok, Val}; 64 | {error, notfound} -> random() 65 | end; 66 | _ -> 67 | {error, empty} 68 | end. 69 | 70 | size() -> 71 | ets:info(?MODULE, size). 72 | 73 | %%%=================================================================== 74 | %%% gen_server callbacks 75 | %%%=================================================================== 76 | init([]) -> 77 | process_flag(trap_exit, true), 78 | ets:new(?MODULE, [named_table, public, 79 | {read_concurrency, true}, 80 | ordered_set]), 81 | {ok, #state{}}. 82 | 83 | handle_call(_Request, _From, State) -> 84 | Reply = ok, 85 | {reply, Reply, State}. 86 | 87 | handle_cast(_Msg, State) -> 88 | {noreply, State}. 89 | 90 | handle_info(_Info, State) -> 91 | {noreply, State}. 92 | 93 | terminate(_Reason, _State) -> 94 | ok. 95 | 96 | code_change(_OldVsn, State, _Extra) -> 97 | {ok, State}. 98 | 99 | %%%=================================================================== 100 | %%% Internal functions 101 | %%%=================================================================== 102 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at contact@process-one.net. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /src/rtb_http.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_http). 19 | -compile([{parse_transform, lager_transform}]). 20 | 21 | %% API 22 | -export([start_link/0, docroot/0, do/1, response_default_headers/0]). 23 | 24 | -include_lib("inets/include/httpd.hrl"). 25 | 26 | %%%=================================================================== 27 | %%% API 28 | %%%=================================================================== 29 | start_link() -> 30 | Opts = httpd_options(), 31 | DocRoot = proplists:get_value(document_root, Opts), 32 | case create_index_html(DocRoot) of 33 | ok -> 34 | Port = proplists:get_value(port, Opts), 35 | case inets:start(httpd, Opts) of 36 | {ok, Pid} -> 37 | lager:info("Accepting HTTP connections on port ~B", [Port]), 38 | {ok, Pid}; 39 | {error, Why} -> 40 | log_error(Why, Port), 41 | Why 42 | end; 43 | {error, _} -> 44 | rtb:halt() 45 | end. 46 | 47 | docroot() -> 48 | ServerRoot = rtb_config:get_option(www_dir), 49 | filename:join(ServerRoot, "data"). 50 | 51 | do(#mod{method = Method, data = Data}) -> 52 | if Method == "GET"; Method == "HEAD" -> 53 | case lists:keyfind(real_name, 1, Data) of 54 | {real_name, {Path, _}} -> 55 | Basename = filename:basename(Path, ".png"), 56 | try list_to_existing_atom(Basename) of 57 | Name -> 58 | case rtb_stats:get_type(Name) of 59 | undefined -> 60 | ok; 61 | Type -> 62 | rtb_stats:flush(Name), 63 | rtb_plot:render(Name, Type) 64 | end 65 | catch _:badarg -> 66 | ok 67 | end; 68 | false -> 69 | ok 70 | end; 71 | true -> 72 | ok 73 | end, 74 | {proceed, Data}. 75 | 76 | response_default_headers() -> 77 | [{"Cache-Control", "no-cache"}]. 78 | 79 | %%%=================================================================== 80 | %%% Internal functions 81 | %%%=================================================================== 82 | httpd_options() -> 83 | ServerRoot = rtb_config:get_option(www_dir), 84 | Port = rtb_config:get_option(www_port), 85 | Domain = rtb_config:get_option(www_domain), 86 | DocRoot = filename:join(ServerRoot, "data"), 87 | [{port, Port}, 88 | {server_root, ServerRoot}, 89 | {document_root, DocRoot}, 90 | {mime_types, [{"html", "text/html"}, 91 | {"png", "image/png"}]}, 92 | {directory_index, ["index.html"]}, 93 | {modules, [mod_alias, ?MODULE, mod_get, mod_head]}, 94 | {script_nocache, true}, 95 | {customize, ?MODULE}, 96 | {server_name, Domain}]. 97 | 98 | create_index_html(DocRoot) -> 99 | Data = ["", 100 | lists:map( 101 | fun(Name) -> 102 | [""] 103 | end, rtb_stats:get_metrics()), 104 | "", 105 | ""], 106 | File = filename:join(DocRoot, "index.html"), 107 | case filelib:ensure_dir(File) of 108 | ok -> 109 | case file:write_file(File, Data) of 110 | ok -> 111 | ok; 112 | {error, Why} = Err -> 113 | lager:critical("Failed to write to ~s: ~s", 114 | [File, file:format_error(Why)]), 115 | Err 116 | end; 117 | {error, Why} = Err -> 118 | lager:critical("Failed to create directory ~s: ~s", 119 | [DocRoot, file:format_error(Why)]), 120 | Err 121 | end. 122 | 123 | script() -> 124 | case rtb_config:get_option(www_refresh) of 125 | 0 -> ""; 126 | N -> script(timer:seconds(N)) 127 | end. 128 | 129 | script(Timeout) -> 130 | "setInterval(function() { 131 | var images = document.images; 132 | for (var i=0; i Code of Conduct 15 | 16 | Help us keep our community open-minded and inclusive. Please read and follow our [Code of Conduct][coc]. 17 | 18 | ## Questions, Bugs, Features 19 | 20 | ### Got a Question or Problem? 21 | 22 | Do not open issues for general support questions as we want to keep GitHub issues for bug reports 23 | and feature requests. You've got much better chances of getting your question answered on dedicated 24 | support platforms, the best being [Stack Overflow][stackoverflow]. 25 | 26 | Stack Overflow is a much better place to ask questions since: 27 | 28 | - there are thousands of people willing to help on Stack Overflow 29 | - questions and answers stay available for public viewing so your question / answer might help 30 | someone else 31 | - Stack Overflow's voting system assures that the best answers are prominently visible. 32 | 33 | To save your and our time, we will systematically close all issues that are requests for general 34 | support and redirect people to the section you are reading right now. 35 | 36 | ### Found an Issue or Bug? 37 | 38 | If you find a bug in the source code, you can help us by submitting an issue to our 39 | [GitHub Repository][github]. Even better, you can submit a Pull Request with a fix. 40 | 41 | ### Missing a Feature? 42 | 43 | You can request a new feature by submitting an issue to our [GitHub Repository][github-issues]. 44 | 45 | If you would like to implement a new feature then consider what kind of change it is: 46 | 47 | * **Major Changes** that you wish to contribute to the project should be discussed first in an 48 | [GitHub issue][github-issues] that clearly outlines the changes and benefits of the feature. 49 | * **Small Changes** can directly be crafted and submitted to the [GitHub Repository][github] 50 | as a Pull Request. See the section about [Pull Request Submission Guidelines](#submit-pr). 51 | 52 | ## Issue Submission Guidelines 53 | 54 | Before you submit your issue search the archive, maybe your question was already answered. 55 | 56 | If your issue appears to be a bug, and hasn't been reported, open a new issue. Help us to maximize 57 | the effort we can spend fixing issues and adding new features, by not reporting duplicate issues. 58 | 59 | The "[new issue][github-new-issue]" form contains a number of prompts that you should fill out to 60 | make it easier to understand and categorize the issue. 61 | 62 | ## Pull Request Submission Guidelines 63 | 64 | By submitting a pull request for a code or doc contribution, you need to have the right 65 | to grant your contribution's copyright license to ProcessOne. Please check [ProcessOne CLA][cla] 66 | for details. 67 | 68 | Before you submit your pull request consider the following guidelines: 69 | 70 | * Search [GitHub][github-pr] for an open or closed Pull Request 71 | that relates to your submission. You don't want to duplicate effort. 72 | * Make your changes in a new git branch: 73 | 74 | ```shell 75 | git checkout -b my-fix-branch master 76 | ``` 77 | * Test your changes and, if relevant, expand the automated test suite. 78 | * Create your patch commit, including appropriate test cases. 79 | * If the changes affect public APIs, change or add relevant documentation. 80 | * Commit your changes using a descriptive commit message. 81 | 82 | ```shell 83 | git commit -a 84 | ``` 85 | Note: the optional commit `-a` command line option will automatically "add" and "rm" edited files. 86 | 87 | * Push your branch to GitHub: 88 | 89 | ```shell 90 | git push origin my-fix-branch 91 | ``` 92 | 93 | * In GitHub, send a pull request to `master` branch. This will trigger the continuous integration and run the test. 94 | We will also notify you if you have not yet signed the [contribution agreement][cla]. 95 | 96 | * If you find that the continunous integration has failed, look into the logs to find out 97 | if your changes caused test failures, the commit message was malformed etc. If you find that the 98 | tests failed or times out for unrelated reasons, you can ping a team member so that the build can be 99 | restarted. 100 | 101 | * If we suggest changes, then: 102 | 103 | * Make the required updates. 104 | * Test your changes and test cases. 105 | * Commit your changes to your branch (e.g. `my-fix-branch`). 106 | * Push the changes to your GitHub repository (this will update your Pull Request). 107 | 108 | You can also amend the initial commits and force push them to the branch. 109 | 110 | ```shell 111 | git rebase master -i 112 | git push origin my-fix-branch -f 113 | ``` 114 | 115 | This is generally easier to follow, but separate commits are useful if the Pull Request contains 116 | iterations that might be interesting to see side-by-side. 117 | 118 | That's it! Thank you for your contribution! 119 | 120 | ## Signing the Contributor License Agreement (CLA) 121 | 122 | Upon submitting a Pull Request, we will ask you to sign our CLA if you haven't done 123 | so before. It's a quick process, we promise, and you will be able to do it all online 124 | 125 | You can read [ProcessOne Contribution License Agreement][cla] in PDF. 126 | 127 | This is part of the legal framework of the open-source ecosystem that adds some red tape, 128 | but protects both the contributor and the company / foundation behind the project. It also 129 | gives us the option to relicense the code with a more permissive license in the future. 130 | 131 | 132 | [coc]: https://github.com/processone/rtb/blob/master/CODE_OF_CONDUCT.md 133 | [stackoverflow]: https://stackoverflow.com/ 134 | [github]: https://github.com/processone/rtb 135 | [github-issues]: https://github.com/processone/rtb/issues 136 | [github-new-issue]: https://github.com/processone/rtb/issues/new 137 | [github-pr]: https://github.com/processone/rtb/pulls 138 | [cla]: https://www.process-one.net/resources/ejabberd-cla.pdf 139 | [license]: https://github.com/processone/rtb/blob/master/LICENSE 140 | -------------------------------------------------------------------------------- /src/mod_xmpp_http.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(mod_xmpp_http). 19 | -compile([{parse_transform, lager_transform}]). 20 | 21 | %% API 22 | -export([upload/4, download/4, format_error/1]). 23 | 24 | -define(BUF_SIZE, 65536). 25 | 26 | -type upload_error() :: {gen_tcp, inet:posix()} | {ssl, term()} | 27 | {http, string()} | {bad_url, string()}. 28 | 29 | %%%=================================================================== 30 | %%% API 31 | %%%=================================================================== 32 | -spec upload(string(), non_neg_integer(), list(), non_neg_integer()) -> 33 | {error, upload_error()}. 34 | upload(URL, Size, ConnOpts, Timeout) -> 35 | case connect(URL, ConnOpts, Timeout) of 36 | {ok, SockMod, Sock, Host, Port, Path, Query} -> 37 | Res = upload_file(SockMod, Sock, Host, Port, Path, Query, Size, Timeout), 38 | SockMod:close(Sock), 39 | Res; 40 | {error, _} = Err -> 41 | Err 42 | end. 43 | 44 | download(URL, Size, ConnOpts, Timeout) -> 45 | case connect(URL, ConnOpts, Timeout) of 46 | {ok, SockMod, Sock, Host, Port, Path, Query} -> 47 | Res = download_file(SockMod, Sock, Host, Port, Path, Query, Size, Timeout), 48 | SockMod:close(Sock), 49 | Res; 50 | {error, _} = Err -> 51 | Err 52 | end. 53 | 54 | -spec format_error(upload_error()) -> string(). 55 | format_error({bad_url, URL}) -> 56 | "Failed to parse URL: " ++ URL; 57 | format_error({http, Reason}) -> 58 | "HTTP failure: " ++ Reason; 59 | format_error(Reason) -> 60 | "HTTP failure: " ++ do_format_error(Reason). 61 | 62 | %%%=================================================================== 63 | %%% Internal functions 64 | %%%=================================================================== 65 | -spec do_format_error(upload_error()) -> string(). 66 | do_format_error({_, closed}) -> 67 | "connection closed"; 68 | do_format_error({_, timeout}) -> 69 | inet:format_error(etimedout); 70 | do_format_error({ssl, Reason}) -> 71 | ssl:format_error(Reason); 72 | do_format_error({gen_tcp, Reason}) -> 73 | case inet:format_error(Reason) of 74 | "unknown POSIX error" -> atom_to_list(Reason); 75 | Txt -> Txt 76 | end. 77 | 78 | connect(URL, ConnOpts, Timeout) -> 79 | case http_uri:parse(URL) of 80 | {ok, {Scheme, _, Host, Port, Path, Query}} 81 | when Scheme == http; Scheme == https -> 82 | SockMod = sockmod(Scheme), 83 | Opts = opts(ConnOpts, Timeout), 84 | case SockMod:connect(Host, Port, Opts, Timeout) of 85 | {ok, Sock} -> 86 | {ok, SockMod, Sock, Host, Port, Path, Query}; 87 | {error, Reason} -> 88 | {error, {SockMod, Reason}} 89 | end; 90 | _ -> 91 | {error, {bad_url, URL}} 92 | end. 93 | 94 | upload_file(SockMod, Sock, Host, Port, Path, Query, Size, Timeout) -> 95 | Hdrs = io_lib:format( 96 | "PUT ~s~s HTTP/1.1\r\n" 97 | "Host: ~s:~B\r\n" 98 | "Content-Type: image/png\r\n" 99 | "Content-Length: ~B\r\n" 100 | "Connection: keep-alive\r\n\r\n", 101 | [Path, Query, Host, Port, Size]), 102 | case SockMod:send(Sock, Hdrs) of 103 | ok -> 104 | Chunk = p1_rand:bytes(?BUF_SIZE), 105 | case upload_data(SockMod, Sock, Size, Chunk) of 106 | ok -> 107 | recv_headers(SockMod, Sock, undefined, Timeout); 108 | {error, _} = Err -> 109 | Err 110 | end; 111 | {error, Reason} -> 112 | {error, {SockMod, Reason}} 113 | end. 114 | 115 | upload_data(_SockMod, _Sock, 0, _Chunk) -> 116 | ok; 117 | upload_data(SockMod, Sock, Size, Chunk) -> 118 | Data = if Size >= ?BUF_SIZE -> 119 | Chunk; 120 | true -> 121 | binary:part(Chunk, 0, Size) 122 | end, 123 | case SockMod:send(Sock, Data) of 124 | ok -> 125 | NewSize = min(Size, ?BUF_SIZE), 126 | upload_data(SockMod, Sock, Size-NewSize, Chunk); 127 | {error, Reason} -> 128 | {error, {SockMod, Reason}} 129 | end. 130 | 131 | download_file(SockMod, Sock, Host, Port, Path, Query, Size, Timeout) -> 132 | Hdrs = io_lib:format( 133 | "GET ~s~s HTTP/1.1\r\n" 134 | "Host: ~s:~B\r\n" 135 | "Connection: keep-alive\r\n\r\n", 136 | [Path, Query, Host, Port]), 137 | case SockMod:send(Sock, Hdrs) of 138 | ok -> 139 | recv_headers(SockMod, Sock, Size, Timeout); 140 | {error, Reason} -> 141 | {error, {SockMod, Reason}} 142 | end. 143 | 144 | recv_body(_SockMod, _Sock, 0, _Timeout) -> 145 | ok; 146 | recv_body(SockMod, Sock, Size, Timeout) -> 147 | ChunkSize = min(Size, ?BUF_SIZE), 148 | case SockMod:recv(Sock, ChunkSize, Timeout) of 149 | {ok, Data} -> 150 | recv_body(SockMod, Sock, Size-size(Data), Timeout); 151 | {error, Reason} -> 152 | {error, {SockMod, Reason}} 153 | end. 154 | 155 | recv_headers(SockMod, Sock, Size, Timeout) -> 156 | case SockMod:recv(Sock, 0, Timeout) of 157 | {ok, {http_response, _, Code, String}} -> 158 | if Code >= 200, Code < 300 -> 159 | recv_headers(SockMod, Sock, Size, Timeout); 160 | true -> 161 | C = integer_to_binary(Code), 162 | Reason = <>, 163 | {error, {http, binary_to_list(Reason)}} 164 | end; 165 | {ok, {http_header, _, 'Content-Length', _, Len}} -> 166 | case binary_to_integer(Len) of 167 | Size -> 168 | recv_headers(SockMod, Sock, Size, Timeout); 169 | Size1 when Size == undefined -> 170 | recv_headers(SockMod, Sock, Size1, Timeout); 171 | _ -> 172 | Txt = "Unexpected Content-Length", 173 | lager:warning("~s: ~s (should be: ~B)", [Txt, Len, Size]), 174 | {error, {http, Txt}} 175 | end; 176 | {ok, {http_header, _, _, _, _}} -> 177 | recv_headers(SockMod, Sock, Size, Timeout); 178 | {ok, {http_error, String}} -> 179 | {error, {http, binary_to_list(iolist_to_binary(String))}}; 180 | {ok, http_eoh} -> 181 | case setopts(SockMod, Sock, [{packet, 0}]) of 182 | ok -> 183 | recv_body(SockMod, Sock, Size, Timeout); 184 | {error, Reason} -> 185 | {error, {SockMod, Reason}} 186 | end; 187 | {error, Reason} -> 188 | {error, {SockMod, Reason}} 189 | end. 190 | 191 | sockmod(http) -> gen_tcp; 192 | sockmod(https) -> ssl. 193 | 194 | setopts(gen_tcp, Sock, Opts) -> 195 | inet:setopts(Sock, Opts); 196 | setopts(ssl, Sock, Opts) -> 197 | ssl:setopts(Sock, Opts). 198 | 199 | opts(Opts, Timeout) -> 200 | [binary, 201 | {packet, http_bin}, 202 | {send_timeout, Timeout}, 203 | {send_timeout_close, true}, 204 | {recbuf, ?BUF_SIZE}, 205 | {sndbuf, ?BUF_SIZE}, 206 | {active, false}|Opts]. 207 | -------------------------------------------------------------------------------- /src/rtb.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb). 19 | -compile([{parse_transform, lager_transform}, 20 | {no_auto_import, [halt/0]}]). 21 | -behaviour(application). 22 | 23 | %% Application callbacks 24 | -export([start/0, start/2, stop/0, stop/1, halt/0, halt/2]). 25 | %% Miscellaneous API 26 | -export([random_server/0, format_list/1, replace/2, cancel_timer/1, 27 | make_pattern/1, set_debug/0]). 28 | 29 | -include("rtb.hrl"). 30 | 31 | -callback load() -> ok | {error, any()}. 32 | -callback start(pos_integer(), 33 | [gen_tcp:option()], 34 | [endpoint()], 35 | boolean()) -> 36 | {ok, pid()} | {error, any()} | ignore. 37 | -callback options() -> [{atom(), any()} | atom()]. 38 | -callback prep_option(atom(), any()) -> {atom(), any()}. 39 | -callback metrics() -> [{atom(), metric()}]. 40 | 41 | -opaque pattern() :: [char() | current | {random, pool | sm} | 42 | {range, pos_integer(), pos_integer()}] | 43 | binary(). 44 | -export_type([pattern/0]). 45 | 46 | %%%=================================================================== 47 | %%% Application callbacks 48 | %%%=================================================================== 49 | start() -> 50 | application:ensure_all_started(?MODULE). 51 | 52 | stop() -> 53 | application:stop(?MODULE). 54 | 55 | start(_StartType, _StartArgs) -> 56 | case start_apps() of 57 | ok -> 58 | case rtb_sup:start_link() of 59 | {ok, SupPid} -> 60 | case rtb_watchdog:start() of 61 | {ok, _} -> 62 | {ok, SupPid}; 63 | _Err -> 64 | halt() 65 | end; 66 | _Err -> 67 | halt() 68 | end; 69 | Err -> 70 | Err 71 | end. 72 | 73 | stop(_State) -> 74 | ok. 75 | 76 | %%%=================================================================== 77 | %%% Miscellaneous functions 78 | %%%=================================================================== 79 | start_apps() -> 80 | try 81 | LogRateLimit = 10000, 82 | ok = application:load(lager), 83 | application:set_env(lager, error_logger_hwm, LogRateLimit), 84 | application:set_env( 85 | lager, handlers, 86 | [{lager_console_backend, [{level, info}]}, 87 | {lager_file_backend, [{file, "log/rtb.log"}, 88 | {level, info}, {size, 0}]}]), 89 | application:set_env(lager, crash_log, "log/crash.log"), 90 | application:set_env(lager, crash_log_size, 0), 91 | {ok, _} = application:ensure_all_started(lager), 92 | lists:foreach( 93 | fun(Handler) -> 94 | lager:set_loghwm(Handler, LogRateLimit) 95 | end, gen_event:which_handlers(lager_event)), 96 | lists:foreach( 97 | fun(App) -> 98 | {ok, _} = application:ensure_all_started(App) 99 | end, [crypto, inets, p1_utils, fast_yaml, oneup, gun]) 100 | catch _:{badmatch, Reason} -> 101 | Reason 102 | end. 103 | 104 | set_debug() -> 105 | lists:foreach( 106 | fun({lager_file_backend, _} = H) -> 107 | lager:set_loglevel(H, debug); 108 | (lager_console_backend = H) -> 109 | lager:set_loglevel(H, debug); 110 | (_) -> 111 | ok 112 | end, gen_event:which_handlers(lager_event)). 113 | 114 | -spec halt() -> no_return(). 115 | halt() -> 116 | application:stop(sasl), 117 | application:stop(lager), 118 | halt(0). 119 | 120 | -spec halt(io:format(), list()) -> no_return(). 121 | halt(Fmt, Args) -> 122 | Txt = io_lib:format(Fmt, Args), 123 | lager:critical("Benchmark failure: ~s", [Txt]), 124 | halt(). 125 | 126 | -spec random_server() -> [endpoint()]. 127 | random_server() -> 128 | Addrs = rtb_config:get_option(servers), 129 | case length(Addrs) of 130 | Len when Len >= 2 -> 131 | Addr = lists:nth(p1_rand:uniform(1, Len), Addrs), 132 | [Addr]; 133 | _ -> 134 | Addrs 135 | end. 136 | 137 | -spec replace(pattern(), pos_integer() | iodata()) -> binary(). 138 | replace(Subj, _) when is_binary(Subj) -> 139 | Subj; 140 | replace(Subj, I) when is_integer(I) -> 141 | replace(Subj, integer_to_list(I)); 142 | replace(Subj, Repl) -> 143 | iolist_to_binary( 144 | lists:map( 145 | fun(current) -> 146 | Repl; 147 | ({random, unique}) -> 148 | integer_to_list( 149 | p1_time_compat:unique_integer([positive])); 150 | ({random, sm}) -> 151 | case rtb_sm:random() of 152 | {ok, {_, I, _}} -> integer_to_list(I); 153 | {error, _} -> Repl 154 | end; 155 | ({range, From, To}) -> 156 | integer_to_list(p1_rand:uniform(From, To)); 157 | (Char) -> 158 | Char 159 | end, Subj)). 160 | 161 | -spec format_list([iodata() | atom()]) -> binary(). 162 | format_list([]) -> 163 | <<>>; 164 | format_list(L) -> 165 | [H|T] = lists:map( 166 | fun(A) when is_atom(A) -> 167 | atom_to_binary(A, latin1); 168 | (B) -> 169 | iolist_to_binary(B) 170 | end, L), 171 | <>. 172 | 173 | -spec cancel_timer(undefined | reference()) -> ok. 174 | cancel_timer(undefined) -> 175 | ok; 176 | cancel_timer(TRef) -> 177 | erlang:cancel_timer(TRef), 178 | receive {timeout, TRef, _} -> ok 179 | after 0 -> ok 180 | end. 181 | 182 | -spec make_pattern(binary()) -> pattern(). 183 | make_pattern(S) -> 184 | Parts = case re:run(S, "\\[([0-9]+)..([0-9]+)\\]") of 185 | {match, [{Pos,Len},{F,FLen},{T,TLen}]} -> 186 | Head = binary:part(S, {0,Pos}), 187 | Tail = binary:part(S, {Pos+Len, size(S)-(Pos+Len)}), 188 | From = binary_to_integer(binary:part(S, {F,FLen})), 189 | To = binary_to_integer(binary:part(S, {T,TLen})), 190 | if From < To -> 191 | [Head, {range, From, To}, Tail]; 192 | From == To -> 193 | [Head, integer_to_binary(From), Tail] 194 | end; 195 | nomatch -> 196 | [S] 197 | end, 198 | Pattern = lists:flatmap( 199 | fun({_, _, _} = Range) -> 200 | [Range]; 201 | (Part) -> 202 | nomatch = binary:match(Part, <<$[>>), 203 | nomatch = binary:match(Part, <<$]>>), 204 | lists:map( 205 | fun($%) -> current; 206 | ($*) -> {random, unique}; 207 | ($?) -> {random, sm}; 208 | (C) -> C 209 | end, binary_to_list(Part)) 210 | end, Parts), 211 | try iolist_to_binary(Pattern) 212 | catch _:_ -> Pattern 213 | end. 214 | 215 | %%%=================================================================== 216 | %%% Internal functions 217 | %%%=================================================================== 218 | format_tail([]) -> 219 | <<>>; 220 | format_tail([S]) -> 221 | <<" and ", S/binary>>; 222 | format_tail([H|T]) -> 223 | <<", ", H/binary, (format_tail(T))/binary>>. 224 | -------------------------------------------------------------------------------- /include/mqtt.hrl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2018 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -define(MQTT_VERSION_4, 4). 19 | -define(MQTT_VERSION_5, 5). 20 | 21 | -record(connect, {proto_level = 4 :: non_neg_integer(), 22 | will :: undefined | publish(), 23 | clean_start = true :: boolean(), 24 | keep_alive = 0 :: non_neg_integer(), 25 | client_id = <<>> :: binary(), 26 | username = <<>> :: binary(), 27 | password = <<>> :: binary(), 28 | will_properties = #{} :: properties(), 29 | properties = #{} :: properties()}). 30 | -record(connack, {session_present = false :: boolean(), 31 | code = success :: reason_code(), 32 | properties = #{} :: properties()}). 33 | 34 | -record(publish, {id :: undefined | non_neg_integer(), 35 | dup = false :: boolean(), 36 | qos = 0 :: qos(), 37 | retain = false :: boolean(), 38 | topic :: binary(), 39 | payload :: binary(), 40 | properties = #{} :: properties(), 41 | meta = #{} :: map()}). 42 | -record(puback, {id :: non_neg_integer(), 43 | code = success :: reason_code(), 44 | properties = #{} :: properties()}). 45 | -record(pubrec, {id :: non_neg_integer(), 46 | code = success :: reason_code(), 47 | properties = #{} :: properties()}). 48 | -record(pubrel, {id :: non_neg_integer(), 49 | code = success :: reason_code(), 50 | properties = #{} :: properties(), 51 | meta = #{} :: map()}). 52 | -record(pubcomp, {id :: non_neg_integer(), 53 | code = success :: reason_code(), 54 | properties = #{} :: properties()}). 55 | 56 | -record(subscribe, {id :: non_neg_integer(), 57 | filters :: [{binary(), sub_opts()}], 58 | properties = #{} :: properties(), 59 | meta = #{} :: map()}). 60 | -record(suback, {id :: non_neg_integer(), 61 | codes :: [reason_code()], 62 | properties = #{} :: properties()}). 63 | 64 | -record(unsubscribe, {id :: non_neg_integer(), 65 | filters :: [binary()], 66 | properties = #{} :: properties(), 67 | meta = #{} :: map()}). 68 | -record(unsuback, {id :: non_neg_integer(), 69 | codes :: [reason_code()], 70 | properties = #{} :: properties()}). 71 | 72 | -record(pingreq, {meta = #{} :: map()}). 73 | -record(pingresp, {}). 74 | 75 | -record(disconnect, {code = 'normal-disconnection' :: reason_code(), 76 | properties = #{} :: properties()}). 77 | 78 | -record(auth, {code = success :: reason_code(), 79 | properties = #{} :: properties()}). 80 | 81 | -record(sub_opts, {qos = 0 :: qos(), 82 | no_local = false :: boolean(), 83 | retain_as_published = false :: boolean(), 84 | retain_handling = 0 :: 0..2}). 85 | 86 | -type qos() :: 0|1|2. 87 | -type sub_opts() :: #sub_opts{}. 88 | -type utf8_pair() :: {binary(), binary()}. 89 | -type properties() :: map(). 90 | -type property() :: assigned_client_identifier | 91 | authentication_data | 92 | authentication_method | 93 | content_type | 94 | correlation_data | 95 | maximum_packet_size | 96 | maximum_qos | 97 | message_expiry_interval | 98 | payload_format_indicator | 99 | reason_string | 100 | receive_maximum | 101 | request_problem_information | 102 | request_response_information | 103 | response_information | 104 | response_topic | 105 | retain_available | 106 | server_keep_alive | 107 | server_reference | 108 | session_expiry_interval | 109 | shared_subscription_available | 110 | subscription_identifier | 111 | subscription_identifiers_available | 112 | topic_alias | 113 | topic_alias_maximum | 114 | user_property | 115 | wildcard_subscription_available | 116 | will_delay_interval. 117 | -type reason_code() :: 'success' | 118 | 'normal-disconnection' | 119 | 'granted-qos-0' | 120 | 'granted-qos-1' | 121 | 'granted-qos-2' | 122 | 'disconnect-with-will-message' | 123 | 'no-matching-subscribers' | 124 | 'no-subscription-existed' | 125 | 'continue-authentication' | 126 | 're-authenticate' | 127 | 'unspecified-error' | 128 | 'malformed-packet' | 129 | 'protocol-error' | 130 | 'implementation-specific-error' | 131 | 'unsupported-protocol-version' | 132 | 'client-identifier-not-valid' | 133 | 'bad-user-name-or-password' | 134 | 'not-authorized' | 135 | 'server-unavailable' | 136 | 'server-busy' | 137 | 'banned' | 138 | 'server-shutting-down' | 139 | 'bad-authentication-method' | 140 | 'keep-alive-timeout' | 141 | 'session-taken-over' | 142 | 'topic-filter-invalid' | 143 | 'topic-name-invalid' | 144 | 'packet-identifier-in-use' | 145 | 'packet-identifier-not-found' | 146 | 'receive-maximum-exceeded' | 147 | 'topic-alias-invalid' | 148 | 'packet-too-large' | 149 | 'message-rate-too-high' | 150 | 'quota-exceeded' | 151 | 'administrative-action' | 152 | 'payload-format-invalid' | 153 | 'retain-not-supported' | 154 | 'qos-not-supported' | 155 | 'use-another-server' | 156 | 'server-moved' | 157 | 'shared-subscriptions-not-supported' | 158 | 'connection-rate-exceeded' | 159 | 'maximum-connect-time' | 160 | 'subscription-identifiers-not-supported' | 161 | 'wildcard-subscriptions-not-supported'. 162 | 163 | -type connect() :: #connect{}. 164 | -type connack() :: #connack{}. 165 | -type publish() :: #publish{}. 166 | -type puback() :: #puback{}. 167 | -type pubrel() :: #pubrel{}. 168 | -type pubrec() :: #pubrec{}. 169 | -type pubcomp() :: #pubcomp{}. 170 | -type subscribe() :: #subscribe{}. 171 | -type suback() :: #suback{}. 172 | -type unsubscribe() :: #unsubscribe{}. 173 | -type unsuback() :: #unsuback{}. 174 | -type pingreq() :: #pingreq{}. 175 | -type pingresp() :: #pingresp{}. 176 | -type disconnect() :: #disconnect{}. 177 | -type auth() :: #auth{}. 178 | 179 | -type mqtt_packet() :: connect() | connack() | publish() | puback() | 180 | pubrel() | pubrec() | pubcomp() | subscribe() | 181 | suback() | unsubscribe() | unsuback() | pingreq() | 182 | pingresp() | disconnect() | auth(). 183 | -type mqtt_version() :: ?MQTT_VERSION_4 | ?MQTT_VERSION_5. 184 | -------------------------------------------------------------------------------- /src/mod_xmpp_proxy65.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(mod_xmpp_proxy65). 19 | -compile([{parse_transform, lager_transform}]). 20 | -behaviour(gen_server). 21 | 22 | %% API 23 | -export([recv/6, connect/6, activate/1, format_error/1]). 24 | 25 | %% gen_server callbacks 26 | -export([init/1, handle_call/3, handle_cast/2, handle_info/2, 27 | terminate/2, code_change/3]). 28 | 29 | -define(BUF_SIZE, 65536). 30 | %%-define(BUF_SIZE, 8192). 31 | %% SOCKS5 stuff 32 | -define(VERSION_5, 5). 33 | -define(AUTH_ANONYMOUS, 0). 34 | -define(CMD_CONNECT, 1). 35 | -define(ATYP_DOMAINNAME, 3). 36 | -define(SUCCESS, 0). 37 | 38 | -type sockmod() :: gen_tcp | ssl. 39 | -type socket() :: gen_tcp:socket() | ssl:sslsocket(). 40 | -type proxy65_error() :: {socks5, atom()} | {sockmod(), atom()} | 41 | crashed | shutdown | init_timeout | 42 | activation_timeout. 43 | 44 | -record(state, {host :: string(), 45 | port :: inet:port_number(), 46 | hash :: binary(), 47 | sockmod :: sockmod(), 48 | socket :: socket(), 49 | opts :: [gen_tcp:option()], 50 | timeout :: non_neg_integer(), 51 | size :: non_neg_integer(), 52 | owner :: pid(), 53 | action :: send | recv}). 54 | 55 | %%%=================================================================== 56 | %%% API 57 | %%%=================================================================== 58 | recv(Host, Port, Hash, Size, ConnOpts, Timeout) -> 59 | start(recv, Host, Port, Hash, Size, ConnOpts, Timeout). 60 | 61 | connect(Host, Port, Hash, Size, ConnOpts, Timeout) -> 62 | start(send, Host, Port, Hash, Size, ConnOpts, Timeout). 63 | 64 | activate(Pid) -> 65 | gen_server:cast(Pid, activate). 66 | 67 | -spec format_error(proxy65_error()) -> string(). 68 | format_error({socks5, unexpected_response}) -> 69 | "Proxy65 failure: unexpected SOCKS5 response"; 70 | format_error(crashed) -> 71 | "Proxy65 failure: connection has been crashed"; 72 | format_error(shutdown) -> 73 | "Proxy65 failure: the system is shutting down"; 74 | format_error(init_timeout) -> 75 | "Proxy65 failure: timed out during initialization"; 76 | format_error(activation_timeout) -> 77 | "Proxy65 failure: timed out waiting for activation"; 78 | format_error(Reason) -> 79 | "Proxy65 failure: " ++ format_socket_error(Reason). 80 | 81 | %%%=================================================================== 82 | %%% gen_server callbacks 83 | %%%=================================================================== 84 | init([Action, Owner, Host, Port, Hash, Size, ConnOpts, Timeout]) -> 85 | erlang:monitor(process, Owner), 86 | {ok, #state{host = Host, port = Port, hash = Hash, 87 | action = Action, size = Size, opts = ConnOpts, 88 | owner = Owner, timeout = Timeout}, Timeout}. 89 | 90 | handle_call(connect, From, #state{host = Host, port = Port, 91 | opts = ConnOpts, action = Action, 92 | hash = Hash, timeout = Timeout, 93 | size = Size} = State) -> 94 | case connect(Host, Port, Hash, ConnOpts, Timeout) of 95 | {ok, SockMod, Sock} -> 96 | State1 = State#state{sockmod = SockMod, socket = Sock}, 97 | gen_server:reply(From, {ok, self()}), 98 | case Action of 99 | recv -> 100 | Result = recv(SockMod, Sock, Size, Timeout), 101 | reply(State1, Result), 102 | {stop, normal, State1}; 103 | send -> 104 | noreply(State1) 105 | end; 106 | {error, _} = Err -> 107 | {stop, normal, Err, State} 108 | end; 109 | handle_call(Request, _From, State) -> 110 | lager:warning("Unexpected call: ~p", [Request]), 111 | noreply(State). 112 | 113 | handle_cast(activate, #state{sockmod = SockMod, socket = Sock, 114 | size = Size} = State) -> 115 | Chunk = p1_rand:bytes(?BUF_SIZE), 116 | Result = send(SockMod, Sock, Size, Chunk), 117 | reply(State, Result), 118 | {stop, normal, State}; 119 | handle_cast(Msg, State) -> 120 | lager:warning("Unexpected cast: ~p", [Msg]), 121 | noreply(State). 122 | 123 | handle_info({'DOWN', _, _, _, _}, State) -> 124 | {stop, normal, State}; 125 | handle_info(timeout, State) -> 126 | Reason = case State#state.socket of 127 | undefined -> init_timeout; 128 | _ -> activation_timeout 129 | end, 130 | reply(State, {error, Reason}), 131 | {stop, normal, State}; 132 | handle_info(Info, State) -> 133 | lager:warning("Unexpected info: ~p", [Info]), 134 | noreply(State). 135 | 136 | terminate(normal, _) -> 137 | ok; 138 | terminate(shutdown, State) -> 139 | reply(State, {error, shutdown}); 140 | terminate(_, State) -> 141 | reply(State, {error, crashed}). 142 | 143 | code_change(_OldVsn, State, _Extra) -> 144 | {ok, State}. 145 | 146 | %%%=================================================================== 147 | %%% Internal functions 148 | %%%=================================================================== 149 | start(Action, Host, Port, Hash, Size, ConnOpts, Timeout) -> 150 | case gen_server:start( 151 | ?MODULE, 152 | [Action, self(), binary_to_list(Host), Port, 153 | Hash, Size, ConnOpts, Timeout], []) of 154 | {ok, Pid} -> 155 | gen_server:call(Pid, connect, 2*Timeout); 156 | {error, _} -> 157 | {error, crashed} 158 | end. 159 | 160 | -spec format_socket_error({sockmod(), atom()}) -> string(). 161 | format_socket_error({_, closed}) -> 162 | "connection closed"; 163 | format_socket_error({_, timeout}) -> 164 | inet:format_error(etimedout); 165 | format_socket_error({ssl, Reason}) -> 166 | ssl:format_error(Reason); 167 | format_socket_error({gen_tcp, Reason}) -> 168 | case inet:format_error(Reason) of 169 | "unknown POSIX error" -> atom_to_list(Reason); 170 | Txt -> Txt 171 | end. 172 | 173 | reply(#state{owner = Owner, action = Action}, Result) -> 174 | Owner ! {proxy65_result, Action, Result}. 175 | 176 | noreply(#state{timeout = Timeout} = State) -> 177 | {noreply, State, Timeout}. 178 | 179 | connect(Host, Port, Hash, ConnOpts, Timeout) -> 180 | Opts = opts(ConnOpts, Timeout), 181 | SockMod = gen_tcp, 182 | try 183 | {ok, Sock} = SockMod:connect(Host, Port, Opts, Timeout), 184 | Init = <>, 185 | InitAck = <>, 186 | Req = <>, 188 | Resp = <>, 190 | ok = SockMod:send(Sock, Init), 191 | {ok, InitAck} = gen_tcp:recv(Sock, size(InitAck)), 192 | ok = gen_tcp:send(Sock, Req), 193 | {ok, Resp} = gen_tcp:recv(Sock, size(Resp)), 194 | {ok, SockMod, Sock} 195 | catch _:{badmatch, {error, Reason}} -> 196 | {error, {SockMod, Reason}}; 197 | _:{badmatch, {ok, _}} -> 198 | {error, {socks5, unexpected_response}} 199 | end. 200 | 201 | send(_SockMod, _Sock, 0, _Chunk) -> 202 | ok; 203 | send(SockMod, Sock, Size, Chunk) -> 204 | Data = if Size >= ?BUF_SIZE -> 205 | Chunk; 206 | true -> 207 | binary:part(Chunk, 0, Size) 208 | end, 209 | case SockMod:send(Sock, Data) of 210 | ok -> 211 | NewSize = receive {'DOWN', _, _, _, _} -> 0 212 | after 0 -> Size - min(Size, ?BUF_SIZE) 213 | end, 214 | send(SockMod, Sock, NewSize, Chunk); 215 | {error, Reason} -> 216 | {error, {SockMod, Reason}} 217 | end. 218 | 219 | recv(_SockMod, _Sock, 0, _Timeout) -> 220 | ok; 221 | recv(SockMod, Sock, Size, Timeout) -> 222 | ChunkSize = min(Size, ?BUF_SIZE), 223 | case SockMod:recv(Sock, ChunkSize, Timeout) of 224 | {ok, Data} -> 225 | NewSize = receive {'DOWN', _, _, _, _} -> 0 226 | after 0 -> Size-size(Data) 227 | end, 228 | recv(SockMod, Sock, NewSize, Timeout); 229 | {error, Reason} -> 230 | {error, {SockMod, Reason}} 231 | end. 232 | 233 | opts(Opts, Timeout) -> 234 | [binary, 235 | {packet, 0}, 236 | {send_timeout, Timeout}, 237 | {send_timeout_close, true}, 238 | {recbuf, ?BUF_SIZE}, 239 | {sndbuf, ?BUF_SIZE}, 240 | {active, false}|Opts]. 241 | -------------------------------------------------------------------------------- /src/mqtt_socket.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(mqtt_socket). 19 | -compile([{parse_transform, lager_transform}]). 20 | 21 | %% API 22 | -export([lookup/2, connect/3, send/2, recv/2, activate/1, close/1]). 23 | -export([format_error/1]). 24 | 25 | -include("rtb.hrl"). 26 | -include_lib("kernel/include/inet.hrl"). 27 | 28 | -define(TCP_SEND_TIMEOUT, timer:seconds(15)). 29 | -define(DNS_TIMEOUT, timer:seconds(5)). 30 | 31 | -type socket_error_reason() :: closed | timeout | inet:posix(). 32 | -type error_reason() :: {socket, socket_error_reason()} | 33 | {dns, inet:posix() | inet_res:res_error()} | 34 | {tls, inet:posix() | atom() | binary()}. 35 | -type socket() :: {websocket, pid()} | 36 | {fast_tls, fast_tls:tls_socket()} | 37 | {gen_tcp, gen_tcp:socket()}. 38 | -export_type([error_reason/0, socket/0]). 39 | 40 | %%%=================================================================== 41 | %%% API 42 | %%%=================================================================== 43 | connect(Addrs, Opts, Time) -> 44 | Timeout = max(0, Time - current_time()) div length(Addrs), 45 | connect(Addrs, Opts, Timeout, {error, {dns, nxdomain}}). 46 | 47 | connect([{EndPoint, Family}|Addrs], Opts, Timeout, _Err) -> 48 | case do_connect(EndPoint, Family, Opts, Timeout) of 49 | {ok, Sock} -> 50 | {ok, Sock}; 51 | {error, _} = Err -> 52 | connect(Addrs, Opts, Timeout, Err) 53 | end; 54 | connect([], _, _, Err) -> 55 | Err. 56 | 57 | do_connect(#endpoint{transport = Type, address = Addr, 58 | port = Port, host = Host, path = Path}, 59 | _Family, Opts, Timeout) when Type == ws; Type == wss -> 60 | lager:debug("Connecting to ~s://~s:~B~s", 61 | [Type, inet_parse:ntoa(Addr), Port, Path]), 62 | Opts1 = case Type of 63 | wss -> [{certfile, rtb_config:get_option(certfile)}|Opts]; 64 | ws -> Opts 65 | end, 66 | case gun:open(Addr, Port, #{transport => transport(Type), 67 | transport_opts => Opts1, 68 | retry => 0}) of 69 | {ok, ConnPid} -> 70 | case gun:await_up(ConnPid, Timeout) of 71 | {ok, _} -> 72 | StreamRef = gun:ws_upgrade( 73 | ConnPid, Path, 74 | [{<<"host">>, Host}], 75 | #{protocols => [{<<"mqtt">>, gun_ws_h}]}), 76 | receive 77 | {gun_upgrade, ConnPid, StreamRef, _, _} -> 78 | {ok, {websocket, ConnPid}}; 79 | {gun_response, ConnPid, StreamRef, _, Status, _} -> 80 | lager:debug("HTTP Upgrade failed with status: ~B", 81 | [Status]), 82 | {error, {socket, eproto}}; 83 | {gun_error, ConnPid, StreamRef, Reason} -> 84 | lager:debug("HTTP Upgrade failed with reason: ~p", 85 | [Reason]), 86 | {error, {socket, closed}} 87 | after Timeout -> 88 | {error, {socket, etimedout}} 89 | end; 90 | {error, Why} -> 91 | {error, {socket, prep_error(Why)}} 92 | end; 93 | {error, Why} -> 94 | lager:error("Failed to initialize HTTP connection: ~p", [Why]), 95 | {error, internal_server_error} 96 | end; 97 | do_connect(#endpoint{address = Addr, transport = Type, port = Port}, 98 | Family, Opts, Timeout) -> 99 | lager:debug("Connecting to ~s://~s:~B", 100 | [Type, inet_parse:ntoa(Addr), Port]), 101 | case gen_tcp:connect(Addr, Port, sockopts(Family, Opts), Timeout) of 102 | {ok, Sock} when Type == tls -> 103 | CertFile = {certfile, rtb_config:get_option(certfile)}, 104 | case fast_tls:tcp_to_tls(Sock, [connect, CertFile]) of 105 | {ok, Sock1} -> 106 | {ok, {fast_tls, Sock1}}; 107 | {error, Why} -> 108 | {error, {tls, Why}} 109 | end; 110 | {ok, Sock} when Type == tcp -> 111 | {ok, {gen_tcp, Sock}}; 112 | {error, Why} -> 113 | {error, {socket, Why}} 114 | end. 115 | 116 | send(Socket, Data) -> 117 | Ret = case Socket of 118 | {websocket, ConnPid} -> 119 | gun:ws_send(ConnPid, {binary, Data}); 120 | {SockMod, Sock} -> 121 | SockMod:send(Sock, Data) 122 | end, 123 | check_sock_result(Socket, Ret). 124 | 125 | recv({fast_tls, Sock}, Data) when Data /= <<>> -> 126 | case fast_tls:recv_data(Sock, Data) of 127 | {ok, _} = OK -> OK; 128 | {error, Reason} when is_atom(Reason) -> {error, {socket, Reason}}; 129 | {error, _} = Err -> Err 130 | end; 131 | recv(_, Data) -> 132 | {ok, Data}. 133 | 134 | activate(Socket) -> 135 | Ret = case Socket of 136 | {websocket, _} -> 137 | ok; 138 | {gen_tcp, Sock} -> 139 | inet:setopts(Sock, [{active, once}]); 140 | {SockMod, Sock} -> 141 | SockMod:setopts(Sock, [{active, once}]) 142 | end, 143 | check_sock_result(Socket, Ret). 144 | 145 | close({websocket, ConnPid}) -> 146 | gun:close(ConnPid); 147 | close({SockMod, Sock}) -> 148 | SockMod:close(Sock). 149 | 150 | format_error({dns, Reason}) -> 151 | format("DNS lookup failed: ~s", [format_inet_error(Reason)]); 152 | format_error({tls, Reason}) -> 153 | format("TLS failed: ~s", [format_tls_error(Reason)]); 154 | format_error({socket, Reason}) -> 155 | format("Connection failed: ~s", [format_inet_error(Reason)]). 156 | 157 | lookup(Addrs, Time) -> 158 | Addrs1 = lists:flatmap( 159 | fun(#endpoint{address = Addr} = EndPoint) when is_tuple(Addr) -> 160 | [{EndPoint, get_addr_type(Addr)}]; 161 | (EndPoint) -> 162 | [{EndPoint, inet6}, {EndPoint, inet}] 163 | end, Addrs), 164 | do_lookup(Addrs1, Time, [], nxdomain). 165 | 166 | do_lookup([{#endpoint{address = IP}, _} = Addr|Addrs], Time, Res, Err) when is_tuple(IP) -> 167 | do_lookup(Addrs, Time, [Addr|Res], Err); 168 | do_lookup([{#endpoint{address = Host} = EndPoint, Family}|Addrs], Time, Res, Err) -> 169 | Timeout = min(?DNS_TIMEOUT, max(0, Time - current_time())), 170 | case inet:gethostbyname(Host, Family, Timeout) of 171 | {ok, HostEntry} -> 172 | Addrs1 = host_entry_to_addrs(HostEntry), 173 | Addrs2 = [{EndPoint#endpoint{address = Addr}, Family} || Addr <- Addrs1], 174 | do_lookup(Addrs, Time, Addrs2 ++ Res, Err); 175 | {error, Why} -> 176 | do_lookup(Addrs, Time, Res, Why) 177 | end; 178 | do_lookup([], _Timeout, [], Err) -> 179 | {error, {dns, Err}}; 180 | do_lookup([], _Timeout, Res, _Err) -> 181 | {ok, Res}. 182 | 183 | %%%=================================================================== 184 | %%% Internal functions 185 | %%%=================================================================== 186 | -spec check_sock_result(socket(), ok | {error, inet:posix()}) -> ok. 187 | check_sock_result(_, ok) -> 188 | ok; 189 | check_sock_result({_, Sock}, {error, Why}) -> 190 | self() ! {tcp_closed, Sock}, 191 | lager:debug("MQTT socket error: ~p", [format_inet_error(Why)]). 192 | 193 | host_entry_to_addrs(#hostent{h_addr_list = AddrList}) -> 194 | lists:filter( 195 | fun(Addr) -> 196 | try get_addr_type(Addr) of 197 | _ -> true 198 | catch _:badarg -> 199 | false 200 | end 201 | end, AddrList). 202 | 203 | get_addr_type({_, _, _, _}) -> inet; 204 | get_addr_type({_, _, _, _, _, _, _, _}) -> inet6; 205 | get_addr_type(_) -> erlang:error(badarg). 206 | 207 | current_time() -> 208 | p1_time_compat:monotonic_time(milli_seconds). 209 | 210 | transport(ws) -> tcp; 211 | transport(wss) -> tls. 212 | 213 | sockopts(Family, Opts) -> 214 | [{active, once}, 215 | {packet, raw}, 216 | {send_timeout, ?TCP_SEND_TIMEOUT}, 217 | {send_timeout_close, true}, 218 | binary, Family|Opts]. 219 | 220 | prep_error({shutdown, Reason}) -> 221 | Reason; 222 | prep_error(Reason) -> 223 | Reason. 224 | 225 | format_inet_error(closed) -> 226 | "connection closed"; 227 | format_inet_error(timeout) -> 228 | format_inet_error(etimedout); 229 | format_inet_error(Reason) when is_atom(Reason) -> 230 | case inet:format_error(Reason) of 231 | "unknown POSIX error" -> atom_to_list(Reason); 232 | Txt -> Txt 233 | end; 234 | format_inet_error(Reason) -> 235 | lists:flatten(io_lib:format("unexpected error: ~p", [Reason])). 236 | 237 | format_tls_error(no_cerfile) -> 238 | "certificate not found"; 239 | format_tls_error(Reason) when is_atom(Reason) -> 240 | format_inet_error(Reason); 241 | format_tls_error(Reason) -> 242 | Reason. 243 | 244 | -spec format(io:format(), list()) -> string(). 245 | format(Fmt, Args) -> 246 | lists:flatten(io_lib:format(Fmt, Args)). 247 | -------------------------------------------------------------------------------- /src/rtb_stats.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_stats). 19 | -compile([{parse_transform, lager_transform}]). 20 | -behaviour(p1_server). 21 | 22 | %% API 23 | -export([start_link/0, incr/1, incr/2, decr/1, decr/2, lookup/1]). 24 | -export([get_type/1, get_metrics/0, flush/1, show_errors/0]). 25 | %% gen_server callbacks 26 | -export([init/1, handle_call/3, handle_cast/2, handle_info/2, 27 | terminate/2, code_change/3]). 28 | 29 | -define(INTERVAL, timer:seconds(1)). 30 | 31 | -include("rtb.hrl"). 32 | 33 | -record(state, {mod :: module(), 34 | time :: integer(), 35 | metrics :: map()}). 36 | 37 | %%%=================================================================== 38 | %%% API 39 | %%%=================================================================== 40 | start_link() -> 41 | p1_server:start_link({local, ?MODULE}, ?MODULE, [], []). 42 | 43 | incr(Metric) -> 44 | incr(Metric, 1). 45 | 46 | incr(Metric, Incr) -> 47 | try ets:lookup_element(?MODULE, Metric, 2) of 48 | Counter -> oneup:inc2(Counter, Incr) 49 | catch _:badarg -> 50 | Counter = oneup:new_counter(), 51 | case ets:insert_new(?MODULE, {Metric, Counter}) of 52 | true -> 53 | oneup:inc2(Counter, Incr); 54 | false -> 55 | Counter2 = ets:lookup_element(?MODULE, Metric, 2), 56 | oneup:inc2(Counter2, Incr) 57 | end 58 | end. 59 | 60 | decr(Metric) -> 61 | decr(Metric, 1). 62 | 63 | decr(Metric, Decr) -> 64 | try ets:lookup_element(?MODULE, Metric, 2) of 65 | Counter -> oneup:inc2(Counter, -Decr) 66 | catch _:badarg -> 67 | ok 68 | end. 69 | 70 | lookup(Key) -> 71 | try ets:lookup_element(?MODULE, Key, 2) of 72 | Counter -> oneup:get(Counter) 73 | catch _:badarg -> 74 | 0 75 | end. 76 | 77 | get_type(Metric) -> 78 | p1_server:call(?MODULE, {get_type, Metric}, infinity). 79 | 80 | get_metrics() -> 81 | p1_server:call(?MODULE, get_metrics, infinity). 82 | 83 | flush(Metric) -> 84 | p1_server:call(?MODULE, {flush, Metric}, infinity). 85 | 86 | show_errors() -> 87 | lists:foreach( 88 | fun({{_, Reason}, Counter}) -> 89 | io:format("~s: ~B~n", [Reason, oneup:get(Counter)]) 90 | end, ets:match_object(?MODULE, {{'error-reason', '_'}, '_'})). 91 | 92 | %%%=================================================================== 93 | %%% gen_server callbacks 94 | %%%=================================================================== 95 | init([]) -> 96 | ets:new(?MODULE, [named_table, public, 97 | {write_concurrency, true}]), 98 | Dir = rtb_config:get_option(stats_dir), 99 | Mod = rtb_config:get_option(module), 100 | case init_metrics(Mod, Dir) of 101 | {ok, Metrics} -> 102 | State = #state{mod = Mod, metrics = Metrics, 103 | time = p1_time_compat:monotonic_time(seconds)}, 104 | erlang:send_after(?INTERVAL, self(), log), 105 | lager:debug("Dumping statistics to ~s every ~B second(s)", 106 | [Dir, ?INTERVAL div 1000]), 107 | {ok, State}; 108 | error -> 109 | rtb:halt() 110 | end. 111 | 112 | handle_call({flush, Name}, _From, State) -> 113 | case maps:get(Name, State#state.metrics, undefined) of 114 | {_, hist, _, File, Fd} -> 115 | case write_hist(Name, File, Fd) of 116 | ok -> ok; 117 | {error, File, Why} -> 118 | lager:critical("Failed to write to ~s: ~s", 119 | [File, file:format_error(Why)]) 120 | end; 121 | _ -> 122 | ok 123 | end, 124 | {reply, ok, State}; 125 | handle_call({get_type, Name}, _From, State) -> 126 | Type = case maps:get(Name, State#state.metrics, undefined) of 127 | undefined -> undefined; 128 | T -> element(2, T) 129 | end, 130 | {reply, Type, State}; 131 | handle_call(get_metrics, _From, State) -> 132 | Metrics = maps:to_list(State#state.metrics), 133 | Names = [Name || {Name, _} <- lists:keysort(2, Metrics)], 134 | {reply, Names, State}; 135 | handle_call(_Request, _From, State) -> 136 | {noreply, State}. 137 | 138 | handle_cast(_Msg, State) -> 139 | {noreply, State}. 140 | 141 | handle_info(log, State) -> 142 | erlang:send_after(?INTERVAL, self(), log), 143 | case log(State) of 144 | ok -> ok; 145 | {error, File, Why} -> 146 | lager:critical("Failed to write to ~s: ~s", 147 | [File, file:format_error(Why)]) 148 | end, 149 | {noreply, State}; 150 | handle_info(_Info, State) -> 151 | {noreply, State}. 152 | 153 | terminate(_Reason, _State) -> 154 | ok. 155 | 156 | code_change(_OldVsn, State, _Extra) -> 157 | {ok, State}. 158 | 159 | %%%=================================================================== 160 | %%% Internal functions 161 | %%%=================================================================== 162 | timestamp(TS) -> 163 | integer_to_list(p1_time_compat:monotonic_time(seconds) - TS). 164 | 165 | write_row(TS, Int, File, Fd) -> 166 | Row = [timestamp(TS), " ", integer_to_list(Int), io_lib:nl()], 167 | case file:write(Fd, Row) of 168 | {error, Why} -> {error, File, Why}; 169 | ok -> ok 170 | end. 171 | 172 | write_hist(Name, File, Fd) -> 173 | {Hist, Sum} = lists:mapfoldl( 174 | fun({{_, X}, Counter}, Acc) -> 175 | Y = oneup:get(Counter), 176 | {{X, Y}, Acc+Y} 177 | end, 0, ets:match_object(?MODULE, {{Name, '_'}, '_'})), 178 | try 179 | {ok, _} = file:position(Fd, bof), 180 | case lists:foldl( 181 | fun({X, Y}, Acc) -> 182 | Percent = Y*100/Sum, 183 | if Percent >= 0.1 -> 184 | Y1 = io_lib:format("~.2f", [Percent]), 185 | Row = [integer_to_list(X), " ", Y1, io_lib:nl()], 186 | ok = file:write(Fd, Row), 187 | true; 188 | true -> 189 | Acc 190 | end 191 | end, false, Hist) of 192 | true -> 193 | ok = file:truncate(Fd); 194 | false -> 195 | ok 196 | end 197 | catch _:{badmatch, {error, Why}} -> 198 | {error, File, Why} 199 | end. 200 | 201 | log(#state{time = TS, metrics = Metrics}) -> 202 | maps:fold( 203 | fun(Name, Val, ok) -> 204 | log(TS, Name, Val); 205 | (_, _, Err) -> 206 | Err 207 | end, ok, Metrics). 208 | 209 | log(TS, Name, {_, Type, Call, File, Fd}) when Type == gauge; Type == counter -> 210 | Val = case Call of 211 | undefined -> lookup(Name); 212 | _ -> Call() 213 | end, 214 | write_row(TS, Val, File, Fd); 215 | log(TS, _Name, {_, rate, {Name, Call}, File, Fd}) -> 216 | New = case Call of 217 | undefined -> lookup(Name); 218 | _ -> Call() 219 | end, 220 | Old = put(Name, New), 221 | Val = case Old of 222 | undefined -> 0; 223 | _ -> abs(Old - New) 224 | end, 225 | write_row(TS, Val, File, Fd); 226 | log(_, _, {_, _, _, _, _}) -> 227 | ok. 228 | 229 | init_metrics(Mod, Dir) -> 230 | init_metrics(Mod:metrics(), Dir, 1, #{}). 231 | 232 | init_metrics([Metric|Metrics], Dir, Pos, Acc) -> 233 | #metric{name = Name, type = Type, call = Call, rate = Rate} = Metric, 234 | File = filename:join(Dir, Name) ++ ".dat", 235 | case init_file(File) of 236 | {ok, Fd} -> 237 | Acc1 = maps:put(Name, {Pos, Type, Call, File, Fd}, Acc), 238 | if Rate andalso Type /= hist -> 239 | RateName = list_to_atom(atom_to_list(Name) ++ "-rate"), 240 | RateFile = filename:join(Dir, RateName) ++ ".dat", 241 | case init_file(RateFile) of 242 | {ok, RateFd} -> 243 | Acc2 = maps:put( 244 | RateName, 245 | {Pos+1, rate, {Name, Call}, RateFile, RateFd}, 246 | Acc1), 247 | init_metrics(Metrics, Dir, Pos+2, Acc2); 248 | error -> 249 | error 250 | end; 251 | true -> 252 | init_metrics(Metrics, Dir, Pos+1, Acc1) 253 | end; 254 | error -> 255 | error 256 | end; 257 | init_metrics([], _, _, Acc) -> 258 | {ok, Acc}. 259 | 260 | init_file(File) -> 261 | case file:open(File, [write, raw]) of 262 | {ok, Fd} -> 263 | case file:write(Fd, ["0 0", io_lib:nl()]) of 264 | ok -> {ok, Fd}; 265 | {error, Why} -> 266 | lager:critical("Failed to write to ~s: ~s", 267 | [File, file:format_error(Why)]), 268 | error 269 | end; 270 | {error, Why} -> 271 | lager:critical("Failed to open ~s for writing: ~s", 272 | [File, file:format_error(Why)]), 273 | error 274 | end. 275 | -------------------------------------------------------------------------------- /src/rtb_watchdog.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_watchdog). 19 | -behaviour(gen_event). 20 | 21 | -author('alexey@process-one.net'). 22 | -author('ekhramtsov@process-one.net'). 23 | 24 | %% API 25 | -export([start/0]). 26 | 27 | %% gen_event callbacks 28 | -export([init/1, handle_event/2, handle_call/2, 29 | handle_info/2, terminate/2, code_change/3]). 30 | 31 | %% We don't use ejabberd logger because lager can be overloaded 32 | %% too and alarm_handler may get stuck. 33 | %%-include("logger.hrl"). 34 | 35 | -define(CHECK_INTERVAL, timer:seconds(30)). 36 | 37 | -record(state, {tref :: reference(), 38 | mref :: reference()}). 39 | -record(proc_stat, {qlen :: non_neg_integer(), 40 | memory :: non_neg_integer(), 41 | initial_call :: mfa(), 42 | current_function :: mfa(), 43 | ancestors :: [pid() | atom()], 44 | application :: pid() | atom(), 45 | name :: pid() | atom()}). 46 | -type state() :: #state{}. 47 | -type proc_stat() :: #proc_stat{}. 48 | 49 | %%%=================================================================== 50 | %%% API 51 | %%%=================================================================== 52 | -spec start() -> ok. 53 | start() -> 54 | application:load(sasl), 55 | application:set_env(sasl, sasl_error_logger, false), 56 | application:start(sasl), 57 | gen_event:add_handler(alarm_handler, ?MODULE, []), 58 | gen_event:swap_handler(alarm_handler, {alarm_handler, swap}, {?MODULE, []}), 59 | application:load(os_mon), 60 | application:set_env(os_mon, start_cpu_sup, false), 61 | application:set_env(os_mon, start_os_sup, false), 62 | application:set_env(os_mon, start_memsup, true), 63 | application:set_env(os_mon, start_disksup, false), 64 | case application:ensure_all_started(os_mon) of 65 | {ok, _} = OK -> 66 | set_oom_watermark(), 67 | OK; 68 | Err -> 69 | Err 70 | end. 71 | 72 | excluded_apps() -> 73 | [os_mon, mnesia, sasl, stdlib, kernel]. 74 | 75 | %%%=================================================================== 76 | %%% gen_event callbacks 77 | %%%=================================================================== 78 | init([]) -> 79 | {ok, #state{}}. 80 | 81 | handle_event({set_alarm, {system_memory_high_watermark, _}}, State) -> 82 | handle_overload(State), 83 | {ok, restart_timer(State)}; 84 | handle_event({clear_alarm, system_memory_high_watermark}, State) -> 85 | cancel_timer(State#state.tref), 86 | {ok, State#state{tref = undefined}}; 87 | handle_event({set_alarm, {process_memory_high_watermark, Pid}}, State) -> 88 | case proc_stat(Pid, get_app_pids()) of 89 | #proc_stat{name = Name} = ProcStat -> 90 | error_logger:warning_msg( 91 | "Process ~p consumes more than 5% of OS memory (~s)", 92 | [Name, format_proc(ProcStat)]), 93 | handle_overload(State), 94 | {ok, State}; 95 | _ -> 96 | {ok, State} 97 | end; 98 | handle_event({clear_alarm, process_memory_high_watermark}, State) -> 99 | {ok, State}; 100 | handle_event(Event, State) -> 101 | error_logger:warning_msg("unexpected event: ~p", [Event]), 102 | {ok, State}. 103 | 104 | handle_call(_Request, State) -> 105 | {ok, {error, badarg}, State}. 106 | 107 | handle_info({timeout, _TRef, handle_overload}, State) -> 108 | handle_overload(State), 109 | {ok, restart_timer(State)}; 110 | handle_info(Info, State) -> 111 | error_logger:warning_msg("unexpected info: ~p", [Info]), 112 | {ok, State}. 113 | 114 | terminate(_Reason, _State) -> 115 | ok. 116 | 117 | code_change(_OldVsn, State, _Extra) -> 118 | {ok, State}. 119 | 120 | %%%=================================================================== 121 | %%% Internal functions 122 | %%%=================================================================== 123 | -spec handle_overload(state()) -> ok. 124 | handle_overload(State) -> 125 | handle_overload(State, processes()). 126 | 127 | -spec handle_overload(state(), [pid()]) -> ok. 128 | handle_overload(_State, Procs) -> 129 | AppPids = get_app_pids(), 130 | {TotalMsgs, ProcsNum, Apps, Stats} = overloaded_procs(AppPids, Procs), 131 | if TotalMsgs >= 10000 -> 132 | SortedStats = lists:reverse(lists:keysort(#proc_stat.qlen, Stats)), 133 | error_logger:warning_msg( 134 | "The system is overloaded with ~b messages " 135 | "queued by ~b process(es) (~b%) " 136 | "from the following applications: ~s; " 137 | "the top processes are:~n~s", 138 | [TotalMsgs, ProcsNum, 139 | round(ProcsNum*100/length(Procs)), 140 | format_apps(Apps), 141 | format_top_procs(SortedStats)]), 142 | kill(SortedStats, round(TotalMsgs/ProcsNum)); 143 | true -> 144 | ok 145 | end, 146 | lists:foreach(fun erlang:garbage_collect/1, Procs). 147 | 148 | -spec get_app_pids() -> map(). 149 | get_app_pids() -> 150 | try application:info() of 151 | Info -> 152 | case lists:keyfind(running, 1, Info) of 153 | {_, Apps} -> 154 | lists:foldl( 155 | fun({Name, Pid}, M) when is_pid(Pid) -> 156 | maps:put(Pid, Name, M); 157 | (_, M) -> 158 | M 159 | end, #{}, Apps); 160 | false -> 161 | #{} 162 | end 163 | catch _:_ -> 164 | #{} 165 | end. 166 | 167 | -spec overloaded_procs(map(), [pid()]) 168 | -> {non_neg_integer(), non_neg_integer(), dict:dict(), [proc_stat()]}. 169 | overloaded_procs(AppPids, AllProcs) -> 170 | lists:foldl( 171 | fun(Pid, {TotalMsgs, ProcsNum, Apps, Stats}) -> 172 | case proc_stat(Pid, AppPids) of 173 | #proc_stat{qlen = QLen, application = App} = Stat 174 | when QLen > 0 -> 175 | {TotalMsgs + QLen, ProcsNum + 1, 176 | dict:update_counter(App, QLen, Apps), 177 | [Stat|Stats]}; 178 | _ -> 179 | {TotalMsgs, ProcsNum, Apps, Stats} 180 | end 181 | end, {0, 0, dict:new(), []}, AllProcs). 182 | 183 | -spec proc_stat(pid(), map()) -> proc_stat() | undefined. 184 | proc_stat(Pid, AppPids) -> 185 | case process_info(Pid, [message_queue_len, 186 | memory, 187 | initial_call, 188 | current_function, 189 | dictionary, 190 | group_leader, 191 | registered_name]) of 192 | [{_, MsgLen}, {_, Mem}, {_, InitCall}, 193 | {_, CurrFun}, {_, Dict}, {_, GL}, {_, Name}] -> 194 | IntLen = proplists:get_value('$internal_queue_len', Dict, 0), 195 | TrueInitCall = proplists:get_value('$initial_call', Dict, InitCall), 196 | Ancestors = proplists:get_value('$ancestors', Dict, []), 197 | Len = IntLen + MsgLen, 198 | App = maps:get(GL, AppPids, kernel), 199 | RegName = case Name of 200 | [] -> Pid; 201 | _ -> Name 202 | end, 203 | #proc_stat{qlen = Len, 204 | memory = Mem, 205 | initial_call = TrueInitCall, 206 | current_function = CurrFun, 207 | ancestors = Ancestors, 208 | application = App, 209 | name = RegName}; 210 | _ -> 211 | undefined 212 | end. 213 | 214 | -spec restart_timer(#state{}) -> #state{}. 215 | restart_timer(State) -> 216 | cancel_timer(State#state.tref), 217 | TRef = erlang:start_timer(?CHECK_INTERVAL, self(), handle_overload), 218 | State#state{tref = TRef}. 219 | 220 | -spec cancel_timer(reference()) -> ok. 221 | cancel_timer(undefined) -> 222 | ok; 223 | cancel_timer(TRef) -> 224 | case erlang:cancel_timer(TRef) of 225 | false -> 226 | receive {timeout, TRef, _} -> ok 227 | after 0 -> ok 228 | end; 229 | _ -> 230 | ok 231 | end. 232 | 233 | -spec format_apps(dict:dict()) -> io:data(). 234 | format_apps(Apps) -> 235 | AppList = lists:reverse(lists:keysort(2, dict:to_list(Apps))), 236 | string:join( 237 | [io_lib:format("~p (~b msgs)", [App, Msgs]) || {App, Msgs} <- AppList], 238 | ", "). 239 | 240 | -spec format_top_procs([proc_stat()]) -> io:data(). 241 | format_top_procs(Stats) -> 242 | Stats1 = lists:sublist(Stats, 5), 243 | string:join( 244 | lists:map( 245 | fun(#proc_stat{name = Name} = Stat) -> 246 | [io_lib:format("** ~w: ", [Name]), format_proc(Stat)] 247 | end,Stats1), 248 | io_lib:nl()). 249 | 250 | -spec format_proc(proc_stat()) -> io:data(). 251 | format_proc(#proc_stat{qlen = Len, memory = Mem, initial_call = InitCall, 252 | current_function = CurrFun, ancestors = Ancs, 253 | application = App}) -> 254 | io_lib:format( 255 | "msgs = ~b, memory = ~b, initial_call = ~s, " 256 | "current_function = ~s, ancestors = ~w, application = ~w", 257 | [Len, Mem, format_mfa(InitCall), format_mfa(CurrFun), Ancs, App]). 258 | 259 | -spec format_mfa(mfa()) -> io:data(). 260 | format_mfa({M, F, A}) when is_atom(M), is_atom(F), is_integer(A) -> 261 | io_lib:format("~s:~s/~b", [M, F, A]); 262 | format_mfa(WTF) -> 263 | io_lib:format("~w", [WTF]). 264 | 265 | -spec kill([proc_stat()], non_neg_integer()) -> ok. 266 | kill(Stats, Threshold) -> 267 | case rtb_config:get_option(oom_killer) of 268 | true -> 269 | do_kill(Stats, Threshold); 270 | false -> 271 | ok 272 | end. 273 | 274 | -spec do_kill([proc_stat()], non_neg_integer()) -> ok. 275 | do_kill(Stats, Threshold) -> 276 | Killed = lists:filtermap( 277 | fun(#proc_stat{qlen = Len, name = Name, application = App}) 278 | when Len >= Threshold -> 279 | case lists:member(App, excluded_apps()) of 280 | true -> 281 | error_logger:warning_msg( 282 | "Unable to kill process ~p from whitelisted " 283 | "application ~p", [Name, App]), 284 | false; 285 | false -> 286 | case kill_proc(Name) of 287 | false -> 288 | false; 289 | Pid -> 290 | maybe_restart_app(App), 291 | {true, Pid} 292 | end 293 | end; 294 | (_) -> 295 | false 296 | end, Stats), 297 | TotalKilled = length(Killed), 298 | if TotalKilled > 0 -> 299 | error_logger:error_msg( 300 | "Killed ~b process(es) consuming more than ~b message(s) each", 301 | [TotalKilled, Threshold]); 302 | true -> 303 | ok 304 | end. 305 | 306 | -spec kill_proc(pid() | atom()) -> false | pid(). 307 | kill_proc(undefined) -> 308 | false; 309 | kill_proc(Name) when is_atom(Name) -> 310 | kill_proc(whereis(Name)); 311 | kill_proc(Pid) -> 312 | exit(Pid, kill), 313 | Pid. 314 | 315 | -spec set_oom_watermark() -> ok. 316 | set_oom_watermark() -> 317 | WaterMark = rtb_config:get_option(oom_watermark), 318 | memsup:set_sysmem_high_watermark(WaterMark/100). 319 | 320 | -spec maybe_restart_app(atom()) -> any(). 321 | maybe_restart_app(lager) -> 322 | application:stop(lager), 323 | application:start(lager); 324 | maybe_restart_app(_) -> 325 | ok. 326 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /src/rtb_config.erl: -------------------------------------------------------------------------------- 1 | %%%------------------------------------------------------------------- 2 | %%% @author Evgeny Khramtsov 3 | %%% @copyright (C) 2002-2019 ProcessOne, SARL. All Rights Reserved. 4 | %%% 5 | %%% Licensed under the Apache License, Version 2.0 (the "License"); 6 | %%% you may not use this file except in compliance with the License. 7 | %%% You may obtain a copy of the License at 8 | %%% 9 | %%% http://www.apache.org/licenses/LICENSE-2.0 10 | %%% 11 | %%% Unless required by applicable law or agreed to in writing, software 12 | %%% distributed under the License is distributed on an "AS IS" BASIS, 13 | %%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | %%% See the License for the specific language governing permissions and 15 | %%% limitations under the License. 16 | %%% 17 | %%%------------------------------------------------------------------- 18 | -module(rtb_config). 19 | -compile([{parse_transform, lager_transform}]). 20 | -behaviour(gen_server). 21 | 22 | %% API 23 | -export([start_link/0, get_option/1]). 24 | -export([options/0, prep_option/2]). 25 | -export([to_bool/1]). 26 | -export([fail_opt_val/2, fail_bad_val/2, fail_unknown_opt/1]). 27 | 28 | %% gen_server callbacks 29 | -export([init/1, handle_call/3, handle_cast/2, handle_info/2, 30 | terminate/2, code_change/3]). 31 | 32 | -include("rtb.hrl"). 33 | 34 | -record(state, {}). 35 | 36 | %%%=================================================================== 37 | %%% API 38 | %%%=================================================================== 39 | start_link() -> 40 | gen_server:start_link({local, ?MODULE}, ?MODULE, [], []). 41 | 42 | get_option(Opt) -> 43 | ets:lookup_element(?MODULE, Opt, 2). 44 | 45 | %%%=================================================================== 46 | %%% gen_server callbacks 47 | %%%=================================================================== 48 | init([]) -> 49 | ets:new(?MODULE, [named_table, public, {read_concurrency, true}]), 50 | case load_config() of 51 | ok -> 52 | case check_limits() of 53 | ok -> 54 | {ok, #state{}}; 55 | {error, _Reason} -> 56 | rtb:halt() 57 | end; 58 | {error, _Reason} -> 59 | rtb:halt() 60 | end. 61 | 62 | handle_call(_Request, _From, State) -> 63 | Reply = ok, 64 | {reply, Reply, State}. 65 | 66 | handle_cast(_Msg, State) -> 67 | {noreply, State}. 68 | 69 | handle_info(_Info, State) -> 70 | {noreply, State}. 71 | 72 | terminate(_Reason, _State) -> 73 | ok. 74 | 75 | code_change(_OldVsn, State, _Extra) -> 76 | {ok, State}. 77 | 78 | %%%=================================================================== 79 | %%% Internal functions 80 | %%%=================================================================== 81 | load_config() -> 82 | case get_config_path() of 83 | {ok, Path} -> 84 | lager:info("Loading configuration from ~s", [Path]), 85 | case parse_yaml(Path) of 86 | {ok, Terms} -> 87 | Cfg = parse_dict(Terms), 88 | do_load_config(Cfg); 89 | {error, Reason} -> 90 | lager:error("Failed to read configuration from ~s: ~s", 91 | [Path, fast_yaml:format_error(Reason)]), 92 | {error, Reason} 93 | end; 94 | {error, _} = Err -> 95 | Err 96 | end. 97 | 98 | do_load_config(Terms) -> 99 | case lists:keyfind(scenario, 1, Terms) of 100 | {_, Scenario} -> 101 | try prep_option(scenario, Scenario) of 102 | {module, Module} -> 103 | case Module:load() of 104 | ok -> 105 | Defined = get_defined(Module), 106 | NewTerms = merge_defaults(Terms, Defined, []), 107 | load_terms(NewTerms); 108 | Err -> 109 | lager:error("Failed to load scenario due to " 110 | "internal error: ~p", [Err]), 111 | {error, load_failed} 112 | end 113 | catch _:_ -> 114 | fail_opt_val(scenario, Scenario) 115 | end; 116 | false -> 117 | rtb:halt("Missing required option: scenario", []) 118 | end. 119 | 120 | -spec parse_yaml(file:filename()) -> {ok, term()} | {error, fast_yaml:yaml_error()}. 121 | parse_yaml(Path) -> 122 | case fast_yaml:decode_from_file(Path) of 123 | {ok, [Document|_]} -> 124 | {ok, Document}; 125 | Other -> 126 | Other 127 | end. 128 | 129 | parse_dict(Terms) -> 130 | lists:map( 131 | fun({Opt, Val}) -> 132 | try {binary_to_atom(Opt, latin1), Val} 133 | catch _:badarg -> 134 | rtb:halt( 135 | "Invalid configuration option: ~s", 136 | [format_val(Opt)]) 137 | end 138 | end, Terms). 139 | 140 | load_terms(Terms) -> 141 | lists:foreach( 142 | fun({Opt, Val, Mod}) -> 143 | lager:debug("Processing option ~s: ~p", [Opt, Val]), 144 | try Mod:prep_option(Opt, Val) of 145 | {NewOpt, NewVal} -> 146 | ets:insert(?MODULE, {NewOpt, NewVal}) 147 | catch _:_ -> 148 | fail_opt_val(Opt, Val) 149 | end 150 | end, Terms). 151 | 152 | merge_defaults([{Opt, Val}|Defined], Predefined, Acc) -> 153 | case lists:keyfind(Opt, 1, Predefined) of 154 | false -> 155 | rtb:halt("Unknown option: ~s", [Opt]); 156 | T -> 157 | case lists:member(Opt, Acc) of 158 | true -> 159 | rtb:halt("Multiple definitions of option: ~s", [Opt]); 160 | false -> 161 | Mod = lists:last(tuple_to_list(T)), 162 | Predefined1 = lists:keyreplace( 163 | Opt, 1, Predefined, {Opt, Val, Mod}), 164 | merge_defaults(Defined, Predefined1, [Opt|Acc]) 165 | end 166 | end; 167 | merge_defaults([], Result, _) -> 168 | case lists:partition(fun(T) -> tuple_size(T) == 2 end, Result) of 169 | {[_|_] = Required, _} -> 170 | rtb:halt("Missing required option(s): ~s", 171 | [rtb:format_list([O || {O, _} <- Required])]); 172 | {[], _} -> 173 | Result 174 | end. 175 | 176 | get_defined(Mod) -> 177 | Globals = options(), 178 | Locals = Mod:options(), 179 | {NewLocals, NewGlobals} = 180 | lists:foldl( 181 | fun(Opt, {L, G}) when is_atom(Opt) -> 182 | case lists:keytake(Opt, 1, G) of 183 | {value, _, G1} -> {L -- [Opt], [Opt|G1]}; 184 | false -> {L, G} 185 | end; 186 | ({Opt, _}, {L, G}) -> 187 | {L, lists:keydelete(Opt, 1, G)}; 188 | (_, Acc) -> 189 | Acc 190 | end, {Locals, Globals}, Locals), 191 | lists:keysort(1, lists:map( 192 | fun({Opt, Val}) -> {Opt, Val, ?MODULE}; 193 | (Opt) -> {Opt, ?MODULE} 194 | end, NewGlobals)) ++ 195 | lists:keysort(1, lists:map( 196 | fun({Opt, Val}) -> {Opt, Val, Mod}; 197 | (Opt) -> {Opt, Mod} 198 | end, NewLocals)). 199 | 200 | get_config_path() -> 201 | case application:get_env(rtb, config) of 202 | {ok, Path} -> 203 | try {ok, iolist_to_binary(Path)} 204 | catch _:_ -> 205 | lager:error("Invalid value of " 206 | "application option 'config': ~p", 207 | [Path]), 208 | {error, einval} 209 | end; 210 | undefined -> 211 | lager:error("Application option 'config' is not set", []), 212 | {error, enoent} 213 | end. 214 | 215 | -spec ulimit_open_files() -> non_neg_integer() | unlimited | unknown. 216 | ulimit_open_files() -> 217 | Output = os:cmd("ulimit -n"), 218 | case string:to_integer(Output) of 219 | {error, _} -> 220 | case Output of 221 | "unlimited" ++ _ -> 222 | unlimited; 223 | _ -> 224 | unknown 225 | end; 226 | {N, _} when is_integer(N) -> 227 | N 228 | end. 229 | 230 | -spec sysctl(string()) -> non_neg_integer() | unknown. 231 | sysctl(Val) -> 232 | Output = os:cmd("/sbin/sysctl -n " ++ Val), 233 | case string:to_integer(Output) of 234 | {error, _} -> 235 | unknown; 236 | {N, _} when is_integer(N) -> 237 | N 238 | end. 239 | 240 | check_files_limit(Capacity) -> 241 | ULimitFiles = ulimit_open_files(), 242 | SysFileMax = sysctl("fs.file-max"), 243 | SysNrOpen = sysctl("fs.nr_open"), 244 | lager:info("Maximum available files from ulimit: ~p", [ULimitFiles]), 245 | lager:info("Maximum available files from fs.file-max: ~p", [SysFileMax]), 246 | lager:info("Maximum available files from fs.nr_open: ~p", [SysNrOpen]), 247 | Limit = lists:min([ULimitFiles, SysFileMax, SysNrOpen]), 248 | if Capacity > Limit -> 249 | lager:critical("Available file descriptors are not enough " 250 | "to run ~B connections", [Capacity]), 251 | {error, system_limit}; 252 | true -> 253 | ok 254 | end. 255 | 256 | check_process_limit(Capacity) -> 257 | Limit = erlang:system_info(process_limit), 258 | lager:info("Maximum available Erlang processes: ~p", [Limit]), 259 | if Capacity > Limit -> 260 | lager:critical("Available processes limit is not enough " 261 | "to run ~B connections: you should increase value of " 262 | "+P emulator flag, see erl(1) manpage for details", 263 | [Capacity]), 264 | {error, system_limit}; 265 | true -> 266 | ok 267 | end. 268 | 269 | check_port_limit(Capacity) -> 270 | Limit = erlang:system_info(port_limit), 271 | lager:info("Maximum available Erlang ports: ~p", [Limit]), 272 | if Capacity > Limit -> 273 | lager:critical("Available ports limit is not enough " 274 | "to run ~B connections: you should increase value of " 275 | "+Q emulator flag, see erl(1) manpage for details", 276 | [Capacity]), 277 | {error, system_limit}; 278 | true -> 279 | ok 280 | end. 281 | 282 | check_limits() -> 283 | Capacity = get_option(capacity), 284 | try 285 | ok = check_files_limit(Capacity), 286 | ok = check_process_limit(Capacity), 287 | ok = check_port_limit(Capacity) 288 | catch _:{badmatch, {error, _} = Err} -> 289 | Err 290 | end. 291 | 292 | -spec to_bool(integer() | binary() | boolean()) -> boolean(). 293 | to_bool(1) -> true; 294 | to_bool(0) -> false; 295 | to_bool(false) -> false; 296 | to_bool(true) -> true; 297 | to_bool(S) when is_binary(S) -> 298 | case string:to_lower(binary_to_list(S)) of 299 | "true" -> true; 300 | "false" -> false 301 | end. 302 | 303 | format_val(I) when is_integer(I) -> 304 | integer_to_list(I); 305 | format_val(S) when is_binary(S) -> 306 | S; 307 | format_val(YAML) -> 308 | try [io_lib:nl(), fast_yaml:encode(YAML)] 309 | catch _:_ -> io_lib:format("~p", [YAML]) 310 | end. 311 | 312 | fail_opt_val(Opt, Val) -> 313 | rtb:halt("Option '~s' has invalid value: ~s", 314 | [Opt, format_val(Val)]). 315 | 316 | fail_bad_val(What, Val) -> 317 | rtb:halt("Invalid ~s: ~s", [What, format_val(Val)]). 318 | 319 | fail_unknown_opt(Opt) -> 320 | rtb:halt("Unknown option: ~s", [Opt]). 321 | 322 | prep_servers([Server|Servers]) -> 323 | case http_uri:parse(binary_to_list(Server)) of 324 | {ok, {Type, _, Host, Port, Path, Query}} 325 | when Port > 0 andalso Port < 65536 -> 326 | Addr = case inet:parse_address(Host) of 327 | {ok, IP} -> IP; 328 | {error, _} -> Host 329 | end, 330 | if Type == tls orelse Type == wss -> 331 | case get_option(certfile) of 332 | undefined -> 333 | rtb:halt("Option 'certfile' is not set " 334 | "but it is assumed by URI ~s", 335 | [Server]); 336 | _ -> 337 | ok 338 | end; 339 | Type == tcp; Type == ws -> 340 | ok; 341 | true -> 342 | rtb:halt("Unsupported URI type: ~s", [Server]) 343 | end, 344 | [#endpoint{transport = Type, 345 | address = Addr, 346 | port = Port, 347 | host = Host, 348 | path = Path ++ Query} 349 | |prep_servers(Servers)] 350 | end; 351 | prep_servers([]) -> 352 | []. 353 | 354 | prep_addresses([IP|IPs], Avail) -> 355 | case inet:parse_address(binary_to_list(IP)) of 356 | {ok, Addr} -> 357 | case lists:member(Addr, Avail) of 358 | true -> 359 | [Addr|prep_addresses(IPs, Avail)]; 360 | false -> 361 | lager:warning("Interface address ~s is not available", 362 | [IP]), 363 | prep_addresses(IPs, Avail) 364 | end; 365 | {error, _} -> 366 | rtb:halt("Not a valid IP address: ~s", [format_val(IP)]) 367 | end; 368 | prep_addresses([], _) -> 369 | []. 370 | 371 | getifaddrs() -> 372 | case inet:getifaddrs() of 373 | {ok, IFList} -> 374 | lists:flatmap( 375 | fun({_, Opts}) -> 376 | [Addr || {addr, Addr} <- Opts] 377 | end, IFList); 378 | {error, Reason} -> 379 | rtb:halt("Failed to get interface addresses: ~s", 380 | [inet:format_error(Reason)]) 381 | end. 382 | 383 | scenario_to_module(M) when is_atom(M) -> 384 | list_to_atom("mod_" ++ atom_to_list(M)); 385 | scenario_to_module(M) -> 386 | erlang:binary_to_atom(<<"mod_", M/binary>>, latin1). 387 | 388 | prep_option(scenario, Scenario) -> 389 | Module = scenario_to_module(Scenario), 390 | case code:ensure_loaded(Module) of 391 | {module, _} -> 392 | {module, Module}; 393 | {error, _} -> 394 | rtb:halt("Unknown scenario: ~s; check if ~s exists", 395 | [Scenario, filename:join(code:lib_dir(rtb), 396 | Module) ++ ".beam"]) 397 | end; 398 | prep_option(interval, 0) -> 399 | lager:warning("The benchmark is in the avalanche mode"), 400 | {interval, 0}; 401 | prep_option(interval, I) when is_integer(I), I>0 -> 402 | lager:info("Arrival rate is ~.1f conn/sec", [1000/I]), 403 | {interval, I}; 404 | prep_option(capacity, C) when is_integer(C), C>0 -> 405 | lager:info("Capacity is ~B sessions", [C]), 406 | {capacity, C}; 407 | prep_option(servers, List) -> 408 | {servers, prep_servers(List)}; 409 | prep_option(bind, List) -> 410 | AvailAddrs = getifaddrs(), 411 | {bind, prep_addresses(List, AvailAddrs)}; 412 | prep_option(stats_dir, Path) -> 413 | Dir = binary_to_list(Path), 414 | case filelib:ensure_dir(filename:join(Dir, "foo")) of 415 | ok -> 416 | {stats_dir, Dir}; 417 | {error, Why} -> 418 | lager:error("Failed to create directory ~s: ~s", 419 | [Dir, file:format_error(Why)]), 420 | erlang:error(badarg) 421 | end; 422 | prep_option(oom_killer, B) -> 423 | {oom_killer, to_bool(B)}; 424 | prep_option(oom_watermark, I) when is_integer(I), I>0, I<100 -> 425 | {oom_watermark, I}; 426 | prep_option(www_dir, Dir) -> 427 | Path = binary_to_list(Dir), 428 | case filelib:ensure_dir(filename:join(Path, "foo")) of 429 | ok -> 430 | {www_dir, Path}; 431 | {error, Why} -> 432 | lager:error("Failed to create directory ~s: ~s", 433 | [Path, file:format_error(Why)]), 434 | erlang:error(badarg) 435 | end; 436 | prep_option(www_port, P) when is_integer(P), P>0, P<65536 -> 437 | {www_port, P}; 438 | prep_option(www_domain, <<_, _/binary>> = D) -> 439 | {www_domain, binary_to_list(D)}; 440 | prep_option(www_refresh, I) when is_integer(I), I>=0 -> 441 | {www_refresh, I}; 442 | prep_option(gnuplot, undefined) -> 443 | Exec = case os:find_executable("gnuplot") of 444 | false -> "gnuplot"; 445 | Path -> Path 446 | end, 447 | prep_option(gnuplot, iolist_to_binary(Exec)); 448 | prep_option(gnuplot, Exec) -> 449 | Path = binary_to_list(Exec), 450 | case string:strip(os:cmd(Path ++ " --version"), right, $\n) of 451 | "gnuplot " ++ Version -> 452 | lager:info("Found ~s ~s", [Path, Version]), 453 | {gnuplot, Path}; 454 | _ -> 455 | lager:critical("Gnuplot was not found", []), 456 | erlang:error(badarg) 457 | end; 458 | prep_option(certfile, undefined) -> 459 | {certfile, undefined}; 460 | prep_option(certfile, CertFile) -> 461 | Path = binary_to_list(CertFile), 462 | case file:open(Path, [read]) of 463 | {ok, Fd} -> 464 | file:close(Fd), 465 | {certfile, Path}; 466 | {error, Reason} -> 467 | lager:critical("Failed to read ~s: ~s", 468 | [Path, file:format_error(Reason)]), 469 | erlang:error(badarg) 470 | end; 471 | prep_option(debug, Debug) -> 472 | case to_bool(Debug) of 473 | false -> 474 | {debug, false}; 475 | true -> 476 | rtb:set_debug(), 477 | {debug, true} 478 | end. 479 | 480 | options() -> 481 | [{bind, []}, 482 | {servers, []}, 483 | {certfile, undefined}, 484 | {stats_dir, <<"stats">>}, 485 | {oom_killer, true}, 486 | {oom_watermark, 80}, 487 | {www_dir, <<"www">>}, 488 | {www_port, 8080}, 489 | {www_domain, <<"localhost">>}, 490 | {www_refresh, 1}, 491 | {gnuplot, undefined}, 492 | {debug, false}, 493 | %% Required options 494 | scenario, 495 | interval, 496 | capacity]. 497 | -------------------------------------------------------------------------------- /c_src/rtb_db.c: -------------------------------------------------------------------------------- 1 | /******************************************************************** 2 | * Copyright (c) 2012-2019 ProcessOne, SARL. All Rights Reserved. 3 | * 4 | * All rights reserved. This program and the accompanying materials 5 | * are made available under the terms of the Eclipse Public License v1.0 6 | * and Eclipse Distribution License v1.0 which accompany this distribution. 7 | * 8 | * The Eclipse Public License is available at 9 | * http://www.eclipse.org/legal/epl-v10.html 10 | * and the Eclipse Distribution License is available at 11 | * http://www.eclipse.org/org/documents/edl-v10.php. 12 | * 13 | * Contributors: 14 | * Evgeny Khramtsov - initial implementation and documentation. 15 | * Roger Light - implementation of password encryption from 16 | * Eclipse Mosquitto project: https://mosquitto.org 17 | *******************************************************************/ 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | #include 28 | #include 29 | 30 | #define BUFSIZE 65535 31 | #define SALT_LEN 12 32 | #define VERSION "0.1.0" 33 | 34 | #define USERS_CSV_FILE "/tmp/users.csv" 35 | #define ROSTERS_CSV_FILE "/tmp/rosters.csv" 36 | #define PASSWD_FILE "/tmp/passwd" 37 | #define USERS_DIR "/tmp/accounts/" 38 | #define ROSTERS_DIR "/tmp/roster/" 39 | 40 | typedef enum {T_EJABBERD = 1, 41 | T_JACKAL, 42 | T_METRONOME, 43 | T_MOSQUITTO, 44 | T_PROSODY} server_type; 45 | typedef enum {T_CSV = 1, T_FLAT} file_type; 46 | 47 | typedef struct { 48 | char *server; 49 | server_type server_type; 50 | file_type file_type; 51 | long int capacity; 52 | char *user; 53 | char *domain; 54 | char *password; 55 | long int roster_size; 56 | } state_t; 57 | 58 | char *timestamp() { 59 | return "1970-01-01 00:00:01"; 60 | } 61 | 62 | char *format_server_type(server_type t) { 63 | switch (t) { 64 | case T_EJABBERD: return "ejabberd"; 65 | case T_JACKAL: return "jackal"; 66 | case T_METRONOME: return "Metronome"; 67 | case T_PROSODY: return "Prosody"; 68 | case T_MOSQUITTO: return "Mosquitto"; 69 | } 70 | abort(); 71 | } 72 | 73 | void replace(char *dst, const char *src, const char c, const char *str) { 74 | int len = strlen(src); 75 | memset(dst, 0, BUFSIZE); 76 | int i = 0, j = 0; 77 | while (ilength+1); 104 | if(!(*encoded)){ 105 | BIO_free_all(b64); 106 | return -1; 107 | } 108 | memcpy(*encoded, bptr->data, bptr->length); 109 | (*encoded)[bptr->length] = '\0'; 110 | BIO_free_all(b64); 111 | 112 | return 0; 113 | } 114 | 115 | int encrypt_password(char *buf, const char *password) 116 | { 117 | int res; 118 | unsigned char salt[SALT_LEN]; 119 | char *salt64 = NULL, *hash64 = NULL; 120 | unsigned char hash[EVP_MAX_MD_SIZE]; 121 | unsigned int hash_len; 122 | const EVP_MD *digest; 123 | #if OPENSSL_VERSION_NUMBER < 0x10100000L 124 | EVP_MD_CTX context; 125 | #else 126 | EVP_MD_CTX *context; 127 | #endif 128 | res = RAND_bytes(salt, SALT_LEN); 129 | if (!res) { 130 | fprintf(stderr, "Insufficient entropy available to perform password generation.\n"); 131 | return -1; 132 | } 133 | res = base64_encode(salt, SALT_LEN, &salt64); 134 | if (res) { 135 | fprintf(stderr, "Unable to encode salt.\n"); 136 | return -1; 137 | } 138 | digest = EVP_get_digestbyname("sha512"); 139 | if (!digest) { 140 | fprintf(stderr, "Unable to create openssl digest.\n"); 141 | return -1; 142 | } 143 | #if OPENSSL_VERSION_NUMBER < 0x10100000L 144 | EVP_MD_CTX_init(&context); 145 | EVP_DigestInit_ex(&context, digest, NULL); 146 | EVP_DigestUpdate(&context, password, strlen(password)); 147 | EVP_DigestUpdate(&context, salt, SALT_LEN); 148 | EVP_DigestFinal_ex(&context, hash, &hash_len); 149 | EVP_MD_CTX_cleanup(&context); 150 | #else 151 | context = EVP_MD_CTX_new(); 152 | EVP_DigestInit_ex(context, digest, NULL); 153 | EVP_DigestUpdate(context, password, strlen(password)); 154 | EVP_DigestUpdate(context, salt, SALT_LEN); 155 | EVP_DigestFinal_ex(context, hash, &hash_len); 156 | EVP_MD_CTX_free(context); 157 | #endif 158 | res = base64_encode(hash, hash_len, &hash64); 159 | if (res) { 160 | fprintf(stderr, "Unable to encode hash.\n"); 161 | return -1; 162 | } 163 | sprintf(buf, "$6$%s$%s", salt64, hash64); 164 | free(salt64); 165 | free(hash64); 166 | return 0; 167 | } 168 | 169 | void mk_user_csv_row(char *row, state_t *state) { 170 | char *user = state->user; 171 | char *domain = state->domain; 172 | char *password = state->password; 173 | char buf[BUFSIZE]; 174 | memset(buf, 0, BUFSIZE); 175 | char *ts = timestamp(); 176 | if (state->server_type == T_EJABBERD) 177 | sprintf(buf, "\"%s\",\"%s\",\"%s\",\"\",\"\",\"0\",\"%s\"\n", 178 | user, domain, password, ts); 179 | else 180 | sprintf(buf, "\"%s\",\"%s\",\"\",\"%s\",\"%s\",\"%s\"\n", 181 | user, password, ts, ts, ts); 182 | replace(row, buf, '%', "%lu"); 183 | } 184 | 185 | void mk_roster_csv_row(char *row, state_t *state) { 186 | char *user = state->user; 187 | char *domain = state->domain; 188 | char buf[BUFSIZE]; 189 | char *ts = timestamp(); 190 | if (state->server_type == T_EJABBERD) 191 | sprintf(buf, 192 | "\"%s\",\"%s\",\"%s@%s\",\"%s\",\"B\",\"N\",\"\",\"N\",\"\",\"item\",\"%s\"\n", 193 | user, domain, user, domain, user, ts); 194 | else 195 | sprintf(buf, 196 | "\"%s\",\"%s@%s\",\"%s\",\"both\",\"\",\"0\",\"0\",\"%s\",\"%s\"\n", 197 | user, user, domain, user, ts, ts); 198 | replace(row, buf, '%', "%lu"); 199 | } 200 | 201 | void mk_user_dat(char *data, state_t *state) { 202 | char *password = state->password; 203 | char buf[BUFSIZE]; 204 | sprintf(buf, "return {\n\t[\"password\"] = \"%s\";\n};\n", password); 205 | replace(data, buf, '%', "%lu"); 206 | } 207 | 208 | void mk_roster_dat(char *data, state_t *state) { 209 | char *user = state->user; 210 | char *domain = state->domain; 211 | char buf[BUFSIZE]; 212 | sprintf(buf, "\t[\"%s@%s\"] = {\n" 213 | "\t\t[\"subscription\"] = \"both\";\n" 214 | "\t\t[\"groups\"] = {};\n" 215 | "\t\t[\"name\"] = \"%s\";\n" 216 | "\t};\n", 217 | user, domain, user); 218 | replace(data, buf, '%', "%lu"); 219 | } 220 | 221 | int generate_users_csv(state_t *state) { 222 | FILE *fd = fopen(USERS_CSV_FILE, "w"); 223 | if (!fd) { 224 | fprintf(stderr, "Failed to open file %s for writing: %s\n", 225 | USERS_CSV_FILE, strerror(errno)); 226 | return errno; 227 | } 228 | 229 | printf("Generating %s... ", USERS_CSV_FILE); 230 | fflush(stdout); 231 | clock_t begin = clock(); 232 | char row[BUFSIZE]; 233 | int i; 234 | mk_user_csv_row(row, state); 235 | for (i=1; i<=state->capacity; i++) { 236 | if (fprintf(fd, row, i, i) < 0) { 237 | fprintf(stderr, "Failed to write to file %s: %s\n", 238 | USERS_CSV_FILE, strerror(errno)); 239 | return errno; 240 | } 241 | } 242 | clock_t end = clock(); 243 | double time_spent = (double)(end - begin) / CLOCKS_PER_SEC; 244 | printf("done in %.3f secs\n", time_spent); 245 | return 0; 246 | } 247 | 248 | int generate_rosters_csv(state_t *state) { 249 | if (!state->roster_size) 250 | return 0; 251 | 252 | FILE *fd = fopen(ROSTERS_CSV_FILE, "w"); 253 | if (!fd) { 254 | fprintf(stderr, "Failed to open file %s for writing: %s\n", 255 | ROSTERS_CSV_FILE, strerror(errno)); 256 | return errno; 257 | } 258 | 259 | printf("Generating %s... ", ROSTERS_CSV_FILE); 260 | fflush(stdout); 261 | clock_t begin = clock(); 262 | int i, j, next, prev; 263 | char row[BUFSIZE]; 264 | mk_roster_csv_row(row, state); 265 | int roster_size = state->roster_size/2; 266 | for (i=1; i<=state->capacity; i++) { 267 | for (j=1; j<=roster_size; j++) { 268 | next = i + j; 269 | next = (next > state->capacity) ? (next % state->capacity): next; 270 | prev = (i <= j) ? state->capacity - (j - 1) : i-j; 271 | if ((fprintf(fd, row, i, next, next) < 0) || 272 | (fprintf(fd, row, i, prev, prev) < 0)) { 273 | fprintf(stderr, "Failed to write to file %s: %s\n", 274 | ROSTERS_CSV_FILE, strerror(errno)); 275 | return errno; 276 | } 277 | } 278 | } 279 | clock_t end = clock(); 280 | double time_spent = (double)(end - begin) / CLOCKS_PER_SEC; 281 | printf("done in %.3f secs\n", time_spent); 282 | return 0; 283 | } 284 | 285 | int generate_user_files(state_t *state) { 286 | int res = mkdir(USERS_DIR, 0755); 287 | if (res && errno != EEXIST) { 288 | fprintf(stderr, "Failed to create directory %s: %s\n", USERS_DIR, strerror(errno)); 289 | return errno; 290 | } 291 | 292 | printf("Generating accounts in %s... ", USERS_DIR); 293 | fflush(stdout); 294 | clock_t begin = clock(); 295 | int i; 296 | FILE *fd; 297 | int dir_len = strlen(USERS_DIR); 298 | char password[BUFSIZE]; 299 | char user[BUFSIZE]; 300 | char file[BUFSIZE]; 301 | mk_user_dat(password, state); 302 | replace(user, state->user, '%', "%lu"); 303 | memset(file, 0, BUFSIZE); 304 | strcpy(user+strlen(user), ".dat"); 305 | strcpy(file, USERS_DIR); 306 | for (i=1; i<=state->capacity; i++) { 307 | sprintf(file+dir_len, user, i); 308 | fd = fopen(file, "w"); 309 | if (!fd) { 310 | fprintf(stderr, "Failed to open file %s for writing: %s\n", file, strerror(errno)); 311 | return errno; 312 | } 313 | if (fprintf(fd, password, i) < 0) { 314 | fprintf(stderr, "Failed to write to file %s: %s\n", file, strerror(errno)); 315 | return errno; 316 | } 317 | fclose(fd); 318 | } 319 | clock_t end = clock(); 320 | double time_spent = (double)(end - begin) / CLOCKS_PER_SEC; 321 | printf("done in %.3f secs\n", time_spent); 322 | return 0; 323 | } 324 | 325 | int generate_roster_files(state_t *state) { 326 | if (!state->roster_size) 327 | return 0; 328 | 329 | int res = mkdir(ROSTERS_DIR, 0755); 330 | if (res && errno != EEXIST) { 331 | fprintf(stderr, "Failed to create directory %s: %s\n", ROSTERS_DIR, strerror(errno)); 332 | return errno; 333 | } 334 | 335 | printf("Generating rosters in %s... ", ROSTERS_DIR); 336 | fflush(stdout); 337 | clock_t begin = clock(); 338 | int i, j, next, prev; 339 | FILE *fd; 340 | int dir_len = strlen(ROSTERS_DIR); 341 | char roster[BUFSIZE]; 342 | char user[BUFSIZE]; 343 | char file[BUFSIZE]; 344 | int roster_size = state->roster_size/2; 345 | mk_roster_dat(roster, state); 346 | replace(user, state->user, '%', "%lu"); 347 | memset(file, 0, BUFSIZE); 348 | strcpy(user+strlen(user), ".dat"); 349 | strcpy(file, ROSTERS_DIR); 350 | for (i=1; i<=state->capacity; i++) { 351 | sprintf(file+dir_len, user, i); 352 | fd = fopen(file, "w"); 353 | if (!fd) { 354 | fprintf(stderr, "Failed to open file %s for writing: %s\n", file, strerror(errno)); 355 | return errno; 356 | } 357 | if (fprintf(fd, 358 | "return {\n\t[false] = {\n\t\t[\"version\"] = 1;\n" 359 | "\t\t[\"pending\"] = {};\n\t};\n") < 0) { 360 | fprintf(stderr, "Failed to write to file %s: %s\n", file, strerror(errno)); 361 | return errno; 362 | } 363 | for (j=1; j<=roster_size; j++) { 364 | next = i + j; 365 | next = (next > state->capacity) ? (next % state->capacity): next; 366 | prev = (i <= j) ? state->capacity - (j - 1) : i-j; 367 | if ((fprintf(fd, roster, next, next) < 0) || 368 | (fprintf(fd, roster, prev, prev) < 0)) { 369 | fprintf(stderr, "Failed to write to file %s: %s\n", file, strerror(errno)); 370 | return errno; 371 | } 372 | } 373 | if (fprintf(fd, "};\n") < 0) { 374 | fprintf(stderr, "Failed to write to file %s: %s\n", file, strerror(errno)); 375 | return errno; 376 | } 377 | fclose(fd); 378 | } 379 | clock_t end = clock(); 380 | double time_spent = (double)(end - begin) / CLOCKS_PER_SEC; 381 | printf("done in %.3f secs\n", time_spent); 382 | return 0; 383 | } 384 | 385 | int generate_passwd(state_t *state) { 386 | int i, res; 387 | FILE *fd = fopen(PASSWD_FILE, "w"); 388 | if (!fd) { 389 | fprintf(stderr, "Failed to open file %s for writing: %s\n", 390 | PASSWD_FILE, strerror(errno)); 391 | return errno; 392 | } 393 | 394 | printf("Generating %s... ", PASSWD_FILE); 395 | fflush(stdout); 396 | clock_t begin = clock(); 397 | char user[BUFSIZE]; 398 | char password[BUFSIZE]; 399 | char buf1[BUFSIZE]; 400 | char buf2[BUFSIZE]; 401 | replace(user, state->user, '%', "%lu"); 402 | replace(password, state->password, '%', "%lu"); 403 | for (i=1; i<=state->capacity; i++) { 404 | sprintf(buf1, password, i); 405 | if ((res = encrypt_password(buf2, buf1))) 406 | return res; 407 | if (fprintf(fd, user, i) < 0 || 408 | fprintf(fd, ":%s\n", buf2) < 0) { 409 | fprintf(stderr, "Failed to write to file %s: %s\n", PASSWD_FILE, strerror(errno)); 410 | return errno; 411 | } 412 | } 413 | clock_t end = clock(); 414 | double time_spent = (double)(end - begin) / CLOCKS_PER_SEC; 415 | printf("done in %.3f secs\n", time_spent); 416 | return 0; 417 | } 418 | 419 | void print_hint(state_t *state) { 420 | int have_rosters = state->roster_size; 421 | if (state->file_type == T_CSV) { 422 | char *mysql_cmd = 423 | " LOAD DATA LOCAL INFILE '%s'\n" 424 | " INTO TABLE %s FIELDS TERMINATED BY ','\n" 425 | " ENCLOSED BY '\"' LINES TERMINATED BY '\\n';\n"; 426 | char *pgsql_cmd = " \\copy %s FROM '%s' WITH CSV QUOTE AS '\"';\n"; 427 | char *sqlite_cmd = ".import '%s' %s\n"; 428 | char *roster_table; 429 | if (state->server_type == T_EJABBERD) 430 | roster_table = "rosterusers"; 431 | else 432 | roster_table = "roster_items"; 433 | 434 | printf("Now execute the following SQL commands:\n"); 435 | printf("** MySQL:\n"); 436 | printf(mysql_cmd, USERS_CSV_FILE, "users"); 437 | if (have_rosters) 438 | printf(mysql_cmd, ROSTERS_CSV_FILE, roster_table); 439 | if (state->server_type == T_EJABBERD) { 440 | printf("** PostgreSQL:\n"); 441 | printf(pgsql_cmd, "users", USERS_CSV_FILE); 442 | if (have_rosters) 443 | printf(pgsql_cmd, roster_table, ROSTERS_CSV_FILE); 444 | printf("** SQLite: \n"); 445 | printf(".mode csv\n"); 446 | printf(sqlite_cmd, USERS_CSV_FILE, "users"); 447 | if (have_rosters) 448 | printf(sqlite_cmd, ROSTERS_CSV_FILE, roster_table); 449 | } 450 | } else if (state->server_type == T_MOSQUITTO) { 451 | printf("Now set 'password_file' option of mosquitto.conf " 452 | "pointing to %s. Something like:\n" 453 | " echo 'allow_anonymous false' >> /etc/mosquitto/mosquitto.conf\n" 454 | " echo 'password_file /tmp/passwd' >> /etc/mosquitto/mosquitto.conf\n", 455 | PASSWD_FILE); 456 | } else { 457 | char domain[BUFSIZE]; 458 | replace(domain, state->domain, '.', "%2e"); 459 | printf("Now copy %s and %s into %s spool directory. Something like:\n" 460 | " sudo rm -rf /var/lib/%s/%s/accounts\n" 461 | " sudo rm -rf /var/lib/%s/%s/roster\n" 462 | " sudo mv %s /var/lib/%s/%s/\n" 463 | " sudo mv %s /var/lib/%s/%s/\n" 464 | " sudo chown %s:%s -R /var/lib/%s/%s\n", 465 | USERS_DIR, ROSTERS_DIR, format_server_type(state->server_type), 466 | state->server, domain, state->server, domain, 467 | USERS_DIR, state->server, domain, 468 | ROSTERS_DIR, state->server, domain, 469 | state->server, state->server, state->server, domain); 470 | } 471 | } 472 | 473 | void print_help() { 474 | printf("Usage: rtb_db -t type -c capacity -u username -p password \n" 475 | " [-f format] [-r roster-size] [-hv]\n" 476 | "Generate database/spool files for XMPP/MQTT servers.\n\n" 477 | " -t, --type type of the server; available values are:\n" 478 | " ejabberd, jackal, metronome, mosquitto, prosody\n" 479 | " -c, --capacity total number of accounts to generate;\n" 480 | " must be an even positive integer\n" 481 | " -u, --username username pattern; must contain '%%' symbol;\n" 482 | " for XMPP servers must be in user@domain format\n" 483 | " -p, --password password pattern; may contain '%%' symbol\n" 484 | " -f, --format format of the storage: sql or flat\n" 485 | " -r, --roster-size number of items in rosters; for XMPP servers only;\n" 486 | " must be an even non-negative integer less than capacity\n" 487 | " -h, --help print this help\n" 488 | " -v, --version print version\n\n"); 489 | } 490 | 491 | int validate_state(state_t *state, char *user) { 492 | if (!state->server_type) { 493 | fprintf(stderr, "Missing required option: -t\n"); 494 | return -1; 495 | } 496 | if (!state->capacity) { 497 | fprintf(stderr, "Missing required option: -c\n"); 498 | return -1; 499 | } 500 | if (!user) { 501 | fprintf(stderr, "Missing required option: -u\n"); 502 | return -1; 503 | } 504 | if (!state->password) { 505 | fprintf(stderr, "Missing required option: -p\n"); 506 | return -1; 507 | } 508 | switch (state->file_type) { 509 | case T_FLAT: 510 | if (state->server_type == T_EJABBERD || state->server_type == T_JACKAL) { 511 | fprintf(stderr, "Unexpected database type for %s: flat\n" 512 | "Currently only 'sql' is supported.\n", 513 | format_server_type(state->server_type)); 514 | return -1; 515 | } 516 | break; 517 | case T_CSV: 518 | if (state->server_type != T_EJABBERD && state->server_type != T_JACKAL) { 519 | fprintf(stderr, "Unexpected database type for %s: sql\n" 520 | "Currenty only 'flat' is supported.\n", 521 | format_server_type(state->server_type)); 522 | return -1; 523 | } 524 | break; 525 | default: 526 | if (state->server_type == T_EJABBERD || state->server_type == T_JACKAL) 527 | state->file_type = T_CSV; 528 | else 529 | state->file_type = T_FLAT; 530 | } 531 | if (state->server_type == T_MOSQUITTO) { 532 | state->user = user; 533 | } else { 534 | char *domain = strchr(user, '@'); 535 | if (domain && domain != user && (strlen(domain) > 1)) { 536 | state->user = calloc(1, strlen(user)); 537 | state->domain = calloc(1, strlen(user)); 538 | if (state->user && state->domain) { 539 | memcpy(state->user, user, domain-user); 540 | strcpy(state->domain, domain+1); 541 | } else { 542 | fprintf(stderr, "Memory failure\n"); 543 | return -1; 544 | } 545 | } else { 546 | fprintf(stderr, "Invalid username: '%s'\n" 547 | "The option must be presented in 'user@domain' format\n", user); 548 | return -1; 549 | } 550 | } 551 | if (!strchr(state->user, '%')) { 552 | fprintf(stderr, "The option 'username' must contain '%%' symbol\n"); 553 | return -1; 554 | } 555 | if (state->server_type != T_MOSQUITTO) { 556 | if (state->roster_size < 0 || 557 | state->roster_size >= state->capacity || 558 | state->roster_size % 2) { 559 | printf("Invalid roster size: %ld\n" 560 | "It must be an even non-negative integer less than capacity.\n", 561 | state->roster_size); 562 | return -1; 563 | } 564 | } else if (state->roster_size) { 565 | fprintf(stderr, "Option roster-size is only allowed for XMPP servers.\n"); 566 | return -1; 567 | } 568 | return 0; 569 | } 570 | 571 | state_t *mk_state(int argc, char *argv[]) { 572 | int opt; 573 | char *user = NULL; 574 | char *endptr; 575 | state_t *state = malloc(sizeof(state_t)); 576 | if (!state) { 577 | fprintf(stderr, "Memory failure\n"); 578 | return NULL; 579 | } 580 | memset(state, 0, sizeof(state_t)); 581 | static struct option long_options[] = { 582 | {"type", required_argument, 0, 't'}, 583 | {"format", required_argument, 0, 'f'}, 584 | {"capacity", required_argument, 0, 'c'}, 585 | {"username", required_argument, 0, 'u'}, 586 | {"password", required_argument, 0, 'p'}, 587 | {"roster-size", required_argument, 0, 'r'}, 588 | {"help", no_argument, 0, 'h'}, 589 | {"version", no_argument, 0, 'v'}, 590 | {0, 0, 0, 0} 591 | }; 592 | while (1) { 593 | int option_index = 0; 594 | opt = getopt_long(argc, argv, "t:f:c:u:p:r:hv", long_options, &option_index); 595 | if (opt == -1) 596 | break; 597 | switch (opt) { 598 | case 't': 599 | if (!strcmp(optarg, "ejabberd")) { 600 | state->server_type = T_EJABBERD; 601 | } else if (!strcmp(optarg, "jackal")) { 602 | state->server_type = T_JACKAL; 603 | } else if (!strcmp(optarg, "prosody")) { 604 | state->server_type = T_PROSODY; 605 | } else if (!strcmp(optarg, "metronome")) { 606 | state->server_type = T_METRONOME; 607 | } else if (!strcmp(optarg, "mosquitto")) { 608 | state->server_type = T_MOSQUITTO; 609 | } else { 610 | fprintf(stderr, 611 | "Unsupported server type: '%s'\n" 612 | "Available types: ejabberd, jackal, metronome, mosquitto and prosody\n", 613 | optarg); 614 | return NULL; 615 | } 616 | state->server = optarg; 617 | break; 618 | case 'f': 619 | if (!strcmp(optarg, "sql")) { 620 | state->file_type = T_CSV; 621 | } else if (!strcmp(optarg, "flat")) { 622 | state->file_type = T_FLAT; 623 | } else { 624 | fprintf(stderr, "Unexpected database type: '%s'\n", optarg); 625 | return NULL; 626 | } 627 | break; 628 | case 'c': 629 | state->capacity = strtol(optarg, NULL, 10); 630 | if (state->capacity <= 0 || state->capacity % 2) { 631 | fprintf(stderr, 632 | "Invalid capacity: '%s'\n" 633 | "It must be an even positive integer\n", optarg); 634 | return NULL; 635 | } 636 | break; 637 | case 'u': 638 | user = optarg; 639 | if (!strlen(user)) { 640 | fprintf(stderr, "Empty username\n"); 641 | return NULL; 642 | } 643 | break; 644 | case 'p': 645 | state->password = optarg; 646 | if (!strlen(state->password)) { 647 | fprintf(stderr, "Empty password\n"); 648 | return NULL; 649 | } 650 | break; 651 | case 'r': 652 | state->roster_size = strtol(optarg, &endptr, 10); 653 | if (endptr == optarg || *endptr != '\0') { 654 | fprintf(stderr, 655 | "Invalid roster-size: '%s'\n" 656 | "It must be an even non-negative integer less than capacity\n", 657 | optarg); 658 | return NULL; 659 | } 660 | break; 661 | case 'v': 662 | printf("rtb_db %s\n", VERSION); 663 | return NULL; 664 | case 'h': 665 | print_help(); 666 | return NULL; 667 | default: 668 | printf("Try '%s --help' for more information.\n", argv[0]); 669 | return NULL; 670 | } 671 | } 672 | 673 | if (validate_state(state, user)) 674 | return NULL; 675 | else 676 | return state; 677 | } 678 | 679 | int main(int argc, char *argv[]) { 680 | int res = -1; 681 | state_t *state = mk_state(argc, argv); 682 | if (state) { 683 | switch (state->server_type) { 684 | case T_EJABBERD: 685 | case T_JACKAL: 686 | res = generate_users_csv(state); 687 | if (!res) { 688 | res = generate_rosters_csv(state); 689 | if (!res) 690 | print_hint(state); 691 | } 692 | break; 693 | case T_METRONOME: 694 | case T_PROSODY: 695 | res = generate_user_files(state); 696 | if (!res) { 697 | res = generate_roster_files(state); 698 | if (!res) 699 | print_hint(state); 700 | } 701 | break; 702 | case T_MOSQUITTO: 703 | res = generate_passwd(state); 704 | if (!res) 705 | print_hint(state); 706 | break; 707 | } 708 | } 709 | free(state); 710 | return res; 711 | } 712 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | RTB: Benchmarking tool to stress real-time protocols 2 | ==================================================== 3 | ![line-plot](https://raw.github.com/processone/rtb/master/img/packets-in-rate.png) 4 | ![box-plot](https://raw.github.com/processone/rtb/master/img/ping-rtt.png) 5 | 6 | The idea of RTB is to provide a benchmarking tool to stress test 7 | XMPP and MQTT servers with the minimum configuration overhead. 8 | Also, at least in the XMPP world, a "golden standard" benchmark 9 | is highly demanded in order to compare server implementations, 10 | avoiding disambiguations as much as possible. RTB is believed 11 | to be used in such role because it has sane defaults (gathered 12 | from statistics of real world servers) and it is able to 13 | stress test all features defined in the 14 | [XMPP Compliance Suite 2019](https://xmpp.org/extensions/xep-0412.html) 15 | 16 | **Table of Contents**: 17 | 1. [Status](#status) 18 | 2. [System requirements](#system-requirements) 19 | 3. [Compiling](#compiling) 20 | 4. [Usage](#usage) 21 | 5. [Database population](#database-population) 22 | 1. [XMPP scenario](#xmpp-scenario) 23 | 2. [MQTT scenario](#mqtt-scenario) 24 | 6. [Configuration](#configuration) 25 | 1. [General parameters](#general-parameters) 26 | 2. [Parameters of the XMPP scenario](#parameters-of-the-xmpp-scenario) 27 | 3. [Parameters of the MQTT scenario](#parameters-of-the-mqtt-scenario) 28 | 4. [Patterns](#patterns) 29 | 30 | # Status 31 | 32 | RTB is in an early stage of development with the following limitations: 33 | - MQTT support is limited to versions 3.1.1 and 5.0 34 | - For XMPP protocol, support for 35 | [Personal Eventing Protocol](https://xmpp.org/extensions/xep-0163.html) 36 | is lacking. 37 | 38 | Also, "sane" defaults and what should be considered a 39 | "golden benchmark" is yet to be discussed within the XMPP and MQTT community. 40 | 41 | However, the tool has been already battle-tested: 42 | [ProcessOne](https://www.process-one.net/) is using the tool to 43 | stress test [ejabberd SaaS](https://ejabberd-saas.com/) deployments. 44 | 45 | # System requirements 46 | 47 | To compile RTB you need: 48 | 49 | - Unix-like OS. Windows is not supported. Only Linux is tested. 50 | - GNU Make. 51 | - GCC 52 | - G++ 53 | - Libexpat ≥ 1.95 54 | - Libyaml ≥ 0.1.4 55 | - Erlang/OTP ≥ 19.0 56 | - OpenSSL ≥ 1.0.0 57 | - Zlib ≥ 1.2.3 58 | - gnuplot ≥ 4.4 59 | 60 | For Debian based distros to install all the dependencies run: 61 | ``` 62 | # apt install gcc g++ make libexpat1-dev libyaml-dev libssl-dev \ 63 | zlib1g-dev gnuplot-nox erlang-nox erlang-dev 64 | ``` 65 | For Arch Linux: 66 | ``` 67 | # pacman -S expat libyaml erlang-nox gnuplot 68 | ``` 69 | For other Linux distros, *BSD and OSX: 70 | 71 | **TODO**: Please create an issue/PR if you know the sequence of packages to install. 72 | 73 | # Compiling 74 | 75 | As usual, the following commands are used to obtain and compile the tool: 76 | ``` 77 | $ git clone https://github.com/processone/rtb.git 78 | $ cd rtb 79 | $ make 80 | ``` 81 | 82 | # Usage 83 | 84 | Once compiled, you will find `rtb.sh`, `rtb.yml.xmpp.example` and 85 | `rtb.yml.mqtt.example` files. Copy either `rtb.yml.mqtt.example` or 86 | `rtb.yml.xmpp.example` to `rtb.yml`, edit it (see section below) and run: 87 | ``` 88 | $ cp rtb.yml.xmpp.example rtb.yml 89 | $ editor rtb.yml 90 | $ ./rtb.sh 91 | ``` 92 | Investigate the output of the script for presence of errors. All errors 93 | are supposed to be self-explanatory and most of them have some hints on 94 | what should be done to fix them. 95 | 96 | To stop the benchmark press `Ctrl+C+C`. In order to start the 97 | benchmark in background append `-detached` switch: 98 | ``` 99 | $ ./rtb.sh -detached 100 | ``` 101 | You will find logs in the `log` directory. To monitor the progress 102 | open the statistics web page: it's located at `http://this.machine.tld:8080` 103 | by default. Edit `www_port` option to change the port if needed. 104 | 105 | # Database population 106 | 107 | During compilation a special utility is created which aims to help 108 | you populating the server's database. It's located at `priv/bin/rtb_db`. 109 | Run `priv/bin/rtb_db --help` to see available options. 110 | 111 | ## XMPP scenario 112 | 113 | The utility is able to generate files for users and rosters in either 114 | CSV format ([ejabberd](https://www.ejabberd.im/) and [jackal](https://github.com/ortuman/jackal)) 115 | or in Lua format ([Metronome](https://metronome.im/)/[Prosody](https://prosody.im/)). 116 | In order to generate files for ejabberd execute something like: 117 | ``` 118 | $ priv/bin/rtb_db -t ejabberd -c 1000 -u user%@domain.tld -p pass% -r 20 119 | ``` 120 | The same, but for Metronome will look like: 121 | ``` 122 | $ priv/bin/rtb_db -t metronome -c 1000 -u user%@domain.tld -p pass% -r 20 123 | 124 | ``` 125 | For Prosody: 126 | ``` 127 | $ priv/bin/rtb_db -t prosody -c 1000 -u user%@domain.tld -p pass% -r 20 128 | 129 | ``` 130 | For jackal: 131 | ``` 132 | $ priv/bin/rtb_db -t jackal -c 1000 -u user%@domain.tld -p pass% -r 20 133 | 134 | ``` 135 | Here 1000 is the total amount of users (must match `capacity` parameter 136 | of the configuration file) and 20 is the number of items in rosters. 137 | Don't provide `-r` option or set it to zero (0) if you don't want to 138 | generate rosters. 139 | 140 | Follow the hint provided by the utility to load generated files 141 | into the server's spool/database. Note that `--username` and 142 | `--password` arguments must match those defined in the configuration file 143 | (see `jid` and `password` parameters). 144 | 145 | ## MQTT scenario 146 | 147 | The utility is also able to generate `passwd` file for 148 | [Mosquitto](https://mosquitto.org/). 149 | In order to generate the file execute something like: 150 | ``` 151 | $ priv/bin/rtb_db -t mosquitto -c 1000 -u user% -p pass% 152 | ``` 153 | Here 1000 is the total amount of users (must match `capacity` parameter 154 | of the configuration file). 155 | 156 | Follow the hint provided by the utility to set up `passwd` file in Mosquitto 157 | configuration. 158 | 159 | Note that `--username` and `--password` arguments must match those defined 160 | in the configuration file (see `username` and `password` parameters). 161 | 162 | # Configuration 163 | 164 | All configuration is performed via editing parameters of `rtb.yml` file. 165 | The file has [YAML](http://en.wikipedia.org/wiki/YAML) syntax. 166 | There are mandatory and optional parameters. 167 | The majority of parameters are optional. 168 | 169 | ## General parameters 170 | 171 | This group of parameters are common for all scenarios. 172 | 173 | ### Mandatory parameters 174 | 175 | - **scenario**: `string()` 176 | 177 | The benchmarking scenario to use. Available values are `mqtt` and `xmpp`. 178 | 179 | - **interval**: `non_neg_integer()` 180 | 181 | The option is used to set a timeout to wait before spawning 182 | the next connection. The value is in **milliseconds**. 183 | 184 | - **capacity**: `pos_integer()` 185 | 186 | The total amount of connections to be spawned, starting from 1. 187 | 188 | - **certfile**: `string()` 189 | 190 | A path to a certificate file. The file MUST contain both a full certficate 191 | chain and a private key in PEM format. 192 | 193 | The option is only mandatory in the case when your scenario is configured 194 | to utilize TLS connections. 195 | 196 | - **servers**: `[uri()]` 197 | 198 | The list of server URIs to connect. The format of the URI must be 199 | `scheme://hostname:port/path` where `scheme` can be `tcp`, `tls`, `ws` or `wss`; 200 | `hostname` can be any DNS name or IP address and `port` is a port number. 201 | Note that `scheme`, `hostname` and `port` parts of the URI are mandatory, where 202 | `path` part is optional and only meaningful when the scheme is `ws` or `wss`. 203 | IPv6 addresses MUST be enclosed in square brackets. 204 | It's highly recommended to use IP addresses in `hostname` 205 | part: excessive DNS lookups may create significant overhead for the 206 | benchmarking tool itself. 207 | 208 | The option is used to set a transport, address and port of the server(s) 209 | being tested. 210 | 211 | The option is only mandatory for MQTT scenario, because there are no well 212 | established mechanisms to locate MQTT servers. 213 | 214 | For XMPP scenario the default is empty list which means server endpoints 215 | will be located according to RFC6120 procedure, that is DNS A/AAAA/SRV lookup 216 | of a domain part of `jid` parameter. 217 | Leaving the default alone is also not recommended for the reason described above. 218 | 219 | An URI from the `servers` list is picked in round-robin manner during 220 | initial connections setup, but it's picked randomly for reconnection attempts. 221 | 222 | Note that WebSockets connections are currently supported by MQTT scenario only. 223 | 224 | Example: 225 | ```yaml 226 | scenario: mqtt 227 | interval: 10 228 | capacity: 10000 229 | certfile: cert.pem 230 | servers: 231 | - tls://127.0.0.1:8883 232 | - tcp://192.168.1.1:1883 233 | - wss://[::1]:443/mqtt 234 | ``` 235 | 236 | ### Optional parameters 237 | 238 | - **bind**: `[ip_address()]` 239 | 240 | The list of IP addresses of local interfaces to bind. The typical 241 | usage of the option is to set several binding interfaces in order 242 | to establish more than 64k outgoing connections from the same machine. 243 | The default is empty list: in this case a binding address will be chosen 244 | automatically by the OS. 245 | 246 | - **stats_dir**: `string()` 247 | 248 | A path to the directory where statistics data will be stored. 249 | The files in the directory are used by `gnuplot` to generate statistics graphs. 250 | The default value is `stats`. 251 | 252 | - **www_dir**: `string()` 253 | 254 | A path to a directory where HTML and image files will be created. 255 | The default is `www`. This is used by the statistics web interface. 256 | 257 | - **www_port**: `pos_integer()` 258 | 259 | A port number to start the statistics web interface at. The default is 8080. 260 | 261 | - **gnuplot**: `string()` 262 | 263 | The path to a gnuplot execution binary. By default RTB is trying to detect 264 | the location of gnuplot automatically. 265 | 266 | - **debug**: `true | false` 267 | 268 | Whether to log debug messages or not. This is only needed to track down 269 | issues of the server or the tool itself. **DON'T** enable in large scale benchmarking. 270 | The default is `false`. 271 | 272 | Example: 273 | ```yaml 274 | bind: 275 | - 192.168.1.1 276 | - 192.168.1.2 277 | - 192.168.1.3 278 | stats_dir: /tmp/rtb/stats 279 | www_dir: /tmp/rtb/www 280 | www_port: 1234 281 | gnuplot: /opt/bin/gnuplot 282 | ``` 283 | 284 | ## Parameters of the XMPP scenario 285 | 286 | This group of parameters are specific to the XMPP scenario only. 287 | The parameters described here are applied per single session. 288 | 289 | ### Mandatory parameters 290 | 291 | - **jid**: `pattern()` 292 | 293 | A pattern for an XMPP address: bare or full. If it's bare, the default 294 | `rtb` resource will be used. Refer to [Patterns](#patterns) section for 295 | the detailed explanation of possible pattern values. 296 | 297 | - **password**: `pattern()` 298 | 299 | The pattern for a password. Refer to [Patterns](#patterns) section for 300 | the detailed explanation of possible pattern values. 301 | 302 | Example: 303 | ```yaml 304 | jid: user%@domain.tld 305 | password: pass% 306 | ``` 307 | 308 | ### Optional parameters 309 | 310 | #### Parameters for timings control. 311 | 312 | - **negotiation_timeout**: `pos_integer() | false` 313 | 314 | A timeout to wait for a stream negotiation to complete. 315 | The value is in **seconds**. It can be set to `false` to disable timeout. 316 | The default is 100 seconds. 317 | 318 | - **connect_timeout**: `pos_integer() | false` 319 | 320 | A timeout to wait for a TCP connection to be established. 321 | The value is in **seconds**. It can be set to `false` to disable timeout. 322 | The default is 100 seconds. 323 | 324 | - **reconnect_interval**: `pos_integer() | false` 325 | 326 | A timeout to wait before another reconnection attempt after previous 327 | connection failure. Initially it is picked randomly between `1` and this 328 | configured value. Then, exponential back off is applied between several 329 | consecutive connection failures. The value is in **seconds**. 330 | It can be set to `false` to disable reconnection attempts completely: 331 | thus the failed session will never be restored. 332 | The default is 60 (1 minute) - the value recommended by 333 | [RFC6120](https://tools.ietf.org/html/rfc6120#section-3.3). 334 | 335 | - **message_interval**: `pos_integer() | false` 336 | 337 | An interval between sending messages. The value is in **seconds**. 338 | It can be set to `false` to disable sending messages completely. 339 | The default is 600 (10 minutes). See also `message_body_size` option. 340 | 341 | - **muc_message_interval**: `pos_integer() | false` 342 | 343 | An interval between sending groupchat messages. The value is in **seconds**. 344 | It can be set to `false` to disable sending groupchat messages completely. 345 | The default is 600 (10 minutes). If there are several MUC rooms configured, 346 | the groupchat message is sent to a randomly chosen one, i.e. RTB doesn't 347 | multicast the message to all joined rooms. 348 | See also `message_body_size` and `muc_rooms` options. 349 | The option doesn't have any effect if `muc_rooms` option is not set. 350 | 351 | - **presence_interval**: `pos_integer() | false` 352 | 353 | An interval between sending presence broadcats. The value is in **seconds**. 354 | It can be set to `false` to disable sending presences completely. 355 | The default is 600 (10 minutes). Note that at the first successful login a 356 | presence broadcast is always sent unless the value is not set to `false`. 357 | 358 | - **disconnect_interval**: `pos_integer() | false` 359 | 360 | An interval to wait before forcing disconnect. The value is in **seconds**. 361 | The default is 600 (10 minutes). If stream management is enabled 362 | (this is the default, see `sm` option), then the session 363 | will be resumed after a random timeout between 1 and the value of 364 | `max` attribute of `` element reported by the server. 365 | Otherwise, the next reconnection attempt will be performed according 366 | to the value and logic of `reconnect_interval`. 367 | 368 | - **proxy65_interval**: `pos_integer() | false` 369 | 370 | An interval between file transfers via Proxy65 service (XEP-0065). 371 | The value is in **seconds**. It can be set to `false` to disable 372 | file transfer completely. The default is 600 (10 minutes). 373 | See also `proxy65_size` option. 374 | 375 | - **http_upload_interval**: `pos_integer() | false` 376 | 377 | An interval between file uploads via HTTP Upload service (XEP-0363). 378 | The value is in **seconds**. It can be set to `false` to disable 379 | file uploads completely. The default is 600 (10 minutes). 380 | See also `http_upload_size` option. 381 | 382 | #### Parameters for payload/size control 383 | 384 | - **message_to**: `pattern()` 385 | 386 | The pattern of a JID to which messages will be sent. By default 387 | a random JID within benchmark capacity is picked (whether it is 388 | already connected or not). Refer to [Patterns](#patterns) section for 389 | the detailed explanation of possible pattern values. 390 | 391 | For example, to send messages to already connected JIDs, set: 392 | ```yaml 393 | message_to: user?@domain.tld 394 | ``` 395 | 396 | - **message_body_size**: `non_neg_integer()` 397 | 398 | The size of `` element of a message in **bytes**. 399 | Only makes sense when `message_interval` is not set to `false`. 400 | The default is 100 bytes. 401 | 402 | - **proxy65_size**: `non_neg_integer()` 403 | 404 | The size of a file to transfer via Proxy65 service in **bytes**. 405 | Only makes sense when `proxy65_interval` is not set to `false`. 406 | The default is 10485760 (10 megabytes). 407 | 408 | - **http_upload_size**: `non_neg_integer()` 409 | 410 | The size of a file to upload via HTTP Upload service in **bytes**. 411 | Only makes sense when `http_upload_interval` is not set to `false`. 412 | There is no default value: the option is only needed to set 413 | if the service doesn't report its maximum file size. 414 | 415 | - **mam_size**: `non_neg_integer()` 416 | 417 | The size of the MAM archive to request from the server. Only makes 418 | sense when `mam` is not set to `false`. The default is 50 (messages). 419 | 420 | - **muc_mam_size**: `non_neg_integer()` 421 | 422 | The size of the MAM archive to request from MUC rooms. Only makes 423 | sense when `muc_mam` is not set to `false`. The default is 50 (messages). 424 | 425 | #### Parameters for enabling/disabling features 426 | 427 | - **starttls**: `true | false` 428 | 429 | Whether to use STARTTLS or not. The default is `true`. 430 | 431 | - **csi**: `true | false` 432 | 433 | Whether to send client state indications or not (XEP-0352). 434 | The default is `true`. 435 | 436 | - **sm**: `true | false` 437 | 438 | Whether to enable stream management with resumption or not (XEP-0198). 439 | The default is `true`. 440 | 441 | - **mam**: `true | false` 442 | 443 | Whether to enable MAM and request MAM archives at login time or not (XEP-0313). 444 | The default is `true`. The requested size of the archive is controlled 445 | by `mam_size` option. 446 | 447 | - **muc_mam**: `true | false` 448 | 449 | Whether to request MAM archives from MUC room or not (XEP-0313). 450 | The default is `true`. The requested size of the archive is controlled 451 | by `muc_mam_size` option. 452 | 453 | - **carbons**: `true | false` 454 | 455 | Whether to enable message carbons or not (XEP-0280). The default is `true`. 456 | 457 | - **blocklist**: `true | false` 458 | 459 | Whether to request block list at login time or not (XEP-0191). 460 | The default is `true`. 461 | 462 | - **roster**: `true | false` 463 | 464 | Whether to request roster at login time or not. The default is `true`. 465 | 466 | - **rosterver**: `true | false` 467 | 468 | Whether to set a roster version attribute in roster request or not. 469 | The default is `true`. 470 | 471 | - **private**: `true | false` 472 | 473 | Whether to request bookmarks from private storage at login time or not (XEP-0049). 474 | The default is `true`. 475 | 476 | #### Miscellaneous parameters 477 | 478 | - **muc_rooms**: `[pattern()]` 479 | 480 | A list of MUC room bare JIDs to join, expressed as a pattern. Refer to 481 | [Patterns](#patterns) section for the detailed explanation of possible pattern values. 482 | The default value is an empty list which means no rooms will be joined. 483 | Example: 484 | ```yaml 485 | muc_rooms: 486 | - large1@conference.domain.tld 487 | - medium[1..10]@conference.domain.tld 488 | - small[1..100]@conference.domain.tld 489 | ``` 490 | 491 | - **sasl_mechanisms**: `[string()]` 492 | 493 | A list of SASL mechanisms to use for authentication. Supported mechanisms are 494 | `PLAIN` and `EXTERNAL`. The appropriate mechanism will be picked automatically. 495 | If several mechanisms found matching then all of them will be tried in the order 496 | defined by the value until the authentication is succeed. The default is `[PLAIN]`. 497 | Note that for `EXTERNAL` authentication you need to have a valid certificate file 498 | defined in the `certfile` option. 499 | 500 | Example: 501 | ```yaml 502 | sasl_mechanisms: 503 | - PLAIN 504 | - EXTERNAL 505 | ``` 506 | 507 | ## Parameters of the MQTT scenario 508 | 509 | This group of parameters are specific to the MQTT scenario only. 510 | The parameters described here are applied per single session. 511 | 512 | ### Mandatory parameters 513 | 514 | - **client_id**: `pattern()` 515 | 516 | A pattern for an MQTT Client ID. Refer to [Patterns](#patterns) section for 517 | the detailed explanation of possible pattern values. 518 | 519 | Example: 520 | ```yaml 521 | client_id: rtb% 522 | ``` 523 | 524 | ### Optional parameters 525 | 526 | #### Authentication parameters 527 | 528 | - **username**: `pattern()` 529 | 530 | A pattern for a user name. Refer to [Patterns](#patterns) section for 531 | the detailed explanation of possible pattern values. 532 | 533 | - **password**: `pattern()` 534 | 535 | The pattern for a password. Refer to [Patterns](#patterns) section for 536 | the detailed explanation of possible pattern values. 537 | 538 | Example: 539 | ```yaml 540 | username: user% 541 | password: pass% 542 | ``` 543 | 544 | #### Parameters for session control 545 | 546 | - **protocol_version**: `string()` 547 | 548 | MQTT protocol version. Can be `3.1.1` or `5.0`. The default is `3.1.1`. 549 | 550 | - **clean_session**: `true | false` 551 | 552 | Whether to set `CleanSession` flag or not. If the value is `true` then 553 | the MQTT session state (subscriptions and message queue) won't be kept 554 | at the server between client reconnections and, thus, no state synchronization 555 | will be performed when the session is re-established. The default is `false`. 556 | 557 | - **will**: `publish_options()` 558 | 559 | The format of a Client Will. The parameter consists of a list of publish 560 | options. See `publish` parameter description. The default is empty will. 561 | 562 | Example: 563 | ```yaml 564 | protocol_version: 5.0 565 | clean_session: true 566 | will: 567 | qos: 2 568 | retain: false 569 | topic: /rtb/? 570 | message: "*" 571 | ``` 572 | 573 | #### Parameters for PUBLISH/SUBSCRIBE 574 | 575 | - **publish**: `publish_options()` 576 | 577 | The format of a PUBLISH message. Only makes sense when `publish_interval` is 578 | not set to `false`. The format is described by a group of sub-options: 579 | 580 | - **qos**: `0..2` 581 | 582 | Quality of Service. The default is 0. 583 | 584 | - **retain**: `true | false` 585 | 586 | Whether the message should be retained or not. The default is `false`. 587 | 588 | - **topic**: `pattern()` 589 | 590 | The pattern for a topic. Refer to [Patterns](#patterns) section for 591 | the detailed explanation of possible pattern values. 592 | 593 | - **message**: `pattern() | non_neg_integer()` 594 | 595 | The pattern or the size of a message payload. If it's an integer 596 | then the payload of the given size will be randomly generated every time 597 | a message is about to be sent. Refer to [Patterns](#patterns) section for 598 | the detailed explanation of possible pattern values. 599 | 600 | Example: 601 | ```yaml 602 | publish: 603 | qos: 1 604 | retain: true 605 | topic: /rtb/? 606 | message: 32 607 | ``` 608 | - **subscribe**: `[{pattern(), 0..2}]` 609 | 610 | The format of a SUBSCRIBE message. It is represented as a list of 611 | topic-filter/QoS pairs. Refer to [Patterns](#patterns) section 612 | for the detailed explanation of possible pattern values. The message is sent 613 | immediately after successful authentication of a newly created session. 614 | The default is empty list, i.e. no SUBSCRIBE messages will be sent. 615 | 616 | Example: 617 | ```yaml 618 | subscribe: 619 | /foo/bar/%: 2 620 | $SYS/#: 1 621 | /rtb/[1..10]: 0 622 | ``` 623 | 624 | - **track_publish_delivery**: `true | false` 625 | 626 | Check if PUBLISH packets reach at least a single destination client. The result 627 | is displayed as `publish-loss` graph in the Web statistics interface. 628 | The default is `false`. 629 | 630 | **NOTE**: the option is only available for MQTT 5.0. See `protocol_version` option. 631 | 632 | #### Parameters for timings control 633 | 634 | - **keep_alive**: `pos_integer()` 635 | 636 | The interval to send keep-alive pings. The value is in **seconds**. 637 | The default is 60 (seconds). 638 | 639 | - **reconnect_interval**: `pos_integer() | false` 640 | 641 | A timeout to wait before another reconnection attempt after previous 642 | connection failure. Initially it is picked randomly between `1` and this 643 | configured value. Then, exponential back off is applied between several 644 | consecutive connection failures. The value is in **seconds**. 645 | It can be set to `false` to disable reconnection attempts completely: 646 | thus the failed session will never be restored. 647 | The default is 60 (1 minute). 648 | 649 | - **disconnect_interval**: `pos_integer() | false` 650 | 651 | An interval to wait before forcing disconnect. The value is in **seconds**. 652 | The default is 600 (10 minutes). The next reconnection attempt will be 653 | performed according to the value and logic of `reconnect_interval`. 654 | 655 | - **publish_interval**: `pos_integer() | false` 656 | 657 | An interval between sending PUBLISH messages. Can be set to `false` in order 658 | to disable sending PUBLISH messages completely. The value is in **seconds**. 659 | The default is 600 (10 minutes). 660 | 661 | ## Patterns 662 | 663 | Many configuration options allow to set patterns as their values. The pattern 664 | is just a regular string with a special treatment of symbols `%`, `*`, `?`, 665 | `[` and `]`. 666 | 667 | ### Current connection identifier 668 | 669 | The symbol '%' is replaced by the current identifier of the connection. For example, 670 | when the pattern of `username` parameter is `user%` and the value of `capacity` 671 | is 5, then the value of `username` will be evaluated into `user1` for first 672 | connection (i.e. the connection with identifier 1), `user2` for the second 673 | connection and `user5` for the last connection. 674 | Patterns with such identifier are supposed to address a connection within 675 | which this pattern is evaluated. 676 | 677 | Example: 678 | ```yaml 679 | jid: user%@domain.tld 680 | client_id: client% 681 | password: pass% 682 | ``` 683 | 684 | ### Random session identifier 685 | 686 | The symbol '?' is replaced by an identifier of random available session. 687 | For example, when there are already spawned 5 connections with 1,3 and 5 688 | connections being fully established (and, thus, internally registered as a "session"), 689 | the pattern `user?` will yield into `user1`, `user3` or `user5`, 690 | but not into `user2` or `user4`. 691 | Patterns with such identifier are supposed to be used for sending/publishing 692 | messages to online users. 693 | 694 | Example: 695 | ```yaml 696 | message_to: user?@domain.tld 697 | publish: 698 | ... 699 | topic: /rtb/topic? 700 | ... 701 | ``` 702 | 703 | ### Unique identifier 704 | 705 | The symbol '*' is replaced by a positive integer. The integer is guaranteed 706 | to be unique within the benchmark lifetime. Note that the integer is **not** 707 | guaranteed to be monotonic. 708 | Patterns with such identifiers are supposed to be used to mark some content 709 | as unique for further tracking (in logs or network dump). 710 | 711 | Example: 712 | ```yaml 713 | publish: 714 | topic: /foo/bar 715 | ... 716 | message: "*" 717 | ``` 718 | 719 | ### Range identifier 720 | 721 | The expression `[X..Y]` where `X` and `Y` are non-negative integers and `X ≤ Y` 722 | is replaced by a random integer between `X` and `Y` (inclusively). 723 | Patterns with such expression are supposed to be used for sending/publishing/subscribing 724 | to a restricted subset of recipients or topics. 725 | 726 | Example: 727 | ```yaml 728 | subscribe: 729 | /rtb/topic/[1..10]: 2 730 | publish: 731 | topic: /rtb/topic/[1..10] 732 | muc_rooms: 733 | - room[1..5]@conference.domain.tld 734 | ``` 735 | --------------------------------------------------------------------------------