├── LICENSE ├── Makefile ├── README.md ├── TODO ├── etc └── emqx_bridge_kafka.config ├── rebar.config ├── rebar.config.script ├── src ├── emqx_bridge_kafka.app.src ├── emqx_bridge_kafka.app.src.script ├── emqx_bridge_kafka.erl ├── emqx_bridge_kafka_app.erl ├── emqx_bridge_kafka_cli.erl └── emqx_bridge_kafka_sup.erl └── test └── emqx_bridge_kafka_SUITE.erl /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | ## shallow clone for speed 2 | 3 | REBAR_GIT_CLONE_OPTIONS += --depth 1 4 | export REBAR_GIT_CLONE_OPTIONS 5 | 6 | REBAR = rebar3 7 | all: compile 8 | 9 | compile: 10 | $(REBAR) compile 11 | 12 | ct: compile 13 | $(REBAR) as test ct -v 14 | 15 | eunit: compile 16 | $(REBAR) as test eunit 17 | 18 | xref: 19 | $(REBAR) xref 20 | 21 | cover: 22 | $(REBAR) cover 23 | 24 | clean: distclean 25 | 26 | distclean: 27 | @rm -rf _build 28 | @rm -f data/app.*.config data/vm.*.args rebar.lock 29 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | emqx_bridge_kafka 2 | ==================== 3 | 4 | This is a template plugin for the EMQ X broker. And you can see [Plugin Development Guide](https://docs.emqx.io/broker/v3/en/plugins.html#plugin-development-template) to learning how to use it. 5 | 6 | Plugin Config 7 | ------------- 8 | 9 | Each plugin should have a 'etc/{plugin_name}.conf|config' file to store application config. 10 | 11 | Authentication and ACL 12 | ---------------------- 13 | 14 | ``` 15 | emqx:hook('client.authenticate', fun ?MODULE:on_client_authenticate/3, [Env]). 16 | emqx:hook('client.check_acl', fun ?MODULE:on_client_check_acl/5, [Env]). 17 | ``` 18 | 19 | Plugin and Hooks 20 | ----------------- 21 | 22 | [Plugin Design](https://docs.emqx.io/broker/v3/en/design.html#plugin-design) 23 | 24 | [Hooks Design](https://docs.emqx.io/broker/v3/en/design.html#hooks-design) 25 | 26 | License 27 | ------- 28 | 29 | Apache License Version 2.0 30 | 31 | Author 32 | ------ 33 | 34 | EMQ X Team. 35 | -------------------------------------------------------------------------------- /TODO: -------------------------------------------------------------------------------- 1 | 1. Add a script to generate plugin project 2 | 2. Upgrade the README.md 3 | 3. Add the plugin development guide 4 | -------------------------------------------------------------------------------- /etc/emqx_bridge_kafka.config: -------------------------------------------------------------------------------- 1 | 2 | [ 3 | {emqx_bridge_kafka, 4 | [{values, [ 5 | {bootstrap_broker, {"192.168.91.182", 9092}}, 6 | {partition_strategy, strict_round_robin}, 7 | {kafka_producer_topic, <<"mqtt_to_kafka">>} 8 | ]} 9 | ]} 10 | ]. 11 | 12 | -------------------------------------------------------------------------------- /rebar.config: -------------------------------------------------------------------------------- 1 | {deps, [ 2 | {ekaf, {git, "https://github.com/helpshift/ekaf", {branch,"master"}}} 3 | ]}. 4 | 5 | {erl_opts, [debug_info]}. 6 | -------------------------------------------------------------------------------- /rebar.config.script: -------------------------------------------------------------------------------- 1 | %%-*- mode: erlang -*- 2 | CONFIG1 = case os:getenv("TRAVIS") of 3 | "true" -> 4 | JobId = os:getenv("TRAVIS_JOB_ID"), 5 | [{coveralls_service_job_id, JobId}, 6 | {coveralls_coverdata, "_build/test/cover/*.coverdata"}, 7 | {coveralls_service_name , "travis-ci"} | CONFIG]; 8 | _ -> 9 | CONFIG 10 | end, 11 | 12 | CUR_BRANCH = os:cmd("git branch | grep -e '^*' | cut -d' ' -f 2") -- "\n", 13 | 14 | MATCH_BRANCH = fun (BranchName) when BranchName =:= "master"; 15 | BranchName =:= "develop" -> 16 | BranchName; 17 | (BranchName) -> 18 | case string:prefix(BranchName, "release") of 19 | nomatch -> "develop"; 20 | _Match -> BranchName 21 | end 22 | end, 23 | 24 | BRANCH = MATCH_BRANCH(CUR_BRANCH), 25 | 26 | DEPS = case lists:keyfind(deps, 1, CONFIG1) of 27 | {_, Deps} -> Deps; 28 | _ -> [] 29 | end, 30 | 31 | UrlPrefix = "https://github.com/emqx/", 32 | 33 | EMQX_DEP = {emqx, {git, UrlPrefix ++ "emqx", {branch, BRANCH}}}, 34 | 35 | NewDeps = [EMQX_DEP | DEPS], 36 | 37 | CONFIG2 = lists:keystore(deps, 1, CONFIG1, {deps, NewDeps}), 38 | 39 | CONFIG2. 40 | -------------------------------------------------------------------------------- /src/emqx_bridge_kafka.app.src: -------------------------------------------------------------------------------- 1 | {application, emqx_bridge_kafka, 2 | [{description, "EMQ X Bridge to Kafka Broker"}, 3 | {vsn, "git"}, 4 | {modules, []}, 5 | {registered, [emqx_bridge_kafka_sup]}, 6 | {applications, [kernel, stdlib, ekaf]}, 7 | {mod, {emqx_bridge_kafka_app,[]}}, 8 | {env, []}, 9 | {licenses, ["Apache-2.0"]}, 10 | {maintainers, ["EMQ X Team "]}, 11 | {links, [{"Homepage", "https://emqx.io/"}, 12 | {"Github", "https://github.com/emqx/emqx_bridge_kafka"} 13 | ]} 14 | ]}. 15 | -------------------------------------------------------------------------------- /src/emqx_bridge_kafka.app.src.script: -------------------------------------------------------------------------------- 1 | %%-*- mode: erlang -*- 2 | %% .app.src.script 3 | 4 | RemoveLeadingV = 5 | fun(Tag) -> 6 | case re:run(Tag, "v\[0-9\]+\.\[0-9\]+\.*") of 7 | nomatch -> 8 | Tag; 9 | {match, _} -> 10 | %% if it is a version number prefixed by 'v' then remove the 'v' 11 | "v" ++ Vsn = Tag, 12 | Vsn 13 | end 14 | end, 15 | 16 | case os:getenv("EMQX_DEPS_DEFAULT_VSN") of 17 | false -> CONFIG; % env var not defined 18 | [] -> CONFIG; % env var set to empty string 19 | Tag -> 20 | [begin 21 | AppConf0 = lists:keystore(vsn, 1, AppConf, {vsn, RemoveLeadingV(Tag)}), 22 | {application, App, AppConf0} 23 | end || Conf = {application, App, AppConf} <- CONFIG] 24 | end. 25 | 26 | -------------------------------------------------------------------------------- /src/emqx_bridge_kafka.erl: -------------------------------------------------------------------------------- 1 | %%-------------------------------------------------------------------- 2 | %% Copyright (c) 2020 EMQ Technologies Co., Ltd. All Rights Reserved. 3 | %% 4 | %% Licensed under the Apache License, Version 2.0 (the "License"); 5 | %% you may not use this file except in compliance with the License. 6 | %% You may obtain a copy of the License at 7 | %% 8 | %% http://www.apache.org/licenses/LICENSE-2.0 9 | %% 10 | %% Unless required by applicable law or agreed to in writing, software 11 | %% distributed under the License is distributed on an "AS IS" BASIS, 12 | %% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | %% See the License for the specific language governing permissions and 14 | %% limitations under the License. 15 | %%-------------------------------------------------------------------- 16 | 17 | -module(emqx_bridge_kafka). 18 | 19 | -include_lib("emqx/include/emqx.hrl"). 20 | 21 | -export([ load/1 22 | , unload/0 23 | ]). 24 | 25 | %% Client Lifecircle Hooks 26 | -export([ on_client_connect/3 27 | , on_client_connack/4 28 | , on_client_connected/3 29 | , on_client_disconnected/4 30 | , on_client_authenticate/3 31 | , on_client_check_acl/5 32 | , on_client_subscribe/4 33 | , on_client_unsubscribe/4 34 | ]). 35 | 36 | %% Session Lifecircle Hooks 37 | -export([ on_session_created/3 38 | , on_session_subscribed/4 39 | , on_session_unsubscribed/4 40 | , on_session_resumed/3 41 | , on_session_discarded/3 42 | , on_session_takeovered/3 43 | , on_session_terminated/4 44 | ]). 45 | 46 | %% Message Pubsub Hooks 47 | -export([ on_message_publish/2 48 | , on_message_delivered/3 49 | , on_message_acked/3 50 | , on_message_dropped/4 51 | ]). 52 | 53 | %% Called when the plugin application start 54 | load(Env) -> 55 | ekaf_init([Env]), 56 | emqx:hook('client.connect', {?MODULE, on_client_connect, [Env]}), 57 | emqx:hook('client.connack', {?MODULE, on_client_connack, [Env]}), 58 | emqx:hook('client.connected', {?MODULE, on_client_connected, [Env]}), 59 | emqx:hook('client.disconnected', {?MODULE, on_client_disconnected, [Env]}), 60 | emqx:hook('client.authenticate', {?MODULE, on_client_authenticate, [Env]}), 61 | emqx:hook('client.check_acl', {?MODULE, on_client_check_acl, [Env]}), 62 | emqx:hook('client.subscribe', {?MODULE, on_client_subscribe, [Env]}), 63 | emqx:hook('client.unsubscribe', {?MODULE, on_client_unsubscribe, [Env]}), 64 | emqx:hook('session.created', {?MODULE, on_session_created, [Env]}), 65 | emqx:hook('session.subscribed', {?MODULE, on_session_subscribed, [Env]}), 66 | emqx:hook('session.unsubscribed',{?MODULE, on_session_unsubscribed, [Env]}), 67 | emqx:hook('session.resumed', {?MODULE, on_session_resumed, [Env]}), 68 | emqx:hook('session.discarded', {?MODULE, on_session_discarded, [Env]}), 69 | emqx:hook('session.takeovered', {?MODULE, on_session_takeovered, [Env]}), 70 | emqx:hook('session.terminated', {?MODULE, on_session_terminated, [Env]}), 71 | emqx:hook('message.publish', {?MODULE, on_message_publish, [Env]}), 72 | emqx:hook('message.delivered', {?MODULE, on_message_delivered, [Env]}), 73 | emqx:hook('message.acked', {?MODULE, on_message_acked, [Env]}), 74 | emqx:hook('message.dropped', {?MODULE, on_message_dropped, [Env]}). 75 | 76 | %%-------------------------------------------------------------------- 77 | %% Client Lifecircle Hooks 78 | %%-------------------------------------------------------------------- 79 | 80 | on_client_connect(ConnInfo = #{clientid := ClientId}, Props, _Env) -> 81 | io:format("Client(~s) connect, ConnInfo: ~p, Props: ~p~n", 82 | [ClientId, ConnInfo, Props]), 83 | {ok, Props}. 84 | 85 | on_client_connack(ConnInfo = #{clientid := ClientId}, Rc, Props, _Env) -> 86 | io:format("Client(~s) connack, ConnInfo: ~p, Rc: ~p, Props: ~p~n", 87 | [ClientId, ConnInfo, Rc, Props]), 88 | {ok, Props}. 89 | 90 | on_client_connected(ClientInfo = #{clientid := ClientId}, ConnInfo, _Env) -> 91 | io:format("Client(~s) connected, ClientInfo:~n~p~n, ConnInfo:~n~p~n", 92 | [ClientId, ClientInfo, ConnInfo]). 93 | 94 | on_client_disconnected(ClientInfo = #{clientid := ClientId}, ReasonCode, ConnInfo, _Env) -> 95 | io:format("Client(~s) disconnected due to ~p, ClientInfo:~n~p~n, ConnInfo:~n~p~n", 96 | [ClientId, ReasonCode, ClientInfo, ConnInfo]). 97 | 98 | on_client_authenticate(_ClientInfo = #{clientid := ClientId}, Result, _Env) -> 99 | io:format("Client(~s) authenticate, Result:~n~p~n", [ClientId, Result]), 100 | {ok, Result}. 101 | 102 | on_client_check_acl(_ClientInfo = #{clientid := ClientId}, Topic, PubSub, Result, _Env) -> 103 | io:format("Client(~s) check_acl, PubSub:~p, Topic:~p, Result:~p~n", 104 | [ClientId, PubSub, Topic, Result]), 105 | {ok, Result}. 106 | 107 | on_client_subscribe(#{clientid := ClientId}, _Properties, TopicFilters, _Env) -> 108 | io:format("Client(~s) will subscribe: ~p~n", [ClientId, TopicFilters]), 109 | {ok, TopicFilters}. 110 | 111 | on_client_unsubscribe(#{clientid := ClientId}, _Properties, TopicFilters, _Env) -> 112 | io:format("Client(~s) will unsubscribe ~p~n", [ClientId, TopicFilters]), 113 | {ok, TopicFilters}. 114 | 115 | %%-------------------------------------------------------------------- 116 | %% Session Lifecircle Hooks 117 | %%-------------------------------------------------------------------- 118 | 119 | on_session_created(#{clientid := ClientId}, SessInfo, _Env) -> 120 | io:format("Session(~s) created, Session Info:~n~p~n", [ClientId, SessInfo]). 121 | 122 | on_session_subscribed(#{clientid := ClientId}, Topic, SubOpts, _Env) -> 123 | io:format("Session(~s) subscribed ~s with subopts: ~p~n", [ClientId, Topic, SubOpts]). 124 | 125 | on_session_unsubscribed(#{clientid := ClientId}, Topic, Opts, _Env) -> 126 | io:format("Session(~s) unsubscribed ~s with opts: ~p~n", [ClientId, Topic, Opts]). 127 | 128 | on_session_resumed(#{clientid := ClientId}, SessInfo, _Env) -> 129 | io:format("Session(~s) resumed, Session Info:~n~p~n", [ClientId, SessInfo]). 130 | 131 | on_session_discarded(_ClientInfo = #{clientid := ClientId}, SessInfo, _Env) -> 132 | io:format("Session(~s) is discarded. Session Info: ~p~n", [ClientId, SessInfo]). 133 | 134 | on_session_takeovered(_ClientInfo = #{clientid := ClientId}, SessInfo, _Env) -> 135 | io:format("Session(~s) is takeovered. Session Info: ~p~n", [ClientId, SessInfo]). 136 | 137 | on_session_terminated(_ClientInfo = #{clientid := ClientId}, Reason, SessInfo, _Env) -> 138 | io:format("Session(~s) is terminated due to ~p~nSession Info: ~p~n", 139 | [ClientId, Reason, SessInfo]). 140 | 141 | %%-------------------------------------------------------------------- 142 | %% Message PubSub Hooks 143 | %%-------------------------------------------------------------------- 144 | 145 | %% Transform message and return 146 | on_message_publish(Message = #message{topic = <<"$SYS/", _/binary>>}, _Env) -> 147 | {ok, Message}; 148 | 149 | %% on_message_publish(Message, _Env) -> 150 | %% io:format("Publish ~s~n", [emqx_message:format(Message)]), 151 | %% {ok, Message}. 152 | 153 | on_message_publish(Message, _Env) -> 154 | io:format("Publish ~s~n", [emqx_message:format(Message)]), 155 | {ok, KafkaTopic} = application:get_env(emqx_bridge_kafka, values), 156 | ProduceTopic = proplists:get_value(kafka_producer_topic, KafkaTopic), 157 | Topic=Message#message.topic, 158 | Payload=Message#message.payload, 159 | Qos=Message#message.qos, 160 | %% Timestamp=Message#message.timestamp, 161 | Json = jsx:encode([ 162 | {type,<<"published">>}, 163 | {topic,Topic}, 164 | {payload,Payload}, 165 | {qos,Qos}, 166 | {cluster_node,node()} 167 | %% ,{ts,emqx_time:now_to_secs(Timestamp)} 168 | ]), 169 | %% ekaf:produce_async(ProduceTopic, Json), 170 | ekaf:produce_async(Topic, Payload), 171 | {ok, Message}. 172 | 173 | 174 | on_message_dropped(#message{topic = <<"$SYS/", _/binary>>}, _By, _Reason, _Env) -> 175 | ok; 176 | on_message_dropped(Message, _By = #{node := Node}, Reason, _Env) -> 177 | io:format("Message dropped by node ~s due to ~s: ~s~n", 178 | [Node, Reason, emqx_message:format(Message)]). 179 | 180 | on_message_delivered(_ClientInfo = #{clientid := ClientId}, Message, _Env) -> 181 | io:format("Message delivered to client(~s): ~s~n", 182 | [ClientId, emqx_message:format(Message)]), 183 | {ok, Message}. 184 | 185 | on_message_acked(_ClientInfo = #{clientid := ClientId}, Message, _Env) -> 186 | io:format("Message acked by client(~s): ~s~n", 187 | [ClientId, emqx_message:format(Message)]). 188 | 189 | %% Called when the plugin application stop 190 | unload() -> 191 | emqx:unhook('client.connect', {?MODULE, on_client_connect}), 192 | emqx:unhook('client.connack', {?MODULE, on_client_connack}), 193 | emqx:unhook('client.connected', {?MODULE, on_client_connected}), 194 | emqx:unhook('client.disconnected', {?MODULE, on_client_disconnected}), 195 | emqx:unhook('client.authenticate', {?MODULE, on_client_authenticate}), 196 | emqx:unhook('client.check_acl', {?MODULE, on_client_check_acl}), 197 | emqx:unhook('client.subscribe', {?MODULE, on_client_subscribe}), 198 | emqx:unhook('client.unsubscribe', {?MODULE, on_client_unsubscribe}), 199 | emqx:unhook('session.created', {?MODULE, on_session_created}), 200 | emqx:unhook('session.subscribed', {?MODULE, on_session_subscribed}), 201 | emqx:unhook('session.unsubscribed',{?MODULE, on_session_unsubscribed}), 202 | emqx:unhook('session.resumed', {?MODULE, on_session_resumed}), 203 | emqx:unhook('session.discarded', {?MODULE, on_session_discarded}), 204 | emqx:unhook('session.takeovered', {?MODULE, on_session_takeovered}), 205 | emqx:unhook('session.terminated', {?MODULE, on_session_terminated}), 206 | emqx:unhook('message.publish', {?MODULE, on_message_publish}), 207 | emqx:unhook('message.delivered', {?MODULE, on_message_delivered}), 208 | emqx:unhook('message.acked', {?MODULE, on_message_acked}), 209 | emqx:unhook('message.dropped', {?MODULE, on_message_dropped}). 210 | 211 | %% Init kafka server parameters 212 | ekaf_init(_Env) -> 213 | application:load(ekaf), 214 | {ok, Values} = application:get_env(emqx_bridge_kafka, values), 215 | BootstrapBroker = proplists:get_value(bootstrap_broker, Values), 216 | PartitionStrategy= proplists:get_value(partition_strategy, Values), 217 | application:set_env(ekaf, ekaf_partition_strategy, PartitionStrategy), 218 | application:set_env(ekaf, ekaf_bootstrap_broker, BootstrapBroker), 219 | {ok, _} = application:ensure_all_started(ekaf), 220 | io:format("Initialized ekaf with ~p~n", [BootstrapBroker]). 221 | -------------------------------------------------------------------------------- /src/emqx_bridge_kafka_app.erl: -------------------------------------------------------------------------------- 1 | %%-------------------------------------------------------------------- 2 | %% Copyright (c) 2020 EMQ Technologies Co., Ltd. All Rights Reserved. 3 | %% 4 | %% Licensed under the Apache License, Version 2.0 (the "License"); 5 | %% you may not use this file except in compliance with the License. 6 | %% You may obtain a copy of the License at 7 | %% 8 | %% http://www.apache.org/licenses/LICENSE-2.0 9 | %% 10 | %% Unless required by applicable law or agreed to in writing, software 11 | %% distributed under the License is distributed on an "AS IS" BASIS, 12 | %% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | %% See the License for the specific language governing permissions and 14 | %% limitations under the License. 15 | %%-------------------------------------------------------------------- 16 | 17 | -module(emqx_bridge_kafka_app). 18 | 19 | -behaviour(application). 20 | 21 | -emqx_plugin(?MODULE). 22 | 23 | -export([ start/2 24 | , stop/1 25 | ]). 26 | 27 | start(_StartType, _StartArgs) -> 28 | {ok, Sup} = emqx_bridge_kafka_sup:start_link(), 29 | emqx_bridge_kafka:load(application:get_all_env()), 30 | {ok, Sup}. 31 | 32 | stop(_State) -> 33 | emqx_bridge_kafka:unload(). 34 | 35 | -------------------------------------------------------------------------------- /src/emqx_bridge_kafka_cli.erl: -------------------------------------------------------------------------------- 1 | %%-------------------------------------------------------------------- 2 | %% Copyright (c) 2020 EMQ Technologies Co., Ltd. All Rights Reserved. 3 | %% 4 | %% Licensed under the Apache License, Version 2.0 (the "License"); 5 | %% you may not use this file except in compliance with the License. 6 | %% You may obtain a copy of the License at 7 | %% 8 | %% http://www.apache.org/licenses/LICENSE-2.0 9 | %% 10 | %% Unless required by applicable law or agreed to in writing, software 11 | %% distributed under the License is distributed on an "AS IS" BASIS, 12 | %% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | %% See the License for the specific language governing permissions and 14 | %% limitations under the License. 15 | %%-------------------------------------------------------------------- 16 | 17 | -module(emqx_bridge_kafka_cli). 18 | 19 | -export([cmd/1]). 20 | 21 | cmd(["arg1", "arg2"]) -> 22 | emqx_ctl:print("ok"); 23 | 24 | cmd(_) -> 25 | emqx_ctl:usage([{"cmd arg1 arg2", "cmd demo"}]). 26 | 27 | -------------------------------------------------------------------------------- /src/emqx_bridge_kafka_sup.erl: -------------------------------------------------------------------------------- 1 | %%-------------------------------------------------------------------- 2 | %% Copyright (c) 2020 EMQ Technologies Co., Ltd. All Rights Reserved. 3 | %% 4 | %% Licensed under the Apache License, Version 2.0 (the "License"); 5 | %% you may not use this file except in compliance with the License. 6 | %% You may obtain a copy of the License at 7 | %% 8 | %% http://www.apache.org/licenses/LICENSE-2.0 9 | %% 10 | %% Unless required by applicable law or agreed to in writing, software 11 | %% distributed under the License is distributed on an "AS IS" BASIS, 12 | %% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | %% See the License for the specific language governing permissions and 14 | %% limitations under the License. 15 | %%-------------------------------------------------------------------- 16 | 17 | -module(emqx_bridge_kafka_sup). 18 | 19 | -behaviour(supervisor). 20 | 21 | -export([start_link/0]). 22 | 23 | -export([init/1]). 24 | 25 | start_link() -> 26 | supervisor:start_link({local, ?MODULE}, ?MODULE, []). 27 | 28 | init([]) -> 29 | {ok, { {one_for_all, 0, 1}, []} }. 30 | 31 | -------------------------------------------------------------------------------- /test/emqx_bridge_kafka_SUITE.erl: -------------------------------------------------------------------------------- 1 | %%-------------------------------------------------------------------- 2 | %% Copyright (c) 2020 EMQ Technologies Co., Ltd. All Rights Reserved. 3 | %% 4 | %% Licensed under the Apache License, Version 2.0 (the "License"); 5 | %% you may not use this file except in compliance with the License. 6 | %% You may obtain a copy of the License at 7 | %% 8 | %% http://www.apache.org/licenses/LICENSE-2.0 9 | %% 10 | %% Unless required by applicable law or agreed to in writing, software 11 | %% distributed under the License is distributed on an "AS IS" BASIS, 12 | %% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | %% See the License for the specific language governing permissions and 14 | %% limitations under the License. 15 | %%-------------------------------------------------------------------- 16 | 17 | -module(emqx_bridge_kafka_SUITE). 18 | 19 | -compile(export_all). 20 | 21 | all() -> []. 22 | 23 | groups() -> []. 24 | --------------------------------------------------------------------------------