├── .travis.yml ├── README.md ├── _config.yml ├── pom.xml └── src ├── main ├── assemblies │ └── plugin.xml ├── java │ └── st │ │ └── malike │ │ └── elasticsearch │ │ └── kafka │ │ └── watch │ │ ├── AddWatcherRestAction.java │ │ ├── ElasticKafkaWatchPlugin.java │ │ ├── RemoveWatcherRestAction.java │ │ ├── SearchWatchersRestAction.java │ │ ├── ViewWatchersRestAction.java │ │ ├── config │ │ └── PluginConfig.java │ │ ├── exception │ │ ├── InvalidCronExpression.java │ │ ├── ReportGenerationNotSupported.java │ │ ├── TemplateFileNotFoundException.java │ │ └── WatchCreationException.java │ │ ├── listener │ │ ├── CreateWatcherListener.java │ │ ├── DeleteWatcherListener.java │ │ ├── DocumentWatcherListener.java │ │ ├── IndexWatcherListener.java │ │ └── ViewWatchersListener.java │ │ ├── model │ │ ├── KafkaEvent.java │ │ └── KafkaWatch.java │ │ ├── service │ │ ├── EventIndexOpsTriggerService.java │ │ ├── KafkaEventGeneratorService.java │ │ ├── KafkaProducerService.java │ │ ├── KafkaWatchService.java │ │ ├── ReportService.java │ │ └── TimeTriggerService.java │ │ └── util │ │ ├── Enums.java │ │ └── JSONResponse.java └── resources │ ├── plugin-descriptor.properties │ └── plugin-security.policy └── test ├── java └── st │ └── malike │ └── elasticsearch │ └── kafka │ └── watch │ ├── ElasticKafkaWatchPluginTest.java │ └── service │ ├── EventIndexOpsTriggerServiceTest.java │ ├── KafkaEventGeneratorServiceTest.java │ ├── KafkaProducerServiceTest.java │ ├── KafkaWatchServiceTest.java │ ├── ReportServiceTest.java │ └── TimeTriggerServiceTest.java └── resources └── log4j.xml /.travis.yml: -------------------------------------------------------------------------------- 1 | language: java 2 | jdk: 3 | - oraclejdk8 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Build Status](https://travis-ci.org/malike/elasticsearch-kafka-watch.svg?branch=master)](https://travis-ci.org/malike/elasticsearch-kafka-watch) 2 | 3 | ## Basic Overview 4 | 5 | This project is a custom watcher for Elasticsearch which works with Apache Kafka by reacting to event in Elasticserch and writing them to Apache Kafka. 6 | 7 | Supports **2** types of triggers. 8 | 9 | #### 1. Time Based Triggers 10 | 11 | This trigger uses *crons* to trigger when an event should be pushed to Apache Kafka based on the watch configuration. 12 |
13 | 14 | #### 2. Event Based Triggers 15 | 16 | This trigger relies on the _IndexListeners_ and _DeleteListeners_. Once data is either created or deleted it triggers all watchers 17 | that meet criteria and pushes the data to Apache Kafka. 18 | 19 |
20 | 21 | ## Install 22 | 23 | 24 | #### Configuration 25 | 26 | Plugin requires a configuration file to know how to connect to Apache Kafka and 27 | also how it would use [elasticsearch report engine](https://malike.github.io/elasticsearch-report-engine) to generate 28 | reports. The file _elasticsearch-kafka-watch.yml_ should be place in _/path/to/elasticsearch/config_ 29 | folder to be picked by plugin. 30 | 31 | [Sample Config File](https://github.com/malike/elasticsearch-kafka-watch/blob/master/SampleConfig/elastcsearch-kafka-watch.yml) 32 | 33 | #### Installation 34 | ``sudo bin/elasticsearch-plugin install [plugin_name] `` 35 | 36 |
37 | 38 | 39 | ## Setup And Requirements 40 | 41 | 42 | ## Usage 43 | 44 | #### 1. Time Based 45 | Create a custom watch with its cron. Events would be generated using the cron. 46 | This is written into Apache Kafka. Any worker/consumer listening on Apache Kafka would react to the event. 47 | 48 | For sending SMS or Email alerts based on events written in Apache Kafka check out [go kafka alert](https://malike.github.io/go-kafka-alert). 49 | 50 | Creating a custom watch for a time based cron expects the following parameters: 51 | 52 | a.
53 | b.
54 | c.
55 | 56 |
57 | 58 | #### 2. Event Based Triggers 59 | 60 | Create a custom watch with and elasticsearch index and query. Once data is written or deleted from the index, it triggers the custom watch to evaluate query 61 | to check if there'll be a _hit_ greater than *0*. 62 | Once this is positive an event is written to Apache Kafka for consumers/workers listening to react. 63 | 64 | For sending SMS or Email alerts based on events written in Apache Kafka check out [go kafka alert](https://malike.github.io/go-kafka-alert). 65 | 66 | Creating a custom watch for a time based cron expects the following parameters: 67 | 68 | a.
69 | b.
70 | c.
71 | 72 | 73 |
74 | 75 | #### 3. Report Scheduling 76 | 77 | This plugin also works with 2 other plugins to schedule reports using elasticsearch as datasource. 78 | 79 | [elasticsearch report engine](https://malike.github.io/elasticsearch-report-engine) and [go kafka alert](https://malike.github.io/go-kafka-alert). The former generates PDF,CSV and HTML reports from elasticsearch using queries. 80 | The later sends the reports as email. PDF and CSV reports can be sent as attachments whiles HTML reports can be sent the email body. 81 | 82 | Reports can be sent using event based triggers or time based triggers. 83 | 84 | 85 | Creating a custom watch for a time based cron expects the following parameters: 86 | 87 | a.
88 | b.
89 | c.
90 | 91 | 92 | ## Supported 93 | 94 | Elasticsearch versions supported by this plugin include : 95 | 96 | | Version | - | 97 | | --------------------- | -------- | 98 | | [Elassticsearch 5.4](https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.zip) | Testing (dev still in progress | 99 | | [Apache Kafka 0.11.0.0](https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz) | Testing (dev still in progress | 100 | 101 | 102 |

 

103 | 104 | ## Benchmark Test 105 | 106 | Measurement on how fast a trigger is sent to Apache Kafka after indexing and deleting data on Elasticsearch. 107 | 108 | System Spec : 109 | 110 |

 

111 | 112 | 113 | 114 | | Number of Events | Type | Trigger Active | Result | 115 | | --------------------- | -------- | -------- | -------- | 116 | | 1 | Indexed| Yes | - | 117 | | 200 | 100 _Indexed_,100 _Deleted_| Yes | - | 118 | | 2,000 | 1,000 _Indexed_, 1,000 _Deleted_| Yes | - | 119 | | 2,000 | 1,000 _Indexed_, 1,000 _Deleted_| No | - | 120 | | 2,000,000 | 1,000,000 _Indexed_, 1,000,000 _Deleted_| Yes | - | 121 | 122 | 123 | 124 | 125 | 126 | 127 |

 

128 | 129 | ## Download 130 | 131 | | Elasticsearch Version | Comments | 132 | | --------------------- | -------- | 133 | | [5.4](https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.zip) | [-]() | 134 | 135 |

 

136 | 137 | ## Installation 138 | 139 | Add the following configuration to your _elasticsearch-kafka-watch.yml_ file located in _/path/to/elasticsearch/config_ 140 | 141 | 142 |

 

143 | 144 | ## Contribute 145 | 146 | Contributions are always welcome! 147 | Please read the [contribution guidelines](CONTRIBUTING.md) first. 148 | 149 | ## Code of Conduct 150 | 151 | Please read [this](CODE_OF_CONDUCT.md). 152 | 153 | ## License 154 | 155 | [GNU General Public License v3.0](https://github.com/malike/elasticsearch-kafka-watch/blob/master/LICENSE) 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-architect -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | elasticsearch-kafka-watch 8 | st.malike.elasticsearch.kafka.watch 9 | 5.4.1-SNAPSHOT 10 | Plugin: ElasticSearch Kafka Watch 11 | Push data into Apache Kafka if provided there's a hit 12 | 13 | 14 | 15 | org.elasticsearch.distribution.zip 16 | false 17 | 5.4.1 18 | 2.8.2 19 | 20 | 21 | 22 | 23 | maven 24 | mvn 25 | http://central.maven.org/maven2/ 26 | 27 | 28 | 29 | 30 | 31 | 32 | maven 33 | http://central.maven.org/maven2/ 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | org.mockito 42 | mockito-core 43 | 1.9.0 44 | test 45 | 46 | 47 | 48 | org.hamcrest 49 | hamcrest-core 50 | 1.3 51 | test 52 | 53 | 54 | 55 | com.jayway.restassured 56 | rest-assured 57 | 2.9.0 58 | test 59 | 60 | 61 | 62 | org.mockito 63 | mockito-all 64 | 1.9.5 65 | test 66 | 67 | 68 | 69 | org.elasticsearch 70 | elasticsearch 71 | ${elasticsearch.version} 72 | provided 73 | 74 | 75 | org.apache.logging.log4j 76 | log4j-api 77 | ${log4j.version} 78 | provided 79 | 80 | 81 | 82 | log4j 83 | log4j 84 | 1.2.17 85 | 86 | 87 | 88 | 89 | org.codelibs 90 | elasticsearch-cluster-runner 91 | 5.4.1.0 92 | test 93 | 94 | 95 | junit 96 | junit 97 | 4.12 98 | test 99 | 100 | 101 | 102 | com.google.code.gson 103 | gson 104 | 2.8.1 105 | 106 | 107 | commons-io 108 | commons-io 109 | 1.3.2 110 | 111 | 112 | commons-beanutils 113 | commons-beanutils-core 114 | 1.8.3 115 | 116 | 117 | commons-lang 118 | commons-lang 119 | 2.6 120 | jar 121 | 122 | 123 | org.apache.httpcomponents 124 | httpclient 125 | 4.5.3 126 | 127 | 128 | org.quartz-scheduler 129 | quartz 130 | 2.2.1 131 | 132 | 133 | org.apache.kafka 134 | kafka-clients 135 | 0.9.0.1 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | src/main/resources 146 | false 147 | 148 | *.properties 149 | 150 | 151 | 152 | 153 | 154 | org.apache.maven.plugins 155 | maven-assembly-plugin 156 | 2.6 157 | 158 | false 159 | ${project.build.directory}/releases/ 160 | 161 | ${basedir}/src/main/assemblies/plugin.xml 162 | 163 | 164 | 165 | 166 | package 167 | 168 | single 169 | 170 | 171 | 172 | 173 | 174 | org.apache.maven.plugins 175 | maven-compiler-plugin 176 | 3.3 177 | 178 | 1.8 179 | 1.8 180 | 181 | 182 | 183 | 184 | org.apache.maven.plugins 185 | maven-surefire-plugin 186 | 2.9 187 | 188 | false 189 | 190 | 191 | 192 | com.carrotsearch.randomizedtesting 193 | junit4-maven-plugin 194 | 2.3.3 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | unit-tests 209 | test 210 | 211 | junit4 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | -------------------------------------------------------------------------------- /src/main/assemblies/plugin.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | plugin 4 | 5 | zip 6 | 7 | false 8 | 9 | 10 | ${project.basedir}/src/main/resources/plugin-descriptor.properties 11 | elasticsearch 12 | true 13 | 14 | 15 | 16 | 17 | elasticsearch 18 | true 19 | true 20 | 21 | 22 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/AddWatcherRestAction.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch; 2 | 3 | import com.google.gson.Gson; 4 | import org.apache.commons.lang.RandomStringUtils; 5 | import org.apache.log4j.Logger; 6 | import org.elasticsearch.action.DocWriteRequest; 7 | import org.elasticsearch.action.index.IndexRequestBuilder; 8 | import org.elasticsearch.client.node.NodeClient; 9 | import org.elasticsearch.common.inject.Inject; 10 | import org.elasticsearch.common.settings.Settings; 11 | import org.elasticsearch.common.xcontent.XContentBuilder; 12 | import org.elasticsearch.common.xcontent.XContentHelper; 13 | import org.elasticsearch.rest.*; 14 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 15 | import st.malike.elasticsearch.kafka.watch.listener.CreateWatcherListener; 16 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 17 | import st.malike.elasticsearch.kafka.watch.util.Enums; 18 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 19 | 20 | import java.io.IOException; 21 | import java.util.Date; 22 | import java.util.LinkedList; 23 | import java.util.List; 24 | import java.util.Map; 25 | 26 | import static org.elasticsearch.rest.RestRequest.Method.POST; 27 | 28 | /** 29 | * @author malike_st 30 | */ 31 | public class AddWatcherRestAction extends BaseRestHandler { 32 | 33 | private static Logger log = Logger.getLogger(AddWatcherRestAction.class); 34 | private static PluginConfig pluginConfig; 35 | 36 | @Inject 37 | public AddWatcherRestAction(Settings settings, RestController controller) { 38 | super(settings); 39 | pluginConfig = new PluginConfig(); 40 | controller.registerHandler(POST, "/_newkafkawatch", this); 41 | } 42 | 43 | @SuppressWarnings("unchecked") 44 | @Override 45 | protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException { 46 | JSONResponse message = new JSONResponse(); 47 | IndexRequestBuilder prepareIndex = client.prepareIndex(pluginConfig.getKafkaWatchElasticsearchIndex(), 48 | pluginConfig.getKafkaWatchElasticsearchType()); 49 | KafkaWatch kafkaWatch = new KafkaWatch(); 50 | if (restRequest.content().length() > 0) { 51 | Map map = XContentHelper.convertToMap(restRequest.content(), false, null).v2(); 52 | if (!map.isEmpty()) { 53 | if (map.containsKey("cron")) { 54 | kafkaWatch.setCron((String) map.get("cron")); 55 | } 56 | if (map.containsKey("channel")) { 57 | List channels = (map.get("channel") instanceof List) ? (List) map.get("channel") 58 | : new LinkedList<>(); 59 | if (map.get("channel") instanceof String) { 60 | channels.add((String) map.get("channel")); 61 | } 62 | kafkaWatch.setChannel(channels); 63 | } 64 | if (map.containsKey("description")) { 65 | kafkaWatch.setDescription((String) map.get("description")); 66 | } 67 | if (map.containsKey("eventType")) { 68 | kafkaWatch.setEventType((String) map.get("eventType")); 69 | } 70 | if (map.containsKey("expectedHit")) { 71 | kafkaWatch.setExpectedHit(Long.parseLong((String) map.get("expectedHit"))); 72 | } 73 | if (map.containsKey("generateReport")) { 74 | kafkaWatch.setGenerateReport((Boolean) map.get("generateReport")); 75 | } 76 | if (map.containsKey("indexName")) { 77 | kafkaWatch.setIndexName((String) map.get("indexName")); 78 | } 79 | if (map.containsKey("query")) { 80 | kafkaWatch.setIndexOpsQuery((String) map.get("query")); 81 | } 82 | if (map.containsKey("reportTemplatePath")) { 83 | kafkaWatch.setReportTemplatePath((String) map.get("reportTemplatePath")); 84 | } 85 | if (map.containsKey("miscData")) { 86 | kafkaWatch.setMiscData((Map) map.get("miscData")); 87 | } 88 | if (map.containsKey("recipient")) { 89 | List recipients = (map.get("recipient") instanceof List) ? (List) map.get("recipient") 90 | : new LinkedList<>(); 91 | if (map.get("recipient") instanceof String) { 92 | recipients.add((String) map.get("recipient")); 93 | } 94 | kafkaWatch.setRecipient(recipients); 95 | } 96 | if (map.containsKey("subject")) { 97 | kafkaWatch.setSubject((String) map.get("subject")); 98 | } 99 | if (map.containsKey("querySymbol")) { 100 | try { 101 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.valueOf( 102 | ((String) map.get("querySymbol")).toUpperCase() 103 | )); 104 | } catch (Exception e) { 105 | 106 | } 107 | } 108 | if (map.containsKey("trigger")) { 109 | try { 110 | kafkaWatch.setTriggerType(Enums.TriggerType.valueOf( 111 | ((String) map.get("trigger")).toUpperCase() 112 | )); 113 | } catch (Exception e) { 114 | 115 | } 116 | } 117 | kafkaWatch.setId(RandomStringUtils.randomAlphabetic(5)); 118 | kafkaWatch.setDateCreated(new Date()); 119 | } else { 120 | return channel -> { 121 | message.setStatus(false); 122 | message.setCount(0L); 123 | message.setMessage(Enums.JSONResponseMessage.MISSING_PARAM.toString()); 124 | XContentBuilder builder = channel.newBuilder(); 125 | builder.startObject(); 126 | message.toXContent(builder, restRequest); 127 | builder.endObject(); 128 | channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 129 | }; 130 | } 131 | } 132 | if (kafkaWatch.getTriggerType() == null || kafkaWatch.getIndexName() == null) { 133 | return channel -> { 134 | message.setStatus(false); 135 | message.setCount(0L); 136 | message.setData("Trigger Type or Index not specified"); 137 | message.setMessage(Enums.JSONResponseMessage.INVALID_DATA.toString()); 138 | XContentBuilder builder = channel.newBuilder(); 139 | builder.startObject(); 140 | message.toXContent(builder, restRequest); 141 | builder.endObject(); 142 | channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 143 | }; 144 | } 145 | if (kafkaWatch.isGenerateReport() && kafkaWatch.getQuerySymbol() == null 146 | && kafkaWatch.getReportTemplatePath() == null) { 147 | return channel -> { 148 | message.setStatus(false); 149 | message.setCount(0L); 150 | message.setData("Watch not configured properly for report"); 151 | message.setMessage(Enums.JSONResponseMessage.NOT_CONFIGURED_FOR_REPORTS.toString()); 152 | XContentBuilder builder = channel.newBuilder(); 153 | builder.startObject(); 154 | message.toXContent(builder, restRequest); 155 | builder.endObject(); 156 | channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 157 | }; 158 | } 159 | Gson gson = new Gson(); 160 | prepareIndex.setOpType(DocWriteRequest.OpType.CREATE); 161 | prepareIndex.setId(kafkaWatch.getId()); 162 | Map m = gson.fromJson(gson.toJson(kafkaWatch), Map.class); 163 | prepareIndex.setSource(m); 164 | return channel -> prepareIndex.execute(new CreateWatcherListener(channel, restRequest, 165 | kafkaWatch, pluginConfig, settings)); 166 | } 167 | } 168 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/ElasticKafkaWatchPlugin.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver; 5 | import org.elasticsearch.cluster.node.DiscoveryNodes; 6 | import org.elasticsearch.common.settings.ClusterSettings; 7 | import org.elasticsearch.common.settings.IndexScopedSettings; 8 | import org.elasticsearch.common.settings.Settings; 9 | import org.elasticsearch.common.settings.SettingsFilter; 10 | import org.elasticsearch.index.IndexModule; 11 | import org.elasticsearch.plugins.ActionPlugin; 12 | import org.elasticsearch.plugins.Plugin; 13 | import org.elasticsearch.rest.RestController; 14 | import org.elasticsearch.rest.RestHandler; 15 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 16 | import st.malike.elasticsearch.kafka.watch.listener.DocumentWatcherListener; 17 | import st.malike.elasticsearch.kafka.watch.listener.IndexWatcherListener; 18 | import st.malike.elasticsearch.kafka.watch.service.KafkaProducerService; 19 | import st.malike.elasticsearch.kafka.watch.service.TimeTriggerService; 20 | 21 | import java.util.Arrays; 22 | import java.util.List; 23 | import java.util.function.Supplier; 24 | 25 | /** 26 | * @author malike_st 27 | */ 28 | public class ElasticKafkaWatchPlugin extends Plugin implements ActionPlugin { 29 | 30 | private static Logger log = Logger.getLogger(ElasticKafkaWatchPlugin.class); 31 | private final TimeTriggerService timeTriggerService; 32 | private final PluginConfig pluginConfig; 33 | private final KafkaProducerService kafkaProducerService; 34 | 35 | 36 | public ElasticKafkaWatchPlugin() throws Exception { 37 | this.pluginConfig = new PluginConfig(); 38 | this.kafkaProducerService = new KafkaProducerService(this.pluginConfig); 39 | this.timeTriggerService = new TimeTriggerService(pluginConfig, kafkaProducerService); 40 | } 41 | 42 | @Override 43 | public List getRestHandlers(Settings settings, 44 | RestController restController, ClusterSettings clusterSettings, 45 | IndexScopedSettings indexScopedSettings, 46 | SettingsFilter settingsFilter, 47 | IndexNameExpressionResolver indexNameExpressionResolver, 48 | Supplier nodesInCluster) { 49 | try { 50 | timeTriggerService.schedule(); 51 | } catch (Exception e) { 52 | log.error("Error starting scheduler. Error " + e.getLocalizedMessage()); 53 | } 54 | return Arrays.asList(new AddWatcherRestAction(settings, restController), 55 | new RemoveWatcherRestAction(settings, restController), 56 | new ViewWatchersRestAction(settings, restController), 57 | new SearchWatchersRestAction(settings, restController)); 58 | } 59 | 60 | @Override 61 | public void onIndexModule(IndexModule indexModule) { 62 | indexModule.addIndexEventListener(new IndexWatcherListener()); 63 | indexModule.addIndexOperationListener(new DocumentWatcherListener(pluginConfig, kafkaProducerService)); 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/RemoveWatcherRestAction.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.action.delete.DeleteRequestBuilder; 5 | import org.elasticsearch.client.node.NodeClient; 6 | import org.elasticsearch.common.inject.Inject; 7 | import org.elasticsearch.common.settings.Settings; 8 | import org.elasticsearch.common.xcontent.XContentBuilder; 9 | import org.elasticsearch.common.xcontent.XContentHelper; 10 | import org.elasticsearch.rest.*; 11 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 12 | import st.malike.elasticsearch.kafka.watch.listener.DeleteWatcherListener; 13 | import st.malike.elasticsearch.kafka.watch.util.Enums; 14 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 15 | 16 | import java.io.IOException; 17 | import java.util.Map; 18 | 19 | import static org.elasticsearch.rest.RestRequest.Method.POST; 20 | 21 | /** 22 | * @author malike_st 23 | */ 24 | public class RemoveWatcherRestAction extends BaseRestHandler { 25 | 26 | private static Logger log = Logger.getLogger(RemoveWatcherRestAction.class); 27 | private static PluginConfig pluginConfig; 28 | 29 | @Inject 30 | public RemoveWatcherRestAction(Settings settings, RestController controller) { 31 | super(settings); 32 | pluginConfig = new PluginConfig(); 33 | controller.registerHandler(POST, "/_removekafkawatch", this); 34 | } 35 | 36 | @SuppressWarnings("unchecked") 37 | @Override 38 | protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException { 39 | JSONResponse message = new JSONResponse(); 40 | String id = null; 41 | if (restRequest.content().length() > 0) { 42 | Map map = XContentHelper.convertToMap(restRequest.content(), false, null).v2(); 43 | if (!map.isEmpty()) { 44 | if (map.containsKey("id")) { 45 | id = (String) map.get("id"); 46 | } 47 | } else { 48 | return channel -> { 49 | message.setStatus(false); 50 | message.setCount(0L); 51 | message.setData("Required Watcher ID not found"); 52 | message.setMessage(Enums.JSONResponseMessage.MISSING_PARAM.toString()); 53 | XContentBuilder builder = channel.newBuilder(); 54 | builder.startObject(); 55 | message.toXContent(builder, restRequest); 56 | builder.endObject(); 57 | channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 58 | }; 59 | } 60 | } 61 | DeleteRequestBuilder prepareDelete = client.prepareDelete(pluginConfig.getKafkaWatchElasticsearchIndex(), 62 | pluginConfig.getKafkaWatchElasticsearchType(), id); 63 | return channel -> prepareDelete.execute(new DeleteWatcherListener(channel, restRequest)); 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/SearchWatchersRestAction.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.action.search.SearchRequestBuilder; 5 | import org.elasticsearch.action.search.SearchType; 6 | import org.elasticsearch.client.node.NodeClient; 7 | import org.elasticsearch.common.inject.Inject; 8 | import org.elasticsearch.common.settings.Settings; 9 | import org.elasticsearch.common.xcontent.XContentBuilder; 10 | import org.elasticsearch.common.xcontent.XContentHelper; 11 | import org.elasticsearch.index.query.QueryBuilders; 12 | import org.elasticsearch.rest.*; 13 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 14 | import st.malike.elasticsearch.kafka.watch.listener.ViewWatchersListener; 15 | import st.malike.elasticsearch.kafka.watch.util.Enums; 16 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 17 | 18 | import java.io.IOException; 19 | import java.util.Map; 20 | 21 | import static org.elasticsearch.rest.RestRequest.Method.POST; 22 | 23 | /** 24 | * @author malike_st 25 | */ 26 | public class SearchWatchersRestAction extends BaseRestHandler { 27 | 28 | private static Logger log = Logger.getLogger(SearchWatchersRestAction.class); 29 | private static PluginConfig pluginConfig; 30 | 31 | @Inject 32 | public SearchWatchersRestAction(Settings settings, RestController controller) { 33 | super(settings); 34 | pluginConfig = new PluginConfig(); 35 | controller.registerHandler(POST, "/_searchkafkawatch", this); 36 | } 37 | 38 | @SuppressWarnings("unchecked") 39 | @Override 40 | protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException { 41 | JSONResponse message = new JSONResponse(); 42 | Integer from = 0; 43 | Integer size = 50; 44 | String query = null; 45 | if (restRequest.content().length() > 0) { 46 | Map map = XContentHelper.convertToMap(restRequest.content(), false, null).v2(); 47 | if (!map.isEmpty()) { 48 | if (map.containsKey("from")) { 49 | from = (Integer) map.get("from"); 50 | } 51 | if (map.containsKey("limit")) { 52 | size = (Integer) map.get("limit"); 53 | } 54 | if (map.containsKey("param")) { 55 | query = (String) map.get("param"); 56 | } 57 | } 58 | } 59 | if (query == null) { 60 | return channel -> { 61 | message.setStatus(false); 62 | message.setCount(0L); 63 | message.setMessage(Enums.JSONResponseMessage.MISSING_PARAM.toString()); 64 | XContentBuilder builder = channel.newBuilder(); 65 | builder.startObject(); 66 | message.toXContent(builder, restRequest); 67 | builder.endObject(); 68 | channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 69 | }; 70 | } 71 | SearchRequestBuilder prepareSearch = client.prepareSearch( 72 | pluginConfig.getKafkaWatchElasticsearchIndex()); 73 | prepareSearch.setFrom(from); 74 | prepareSearch.setSize(size); 75 | prepareSearch.setSearchType(SearchType.DFS_QUERY_THEN_FETCH); 76 | prepareSearch.setTypes(pluginConfig.getKafkaWatchElasticsearchType()); 77 | prepareSearch.setQuery(QueryBuilders.wrapperQuery(query)); 78 | return channel -> prepareSearch.execute(new ViewWatchersListener(channel, restRequest)); 79 | } 80 | 81 | } 82 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/ViewWatchersRestAction.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.action.search.SearchRequestBuilder; 5 | import org.elasticsearch.action.search.SearchType; 6 | import org.elasticsearch.client.node.NodeClient; 7 | import org.elasticsearch.common.inject.Inject; 8 | import org.elasticsearch.common.settings.Settings; 9 | import org.elasticsearch.common.xcontent.XContentHelper; 10 | import org.elasticsearch.index.query.QueryBuilders; 11 | import org.elasticsearch.rest.BaseRestHandler; 12 | import org.elasticsearch.rest.RestController; 13 | import org.elasticsearch.rest.RestRequest; 14 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 15 | import st.malike.elasticsearch.kafka.watch.listener.ViewWatchersListener; 16 | 17 | import java.io.IOException; 18 | import java.util.Map; 19 | 20 | import static org.elasticsearch.rest.RestRequest.Method.POST; 21 | 22 | /** 23 | * @author malike_st 24 | */ 25 | public class ViewWatchersRestAction extends BaseRestHandler { 26 | 27 | private static Logger log = Logger.getLogger(ViewWatchersRestAction.class); 28 | private static PluginConfig pluginConfig; 29 | 30 | @Inject 31 | public ViewWatchersRestAction(Settings settings, RestController controller) { 32 | super(settings); 33 | pluginConfig = new PluginConfig(); 34 | controller.registerHandler(POST, "/_listkafkawatch", this); 35 | } 36 | 37 | @SuppressWarnings("unchecked") 38 | @Override 39 | protected RestChannelConsumer prepareRequest(RestRequest restRequest, NodeClient client) throws IOException { 40 | //limit and offset 41 | Integer from = 0; 42 | Integer size = 50; 43 | if (restRequest.content().length() > 0) { 44 | Map map = XContentHelper.convertToMap(restRequest.content(), false, null).v2(); 45 | if (!map.isEmpty()) { 46 | if (map.containsKey("from")) { 47 | from = (Integer) map.get("from"); 48 | } 49 | if (map.containsKey("limit")) { 50 | size = (Integer) map.get("limit"); 51 | } 52 | } 53 | } 54 | SearchRequestBuilder prepareSearch = client.prepareSearch( 55 | pluginConfig.getKafkaWatchElasticsearchIndex()); 56 | prepareSearch.setFrom(from); 57 | prepareSearch.setSize(size); 58 | prepareSearch.setSearchType(SearchType.DFS_QUERY_THEN_FETCH); 59 | prepareSearch.setTypes(pluginConfig.getKafkaWatchElasticsearchType()); 60 | prepareSearch.setQuery(QueryBuilders.matchAllQuery()); 61 | return channel -> prepareSearch.execute(new ViewWatchersListener(channel, restRequest)); 62 | } 63 | 64 | 65 | } 66 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/config/PluginConfig.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.config; 2 | 3 | /** 4 | * @autor malike_st 5 | */ 6 | public class PluginConfig { 7 | 8 | private final String KAFKA_WATCH_ELASTICSEARCH_INDEX = "kafka.watch.elasticsearch.index"; 9 | private final String KAFKA_WATCH_ELASTICSEARCH_TYPE = "kafka.watch.elasticsearch.type"; 10 | private final String KAFKA_WATCH_BOOTSTRAP_SERVERS = "localhost:9092"; 11 | private final String KAFKA_WATCH_TOPIC = "kafka.watch.topic"; 12 | private final String KAFKA_WATCH_DISABLE = "kafka.watch.disable"; 13 | private final String REPORT_ENGINE_ENDPOINT = "report.engine.endpoint"; 14 | private final String REPORT_ENGINE_DISABLE = "report.engine.disable"; 15 | 16 | 17 | public String getKafkaWatchElasticsearchIndex() { 18 | 19 | return KAFKA_WATCH_ELASTICSEARCH_INDEX; 20 | } 21 | 22 | public String getKafkaWatchElasticsearchType() { 23 | return KAFKA_WATCH_ELASTICSEARCH_TYPE; 24 | } 25 | 26 | public String getKafkaWatchBootstrapServers() { 27 | return KAFKA_WATCH_BOOTSTRAP_SERVERS; 28 | } 29 | 30 | public String getKafkaWatchTopic() { 31 | return KAFKA_WATCH_TOPIC; 32 | } 33 | 34 | public Boolean getKafkaWatchDisable() { 35 | return KAFKA_WATCH_DISABLE.toLowerCase().equals("false"); 36 | } 37 | 38 | public Boolean getReportEngineDisable() { 39 | return REPORT_ENGINE_DISABLE.toLowerCase().equals("false"); 40 | } 41 | 42 | public String getReportEngineEndpoint() { 43 | 44 | return REPORT_ENGINE_ENDPOINT; 45 | } 46 | 47 | 48 | } 49 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/exception/InvalidCronExpression.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.exception; 2 | 3 | /** 4 | * @author malike_st 5 | */ 6 | public class InvalidCronExpression extends Exception { 7 | 8 | public InvalidCronExpression() { 9 | super(); 10 | } 11 | 12 | public InvalidCronExpression(String message) { 13 | super(message); 14 | } 15 | 16 | public InvalidCronExpression(String message, Throwable cause) { 17 | super(message, cause); 18 | } 19 | 20 | public InvalidCronExpression(Throwable cause) { 21 | super(cause); 22 | } 23 | 24 | public InvalidCronExpression(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) { 25 | super(message, cause, enableSuppression, writableStackTrace); 26 | } 27 | } 28 | 29 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/exception/ReportGenerationNotSupported.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.exception; 2 | 3 | /** 4 | * @autor malike_st 5 | */ 6 | public class ReportGenerationNotSupported extends Exception { 7 | 8 | public ReportGenerationNotSupported() { 9 | super(); 10 | } 11 | 12 | public ReportGenerationNotSupported(String message) { 13 | super(message); 14 | } 15 | 16 | public ReportGenerationNotSupported(String message, Throwable cause) { 17 | super(message, cause); 18 | } 19 | 20 | public ReportGenerationNotSupported(Throwable cause) { 21 | super(cause); 22 | } 23 | 24 | protected ReportGenerationNotSupported(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) { 25 | super(message, cause, enableSuppression, writableStackTrace); 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/exception/TemplateFileNotFoundException.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.exception; 2 | 3 | /** 4 | * @autor malike_st 5 | */ 6 | public class TemplateFileNotFoundException extends Exception { 7 | public TemplateFileNotFoundException() { 8 | super(); 9 | } 10 | 11 | public TemplateFileNotFoundException(String message) { 12 | super(message); 13 | } 14 | 15 | public TemplateFileNotFoundException(String message, Throwable cause) { 16 | super(message, cause); 17 | } 18 | 19 | public TemplateFileNotFoundException(Throwable cause) { 20 | super(cause); 21 | } 22 | 23 | public TemplateFileNotFoundException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) { 24 | super(message, cause, enableSuppression, writableStackTrace); 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/exception/WatchCreationException.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.exception; 2 | 3 | /** 4 | * @autor malike_st 5 | */ 6 | public class WatchCreationException extends Exception { 7 | 8 | public WatchCreationException() { 9 | super(); 10 | } 11 | 12 | public WatchCreationException(String message) { 13 | super(message); 14 | } 15 | 16 | public WatchCreationException(String message, Throwable cause) { 17 | super(message, cause); 18 | } 19 | 20 | public WatchCreationException(Throwable cause) { 21 | super(cause); 22 | } 23 | 24 | public WatchCreationException(String message, Throwable cause, boolean enableSuppression, boolean writableStackTrace) { 25 | super(message, cause, enableSuppression, writableStackTrace); 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/listener/CreateWatcherListener.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.listener; 2 | 3 | import com.google.gson.Gson; 4 | import org.apache.log4j.Logger; 5 | import org.elasticsearch.ElasticsearchException; 6 | import org.elasticsearch.action.ActionListener; 7 | import org.elasticsearch.action.index.IndexResponse; 8 | import org.elasticsearch.common.settings.Settings; 9 | import org.elasticsearch.common.xcontent.XContentBuilder; 10 | import org.elasticsearch.rest.BytesRestResponse; 11 | import org.elasticsearch.rest.RestChannel; 12 | import org.elasticsearch.rest.RestRequest; 13 | import org.elasticsearch.rest.RestStatus; 14 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 15 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 16 | import st.malike.elasticsearch.kafka.watch.service.KafkaProducerService; 17 | import st.malike.elasticsearch.kafka.watch.service.TimeTriggerService; 18 | import st.malike.elasticsearch.kafka.watch.util.Enums; 19 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 20 | 21 | import java.io.IOException; 22 | 23 | /** 24 | * @autor malike_st 25 | */ 26 | public class CreateWatcherListener implements ActionListener { 27 | 28 | private static Logger log = Logger.getLogger(CreateWatcherListener.class); 29 | private final RestChannel restChannel; 30 | private final KafkaWatch kafkaWatch; 31 | private final RestRequest restRequest; 32 | private TimeTriggerService timeTriggerService; 33 | 34 | public CreateWatcherListener(RestChannel restChannel, RestRequest restRequest, 35 | KafkaWatch kafkaWatch, PluginConfig pluginConfig, Settings settings) throws Exception { 36 | this.restChannel = restChannel; 37 | this.restRequest = restRequest; 38 | this.kafkaWatch = kafkaWatch; 39 | this.timeTriggerService = new TimeTriggerService(pluginConfig, new KafkaProducerService(pluginConfig)); 40 | } 41 | 42 | @Override 43 | public void onResponse(IndexResponse indexResponse) { 44 | JSONResponse message = new JSONResponse(); 45 | try { 46 | XContentBuilder builder = restChannel.newBuilder(); 47 | if (indexResponse.getResult().getLowercase().equals("created")) { 48 | message.setStatus(true); 49 | message.setCount(1L); 50 | message.setData(new Gson().toJson(kafkaWatch)); 51 | message.setMessage(Enums.JSONResponseMessage.SUCCESS.toString()); 52 | builder.startObject(); 53 | message.toXContent(builder, restRequest); 54 | builder.endObject(); 55 | 56 | if (!kafkaWatch.getTriggerType().equals(Enums.TriggerType.INDEX_OPS)) { 57 | timeTriggerService.addJob(kafkaWatch); 58 | } 59 | } else { 60 | message.setStatus(false); 61 | message.setCount(0L); 62 | message.setData(indexResponse); 63 | message.setMessage(Enums.JSONResponseMessage.ERROR.toString()); 64 | builder.startObject(); 65 | message.toXContent(builder, restRequest); 66 | builder.endObject(); 67 | } 68 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 69 | } catch (Exception e) { 70 | try { 71 | XContentBuilder builder = restChannel.newBuilder(); 72 | builder.startObject(); 73 | message.setData(e.getLocalizedMessage()); 74 | message.toXContent(builder, restRequest); 75 | message.setStatus(false); 76 | builder.endObject(); 77 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 78 | } catch (IOException ex) { 79 | onFailure(e); 80 | } 81 | } 82 | } 83 | 84 | @Override 85 | public void onFailure(Exception e) { 86 | log.error("Error creating new watcher " + e.getLocalizedMessage()); 87 | throw new ElasticsearchException("Exception :", e.getLocalizedMessage()); 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/listener/DeleteWatcherListener.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.listener; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.ElasticsearchException; 5 | import org.elasticsearch.action.ActionListener; 6 | import org.elasticsearch.action.delete.DeleteResponse; 7 | import org.elasticsearch.common.xcontent.XContentBuilder; 8 | import org.elasticsearch.rest.BytesRestResponse; 9 | import org.elasticsearch.rest.RestChannel; 10 | import org.elasticsearch.rest.RestRequest; 11 | import org.elasticsearch.rest.RestStatus; 12 | import st.malike.elasticsearch.kafka.watch.util.Enums; 13 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 14 | 15 | import java.io.IOException; 16 | 17 | 18 | /** 19 | * @autor malike_st 20 | */ 21 | public class DeleteWatcherListener implements ActionListener { 22 | 23 | private static Logger log = Logger.getLogger(DeleteWatcherListener.class); 24 | 25 | private final RestChannel restChannel; 26 | private final RestRequest restRequest; 27 | 28 | public DeleteWatcherListener(RestChannel restChannel, RestRequest restRequest) { 29 | this.restChannel = restChannel; 30 | this.restRequest = restRequest; 31 | } 32 | 33 | @Override 34 | public void onResponse(DeleteResponse deleteResponse) { 35 | JSONResponse message = new JSONResponse(); 36 | try { 37 | XContentBuilder builder = restChannel.newBuilder(); 38 | if (deleteResponse.getResult().getLowercase().equals("deleted")) { 39 | message.setStatus(true); 40 | message.setCount(1L); 41 | message.setMessage(Enums.JSONResponseMessage.SUCCESS.toString()); 42 | builder.startObject(); 43 | message.toXContent(builder, restRequest); 44 | builder.endObject(); 45 | } else { 46 | message.setStatus(false); 47 | message.setCount(0L); 48 | message.setData(deleteResponse); 49 | if (deleteResponse.getResult().getLowercase().equals("not_found")) { 50 | message.setMessage(Enums.JSONResponseMessage.DATA_NOT_FOUND.toString()); 51 | } else { 52 | message.setMessage(Enums.JSONResponseMessage.ERROR.toString()); 53 | } 54 | builder.startObject(); 55 | message.toXContent(builder, restRequest); 56 | builder.endObject(); 57 | } 58 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 59 | } catch (IOException e) { 60 | try { 61 | XContentBuilder builder = restChannel.newBuilder(); 62 | builder.startObject(); 63 | message.setStatus(false); 64 | message.setData(e.getLocalizedMessage()); 65 | message.toXContent(builder, restRequest); 66 | builder.endObject(); 67 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 68 | } catch (IOException ex) { 69 | throw new ElasticsearchException("Exception :", e.getLocalizedMessage()); 70 | } 71 | } 72 | } 73 | 74 | @Override 75 | public void onFailure(Exception e) { 76 | log.error("Error deleting watcher " + e.getLocalizedMessage()); 77 | throw new ElasticsearchException("Exception :", e.getLocalizedMessage()); 78 | } 79 | } 80 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/listener/DocumentWatcherListener.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.listener; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.index.engine.Engine; 5 | import org.elasticsearch.index.shard.IndexingOperationListener; 6 | import org.elasticsearch.index.shard.ShardId; 7 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 8 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 9 | import st.malike.elasticsearch.kafka.watch.service.EventIndexOpsTriggerService; 10 | import st.malike.elasticsearch.kafka.watch.service.KafkaEventGeneratorService; 11 | import st.malike.elasticsearch.kafka.watch.service.KafkaProducerService; 12 | import st.malike.elasticsearch.kafka.watch.service.KafkaWatchService; 13 | 14 | import java.util.List; 15 | 16 | /** 17 | * @author malike_st 18 | */ 19 | public class DocumentWatcherListener implements IndexingOperationListener { 20 | 21 | private static Logger log = Logger.getLogger(DocumentWatcherListener.class); 22 | private final PluginConfig pluginConfig; 23 | private final KafkaProducerService kafkaProducerService; 24 | EventIndexOpsTriggerService eventIndexOpsTriggerService = new EventIndexOpsTriggerService(); 25 | KafkaWatchService kafkaWatchService = new KafkaWatchService(); 26 | KafkaEventGeneratorService kafkaEventGeneratorService = new KafkaEventGeneratorService(); 27 | public DocumentWatcherListener(PluginConfig pluginConfig, KafkaProducerService kafkaProducerService) { 28 | this.pluginConfig = pluginConfig; 29 | this.kafkaProducerService = kafkaProducerService; 30 | } 31 | 32 | @Override 33 | public void postIndex(ShardId shardId, Engine.Index index, Engine.IndexResult result) { 34 | if ((!shardId.getIndexName().equals(pluginConfig.getKafkaWatchElasticsearchIndex()) 35 | && (!pluginConfig.getKafkaWatchDisable()))) { 36 | 37 | List kafkaWatchList = kafkaWatchService.searchWatchByIndex(shardId.getIndexName()); 38 | if (kafkaWatchList != null && kafkaWatchList.isEmpty()) { 39 | for (KafkaWatch kafkaWatch : kafkaWatchList) { 40 | if (eventIndexOpsTriggerService.evaluateRuleForEvent(shardId.getIndexName(), index, result, kafkaWatch)) { 41 | kafkaProducerService.send(kafkaEventGeneratorService.generate(kafkaWatch)); 42 | log.info("New trigger : Document Created " + index.source().utf8ToString()); 43 | } else { 44 | log.info("Trigger did not meet requirements to be pushed to Apache Kafka " + index.source().utf8ToString()); 45 | } 46 | } 47 | } 48 | } 49 | } 50 | 51 | 52 | @Override 53 | public void postDelete(ShardId shardId, Engine.Delete delete, Engine.DeleteResult result) { 54 | if ((!shardId.getIndexName().equals(pluginConfig.getKafkaWatchElasticsearchIndex()) 55 | && (!pluginConfig.getKafkaWatchDisable()))) { 56 | 57 | List kafkaWatchList = kafkaWatchService.searchWatchByIndex(shardId.getIndexName()); 58 | if (kafkaWatchList != null && kafkaWatchList.isEmpty()) { 59 | for (KafkaWatch kafkaWatch : kafkaWatchList) { 60 | if (eventIndexOpsTriggerService.evaluateRuleForEvent(shardId.getIndexName(), delete, result, kafkaWatch)) { 61 | kafkaProducerService.send(kafkaEventGeneratorService.generate(kafkaWatch)); 62 | log.info("New trigger : Document deleted " + delete.id()); 63 | } else { 64 | log.info("Trigger did not meet requirements to be pushed to Apache Kafka" + delete.id() 65 | + " Index Name : " + shardId.getIndexName()); 66 | } 67 | } 68 | } 69 | } 70 | } 71 | 72 | 73 | } 74 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/listener/IndexWatcherListener.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.listener; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.index.Index; 5 | import org.elasticsearch.index.IndexService; 6 | import org.elasticsearch.index.IndexSettings; 7 | import org.elasticsearch.index.shard.IndexEventListener; 8 | import org.elasticsearch.indices.cluster.IndicesClusterStateService; 9 | 10 | /** 11 | * @author malike_st 12 | */ 13 | public class IndexWatcherListener implements IndexEventListener { 14 | 15 | private static Logger log = Logger.getLogger(DocumentWatcherListener.class); 16 | 17 | @Override 18 | public void afterIndexCreated(IndexService indexService) { 19 | log.info("New trigger : Index Created " + indexService.getMetaData().getIndex().getName()); 20 | } 21 | 22 | @Override 23 | public void afterIndexRemoved(Index index, IndexSettings indexSettings, 24 | IndicesClusterStateService.AllocatedIndices.IndexRemovalReason reason) { 25 | log.info("New trigger : Index deleted " + index.getName()); 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/listener/ViewWatchersListener.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.listener; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.ElasticsearchException; 5 | import org.elasticsearch.action.ActionListener; 6 | import org.elasticsearch.action.search.SearchResponse; 7 | import org.elasticsearch.common.xcontent.XContentBuilder; 8 | import org.elasticsearch.index.IndexNotFoundException; 9 | import org.elasticsearch.rest.BytesRestResponse; 10 | import org.elasticsearch.rest.RestChannel; 11 | import org.elasticsearch.rest.RestRequest; 12 | import org.elasticsearch.rest.RestStatus; 13 | import org.elasticsearch.search.SearchHit; 14 | import org.elasticsearch.search.SearchHits; 15 | import st.malike.elasticsearch.kafka.watch.util.Enums; 16 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 17 | 18 | import java.io.IOException; 19 | import java.util.LinkedList; 20 | import java.util.List; 21 | import java.util.Map; 22 | 23 | /** 24 | * @autor malike_st 25 | */ 26 | public class ViewWatchersListener implements ActionListener { 27 | 28 | private static Logger log = Logger.getLogger(ViewWatchersListener.class); 29 | 30 | private final RestChannel restChannel; 31 | private final RestRequest restRequest; 32 | 33 | public ViewWatchersListener(RestChannel restChannel, RestRequest restRequest) { 34 | this.restChannel = restChannel; 35 | this.restRequest = restRequest; 36 | } 37 | 38 | @Override 39 | public void onResponse(SearchResponse searchResponse) { 40 | JSONResponse message = new JSONResponse(); 41 | try { 42 | XContentBuilder builder = restChannel.newBuilder(); 43 | List dataList = extractData(searchResponse); 44 | message.setStatus(true); 45 | message.setCount(Long.valueOf(searchResponse.getHits().getTotalHits())); 46 | message.setData(dataList); 47 | message.setMessage(Enums.JSONResponseMessage.SUCCESS.toString()); 48 | builder.startObject(); 49 | message.toXContent(builder, restRequest); 50 | builder.endObject(); 51 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 52 | } catch (IOException e) { 53 | try { 54 | XContentBuilder builder = restChannel.newBuilder(); 55 | builder.startObject(); 56 | message.setData(e.getLocalizedMessage()); 57 | message.toXContent(builder, restRequest); 58 | builder.endObject(); 59 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 60 | } catch (IOException ex) { 61 | throw new ElasticsearchException("Exception :", e.getLocalizedMessage()); 62 | } 63 | } 64 | } 65 | 66 | @Override 67 | public void onFailure(Exception e) { 68 | log.error("Error loading watchers " + e.getLocalizedMessage()); 69 | try { 70 | if (e instanceof IndexNotFoundException) { 71 | JSONResponse message = new JSONResponse(); 72 | XContentBuilder builder = restChannel.newBuilder(); 73 | builder.startObject(); 74 | message.setStatus(true); 75 | message.setCount(0L); 76 | message.setMessage(Enums.JSONResponseMessage.SUCCESS.toString()); 77 | message.toXContent(builder, restRequest); 78 | builder.endObject(); 79 | restChannel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); 80 | } 81 | } catch (Exception ex) { 82 | } 83 | throw new ElasticsearchException("Exception :", e.getLocalizedMessage()); 84 | } 85 | 86 | public List extractData(SearchResponse response) { 87 | List data = new LinkedList<>(); 88 | SearchHits hits = response.getHits(); 89 | try { 90 | for (SearchHit hit : hits) { 91 | Map sourceMap = hit.getSourceAsMap(); 92 | data.add(sourceMap); 93 | } 94 | } catch (Exception e) { 95 | } 96 | return data; 97 | } 98 | 99 | } 100 | 101 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/model/KafkaEvent.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.model; 2 | 3 | import java.util.Date; 4 | import java.util.List; 5 | import java.util.Map; 6 | 7 | /** 8 | * @author malike_st 9 | */ 10 | public class KafkaEvent { 11 | 12 | private String eventId; 13 | private String subject; 14 | private Map channel; 15 | private List recipient; 16 | private Map unmappedData; 17 | private String eventType; 18 | private String description; 19 | private Date dateCreated; 20 | 21 | public String getEventId() { 22 | return eventId; 23 | } 24 | 25 | public void setEventId(String eventId) { 26 | this.eventId = eventId; 27 | } 28 | 29 | public String getSubject() { 30 | return subject; 31 | } 32 | 33 | public void setSubject(String subject) { 34 | this.subject = subject; 35 | } 36 | 37 | public Map getChannel() { 38 | return channel; 39 | } 40 | 41 | public void setChannel(Map channel) { 42 | this.channel = channel; 43 | } 44 | 45 | public List getRecipient() { 46 | return recipient; 47 | } 48 | 49 | public void setRecipient(List recipient) { 50 | this.recipient = recipient; 51 | } 52 | 53 | public Map getUnmappedData() { 54 | return unmappedData; 55 | } 56 | 57 | public void setUnmappedData(Map unmappedData) { 58 | this.unmappedData = unmappedData; 59 | } 60 | 61 | public String getEventType() { 62 | return eventType; 63 | } 64 | 65 | public void setEventType(String eventType) { 66 | this.eventType = eventType; 67 | } 68 | 69 | public String getDescription() { 70 | return description; 71 | } 72 | 73 | public void setDescription(String description) { 74 | this.description = description; 75 | } 76 | 77 | public Date getDateCreated() { 78 | return dateCreated; 79 | } 80 | 81 | public void setDateCreated(Date dateCreated) { 82 | this.dateCreated = dateCreated; 83 | } 84 | } 85 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/model/KafkaWatch.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.model; 2 | 3 | import org.elasticsearch.common.xcontent.ToXContent; 4 | import org.elasticsearch.common.xcontent.XContentBuilder; 5 | import st.malike.elasticsearch.kafka.watch.util.Enums; 6 | 7 | import java.io.IOException; 8 | import java.util.Date; 9 | import java.util.List; 10 | import java.util.Map; 11 | 12 | /** 13 | * @author malike_st 14 | */ 15 | public class KafkaWatch implements ToXContent { 16 | 17 | private String id; 18 | private String cron; 19 | private String eventType; 20 | private String subject; 21 | private String description; 22 | private List recipient; 23 | private List channel; 24 | private Enums.TriggerType triggerType; 25 | private String indexOpsQuery; 26 | private Enums.QuerySymbol querySymbol; 27 | private Long expectedHit; 28 | private String indexName; 29 | private boolean generateReport; 30 | private String reportFormat; 31 | private String reportTemplatePath; 32 | private Map miscData; 33 | private Date dateCreated; 34 | 35 | 36 | public String getId() { 37 | return id; 38 | } 39 | 40 | public void setId(String id) { 41 | this.id = id; 42 | } 43 | 44 | public String getCron() { 45 | return cron; 46 | } 47 | 48 | public void setCron(String cron) { 49 | this.cron = cron; 50 | } 51 | 52 | public String getEventType() { 53 | return eventType; 54 | } 55 | 56 | public void setEventType(String eventType) { 57 | this.eventType = eventType; 58 | } 59 | 60 | public String getSubject() { 61 | return subject; 62 | } 63 | 64 | public void setSubject(String subject) { 65 | this.subject = subject; 66 | } 67 | 68 | public String getDescription() { 69 | return description; 70 | } 71 | 72 | public void setDescription(String description) { 73 | this.description = description; 74 | } 75 | 76 | public List getRecipient() { 77 | return recipient; 78 | } 79 | 80 | public void setRecipient(List recipient) { 81 | this.recipient = recipient; 82 | } 83 | 84 | public List getChannel() { 85 | return channel; 86 | } 87 | 88 | public void setChannel(List channel) { 89 | this.channel = channel; 90 | } 91 | 92 | public Enums.TriggerType getTriggerType() { 93 | return triggerType; 94 | } 95 | 96 | public void setTriggerType(Enums.TriggerType triggerType) { 97 | this.triggerType = triggerType; 98 | } 99 | 100 | public String getIndexOpsQuery() { 101 | return indexOpsQuery; 102 | } 103 | 104 | public void setIndexOpsQuery(String indexOpsQuery) { 105 | this.indexOpsQuery = indexOpsQuery; 106 | } 107 | 108 | public Enums.QuerySymbol getQuerySymbol() { 109 | return querySymbol; 110 | } 111 | 112 | public void setQuerySymbol(Enums.QuerySymbol querySymbol) { 113 | this.querySymbol = querySymbol; 114 | } 115 | 116 | public Long getExpectedHit() { 117 | return expectedHit; 118 | } 119 | 120 | public void setExpectedHit(Long expectedHit) { 121 | this.expectedHit = expectedHit; 122 | } 123 | 124 | public String getIndexName() { 125 | return indexName; 126 | } 127 | 128 | public void setIndexName(String indexName) { 129 | this.indexName = indexName; 130 | } 131 | 132 | public boolean isGenerateReport() { 133 | return generateReport; 134 | } 135 | 136 | public void setGenerateReport(boolean generateReport) { 137 | this.generateReport = generateReport; 138 | } 139 | 140 | public Map getMiscData() { 141 | return miscData; 142 | } 143 | 144 | public void setMiscData(Map miscData) { 145 | this.miscData = miscData; 146 | } 147 | 148 | public String getReportFormat() { 149 | return reportFormat; 150 | } 151 | 152 | public void setReportFormat(String reportFormat) { 153 | this.reportFormat = reportFormat; 154 | } 155 | 156 | public String getReportTemplatePath() { 157 | return reportTemplatePath; 158 | } 159 | 160 | public void setReportTemplatePath(String reportTemplatePath) { 161 | this.reportTemplatePath = reportTemplatePath; 162 | } 163 | 164 | public Date getDateCreated() { 165 | return dateCreated; 166 | } 167 | 168 | public void setDateCreated(Date dateCreated) { 169 | this.dateCreated = dateCreated; 170 | } 171 | 172 | 173 | public XContentBuilder toXContent(XContentBuilder xContentBuilder, ToXContent.Params params) 174 | throws IOException { 175 | xContentBuilder.field("id", id); 176 | xContentBuilder.field("dateCreated", dateCreated); 177 | xContentBuilder.field("generateReport", generateReport); 178 | if (this.cron != null || !this.cron.isEmpty()) { 179 | xContentBuilder.field("cron", cron); 180 | } 181 | if (this.eventType != null || !this.eventType.isEmpty()) { 182 | xContentBuilder.field("eventType", eventType); 183 | } 184 | if (this.subject != null || !this.subject.isEmpty()) { 185 | xContentBuilder.field("subject", subject); 186 | } 187 | if (this.description != null || !this.description.isEmpty()) { 188 | xContentBuilder.field("description", description); 189 | } 190 | if (this.recipient != null || !this.recipient.isEmpty()) { 191 | xContentBuilder.field("recipient", recipient); 192 | } 193 | if (this.channel != null || !this.channel.isEmpty()) { 194 | xContentBuilder.field("channel", channel); 195 | } 196 | if (this.triggerType != null) { 197 | xContentBuilder.field("triggerType", triggerType); 198 | } 199 | if (this.indexOpsQuery != null || !this.indexOpsQuery.isEmpty()) { 200 | xContentBuilder.field("indexOpsQuery", indexOpsQuery); 201 | } 202 | 203 | if (this.querySymbol != null) { 204 | xContentBuilder.field("querySymbol", querySymbol); 205 | } 206 | if (this.indexName != null || !this.indexName.isEmpty()) { 207 | xContentBuilder.field("indexName", indexName); 208 | } 209 | if (this.miscData != null || !this.miscData.isEmpty()) { 210 | xContentBuilder.field("miscData", miscData); 211 | } 212 | if (this.reportTemplatePath != null || !this.reportTemplatePath.isEmpty()) { 213 | xContentBuilder.field("reportTemplatePath", reportTemplatePath); 214 | } 215 | return xContentBuilder; 216 | } 217 | } 218 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/service/EventIndexOpsTriggerService.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.elasticsearch.index.engine.Engine; 4 | import org.elasticsearch.search.SearchHits; 5 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 6 | 7 | /** 8 | * @author malike_st 9 | */ 10 | public class EventIndexOpsTriggerService { 11 | 12 | private KafkaEventGeneratorService kafkaEventGeneratorService = new KafkaEventGeneratorService(); 13 | private KafkaWatchService kafkaWatchService = new KafkaWatchService(); 14 | 15 | 16 | public boolean evaluateRuleForEvent(String indexName, Engine.Index index, 17 | Engine.IndexResult indexResult, KafkaWatch kafkaWatch) { 18 | if (indexResult.isCreated()) { 19 | return evaluateRule(indexName, kafkaWatch); 20 | } 21 | return false; 22 | } 23 | 24 | public boolean evaluateRuleForEvent(String indexName, Engine.Delete delete, 25 | Engine.DeleteResult deleteResult, KafkaWatch kafkaWatch) { 26 | if (deleteResult.isFound()) { 27 | return evaluateRule(indexName, kafkaWatch); 28 | } 29 | return false; 30 | } 31 | 32 | 33 | private boolean evaluateRule(String indexName, KafkaWatch kafkaWatch) { 34 | if (kafkaWatch == null) { 35 | return false; 36 | } 37 | if (!kafkaWatch.getIndexName().equalsIgnoreCase(indexName)) { 38 | return false; 39 | } 40 | if (kafkaWatch.getExpectedHit() == 0) { 41 | return true; 42 | } 43 | SearchHits response = kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery()); 44 | if (response == null) { 45 | return false; 46 | } 47 | int compared = Long.valueOf(response.getTotalHits()).compareTo( 48 | kafkaWatch.getExpectedHit() 49 | ); 50 | switch (kafkaWatch.getQuerySymbol()) { 51 | case EQUAL_TO: 52 | return compared == 0; 53 | case GREATER_THAN_OR_EQUAL_TO: 54 | return compared >= 0; 55 | case LESS_THAN_OR_EQUAL_TO: 56 | return compared <= 0; 57 | } 58 | return false; 59 | } 60 | 61 | 62 | } 63 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/service/KafkaEventGeneratorService.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import st.malike.elasticsearch.kafka.watch.model.KafkaEvent; 4 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 5 | 6 | /** 7 | * @autor malike_st 8 | */ 9 | public class KafkaEventGeneratorService { 10 | 11 | public KafkaEvent generate(KafkaWatch kafkaWatch) { 12 | 13 | throw new UnsupportedOperationException(); 14 | 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/service/KafkaProducerService.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import com.google.gson.Gson; 4 | import org.apache.kafka.clients.producer.KafkaProducer; 5 | import org.apache.kafka.clients.producer.Producer; 6 | import org.apache.kafka.clients.producer.ProducerRecord; 7 | import org.apache.log4j.Logger; 8 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 9 | import st.malike.elasticsearch.kafka.watch.model.KafkaEvent; 10 | 11 | import java.util.Properties; 12 | 13 | /** 14 | * @autor malike_st 15 | */ 16 | public class KafkaProducerService { 17 | 18 | private static Logger log = Logger.getLogger(KafkaProducerService.class); 19 | private static PluginConfig pluginConfig=new PluginConfig(); 20 | private final Producer producer; 21 | 22 | public KafkaProducerService(PluginConfig pluginConfigs) { 23 | Properties props = new Properties(); 24 | props.put("bootstrap.servers", pluginConfig.getKafkaWatchBootstrapServers()); 25 | props.put("acks", "all"); 26 | props.put("retries", 0); 27 | props.put("batch.size", 16384); 28 | props.put("linger.ms", 1); 29 | props.put("buffer.memory", 33554432); 30 | props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 31 | props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 32 | producer = new KafkaProducer<>(props); 33 | } 34 | 35 | 36 | public void send(KafkaEvent event) { 37 | producer.send(new ProducerRecord<>( 38 | pluginConfig.getKafkaWatchTopic(), 39 | new Gson().toJson(event))); 40 | } 41 | 42 | } 43 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/service/KafkaWatchService.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.elasticsearch.search.SearchHits; 5 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 6 | 7 | import java.util.List; 8 | 9 | /** 10 | * @autor malike_st 11 | */ 12 | public class KafkaWatchService { 13 | 14 | private static Logger log = Logger.getLogger(KafkaWatchService.class); 15 | 16 | public SearchHits executeWatchQuery(String query) { 17 | return null; 18 | } 19 | 20 | public List searchWatchByIndex(String index) { 21 | return null; 22 | } 23 | 24 | public List findAllWatch() { 25 | return null; 26 | } 27 | 28 | public KafkaWatch findById(String key) { 29 | return null; 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/service/ReportService.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import com.google.gson.Gson; 4 | import org.apache.commons.codec.Charsets; 5 | import org.apache.commons.io.IOUtils; 6 | import org.apache.http.HttpResponse; 7 | import org.apache.http.NameValuePair; 8 | import org.apache.http.client.HttpClient; 9 | import org.apache.http.client.entity.UrlEncodedFormEntity; 10 | import org.apache.http.client.methods.HttpPost; 11 | import org.apache.http.impl.client.HttpClientBuilder; 12 | import org.apache.http.message.BasicNameValuePair; 13 | import org.apache.log4j.Logger; 14 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 15 | import st.malike.elasticsearch.kafka.watch.exception.ReportGenerationNotSupported; 16 | import st.malike.elasticsearch.kafka.watch.exception.TemplateFileNotFoundException; 17 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 18 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 19 | 20 | import java.io.File; 21 | import java.util.ArrayList; 22 | import java.util.List; 23 | 24 | /** 25 | * @autor malike_st 26 | */ 27 | public class ReportService { 28 | 29 | private static Logger log = Logger.getLogger(ReportService.class); 30 | private final PluginConfig pluginConfig; 31 | private final KafkaProducerService kafkaProducerService; 32 | HttpClient client = HttpClientBuilder.create().build(); 33 | Gson gson = new Gson(); 34 | public ReportService(PluginConfig pluginConfig, KafkaProducerService kafkaProducerService) { 35 | this.pluginConfig = pluginConfig; 36 | this.kafkaProducerService = kafkaProducerService; 37 | } 38 | 39 | public String getReport(KafkaWatch kafkaWatch) throws TemplateFileNotFoundException, ReportGenerationNotSupported { 40 | if (pluginConfig.getReportEngineDisable()) { 41 | throw new ReportGenerationNotSupported("Report generation not supported"); 42 | } 43 | if (kafkaWatch == null) { 44 | return null; 45 | } 46 | if (validateReportFile(kafkaWatch)) { 47 | return executeService(kafkaWatch.getIndexName(), 48 | kafkaWatch.getIndexOpsQuery(), kafkaWatch.getReportFormat(), 49 | kafkaWatch.getReportTemplatePath()); 50 | } else { 51 | throw new TemplateFileNotFoundException("Report template not found"); 52 | } 53 | } 54 | 55 | public String executeService(String index, String query, 56 | String format, String templateFile) { 57 | 58 | try { 59 | HttpPost post = new HttpPost(pluginConfig.getReportEngineEndpoint()); 60 | 61 | post.setHeader("Content-Type", "application/json"); 62 | 63 | List urlParameters = new ArrayList<>(); 64 | urlParameters.add(new BasicNameValuePair("format", (format == null 65 | || format.isEmpty()) ? "PDF" : format)); 66 | urlParameters.add(new BasicNameValuePair("index", index)); 67 | urlParameters.add(new BasicNameValuePair("returnAs", "PLAIN")); 68 | urlParameters.add(new BasicNameValuePair("template", templateFile)); 69 | urlParameters.add(new BasicNameValuePair("query", query)); 70 | 71 | post.setEntity(new UrlEncodedFormEntity(urlParameters)); 72 | 73 | HttpResponse response = client.execute(post); 74 | if (response.getStatusLine().getStatusCode() == 200) { 75 | String responseString = IOUtils.toString(response.getEntity().getContent(), 76 | Charsets.UTF_8.toString()); 77 | JSONResponse jsonResponse = gson.fromJson(responseString, JSONResponse.class); 78 | if (jsonResponse.getStatus()) { 79 | return (String) jsonResponse.getData(); 80 | } else { 81 | log.error("Error generating report. Response is " + gson.toJson(response.getEntity().getContent())); 82 | } 83 | } else { 84 | log.error("Error generating report. Status Code is " + response.getStatusLine().getStatusCode()); 85 | } 86 | 87 | } catch (Exception e) { 88 | } 89 | return null; 90 | } 91 | 92 | public boolean validateReportFile(KafkaWatch kafkaWatch) throws TemplateFileNotFoundException { 93 | File file = new File(kafkaWatch.getReportTemplatePath()); 94 | if (!(file.exists() && !file.isDirectory())) { 95 | throw new TemplateFileNotFoundException("Report template not found"); 96 | } 97 | return true; 98 | } 99 | 100 | } 101 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/service/TimeTriggerService.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.apache.log4j.Logger; 4 | import org.quartz.*; 5 | import org.quartz.impl.JobDetailImpl; 6 | import org.quartz.impl.StdSchedulerFactory; 7 | import org.quartz.impl.triggers.CronTriggerImpl; 8 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 9 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 10 | 11 | import java.util.List; 12 | 13 | /** 14 | * @author malike_st 15 | */ 16 | public class TimeTriggerService { 17 | private static Logger log = Logger.getLogger(TimeTriggerService.class); 18 | private final PluginConfig pluginConfig; 19 | private final KafkaProducerService kafkaProducerService; 20 | 21 | private KafkaWatchService kafkaWatchService = new KafkaWatchService(); 22 | private Scheduler scheduler; 23 | private JobDetailImpl jobDetail; 24 | 25 | public TimeTriggerService(PluginConfig pluginConfig, KafkaProducerService kafkaProducerService) 26 | throws Exception { 27 | this.pluginConfig = pluginConfig; 28 | this.kafkaProducerService = kafkaProducerService; 29 | schedule(); 30 | } 31 | 32 | public void schedule() throws Exception { 33 | 34 | SchedulerFactory schedulerFactory = new StdSchedulerFactory(); 35 | scheduler = schedulerFactory.getScheduler(); 36 | 37 | jobDetail = new JobDetailImpl(); 38 | jobDetail.setGroup("Kafka-Elasticsearch"); 39 | jobDetail.setName("Kafka-Elasticsearch"); 40 | jobDetail.setJobClass(SchedulerJob.class); 41 | 42 | 43 | List watches = kafkaWatchService.findAllWatch(); 44 | if (watches != null && !watches.isEmpty()) { 45 | for (KafkaWatch watch : watches) { 46 | addJob(watch); 47 | } 48 | } 49 | 50 | scheduler.start(); 51 | } 52 | 53 | public JobDetail addJob(KafkaWatch kafkaWatch) throws Exception { 54 | 55 | CronTriggerImpl cronTrigger = new CronTriggerImpl(); 56 | cronTrigger.setCronExpression(kafkaWatch.getCron()); 57 | cronTrigger.setName(kafkaWatch.getId()); 58 | cronTrigger.setGroup(kafkaWatch.getId()); 59 | 60 | if (scheduler == null) { 61 | schedule(); 62 | } 63 | 64 | scheduler.scheduleJob(jobDetail, cronTrigger); 65 | if (!scheduler.isStarted()) { 66 | scheduler.start(); 67 | } 68 | 69 | return scheduler.getJobDetail(new JobKey(kafkaWatch.getId())); 70 | } 71 | 72 | 73 | public void deleteJob(KafkaWatch kafkaWatch) throws Exception { 74 | if (kafkaWatch == null) { 75 | return; 76 | } 77 | JobDetail jobDetail = scheduler.getJobDetail(new JobKey(kafkaWatch.getId())); 78 | if (jobDetail != null) { 79 | scheduler.deleteJob(new JobKey(kafkaWatch.getId())); 80 | } 81 | } 82 | 83 | 84 | class SchedulerJob implements Job { 85 | 86 | KafkaWatchService kafkaWatchService = new KafkaWatchService(); 87 | KafkaEventGeneratorService kafkaEventGeneratorService = new KafkaEventGeneratorService(); 88 | 89 | @Override 90 | public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException { 91 | 92 | String key = jobExecutionContext.getJobDetail().getKey().toString(); 93 | KafkaWatch watch = kafkaWatchService.findById(key); 94 | kafkaProducerService.send(kafkaEventGeneratorService.generate(watch)); 95 | } 96 | } 97 | } 98 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/util/Enums.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.util; 2 | 3 | /** 4 | * @author malike_st 5 | */ 6 | public class Enums { 7 | 8 | public enum JSONResponseMessage { 9 | SUCCESS, 10 | ERROR, 11 | MISSING_PARAM, 12 | INVALID_DATA, 13 | NOT_CONFIGURED_FOR_REPORTS, 14 | DATA_NOT_FOUND 15 | } 16 | 17 | public enum TriggerType { 18 | TIME, 19 | INDEX_OPS 20 | } 21 | 22 | public enum QuerySymbol { 23 | EQUAL_TO, 24 | GREATER_THAN_OR_EQUAL_TO, 25 | LESS_THAN_OR_EQUAL_TO, 26 | } 27 | 28 | 29 | } 30 | -------------------------------------------------------------------------------- /src/main/java/st/malike/elasticsearch/kafka/watch/util/JSONResponse.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.util; 2 | 3 | import org.elasticsearch.common.xcontent.ToXContent; 4 | import org.elasticsearch.common.xcontent.XContentBuilder; 5 | 6 | import java.io.IOException; 7 | 8 | /** 9 | * @author malike_st 10 | */ 11 | public class JSONResponse implements ToXContent { 12 | 13 | private String message; 14 | private Boolean status; 15 | private Object data; 16 | private Long count; 17 | 18 | public String getMessage() { 19 | return message; 20 | } 21 | 22 | public void setMessage(String message) { 23 | this.message = message; 24 | } 25 | 26 | public Boolean getStatus() { 27 | return status; 28 | } 29 | 30 | public void setStatus(Boolean status) { 31 | this.status = status; 32 | } 33 | 34 | public Object getData() { 35 | return data; 36 | } 37 | 38 | public void setData(Object data) { 39 | this.data = data; 40 | } 41 | 42 | public Long getCount() { 43 | return count; 44 | } 45 | 46 | public void setCount(Long count) { 47 | this.count = count; 48 | } 49 | 50 | @Override 51 | public XContentBuilder toXContent(XContentBuilder xContentBuilder, Params params) throws IOException { 52 | return xContentBuilder.field("message", message) 53 | .field("status", status) 54 | .field("count", count) 55 | .field("data", data); 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /src/main/resources/plugin-descriptor.properties: -------------------------------------------------------------------------------- 1 | description=${project.description} 2 | version=${project.version} 3 | name=${project.artifactId} 4 | classname=st.malike.elasticsearch.kafka.watch.ElasticKafkaWatchPlugin 5 | java.version=1.8 6 | elasticsearch.version=${elasticsearch.version} -------------------------------------------------------------------------------- /src/main/resources/plugin-security.policy: -------------------------------------------------------------------------------- 1 | //permission to access template files from file system 2 | 3 | grant{ 4 | permission java.io.FilePermission "config/elastic-kafka-watch.yml", "read,write" 5 | } -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/ElasticKafkaWatchPluginTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch; 2 | 3 | import com.google.gson.Gson; 4 | import com.jayway.restassured.response.ValidatableResponse; 5 | import org.apache.commons.lang.RandomStringUtils; 6 | import org.codelibs.elasticsearch.runner.ElasticsearchClusterRunner; 7 | import org.elasticsearch.common.settings.Settings; 8 | import org.elasticsearch.node.Node; 9 | import org.hamcrest.Matchers; 10 | import org.junit.AfterClass; 11 | import org.junit.Before; 12 | import org.junit.BeforeClass; 13 | import org.junit.Test; 14 | import org.junit.runner.RunWith; 15 | import org.junit.runners.BlockJUnit4ClassRunner; 16 | import st.malike.elasticsearch.kafka.watch.util.Enums; 17 | 18 | import java.io.IOException; 19 | import java.util.HashMap; 20 | import java.util.Map; 21 | 22 | import static com.jayway.restassured.RestAssured.given; 23 | 24 | /** 25 | * @author malike_st 26 | */ 27 | @RunWith(BlockJUnit4ClassRunner.class) 28 | public class ElasticKafkaWatchPluginTest { 29 | 30 | 31 | private static final String KAFKA_WATCH_ELASTICSEARCH_TYPE = "kafka.watch.elasticsearch.type"; 32 | private static final String KAFKA_WATCH_BOOTSTRAP_SERVERS = "localhost:9092"; 33 | private static final String KAFKA_WATCH_TOPIC = "kafka.watch.topic"; 34 | private static final String KAFKA_WATCH_DISABLE = "kafka.watch.disable"; 35 | private static final String REPORT_ENGINE_ENDPOINT = "report.engine.endpoint"; 36 | private static final String REPORT_ENGINE_DISABLE = "report.engine.disable"; 37 | private static final String INDEX_NAME = "kafka_watch"; 38 | private static final String INDEX_TYPE = "kafka_topic"; 39 | private static final String CLUSTER_NAME = "DUMMY_CLUSTER"; 40 | private static final String CLUSTER_HOST_ADDRESS = "localhost:9201-9210"; 41 | private static Node node; 42 | private static ElasticsearchClusterRunner runner; 43 | String queryString = "{" 44 | + " \"match\": {" 45 | + " \"description\": \"SUBSCRIPTION\"" 46 | + " }" 47 | + "}"; 48 | private Map param; 49 | 50 | 51 | 52 | @BeforeClass 53 | public static void setUp() throws IOException { 54 | 55 | runner = new ElasticsearchClusterRunner(); 56 | 57 | runner.onBuild(new ElasticsearchClusterRunner.Builder() { 58 | 59 | @Override 60 | public void build(final int number, final Settings.Builder settingsBuilder) { 61 | settingsBuilder.put("http.cors.allow-origin", "*"); 62 | settingsBuilder.put("http.cors.enabled", true); 63 | settingsBuilder.putArray("discovery.zen.ping.unicast.hosts", CLUSTER_HOST_ADDRESS); 64 | } 65 | }).build(ElasticsearchClusterRunner.newConfigs().clusterName(CLUSTER_NAME).numOfNode(1) 66 | .pluginTypes("st.malike.elasticsearch.kafka.watch.ElasticKafkaWatchPlugin")); 67 | 68 | runner.ensureYellow(); 69 | 70 | // create an index 71 | runner.createIndex(INDEX_NAME, (Settings) null); 72 | 73 | 74 | runner.refresh(); 75 | 76 | node = runner.node(); 77 | } 78 | 79 | @AfterClass 80 | public static void tearDown() throws IOException { 81 | runner.close(); 82 | runner.clean(); 83 | } 84 | 85 | 86 | @Before 87 | public void setUpTest() { 88 | param = new HashMap(); 89 | runner.deleteIndex(INDEX_NAME); 90 | runner.createIndex(INDEX_NAME, (Settings) null); 91 | 92 | } 93 | 94 | @Test 95 | public void addDummyData() { 96 | given() 97 | .log().all().contentType("application/json") 98 | .body("{" + 99 | " \"user\" : \"kimchy\",\n" + 100 | " \"post_date\" : \"2009-11-15T14:12:12\",\n" + 101 | " \"message\" : \"trying out Elasticsearch\"\n" + 102 | "}") 103 | .when() 104 | .post("http://localhost:9201/" + INDEX_NAME + "/" + INDEX_TYPE + "/") 105 | .then() 106 | .statusCode(201); 107 | } 108 | 109 | @Test 110 | public void addNewWatcherEmptyParams() { 111 | given() 112 | .log().all().contentType("application/json") 113 | .body(new Gson().toJson(param)) 114 | .when() 115 | .post("http://localhost:9201/_newkafkawatch") 116 | .then() 117 | .statusCode(200) 118 | .body("status", Matchers.is(false)) 119 | .body("message", Matchers.is(Enums.JSONResponseMessage.MISSING_PARAM.toString())); 120 | } 121 | 122 | @Test 123 | public void addNewWatcherMissingParam() { 124 | 125 | param.put("querySymbol", ""); 126 | param.put("expectedHit", "5"); 127 | 128 | given() 129 | .log().all().contentType("application/json") 130 | .body(new Gson().toJson(param)) 131 | .when() 132 | .post("http://localhost:9201/_newkafkawatch") 133 | .then() 134 | .statusCode(200) 135 | .body("status", Matchers.is(false)) 136 | .body("message", Matchers.is(Enums.JSONResponseMessage.INVALID_DATA.toString())); 137 | } 138 | 139 | @Test 140 | public void addNewWatcherForReportWithoutTemplate() { 141 | 142 | 143 | param.put("eventType", "SUBSCRIPTION_REPORT"); 144 | param.put("description", "Send reports for subscription daily"); 145 | param.put("channel", "EMAIL"); 146 | param.put("cron", "0 0 9 * * MON-FRI"); 147 | param.put("query", queryString); 148 | param.put("generateReport", true); 149 | param.put("trigger", Enums.TriggerType.TIME.toString()); 150 | param.put("indexName", INDEX_NAME); 151 | 152 | 153 | given() 154 | .log().all().contentType("application/json") 155 | .body(new Gson().toJson(param)) 156 | .when() 157 | .post("http://localhost:9201/_newkafkawatch") 158 | .then() 159 | .statusCode(200) 160 | .body("status", Matchers.is(false)) 161 | .body("message", Matchers.is(Enums.JSONResponseMessage.NOT_CONFIGURED_FOR_REPORTS.toString())); 162 | } 163 | 164 | @Test 165 | public void addNewWatcherForReportWithoutTemplateOrQuery() { 166 | 167 | 168 | param.put("eventType", "SUBSCRIPTION_REPORT"); 169 | param.put("description", "Send reports for subscription daily"); 170 | param.put("channel", "EMAIL"); 171 | param.put("cron", "0 0 9 * * MON-FRI"); 172 | param.put("generateReport", true); 173 | param.put("trigger", Enums.TriggerType.TIME.toString()); 174 | param.put("indexName", INDEX_NAME); 175 | 176 | 177 | given() 178 | .log().all().contentType("application/json") 179 | .body(new Gson().toJson(param)) 180 | .when() 181 | .post("http://localhost:9201/_newkafkawatch") 182 | .then() 183 | .statusCode(200) 184 | .body("status", Matchers.is(false)) 185 | .body("message", Matchers.is(Enums.JSONResponseMessage.NOT_CONFIGURED_FOR_REPORTS.toString())); 186 | } 187 | 188 | @Test 189 | public void addNewWatcherReportKafkaWatch() { 190 | 191 | param.put("eventType", "SUBSCRIPTION_REPORT"); 192 | param.put("description", "Send reports for subscription daily"); 193 | param.put("channel", "EMAIL"); 194 | param.put("cron", "0 0 9 * * MON-FRI"); 195 | param.put("query", queryString); 196 | param.put("generateReport", true); 197 | param.put("reportTemplatePath", "/home/malike/dev/report.jrxml"); 198 | param.put("trigger", Enums.TriggerType.TIME.toString()); 199 | param.put("indexName", INDEX_NAME); 200 | 201 | 202 | given() 203 | .log().all().contentType("application/json") 204 | .body(new Gson().toJson(param)) 205 | .when() 206 | .post("http://localhost:9201/_newkafkawatch") 207 | .then() 208 | .statusCode(200) 209 | .body("status", Matchers.is(true)) 210 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 211 | } 212 | 213 | @Test 214 | public void addNewWatcher() { 215 | 216 | param.put("eventType", "SUBSCRIPTION"); 217 | param.put("description", "Send welcome notification for every subscription created"); 218 | param.put("channel", "SMS"); 219 | param.put("trigger", "INDEX_OPS"); 220 | param.put("indexName", INDEX_NAME); 221 | 222 | 223 | given() 224 | .log().all().contentType("application/json") 225 | .body(new Gson().toJson(param)) 226 | .when() 227 | .post("http://localhost:9201/_newkafkawatch") 228 | .then() 229 | .statusCode(200) 230 | .body("status", Matchers.is(true)) 231 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 232 | } 233 | 234 | @Test 235 | public void removeWatcherNoParams() { 236 | given() 237 | .log().all().contentType("application/json") 238 | .body(new Gson().toJson(param)) 239 | .when() 240 | .post("http://localhost:9201/_removekafkawatch") 241 | .then() 242 | .statusCode(200) 243 | .body("status", Matchers.is(false)) 244 | .body("message", Matchers.is(Enums.JSONResponseMessage.MISSING_PARAM.toString())); 245 | } 246 | 247 | @Test 248 | public void removeWatcherUnknownWatcherId() { 249 | 250 | param.put("id", RandomStringUtils.randomAlphanumeric(5)); 251 | 252 | given() 253 | .log().all().contentType("application/json") 254 | .body(new Gson().toJson(param)) 255 | .when() 256 | .post("http://localhost:9201/_removekafkawatch") 257 | .then() 258 | .statusCode(200) 259 | .body("status", Matchers.is(false)) 260 | .body("message", Matchers.is(Enums.JSONResponseMessage.DATA_NOT_FOUND.toString())); 261 | } 262 | 263 | @Test 264 | public void removeWatcher() { 265 | 266 | param.put("eventType", "SUBSCRIPTION"); 267 | param.put("description", "Send welcome notification for every subscription created"); 268 | param.put("channel", "SMS"); 269 | param.put("trigger", "INDEX_OPS"); 270 | param.put("indexName", INDEX_NAME); 271 | 272 | 273 | ValidatableResponse validatableResponse = given() 274 | .log().all().contentType("application/json") 275 | .body(new Gson().toJson(param)) 276 | .when() 277 | .post("http://localhost:9201/_newkafkawatch") 278 | .then() 279 | .statusCode(200); 280 | 281 | Map response = new Gson().fromJson((String) validatableResponse.extract().body() 282 | .jsonPath().get("data"), Map.class); 283 | param.put("id", response.get("id")); 284 | 285 | given() 286 | .log().all().contentType("application/json") 287 | .body(new Gson().toJson(param)) 288 | .when() 289 | .post("http://localhost:9201/_removekafkawatch") 290 | .then() 291 | .statusCode(200) 292 | .body("status", Matchers.is(true)) 293 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 294 | } 295 | 296 | @Test 297 | public void viewWatchersWithNoWatchersCreated() { 298 | given() 299 | .log().all().contentType("application/json") 300 | .body(new Gson().toJson(param)) 301 | .when() 302 | .post("http://localhost:9201/_listkafkawatch") 303 | .then() 304 | .statusCode(200) 305 | .body("status", Matchers.is(true)) 306 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 307 | } 308 | 309 | @Test 310 | public void viewWatchers() { 311 | 312 | 313 | param.put("eventType", "SUBSCRIPTION"); 314 | param.put("description", "Send welcome notification for every subscription created"); 315 | param.put("channel", "SMS"); 316 | param.put("trigger", "INDEX_OPS"); 317 | param.put("indexName", INDEX_NAME); 318 | 319 | 320 | given() 321 | .log().all().contentType("application/json") 322 | .body(new Gson().toJson(param)) 323 | .when() 324 | .post("http://localhost:9201/_newkafkawatch") 325 | .then() 326 | .statusCode(200) 327 | .body("status", Matchers.is(true)); 328 | 329 | //to refresh data... for fetch 330 | runner.flush(); 331 | runner.refresh(); 332 | 333 | param = new HashMap<>(); 334 | 335 | given() 336 | .log().all().contentType("application/json") 337 | .body(new Gson().toJson(param)) 338 | .when() 339 | .post("http://localhost:9201/_listkafkawatch") 340 | .then() 341 | .statusCode(200) 342 | .body("status", Matchers.is(true)) 343 | .body("data[0].eventType", Matchers.is("SUBSCRIPTION")) 344 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 345 | } 346 | 347 | @Test 348 | public void searchWatchersNoQuery() { 349 | given() 350 | .log().all().contentType("application/json") 351 | .body(new Gson().toJson(param)) 352 | .when() 353 | .post("http://localhost:9201/_searchkafkawatch") 354 | .then() 355 | .statusCode(200) 356 | .body("status", Matchers.is(false)) 357 | .body("message", Matchers.is(Enums.JSONResponseMessage.MISSING_PARAM.toString())); 358 | } 359 | 360 | @Test 361 | public void searchWatchers() { 362 | 363 | 364 | param.put("eventType", "SUBSCRIPTION"); 365 | param.put("description", "Send welcome notification for every subscription created"); 366 | param.put("channel", "SMS"); 367 | param.put("trigger", "INDEX_OPS"); 368 | param.put("indexName", INDEX_NAME); 369 | 370 | 371 | given() 372 | .log().all().contentType("application/json") 373 | .body(new Gson().toJson(param)) 374 | .when() 375 | .post("http://localhost:9201/_newkafkawatch") 376 | .then() 377 | .statusCode(200) 378 | .body("status", Matchers.is(true)); 379 | 380 | 381 | String queryString = "{" 382 | + " \"match\": {" 383 | + " \"eventType\": \"SUBSCRIPTION\"" 384 | + " }" 385 | + "}"; 386 | 387 | param.put("param", queryString); 388 | 389 | 390 | given() 391 | .log().all().contentType("application/json") 392 | .body(new Gson().toJson(param)) 393 | .when() 394 | .post("http://localhost:9201/_searchkafkawatch") 395 | .then() 396 | .statusCode(200) 397 | .body("status", Matchers.is(true)) 398 | .body("data[0].eventType", Matchers.is("SUBSCRIPTION")) 399 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 400 | } 401 | 402 | 403 | @Test 404 | public void searchWatchersByIndex() { 405 | 406 | 407 | param.put("eventType", "SUBSCRIPTION"); 408 | param.put("description", "Send welcome notification for every subscription created"); 409 | param.put("channel", "SMS"); 410 | param.put("trigger", "INDEX_OPS"); 411 | param.put("indexName", INDEX_NAME); 412 | 413 | 414 | given() 415 | .log().all().contentType("application/json") 416 | .body(new Gson().toJson(param)) 417 | .when() 418 | .post("http://localhost:9201/_newkafkawatch") 419 | .then() 420 | .statusCode(200) 421 | .body("status", Matchers.is(true)); 422 | 423 | //to refresh data... for fetch 424 | runner.flush(); 425 | runner.refresh(); 426 | 427 | 428 | String queryString = "{" 429 | + " \"match\": {" 430 | + " \"indexName\": \"" + INDEX_NAME + "\"" 431 | + " }" 432 | + "}"; 433 | 434 | param.put("param", queryString); 435 | 436 | 437 | given() 438 | .log().all().contentType("application/json") 439 | .body(new Gson().toJson(param)) 440 | .when() 441 | .post("http://localhost:9201/_searchkafkawatch") 442 | .then() 443 | .statusCode(200) 444 | .body("status", Matchers.is(true)) 445 | .body("data[0].eventType", Matchers.is("SUBSCRIPTION")) 446 | .body("message", Matchers.is(Enums.JSONResponseMessage.SUCCESS.toString())); 447 | } 448 | 449 | 450 | @Test 451 | public void testEventTriggerAfterDocumentIndexed() { 452 | 453 | given() 454 | .log().all().contentType("application/json") 455 | .body(new Gson().toJson(param)) 456 | .when() 457 | .post("http://localhost:9201/_newkafkawatch") 458 | .then() 459 | .statusCode(200) 460 | .body("status", Matchers.is(false)) 461 | .body("message", Matchers.is(Enums.JSONResponseMessage.MISSING_PARAM.toString())); 462 | } 463 | 464 | @Test 465 | public void testEventNotTriggerAfterDocumentIndexed() { 466 | 467 | given() 468 | .log().all().contentType("application/json") 469 | .body(new Gson().toJson(param)) 470 | .when() 471 | .post("http://localhost:9201/_newkafkawatch") 472 | .then() 473 | .statusCode(200) 474 | .body("status", Matchers.is(false)) 475 | .body("message", Matchers.is(Enums.JSONResponseMessage.MISSING_PARAM.toString())); 476 | } 477 | 478 | @Test 479 | public void testEventTriggerBySchedule() { 480 | 481 | given() 482 | .log().all().contentType("application/json") 483 | .body(new Gson().toJson(param)) 484 | .when() 485 | .post("http://localhost:9201/_newkafkawatch") 486 | .then() 487 | .statusCode(200) 488 | .body("status", Matchers.is(false)) 489 | .body("message", Matchers.is(Enums.JSONResponseMessage.MISSING_PARAM.toString())); 490 | } 491 | 492 | 493 | } 494 | -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/service/EventIndexOpsTriggerServiceTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.apache.commons.lang.RandomStringUtils; 4 | import org.elasticsearch.index.engine.Engine; 5 | import org.elasticsearch.search.SearchHits; 6 | import org.junit.Assert; 7 | import org.junit.Before; 8 | import org.junit.Test; 9 | import org.junit.runner.RunWith; 10 | import org.mockito.InjectMocks; 11 | import org.mockito.Mock; 12 | import org.mockito.Mockito; 13 | import org.mockito.Spy; 14 | import org.mockito.runners.MockitoJUnitRunner; 15 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 16 | import st.malike.elasticsearch.kafka.watch.util.Enums; 17 | 18 | import java.util.Arrays; 19 | import java.util.Date; 20 | 21 | /** 22 | * @autor malike_st 23 | */ 24 | @RunWith(MockitoJUnitRunner.class) 25 | public class EventIndexOpsTriggerServiceTest { 26 | 27 | @Mock 28 | private SearchHits searchHits; 29 | @Spy 30 | @InjectMocks 31 | private EventIndexOpsTriggerService eventIndexOpsTriggerService; 32 | @Mock 33 | private Engine.Index index; 34 | @Mock 35 | private Engine.IndexResult indexResult; 36 | @Mock 37 | private Engine.Delete delete; 38 | @Mock 39 | private Engine.DeleteResult deleteResult; 40 | @Mock 41 | private KafkaWatchService kafkaWatchService; 42 | private KafkaWatch kafkaWatch; 43 | private String INDEX_NAME = "TEST"; 44 | 45 | 46 | @Before 47 | public void setUp() throws Exception { 48 | 49 | kafkaWatch = new KafkaWatch(); 50 | kafkaWatch.setId(RandomStringUtils.randomAlphanumeric(5)); 51 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.GREATER_THAN_OR_EQUAL_TO); 52 | kafkaWatch.setSubject("Random Kafka Watch"); 53 | kafkaWatch.setDateCreated(new Date()); 54 | kafkaWatch.setTriggerType(Enums.TriggerType.INDEX_OPS); 55 | kafkaWatch.setChannel(Arrays.asList("SMS", "EMAIL")); 56 | kafkaWatch.setDescription("Random Kafka Watch To Test"); 57 | kafkaWatch.setEventType("SUBSCRIPTION"); 58 | kafkaWatch.setReportTemplatePath("/home/malike/devfiles/report.jrxml"); 59 | kafkaWatch.setGenerateReport(true); 60 | kafkaWatch.setIndexName("Test"); 61 | kafkaWatch.setExpectedHit(2L); 62 | kafkaWatch.setReportFormat("HTML"); 63 | kafkaWatch.setRecipient(Arrays.asList("233201234567", "st.malike@gmail.com")); 64 | 65 | 66 | } 67 | 68 | 69 | @Test 70 | public void testEvaluateRuleForIndexCreateWithNoWatch() { 71 | 72 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(INDEX_NAME, 73 | index, indexResult, null); 74 | Assert.assertFalse(rule); 75 | } 76 | 77 | @Test 78 | public void testEvaluateRuleForIndexCreateWithUnmatchingIndexName() { 79 | 80 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(INDEX_NAME + "TEST", 81 | index, indexResult, kafkaWatch); 82 | Assert.assertFalse(rule); 83 | } 84 | 85 | @Test 86 | public void testEvaluateRuleForIndexCreateGreaterOrEqualTo() { 87 | 88 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 89 | .thenReturn(searchHits); 90 | Mockito.when(searchHits.getTotalHits()).thenReturn(kafkaWatch.getExpectedHit()); 91 | Mockito.when(indexResult.isCreated()).thenReturn(true); 92 | 93 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 94 | index, indexResult, kafkaWatch); 95 | Assert.assertTrue(rule); 96 | } 97 | 98 | @Test 99 | public void testEvaluateRuleForIndexCreateEqualTo() { 100 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.EQUAL_TO); 101 | 102 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 103 | .thenReturn(searchHits); 104 | Mockito.when(searchHits.getTotalHits()).thenReturn(kafkaWatch.getExpectedHit()); 105 | Mockito.when(indexResult.isCreated()).thenReturn(true); 106 | 107 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 108 | index, indexResult, kafkaWatch); 109 | Assert.assertTrue(rule); 110 | } 111 | 112 | @Test 113 | public void testEvaluateRuleForIndexCreateLessOrEqualTo() { 114 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.LESS_THAN_OR_EQUAL_TO); 115 | 116 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 117 | .thenReturn(searchHits); 118 | Mockito.when(searchHits.getTotalHits()).thenReturn(kafkaWatch.getExpectedHit()); 119 | Mockito.when(indexResult.isCreated()).thenReturn(true); 120 | 121 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 122 | index, indexResult, kafkaWatch); 123 | Assert.assertTrue(rule); 124 | } 125 | 126 | @Test 127 | public void testEvaluateRuleForIndexCreateNotEqualTOHit() { 128 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.EQUAL_TO); 129 | 130 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 131 | .thenReturn(searchHits); 132 | Mockito.when(searchHits.getTotalHits()).thenReturn(15L); 133 | 134 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 135 | index, indexResult, kafkaWatch); 136 | Assert.assertFalse(rule); 137 | } 138 | 139 | @Test 140 | public void testEvaluateRuleForDeleteCreateWithNoWatch() { 141 | 142 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(INDEX_NAME, 143 | delete, deleteResult, null); 144 | Assert.assertFalse(rule); 145 | } 146 | 147 | @Test 148 | public void testEvaluateRuleForDeleteCreateWithUnmatchingIndexName() { 149 | 150 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(INDEX_NAME + "T", 151 | delete, deleteResult, kafkaWatch); 152 | Assert.assertFalse(rule); 153 | } 154 | 155 | @Test 156 | public void testEvaluateRuleForIndexDeletedGreaterOrEqualTo() { 157 | 158 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 159 | .thenReturn(searchHits); 160 | Mockito.when(searchHits.getTotalHits()).thenReturn(kafkaWatch.getExpectedHit()); 161 | Mockito.when(deleteResult.isFound()).thenReturn(true); 162 | 163 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 164 | delete, deleteResult, kafkaWatch); 165 | Assert.assertTrue(rule); 166 | } 167 | 168 | @Test 169 | public void testEvaluateRuleForIndexDeleteEqualTo() { 170 | 171 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.EQUAL_TO); 172 | 173 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 174 | .thenReturn(searchHits); 175 | Mockito.when(searchHits.getTotalHits()).thenReturn(kafkaWatch.getExpectedHit()); 176 | Mockito.when(deleteResult.isFound()).thenReturn(true); 177 | 178 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 179 | delete, deleteResult, kafkaWatch); 180 | Assert.assertTrue(rule); 181 | } 182 | 183 | @Test 184 | public void testEvaluateRuleForIndexDeleteLessOrEqualTo() { 185 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.LESS_THAN_OR_EQUAL_TO); 186 | 187 | Mockito.when(kafkaWatchService.executeWatchQuery(kafkaWatch.getIndexOpsQuery())) 188 | .thenReturn(searchHits); 189 | Mockito.when(searchHits.getTotalHits()).thenReturn(kafkaWatch.getExpectedHit()); 190 | Mockito.when(deleteResult.isFound()).thenReturn(true); 191 | 192 | Boolean rule = eventIndexOpsTriggerService.evaluateRuleForEvent(kafkaWatch.getIndexName(), 193 | delete, deleteResult, kafkaWatch); 194 | Assert.assertTrue(rule); 195 | } 196 | 197 | 198 | } 199 | -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/service/KafkaEventGeneratorServiceTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.junit.After; 4 | import org.junit.Before; 5 | import org.junit.Test; 6 | import org.junit.runner.RunWith; 7 | import org.mockito.InjectMocks; 8 | import org.mockito.runners.MockitoJUnitRunner; 9 | 10 | /** 11 | * @autor malike_st 12 | */ 13 | @RunWith(MockitoJUnitRunner.class) 14 | public class KafkaEventGeneratorServiceTest { 15 | 16 | @InjectMocks 17 | private KafkaEventGeneratorService kafkaEventGeneratorService; 18 | 19 | @Before 20 | public void setUp() throws Exception { 21 | 22 | } 23 | 24 | @After 25 | public void tearDown() throws Exception { 26 | 27 | } 28 | 29 | @Test 30 | public void generate() throws Exception { 31 | 32 | } 33 | } -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/service/KafkaProducerServiceTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.apache.kafka.clients.producer.Producer; 4 | import org.junit.Before; 5 | import org.junit.Test; 6 | import org.junit.runner.RunWith; 7 | import org.mockito.InjectMocks; 8 | import org.mockito.Mock; 9 | import org.mockito.Mockito; 10 | import org.mockito.Spy; 11 | import org.mockito.runners.MockitoJUnitRunner; 12 | import st.malike.elasticsearch.kafka.watch.model.KafkaEvent; 13 | 14 | /** 15 | * @autor malike_st 16 | */ 17 | @RunWith(MockitoJUnitRunner.class) 18 | public class KafkaProducerServiceTest { 19 | 20 | @InjectMocks 21 | @Spy 22 | private KafkaProducerService kafkaProducerService; 23 | @Mock 24 | private Producer producer; 25 | @Mock 26 | private KafkaEvent kafkaEvent; 27 | 28 | 29 | @Before 30 | public void setUp() throws Exception { 31 | 32 | } 33 | 34 | @Test 35 | public void testSend() { 36 | 37 | kafkaProducerService.send(kafkaEvent); 38 | Mockito.verify(kafkaProducerService, Mockito.times(1)).send(Mockito.any()); 39 | 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/service/KafkaWatchServiceTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.junit.After; 4 | import org.junit.Before; 5 | import org.junit.Test; 6 | import org.junit.runner.RunWith; 7 | import org.mockito.runners.MockitoJUnitRunner; 8 | 9 | /** 10 | * @autor malike_st 11 | */ 12 | @RunWith(MockitoJUnitRunner.class) 13 | public class KafkaWatchServiceTest { 14 | 15 | @Before 16 | public void setUp() throws Exception { 17 | 18 | } 19 | 20 | @After 21 | public void tearDown() throws Exception { 22 | 23 | } 24 | 25 | @Test 26 | public void executeWatchQuery() throws Exception { 27 | 28 | } 29 | 30 | @Test 31 | public void searchWatchByIndex() throws Exception { 32 | 33 | } 34 | 35 | @Test 36 | public void findById() { 37 | 38 | } 39 | } -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/service/ReportServiceTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import com.google.gson.Gson; 4 | import org.apache.commons.codec.Charsets; 5 | import org.apache.commons.lang.RandomStringUtils; 6 | import org.apache.http.HttpResponse; 7 | import org.apache.http.ProtocolVersion; 8 | import org.apache.http.StatusLine; 9 | import org.apache.http.client.HttpClient; 10 | import org.apache.http.entity.StringEntity; 11 | import org.apache.http.message.BasicHttpResponse; 12 | import org.junit.Assert; 13 | import org.junit.Before; 14 | import org.junit.Test; 15 | import org.junit.runner.RunWith; 16 | import org.mockito.InjectMocks; 17 | import org.mockito.Mock; 18 | import org.mockito.Mockito; 19 | import org.mockito.Spy; 20 | import org.mockito.runners.MockitoJUnitRunner; 21 | import st.malike.elasticsearch.kafka.watch.config.PluginConfig; 22 | import st.malike.elasticsearch.kafka.watch.exception.TemplateFileNotFoundException; 23 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 24 | import st.malike.elasticsearch.kafka.watch.util.Enums; 25 | import st.malike.elasticsearch.kafka.watch.util.JSONResponse; 26 | 27 | import java.util.Arrays; 28 | import java.util.Date; 29 | 30 | /** 31 | * @autor malike_st 32 | */ 33 | @RunWith(MockitoJUnitRunner.class) 34 | public class ReportServiceTest { 35 | 36 | HttpResponse httpResponse; 37 | JSONResponse jsonResponse; 38 | @InjectMocks 39 | @Spy 40 | private ReportService reportService; 41 | @Mock 42 | private HttpClient httpClient; 43 | @Mock 44 | private StatusLine statusLine; 45 | @Mock 46 | private PluginConfig pluginConfig; 47 | private KafkaWatch kafkaWatch; 48 | private String HTML = "TestSample "; 49 | 50 | @Before 51 | public void setUp() throws Exception { 52 | kafkaWatch = new KafkaWatch(); 53 | kafkaWatch.setId(RandomStringUtils.randomAlphanumeric(5)); 54 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.GREATER_THAN_OR_EQUAL_TO); 55 | kafkaWatch.setSubject("Random Kafka Watch"); 56 | kafkaWatch.setDateCreated(new Date()); 57 | kafkaWatch.setTriggerType(Enums.TriggerType.INDEX_OPS); 58 | kafkaWatch.setChannel(Arrays.asList("SMS", "EMAIL")); 59 | kafkaWatch.setDescription("Random Kafka Watch To Test"); 60 | kafkaWatch.setEventType("SUBSCRIPTION"); 61 | kafkaWatch.setReportTemplatePath("/home/malike/devfiles/report.jrxml"); 62 | kafkaWatch.setGenerateReport(true); 63 | kafkaWatch.setIndexName("Test"); 64 | kafkaWatch.setExpectedHit(0L); 65 | kafkaWatch.setReportFormat("HTML"); 66 | kafkaWatch.setRecipient(Arrays.asList("233201234567", "st.malike@gmail.com")); 67 | 68 | 69 | jsonResponse = new JSONResponse(); 70 | jsonResponse.setData(HTML); 71 | jsonResponse.setCount(1L); 72 | jsonResponse.setStatus(true); 73 | jsonResponse.setMessage("SUCCESS"); 74 | 75 | String response = new Gson().toJson(jsonResponse); 76 | StringEntity httpEntity = new StringEntity(response); 77 | httpEntity.setContentType("application/json"); 78 | httpEntity.setContentEncoding(Charsets.UTF_8.name()); 79 | httpEntity.setChunked(false); 80 | httpResponse = new BasicHttpResponse(new ProtocolVersion("HTTP", 1, 1), 200, "Test"); 81 | 82 | httpResponse.setStatusCode(200); 83 | httpResponse.setEntity(httpEntity); 84 | } 85 | 86 | @Test 87 | public void testGenerateReport() throws Exception { 88 | 89 | Mockito.when(statusLine.getStatusCode()).thenReturn(200); 90 | Mockito.doReturn(HTML).when(reportService).executeService(Mockito.any(), 91 | Mockito.any(), Mockito.any(), Mockito.any()); 92 | Mockito.doReturn(true).when(reportService).validateReportFile(kafkaWatch); 93 | 94 | 95 | Assert.assertTrue(reportService.getReport(kafkaWatch).equals(HTML)); 96 | 97 | } 98 | 99 | @Test(expected = TemplateFileNotFoundException.class) 100 | public void testGenerateReportTemplateFileNotFound() throws Exception { 101 | 102 | Mockito.when(httpClient.execute(Mockito.any())).thenReturn(httpResponse); 103 | 104 | reportService.getReport(kafkaWatch); 105 | } 106 | 107 | 108 | } 109 | -------------------------------------------------------------------------------- /src/test/java/st/malike/elasticsearch/kafka/watch/service/TimeTriggerServiceTest.java: -------------------------------------------------------------------------------- 1 | package st.malike.elasticsearch.kafka.watch.service; 2 | 3 | import org.apache.commons.lang.RandomStringUtils; 4 | import org.junit.After; 5 | import org.junit.Assert; 6 | import org.junit.Before; 7 | import org.junit.Test; 8 | import org.junit.runner.RunWith; 9 | import org.mockito.InjectMocks; 10 | import org.mockito.Mock; 11 | import org.mockito.Mockito; 12 | import org.mockito.Spy; 13 | import org.mockito.runners.MockitoJUnitRunner; 14 | import org.quartz.JobKey; 15 | import org.quartz.Scheduler; 16 | import org.quartz.SchedulerFactory; 17 | import org.quartz.impl.JobDetailImpl; 18 | import org.quartz.impl.triggers.CronTriggerImpl; 19 | import st.malike.elasticsearch.kafka.watch.model.KafkaWatch; 20 | import st.malike.elasticsearch.kafka.watch.util.Enums; 21 | 22 | import java.util.Arrays; 23 | import java.util.Date; 24 | import java.util.LinkedList; 25 | 26 | /** 27 | * @autor malike_st 28 | */ 29 | @RunWith(MockitoJUnitRunner.class) 30 | public class TimeTriggerServiceTest { 31 | 32 | @InjectMocks 33 | @Spy 34 | private TimeTriggerService timeTriggerService; 35 | @Mock 36 | private KafkaWatchService kafkaWatchService; 37 | @Mock 38 | private Scheduler scheduler; 39 | @Mock 40 | private SchedulerFactory schedulerFactory; 41 | private KafkaWatch kafkaWatch; 42 | 43 | 44 | @Before 45 | public void setUp() throws Exception { 46 | kafkaWatch = new KafkaWatch(); 47 | kafkaWatch.setCron("0/20 * * * * ?"); 48 | kafkaWatch.setId(RandomStringUtils.randomAlphanumeric(5)); 49 | kafkaWatch.setQuerySymbol(Enums.QuerySymbol.GREATER_THAN_OR_EQUAL_TO); 50 | kafkaWatch.setSubject("Random Kafka Watch"); 51 | kafkaWatch.setDateCreated(new Date()); 52 | kafkaWatch.setTriggerType(Enums.TriggerType.INDEX_OPS); 53 | kafkaWatch.setChannel(Arrays.asList("SMS", "EMAIL")); 54 | kafkaWatch.setDescription("Random Kafka Watch To Test"); 55 | kafkaWatch.setEventType("SUBSCRIPTION"); 56 | kafkaWatch.setIndexName("Test"); 57 | kafkaWatch.setExpectedHit(1L); 58 | kafkaWatch.setRecipient(Arrays.asList("233201234567", "st.malike@gmail.com")); 59 | 60 | } 61 | 62 | @After 63 | public void tearDown() throws Exception { 64 | 65 | } 66 | 67 | @Test 68 | public void schedule() throws Exception { 69 | 70 | Mockito.when(scheduler.isStarted()).thenReturn(true); 71 | Mockito.when(kafkaWatchService.findAllWatch()).thenReturn(new LinkedList()); 72 | timeTriggerService.schedule(); 73 | 74 | Assert.assertTrue(scheduler.isStarted()); 75 | 76 | } 77 | 78 | @Test 79 | public void addJob() throws Exception { 80 | 81 | Mockito.when(scheduler.getJobDetail(new JobKey(kafkaWatch.getId()))).thenReturn(new JobDetailImpl()); 82 | Mockito.when(scheduler.scheduleJob(Mockito.any(JobDetailImpl.class), Mockito.any(CronTriggerImpl.class))) 83 | .thenReturn(new Date()); 84 | 85 | Mockito.when(scheduler.getJobDetail(new JobKey(kafkaWatch.getId()))).thenReturn(new JobDetailImpl()); 86 | 87 | timeTriggerService.addJob(kafkaWatch); 88 | 89 | } 90 | 91 | 92 | @Test 93 | public void deleteJob() throws Exception { 94 | 95 | Mockito.when(scheduler.getJobDetail(new JobKey(kafkaWatch.getId()))).thenReturn(new JobDetailImpl()); 96 | 97 | timeTriggerService.deleteJob(kafkaWatch); 98 | 99 | } 100 | 101 | 102 | } -------------------------------------------------------------------------------- /src/test/resources/log4j.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | --------------------------------------------------------------------------------