kafka-cascade is a lightweight library built on top of kafkajs to provide automatic message reprocessing from services utilizing Kafka.
128 | The basic flow of how kafka-cascade operates is as follows:
129 | 
130 | Basic Usage
131 | To create a kafka-cascade service you need to import the library:
132 | const cascade = require('kafka');
133 | const service = cascade.service(kafka, 'example-topic', 'example-group', serviceCB, successCB, dlqCB);
134 |
135 | The kafka object is created from kafkajs and is passed into the service to create the consumer and producer required to process messages.
136 | Callbacks
137 | The service callback should have the following signature:
138 | serviceCB(msg, resolve, reject)
139 | msg is the kafka message that the consumer is listening for. resolve and reject are the callbacks to provide message reprocessing.
140 | resolve takes the msg as an argument and will call the success callback which can receive the msg as it's argument.
141 | reject has the following signature: reject(msg, status = ''). Reject will either reprocess the message or call the dead letter queue (DLQ) callback which can receive the msg as it's argument.
142 | Retry Strategies
143 | kafka-cascade supports two different retry strategies that can be set by calling .setDefaultRoute on the returned service.
144 | Levels: specifies how many times a message should be retried before calling the DLQ callback. Each level can have it's own retry strategy.
145 |
146 | - Timeout Limit: specifies how long to wait before retrying the message
147 | - Batch LimitL specifies how many messages the producer should wait for before sending all of the messages to be retried at once
148 |
149 | Routes
150 | kafka-cascade supports setting routes on the message retries for different status codes. By calling .setRoute on the service, kafka-cascade can utilize that route by suppling the status code when calling reject from the service callback
151 | Example
152 | const cascade = require('kafka-cascade');
153 |
154 | // create the service callback to simulate some time consumming task
155 | const serviceCB = (msg, resolve, reject) => {
156 | performTimeConsumingTask(msg)
157 | .then(res => resolve(res))
158 | .catch(error => {
159 | if(error === 'timeout') reject(msg, error);
160 | else reject(msg);
161 | });
162 | };
163 |
164 | // the service callback will be called with any arguments passed in through the resolve callback
165 | const successCB = (res) => {
166 | console.log('success:', res);
167 | };
168 |
169 | // the dead letter queue call back will be called when a message is out of retries
170 | const dlqCB = (msg) => {
171 | console.log('failed:', msg);
172 | }
173 |
174 | // create a service with previously established kafka object
175 | const service = await cascade.service(kafka, 'example-topic', 'example-group', serviceCB, successCB, dlqCB);
176 |
177 | // establish a default route with a 5 levels and a timeout strategy
178 | await service.setDefaultRoute(5, {timeoutLimit: [500, 1000, 2000, 4000, 8000]});
179 |
180 | // establish a 'timeout' route which goes directly to the DLQ
181 | await service.setRoute('timeout', 0);
182 |
183 | // connect and start the service
184 | await service.connect();
185 | await service.run();
186 |
187 | Events
188 | The following events can be registered onto the CascadeService object
189 | connect()
190 | Emitted when the service succesfully connects to Kafka
191 | disconnect()
192 | Emitted when the service succesfully connects to Kafka
193 | run()
194 | Emitted when the service succesfully executes consumer.run
195 | stop()
196 | Emitted when the service succesfully stops
197 | pause()
198 | Emitted when the service is paused
199 | resume()
200 | Emitted when the service is resumed
201 | receive(msg)
202 | Emitted when the service receives a Kafka message on the base topic
203 | success(msg)
204 | Emitted when the service callback resolves a message
205 | retry(msg)
206 | Emitted when a rejected message is retried
207 | dlq(msg)
208 | Emitted when a reject message is sent to the DLQ
209 | error(error)
210 | Emitted on error within the library
211 | serviceError(error)
212 | Emitted when a uncaught error occurs within the service callback
213 |