A Node.js client and a metric visualizer for ksqlDB
11 |
12 |
13 | ## About
14 |
15 | Need to run stream processing workloads on ksqlDB in Node.JS? Our lightweight **Node.js client** can help.
16 |
17 | Check out [ksQlient](./ksQlient/)
18 |
19 | Need to visualize ksqlDB query metrics to diagnose bottleneck issues? Try out our **metric visualizer**.
20 |
21 | Check out [ksqLight](./ksqLight/)
22 |
--------------------------------------------------------------------------------
/ksQlient/.npmignore:
--------------------------------------------------------------------------------
1 | #Test Files
2 | /__tests__/
3 | .yml
4 | /static/
5 |
--------------------------------------------------------------------------------
/ksQlient/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2022 OSLabs Beta
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/ksQlient/README.md:
--------------------------------------------------------------------------------
1 | # ksQlient (formerly ksqlDB-JS)
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
A native Node.js client for ksqlDB
10 |
11 |
12 | ## About the Project
13 |
14 | ksQlient is a **lightweight Node.js client for ksqlDB**, a database for streaming applications leveraging Kafka infrastructure under the hood.
15 |
16 | With our client, you can deploy stream-processing workloads on ksqlDB from within JS applications using simple, declarative SQL statements.
17 |
18 | Sample use cases:
19 |
20 | 1. Build applications that respond immediately to events.
21 | 2. Craft materialized views over streams.
22 | 3. Receive real-time push updates, or pull current state on demand.
23 |
24 | ## Table of Contents
25 |
26 | - [About the project](#about)
27 | - [Getting Started](#getting-started)
28 | - [Usage](#usage)
29 | - [Features](#features)
30 | - [Developers](#developers)
31 | - [Contributions](#contributions)
32 | - [License](#license)
33 |
34 | ## Getting Started
35 |
36 | The client is available on Node package manager (npm) ([link](https://www.npmjs.com/package/ksqlient))
37 |
38 | ```bash
39 |
40 | npm install ksqlient
41 |
42 | ```
43 |
44 | ## Usage
45 |
46 | Create a client in the application file:
47 |
48 | ```javascript
49 | const ksqldb = require("ksqlient");
50 | const client = new ksqldb({ ksqldbURL: "" });
51 | ```
52 |
53 | To run tests, initiate Docker containers included in yaml file:
54 |
55 | ```bash
56 | docker-compose up
57 | npm test
58 | ```
59 |
60 | ## Features
61 |
62 | ### Create a pull query
63 |
64 | ```javascript
65 | client.pull("SELECT * FROM myTable;");
66 | ```
67 |
68 | ### Create a push query (persistent query that subscribes to a stream)
69 |
70 | ```javascript
71 | client.push("SELECT * FROM myTable EMIT CHANGES;", (data) => {
72 | console.log(data);
73 | });
74 | ```
75 |
76 | ### Terminate persistent query (e.g. push query)
77 |
78 | ```javascript
79 | client.terminate(queryId);
80 | ```
81 |
82 | ### Insert rows of data into a stream
83 |
84 | ```javascript
85 | client.insertStream("myTable", [
86 | { name: "jack", email: "123@mail.com", age: 25 },
87 | { name: "john", email: "456@mail.com", age: 20 },
88 | ]);
89 | ```
90 |
91 | ### List streams/queries
92 |
93 | ```javascript
94 | client.ksql("LIST STREAMS;");
95 | ```
96 |
97 | ### Create table/streams
98 |
99 | ```javascript
100 | client.createStream(
101 | "testStream",
102 | (columnsType = ["name VARCHAR", "email varchar", "age INTEGER"]),
103 | (topic = "testTopic"),
104 | (value_format = "json"),
105 | (partitions = 1)
106 | );
107 | ```
108 |
109 | ### For custom SQL statements including complex joins use the .ksql method
110 |
111 | ```javascript
112 | client.ksql("DROP STREAM IF EXISTS testStream;");
113 | ```
114 |
115 | ### SQL Query builder
116 |
117 | Feel free to use the built-in query builder to parametrize any SQL query to avoid SQL injection.
118 |
119 | ```javascript
120 | const builder = new queryBuilder();
121 | const query = "SELECT * FROM table WHERE id = ? AND size = ?";
122 | const finishedQuery = builder.build(query, 123, "middle");
123 |
124 | client.ksql(finishedQuery);
125 | ```
126 |
127 | ### Create a table (materialized view) from a source stream
128 |
129 | ```javascript
130 | client.createTableAs(
131 | "testTable",
132 | "sourceStream",
133 | (selectArray = ["name", "LATEST_BY_OFFSET(age) AS recentAge"]),
134 | (propertiesObj = { topic: "newTestTopic" }),
135 | (conditionsObj = { WHERE: "age >= 21", GROUP_BY: "name" })
136 | );
137 | ```
138 |
139 | ### Create a stream based on an existing stream
140 |
141 | ```javascript
142 | client.createStreamAs(
143 | "testStream",
144 | (selectColumns = ["name", "age"]),
145 | "sourceStream",
146 | (propertiesObj = {
147 | kafka_topic: "testTopic",
148 | value_format: "json",
149 | partitions: 1,
150 | }),
151 | (conditions = "age > 50")
152 | );
153 | ```
154 |
155 | ### Pull stream data between two timestamps
156 |
157 | ```javascript
158 | client.pullFromTo(
159 | "TESTSTREAM",
160 | "America/Los_Angeles",
161 | (from = ["2022-01-01", "00", "00", "00"]),
162 | (to = ["2022-01-01", "00", "00", "00"])
163 | );
164 | ```
165 |
166 | ### Troubleshooting methods to inspect server metrics
167 |
168 | - inspectServerStatus
169 | - inspectQueryStatus
170 | - inspectClusterStatus
171 |
172 | ## Use Case
173 |
174 | We have built a [demo app](https://github.com/stabRabbitDemo/app) to showcase how ksQlient can be used to create a streaming application .
175 |
176 | ## Developers
177 |
178 | - Javan Ang - [GitHub](https://github.com/javanang) | [LinkedIn](https://www.linkedin.com/in/javanang/)
179 | - Michael Snyder - [GitHub](https://github.com/MichaelCSnyder) | [LinkedIn](https://www.linkedin.com/in/michaelcharlessnyder/)
180 | - Jonathan Luu - [GitHub](https://github.com/jonathanluu17) | [LinkedIn](https://www.linkedin.com/in/jonathanluu17/)
181 | - Matthew Xing - [GitHub](https://github.com/matthewxing1) | [LinkedIn](https://www.linkedin.com/in/matthew-xing/)
182 | - Gerry Bong - [GitHub](https://github.com/ggbong734) | [LinkedIn](https://www.linkedin.com/in/gerry-bong-71137420/)
183 |
184 | ## Contributions
185 |
186 | Contributions to the code, examples, documentation, etc. are very much appreciated.
187 |
188 | - Please report issues and bugs directly in this [GitHub project](https://github.com/oslabs-beta/ksqljs/issues).
189 |
190 | ## License
191 |
192 | This product is licensed under the MIT License - see the LICENSE.md file for details.
193 |
194 | This is an open source product.
195 |
196 | This product is accelerated by OS Labs.
197 |
198 | ksqlDB is licensed under the [Confluent Community License](https://github.com/confluentinc/ksql/blob/master/LICENSE).
199 |
200 | _Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the [Apache Software Foundation](https://www.apache.org/)_.
201 |
--------------------------------------------------------------------------------
/ksQlient/__tests__/integrationtests.js:
--------------------------------------------------------------------------------
1 | const ksqldb = require('../ksqldb/ksqldb');
2 | // Pre-requisite: start a docker container
3 | /* To add to README: Prior to running test with 'npm test', please start the ksqlDB
4 | server using the command 'docker-compose up'. This will spin up a ksqlDB server on
5 | 'http://localhost:8088'. If the command was run before, the created container might
6 | need to be removed first.
7 | */
8 |
9 | // ** INTEGRATION TEST INSTRUCTIONS **
10 |
11 | // Prior to running the test files, please ensure an instance of the ksqldb server is running
12 | // Steps to starting the ksqldb server can be found here: (https://ksqldb.io/quickstart.html)
13 | // Once the ksqlDB server is running, tests can be run with terminal line: (npm test)
14 |
15 | describe('--Integration Tests--', () => {
16 | describe('--Method Tests--', () => {
17 | beforeAll((done) => {
18 | client = new ksqldb({ ksqldbURL: 'http://localhost:8088' });
19 | done();
20 | });
21 |
22 | afterAll(async () => {
23 | await client.ksql('DROP STREAM IF EXISTS TESTJESTSTREAM DELETE TOPIC;');
24 | })
25 |
26 | it('.createStream properly creates a stream', async () => {
27 | await client.ksql('DROP STREAM IF EXISTS TESTJESTSTREAM DELETE TOPIC;')
28 | const result = await client.createStream('TESTJESTSTREAM', ['name VARCHAR', 'email varchar', 'age INTEGER'], 'testJestTopic', 'json', 1);
29 | const streams = await client.ksql('LIST STREAMS;');
30 | const allStreams = streams.streams;
31 | let streamExists = false;
32 | for (let i = 0; i < allStreams.length; i++) {
33 | if (allStreams[i].name === "TESTJESTSTREAM") {
34 | streamExists = true;
35 | break;
36 | }
37 | }
38 | expect(streamExists).toEqual(true);
39 | })
40 |
41 | it('.push properly creates a push query', async () => {
42 | let pushActive = false;
43 | await client.push('SELECT * FROM TESTJESTSTREAM EMIT CHANGES LIMIT 1;', async (data) => {
44 | if (JSON.parse(data).queryId) {
45 | pushActive = true;
46 | }
47 | expect(pushActive).toEqual(true)
48 | });
49 | })
50 |
51 | it('.terminate properly terminates a push query', () => {
52 | client.push('SELECT * FROM TESTJESTSTREAM EMIT CHANGES LIMIT 3;', async (data) => {
53 | const terminateRes = await client.terminate(JSON.parse(data).queryId);
54 | expect(terminateRes.wasTerminated).toEqual(true);
55 | })
56 | })
57 |
58 | it('.insertStream properly inserts a row into a stream', async () => {
59 |
60 | const data = [];
61 | await client.push('SELECT * FROM TESTJESTSTREAM EMIT CHANGES;', async (chunk) => {
62 | data.push(JSON.parse(chunk));
63 | if (data[1]) {
64 | client.terminate(data[0].queryId);
65 | expect(data[1]).toEqual(["stab-rabbit", "123@mail.com", 100])
66 | }
67 | });
68 | const response = await client.insertStream('TESTJESTSTREAM', [
69 | { "name": "stab-rabbit", "email": "123@mail.com", "age": 100 }
70 | ]);
71 | })
72 |
73 | it('.pull receives the correct data from a pull query', async () => {
74 | const pullData = await client.pull("SELECT * FROM TESTJESTSTREAM;");
75 | expect(pullData[1]).toEqual(["stab-rabbit", "123@mail.com", 100]);
76 | })
77 |
78 | it('.pullFromTo receives all the data', async () => {
79 | const pullData = await client.pull("SELECT * FROM TESTJESTSTREAM;");
80 | const data = await client.pullFromTo('TESTJESTSTREAM', 'America/Los_Angeles', ['2022-01-01', '00', '00', '00']);
81 | const expectPullData = pullData[1];
82 | const expectData = data[0].slice(0, 3);
83 | expect(expectPullData).toEqual(expectData);
84 | })
85 |
86 | describe('--Materialized Streams and Tables--', () => {
87 | beforeAll(async () => {
88 | await client.ksql('DROP STREAM IF EXISTS testAsStream;')
89 | await client.ksql('DROP TABLE IF EXISTS testAsTable;');
90 | await client.ksql('DROP STREAM IF EXISTS newTestStream DELETE TOPIC;');
91 | await client.createStream('newTestStream', ['name VARCHAR', 'age INTEGER'], 'newTestTopic', 'json', 1);
92 | });
93 |
94 | afterAll(async () => {
95 | await client.ksql('DROP STREAM IF EXISTS newTestStream DELETE TOPIC;');
96 | })
97 |
98 | describe('--Materialized Streams Tests--', () => {
99 | beforeAll(async () => {
100 | // await client.ksql('DROP STREAM IF EXISTS testAsStream;')
101 | // await client.ksql('DROP STREAM IF EXISTS newTestStream DELETE TOPIC;');
102 |
103 | // await client.createStream('newTestStream', ['name VARCHAR', 'age INTEGER'], 'newTestTopic', 'json', 1);
104 | testAsQueryId = await client.createStreamAs('testAsStream', ['name', 'age'], 'newTestStream', {
105 | kafka_topic: 'newTestTopic',
106 | value_format: 'json',
107 | partitions: 1
108 | }, 'age > 50');
109 | })
110 |
111 | afterAll(async () => {
112 | await client.ksql('DROP STREAM IF EXISTS testAsStream;')
113 | // await client.ksql('DROP STREAM IF EXISTS newTestStream DELETE TOPIC;');
114 | })
115 |
116 | it('creates materialized stream', async () => {
117 | let streamFound = false;
118 | const { streams } = await client.ksql('LIST STREAMS;');
119 |
120 | for (let i = 0; i < streams.length; i++) {
121 | if (streams[i].name, streams[i].name === 'TESTASSTREAM') {
122 | streamFound = true;
123 | break;
124 | }
125 | }
126 | expect(streamFound).toBe(true);
127 | });
128 | });
129 |
130 |
131 | describe('--Materialized Tables Tests--', () => {
132 | beforeAll(async () => {
133 | await client.createTableAs('testAsTable', 'newTestStream', ['name', 'LATEST_BY_OFFSET(age) AS recentAge'], { topic: 'newTestTopic' }, { WHERE: 'age >= 21', GROUP_BY: 'name' });
134 | });
135 | afterAll(async () => {
136 | await client.ksql('DROP TABLE IF EXISTS testAsTable;');
137 | // await client.ksql('DROP TABLE IF EXISTS TABLEOFSTREAM DELETE TOPIC;')
138 | // await client.ksql('DROP STREAM IF EXISTS NEWTESTSTREAM DELETE TOPIC;')
139 | })
140 |
141 | it('creates a materialized table view of a stream', async () => {
142 | const { tables } = await client.ksql('LIST TABLES;');
143 | let tableFound = false;
144 | for (let i = 0; i < tables.length; i++) {
145 | if (tables[i].name === 'TESTASTABLE') {
146 | tableFound = true;
147 | break;
148 | }
149 | }
150 | expect(tableFound).toEqual(true);
151 | })
152 | })
153 | })
154 | })
155 |
156 |
157 | describe('--Health Tests--', () => {
158 | beforeAll((done) => {
159 | client = new ksqldb({ ksqldbURL: 'http://localhost:8088' });
160 | done();
161 | });
162 |
163 | afterAll(async () => {
164 | await client.ksql('DROP STREAM IF EXISTS TESTSTREAM2;');
165 | })
166 |
167 | it('.inspectQueryStatus checks if a stream is created successfully', async () => {
168 | const streamName = 'TESTSTREAM2'
169 | const create = await client.ksql(`CREATE STREAM IF NOT EXISTS ${streamName}
170 | (name VARCHAR,
171 | email varchar,
172 | age INTEGER)
173 | WITH (
174 | KAFKA_TOPIC = 'testJestTopic',
175 | VALUE_FORMAT = 'json',
176 | PARTITIONS = 1
177 | );`);
178 | const commandId = create ? create.commandId : `stream/${streamName}/create`;
179 | const status = await client.inspectQueryStatus(commandId);
180 | // response should be { status: 'SUCCESS', message: 'Stream created', queryId: null }
181 | expect(status.data).toEqual(expect.objectContaining({
182 | status: expect.any(String),
183 | message: expect.any(String),
184 | queryId: null
185 | }));
186 | })
187 |
188 | it('.inspectServerInfo returns the server info and status', async () => {
189 | const status = await client.inspectServerInfo();
190 | // should return something like: {
191 | // KsqlServerInfo: {
192 | // version: '0.25.1',
193 | // kafkaClusterId: '0Yxd6N5OSKGDUalltPWvXg',
194 | // ksqlServiceId: 'default_',
195 | // serverStatus: 'RUNNING'
196 | // }
197 | // }
198 | expect(status.data).toEqual(expect.objectContaining({
199 | KsqlServerInfo: expect.objectContaining({
200 | version: expect.any(String),
201 | kafkaClusterId: expect.any(String),
202 | serverStatus: expect.any(String)
203 | })
204 | }));
205 | })
206 |
207 | it('.inspectServerHealth returns the server health', async () => {
208 | const status = await client.inspectServerHealth();
209 | // should return something like: {
210 | // isHealthy: true,
211 | // details: {
212 | // metastore: { isHealthy: true },
213 | // kafka: { isHealthy: true },
214 | // commandRunner: { isHealthy: true }
215 | // }
216 | // }
217 | expect(status.data).toEqual(expect.objectContaining({
218 | isHealthy: expect.any(Boolean),
219 | details: expect.objectContaining({
220 | metastore: expect.anything(),
221 | kafka: expect.anything(),
222 | commandRunner: expect.anything()
223 | })
224 | })
225 | );
226 | })
227 |
228 | it('.inspectClusterStatus returns the cluster status', async () => {
229 | const status = await client.inspectClusterStatus();
230 | // should return something like: {
231 | // clusterStatus: {
232 | // 'ksqldb-server:8088': {
233 | // hostAlive: true,
234 | // lastStatusUpdateMs: 1653164479237,
235 | // activeStandbyPerQuery: [Object],
236 | // hostStoreLags: [Object]
237 | // }
238 | // }}
239 | expect(status.data).toEqual(expect.objectContaining({
240 | clusterStatus: expect.anything()
241 | })
242 | );
243 | })
244 |
245 | it('.isValidProperty returns true if a server configuration property is not prohibited from setting', async () => {
246 | const status = await client.isValidProperty('test');
247 | // should return true
248 | expect(status.data).toEqual(true);
249 | })
250 |
251 | // it('isValidProperty returns an error if the server property is prohibited from setting', async () => {
252 | // const status = await client.isValidProperty('ksql.connect.url');
253 | // // should return something like
254 | // // {
255 | // // "@type": "generic_error",
256 | // // "error_code": 40000,
257 | // // "message": "One or more properties overrides set locally are prohibited by the KSQL server (use UNSET to reset their default value): [ksql.service.id]"
258 | // // }
259 | // expect(status.data).toEqual(expect.objectContaining({
260 | // type: expect.any(String),
261 | // error_code: expect.any(Number),
262 | // message: expect.any(String),
263 | // }));
264 | // })
265 | })
266 | })
267 |
--------------------------------------------------------------------------------
/ksQlient/__tests__/queryBuilderTests.js:
--------------------------------------------------------------------------------
1 | const queryBuilder = require('../ksqldb/queryBuilder.js');
2 | const { QueryBuilderError, EmptyQueryError, NumParamsError, InappropriateStringParamError } = require('../ksqldb/customErrors.js');
3 |
4 |
5 | describe('Unit tests for query builder class', () => {
6 | let builder;
7 | let query;
8 |
9 | beforeAll((done) => {
10 | builder = new queryBuilder();
11 | query = 'SELECT * FROM table WHERE id = ? AND size = ?';
12 | done();
13 | });
14 |
15 | describe('Normal use case', () => {
16 | it('properly adds params into a sql query', () => {
17 | const finishedQuery = builder.build(query, 123, "middle");
18 | expect(finishedQuery).toEqual("SELECT * FROM table WHERE id = 123 AND size = 'middle'");
19 | });
20 | });
21 |
22 | describe('Error testing', () => {
23 | it('throws an error if number of params is different from number of question marks', () => {
24 | expect(() => { builder.build(query, 123, "middle", "extra") }).toThrow(NumParamsError);
25 | });
26 |
27 | it('throws an error if the query is empty', () => {
28 | const query = '';
29 | expect(() => { builder.build(query, 123) }).toThrow(EmptyQueryError);
30 | });
31 |
32 | it('throws an error if an object is passed in as a param', () => {
33 | expect(() => { builder.build(query, 123, { "middle": "size" }) }).toThrow(QueryBuilderError);
34 | });
35 | });
36 |
37 | describe('SQL injection', () => {
38 | it("prevents 'OR 1=1' SQL injection by escaping single quotations ", () => {
39 | // https://stackoverflow.com/questions/5139770/escape-character-in-sql-server
40 | const finishedQuery = builder.build(query, 123, "middle' OR 1=1");
41 | expect(finishedQuery).toEqual("SELECT * FROM table WHERE id = 123 AND size = 'middle'' OR 1=1'");
42 | });
43 |
44 | it("prevents (middle' OR 'a'=a') SQL injection by escaping single quotations ", () => {
45 | const finishedQuery = builder.build(query, 123, "middle' OR 'a'='a",);
46 | expect(finishedQuery).toEqual("SELECT * FROM table WHERE id = 123 AND size = 'middle'' OR ''a''=''a'");
47 | });
48 |
49 | it("throws an error if user tries to add a semicolon into a string param not wrapped in quotes", () => {
50 | expect(() => { builder.build(query, ['123; DROP tables WHERE size = '], 'middle') }).toThrow(InappropriateStringParamError);
51 | });
52 | });
53 | });
54 |
--------------------------------------------------------------------------------
/ksQlient/docker-compose.yml:
--------------------------------------------------------------------------------
1 | ---
2 | version: "3.2"
3 |
4 | services:
5 | zookeeper:
6 | image: confluentinc/cp-zookeeper:7.0.1
7 | hostname: zookeeper
8 | container_name: zookeeper
9 | ports:
10 | - "2181:2181"
11 | environment:
12 | ZOOKEEPER_CLIENT_PORT: 2181
13 | ZOOKEEPER_TICK_TIME: 2000
14 |
15 | broker:
16 | image: confluentinc/cp-kafka:7.0.1
17 | hostname: broker
18 | container_name: broker
19 | depends_on:
20 | - zookeeper
21 | ports:
22 | - "29092:29092"
23 | environment:
24 | KAFKA_BROKER_ID: 1
25 | KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
26 | KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
27 | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
28 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
29 | KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
30 | KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
31 | KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
32 |
33 | ksqldb-server:
34 | image: confluentinc/ksqldb-server:0.25.1
35 | hostname: ksqldb-server
36 | container_name: ksqldb-server
37 | volumes:
38 | - type: bind
39 | source: ./
40 | target: /home/appuser
41 | depends_on:
42 | - broker
43 | ports:
44 | - "8088:8088"
45 | - "1090:1099"
46 | environment:
47 | KSQL_LISTENERS: http://0.0.0.0:8088
48 | KSQL_BOOTSTRAP_SERVERS: broker:9092
49 | KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
50 | KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
51 | # KSQL_KSQL_OPTS: "-Djava.security.auth.login.config=/jaas_config.file"
52 | # KSQL_AUTHENTICATION_METHOD: BASIC
53 | # KSQL_AUTHENTICATION_REALM: KsqlServer-Props
54 | # KSQL_AUTHENTICATION_ROLES: admin,developer,user
55 | # KSQL_SSL_CLIENT_AUTHENTICATION: NONE
56 | # KSQL_SSL_TRUSTSTORE_LOCATION: ksqldb_server_config/kafka.server.truststore.jks
57 | # KSQL_SSL_TRUSTSTORE_PASSWORD: ${SSL_PASSWORD}
58 | # KSQL_SSL_KEYSTORE_LOCATION: ksqldb_server_config/kafka.server.keystore.jks
59 | # KSQL_SSL_KEYSTORE_PASSWORD: ${SSL_PASSWORD}
60 | # KSQL_SSL_KEY_PASSWORD: ${SSL_PASSWORD}
61 | KSQL_KSQL_HEARTBEAT_ENABLE: "true"
62 | KSQL_KSQL_LAG_REPORTING_ENABLE: "true"
63 |
64 | ksqldb-cli:
65 | image: confluentinc/ksqldb-cli:0.25.1
66 | container_name: ksqldb-cli
67 | depends_on:
68 | - broker
69 | - ksqldb-server
70 | entrypoint: /bin/sh
71 | tty: true
--------------------------------------------------------------------------------
/ksQlient/ksqldb/customErrors.js:
--------------------------------------------------------------------------------
1 | /** This file contains custom Node.js errors that extends the built-in Error class
2 | * REF: https://rclayton.silvrback.com/custom-errors-in-node-js
3 | * REF: https://futurestud.io/tutorials/node-js-create-your-custom-error
4 | */
5 |
6 | // used to wrap error received from ksqlDB server
7 | class ksqlDBError extends Error {
8 | constructor(error) {
9 | super(error.message)
10 |
11 | // Ensure the name of this error is the same as the class name
12 | this.name = this.constructor.name
13 |
14 | // capturing the stack trace keeps the reference to your error class
15 | Error.captureStackTrace(this, this.constructor);
16 |
17 | // you may also assign additional properties to your error
18 | //this.status = 404
19 | Object.keys(error).forEach(property => {this[property] = error[property]});
20 | }
21 | }
22 |
23 | // for returning error related to use of queryBuilder class
24 | class QueryBuilderError extends Error {
25 | constructor(message) {
26 | super(message)
27 | this.name = this.constructor.name
28 | Error.captureStackTrace(this, this.constructor);
29 | }
30 | }
31 |
32 | class EmptyQueryError extends QueryBuilderError {
33 | constructor() {
34 | super('Query should not be empty, undefined, or null');
35 | }
36 | }
37 |
38 | class NumParamsError extends QueryBuilderError {
39 | constructor(message) {
40 | super(message);
41 | }
42 | }
43 |
44 | class InappropriateStringParamError extends QueryBuilderError {
45 | constructor(message) {
46 | super(message);
47 | }
48 | }
49 |
50 | class invalidArgumentTypes extends Error {
51 | constructor(message) {
52 | super(message);
53 |
54 | this.name = this.constructor.name;
55 | // necessary?
56 | Error.captureStackTrace(this, this.constructor);
57 |
58 | }
59 | }
60 |
61 | module.exports = {
62 | ksqlDBError,
63 | QueryBuilderError,
64 | EmptyQueryError,
65 | NumParamsError,
66 | InappropriateStringParamError,
67 | invalidArgumentTypes
68 | };
69 |
--------------------------------------------------------------------------------
/ksQlient/ksqldb/ksqldb.js:
--------------------------------------------------------------------------------
1 | const axios = require("axios");
2 | const https = require('node:https');
3 | const http2 = require("http2");
4 | const { ksqlDBError } = require("./customErrors.js");
5 | const validateInputs = require('./validateInputs.js');
6 | const queryBuilder = require('./queryBuilder.js');
7 | const builder = new queryBuilder();
8 |
9 | class ksqldb {
10 | /**
11 | * Constructor
12 | * @param {object} config
13 | *
14 | * Config object can have these properties
15 | *
16 | * ksqldbURL: Connection URL or address
17 | *
18 | * API: Username or API key for basic authentication
19 | *
20 | * secret: Password or secret for basic authentication
21 | *
22 | * httpsAgent: httpsAgent for setting TLS properties
23 | */
24 | constructor(config) {
25 | this.ksqldbURL = config.ksqldbURL;
26 | this.API = config.API ? config.API : null;
27 | this.secret = config.secret ? config.secret : null;
28 | this.httpsAgentAxios = config.httpsAgent ? new https.Agent(config.httpsAgent) : null;
29 | this.httpsAgentHttp2 = config.httpsAgent ? config.httpsAgent : null;
30 | }
31 |
32 | /**
33 | * Executes a pull query and returns the results.
34 | *
35 | *
This method may be used to execute pull queries, and returns an array containing all the data
36 | * received. The first value of the array will be an object containing ->
37 | * queryId: a string that contains the id of the stream that the pull query is being executed upon.
38 | * columnNames: an array that contains the names of the columns in the format of strings.
39 | * columnTypes: an array that contains the names of the columnTypes in the format of strings.
40 | *
41 | * Any subsequent values of the array are arrays that contain the data received.
42 | *
43 | *
If user input is used to build the query, please use the queryBuilder method to protect against sql injection.
44 | *
45 | * @param {string} query sql statement of query to execute
46 | * @return {Promise} a promise that completes once the server response is received, and contains the query
47 | * result if successful.
48 | *
49 | * Example: [{object that contains the metadata}, [data], [data], ...}]
50 | */
51 | pull = (query) => {
52 | validateInputs([query, 'string', 'query']);
53 |
54 | const validatedQuery = builder.build(query);
55 |
56 | return axios
57 | .post(this.ksqldbURL + "/query-stream",
58 | {
59 | sql: validatedQuery,
60 | },
61 | {
62 | headers:
63 | this.API && this.secret ?
64 | {
65 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
66 | }
67 | :
68 | {},
69 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
70 | })
71 | .then((res) => res.data)
72 | .catch((error) => {
73 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
74 | });
75 | }
76 |
77 | /**
78 | * Executes a push query, and returns the results one row at a time.
79 | *
80 | *
This method may be used to issue a push query against a stream, with the first piece of data being an object
81 | * containing queryId, the id of the push query that can be used to terminate the push query being executed.
82 | * Otherwise, the push query will continuously run until terminated, returning results one at a time.
83 | *
84 | *
If user input is used to build the query, please use the queryBuilder method to protect against sql injection.
85 | *
86 | * @param {string} query sql statement of query to execute
87 | * @param {function} cb a callback function that is ran against each piece of data returned.
88 | * @return {Promise} a promise that completes once the server response is received, and contains the query
89 | * result if successful.
90 | */
91 | push(query, cb) {
92 | validateInputs([query, 'string', 'query', true], [cb, 'function', 'cb', true]);
93 | const validatedQuery = builder.build(query);
94 |
95 | return new Promise((resolve, reject) => {
96 | let sentQueryId = false;
97 | const session = http2.connect(
98 | this.ksqldbURL,
99 | this.httpsAgentHttp2 ? this.httpsAgentHttp2 : {}
100 | );
101 |
102 | session.on("error", (err) => reject(err));
103 |
104 | const req = session.request(
105 | this.secret && this.API ?
106 | {
107 | ":path": "/query-stream",
108 | ":method": "POST",
109 | "Authorization": this.API && this.secret ? `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}` : '',
110 | }
111 | :
112 | {
113 | ":path": "/query-stream",
114 | ":method": "POST",
115 | }
116 | );
117 |
118 | const reqBody = {
119 | sql: validatedQuery,
120 | Accept: "application/json, application/vnd.ksqlapi.delimited.v1",
121 | };
122 |
123 | req.write(JSON.stringify(reqBody), "utf8");
124 | req.end();
125 | req.setEncoding("utf8");
126 |
127 | req.on("data", (chunk) => {
128 | // check for chunk containing errors
129 | if (JSON.parse(chunk)['@type']?.includes('error')) throw new ksqlDBError(JSON.parse(chunk));
130 | // continue if chunk indicates a healthy response
131 | if (!sentQueryId) {
132 | sentQueryId = true;
133 | cb(chunk);
134 | resolve(JSON.parse(chunk)?.queryId)
135 | }
136 | else {
137 | cb(chunk);
138 | }
139 | });
140 |
141 | req.on("end", () => session.close());
142 | })
143 | }
144 |
145 | /**
146 | * Executes a terminate query that ends a push query.
147 | *
148 | *
This method may be used to end an active push query, and returns an object signifying whether the push query was
149 | * terminated properly or not.
150 | *
151 | *
This method is sql injection protected with the use of queryBuilder.
152 | *
153 | * @param {string} queryId a string that is the id of the push query to be terminated.
154 | * @return {Promise} a promise that completes once the server response is received, and is an object that signifies
155 | * if the termination was successful.
156 | */
157 | terminate(queryId) {
158 | validateInputs([queryId, 'string', 'queryId']);
159 |
160 | return axios.post(this.ksqldbURL + '/ksql',
161 | {
162 | ksql: `TERMINATE ${queryId};`
163 | },
164 | {
165 | headers:
166 | this.API && this.secret ?
167 | {
168 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
169 | }
170 | :
171 | {},
172 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
173 | })
174 | .then(res => res.data[0])
175 | .catch(error => {
176 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
177 | });
178 | }
179 |
180 | /**
181 | * Executes a query and returns the result(s).
182 | *
183 | *
This method may be used to issue custom sql queries against ksqldb without constraints.
184 | *
185 | *
If user input is used to build the query, please use the queryBuilder method to protect against sql injection.
186 | *
187 | * @param {string} query statement of a query to execute.
188 | * @return {Promise} a promise that completes once the server response is received, and returns the requested data.
189 | */
190 | ksql(query) {
191 | validateInputs([query, 'string', 'query']);
192 |
193 | const validatedQuery = builder.build(query);
194 |
195 | return axios.post(this.ksqldbURL + '/ksql',
196 | {
197 | ksql: validatedQuery
198 | },
199 | {
200 | headers:
201 | this.API && this.secret ?
202 | {
203 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
204 | }
205 | :
206 | {},
207 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
208 | })
209 | .then(res => res.data[0])
210 | .catch(error => {
211 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
212 | });
213 | }
214 |
215 | /**
216 | * Executes a query to create a stream.
217 | *
218 | *
This method is used to create a stream.
219 | *
220 | *
This method is sql injection protected with the use of queryBuilder.
221 | *
222 | * @param {string} name the name of the stream to be created.
223 | * @param {array} columnsType an array that contains the name of the columns and the associated types e.g [name VARCHAR, age INTEGER, ...]
224 | * @param {string} topic the name of the topic the stream is listening to. The topic is created if it does not currently exist.
225 | * @param {string} value_format a string specifying the value format.
226 | * @param {integer} partitions the number of partitions the stream should have.
227 | * @param {integer} key the key of the string.
228 | * @return {Promise} a promise that completes once the server response is received, and returns a response object.
229 | */
230 | createStream(name, columnsType, topic, value_format = 'json', partitions = 1, key) {
231 | validateInputs([name, 'string', 'name', true], [columnsType, 'array', 'columnsType', true], [topic, 'string', 'topic'], [partitions, 'number', 'partitions']);
232 |
233 | const columnsTypeString = columnsType.reduce((result, currentType) => result + ', ' + currentType);
234 | const query = `CREATE STREAM ${name} (${columnsTypeString}) WITH (kafka_topic='${topic}', value_format='${value_format}', partitions=${partitions});`;
235 |
236 | return axios.post(this.ksqldbURL + '/ksql', { ksql: query }, {
237 | headers:
238 | this.API && this.secret ?
239 | {
240 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
241 | }
242 | :
243 | {},
244 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
245 | })
246 | .catch(error => {
247 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
248 | });
249 | }
250 |
251 | /**
252 | *
253 | * @param {string} streamName - the name of the stream to be created
254 | * @param {string[]} selectColumns - the columns from the underlying stream to be included in the new materialized stream
255 | * @param {string} sourceStream - the underlying stream from which the new materialized stream will be created
256 | * @param {object} propertiesObj - an object whose keys are property names and values are the associated values
257 | * @param {string} conditions - a string containing the conditional statement (i.e., the 'WHERE' statement)
258 | * @param {string} partitionBy - column by which data will be distributed
259 | * @returns {Promise} - a promise that completes once the server response is received, and returns a query ID
260 | */
261 | createStreamAs = (streamName, selectColumns, sourceStream, propertiesObj, conditions, partitionBy) => {
262 | validateInputs([streamName, 'string', 'streamName', true], [selectColumns, 'array', 'selectColumns', true], [sourceStream, 'string', 'sourceStream', true], [propertiesObj, 'object', 'propertiesObj'], [conditions, 'string', 'conditions'], [partitionBy, 'string', 'partitionBy']);
263 |
264 | const propertiesArgs = [];
265 | const selectColStr = selectColumns.reduce((result, current) => result + ', ' + current);
266 | // begin with first consistent portion of query
267 | let builderQuery = 'CREATE STREAM ? ';
268 |
269 | // include properties in query if provided
270 | if (Object.keys(propertiesObj).length > 0) {
271 | builderQuery += 'WITH (';
272 | for (const [key, value] of Object.entries(propertiesObj)) {
273 | const justStarted = builderQuery[builderQuery.length - 1] === '(';
274 |
275 | if (!justStarted) builderQuery += ', ';
276 | builderQuery += '? = ?';
277 | propertiesArgs.push([key], value);
278 | };
279 | builderQuery += ') ';
280 | }
281 |
282 | // continue building the query to be sent to the builder
283 | builderQuery += `AS SELECT ${selectColStr} FROM ? `;
284 | if (conditions.indexOf(';') === -1) builderQuery += `WHERE ${conditions} `;
285 | builderQuery += partitionBy || '';
286 | builderQuery += 'EMIT CHANGES;'
287 |
288 | // utilize query with variables to build actual query
289 | const query = builder.build(builderQuery, [streamName], ...propertiesArgs, [sourceStream]);
290 |
291 | return axios.post(this.ksqldbURL + '/ksql', { ksql: query }, {
292 | headers:
293 | this.API && this.secret ?
294 | {
295 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
296 | }
297 | :
298 | {},
299 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
300 | })
301 | .then(res => res.data[0].commandStatus.queryId)
302 | .catch(error => {
303 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
304 | });
305 | }
306 |
307 | //---------------------Create tables-----------------
308 | /**
309 | * Executes a query to create a table.
310 | *
311 | *
This method is used to create a table.
312 | *
313 | *
This method is sql injection protected with the use of queryBuilder.
314 | *
315 | * @param {string} name the name of the table to be created.
316 | * @param {array} columnsType an array that contains the name of the columns and the associated types e.g [name VARCHAR, age INTEGER, ...]
317 | * @param {string} topic name of the topic the table is listening to. The topic is created if it does not currently exist.
318 | * @param {string} value_format string specifying the value format.
319 | * @param {integer} partitions number of partitions the table should have.
320 | * @return {Promise} a promise that completes once the server response is received, and returns a response object.
321 | */
322 | createTable = (name, columnsType, topic, value_format = 'json', partitions) => {
323 | validateInputs([name, 'string', 'name', true], [columnsType, 'array', 'columnsType', true], [topic, 'string', 'topic', true], [value_format, 'string', 'value_format', true], [partitions, 'number', 'partitions']);
324 |
325 | const columnsTypeString = columnsType.reduce((result, currentType) => result + ', ' + currentType);
326 | const query = `CREATE TABLE ${name} (${columnsTypeString}) WITH (kafka_topic='${topic}', value_format='${value_format}', partitions=${partitions});`
327 |
328 | axios.post(this.ksqldbURL + '/ksql',
329 | {
330 | ksql: query
331 | },
332 | {
333 | headers:
334 | this.API && this.secret ?
335 | {
336 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
337 | }
338 | :
339 | {},
340 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
341 | })
342 | .catch(error => {
343 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
344 | });
345 | }
346 |
347 | //---------------------Create tables as select-----------------
348 | /**
349 | * Execute a query to create a new materialized table view of an existing table or stream
350 | *
351 | *
This method is used to create a materialized table view
352 | *
353 | *
This method is sql injection protected with the use of queryBuilder.
354 | *
355 | * @param {string} tableName name of the table to be created
356 | * @param {string} source name of the source stream / table materialized view is based on
357 | * @param {array} selectArray an array that contains the values (strings, aggregate functions) of the columns for the materialized view table
358 | * @param {object} propertiesObj an object containing key value pairs for supported table properties e.g {topic: 'myTopic', value_format: 'json', partitions: '1'}. {} for default values
359 | * @param {object} conditionsObj an object containing key value pairs for supported query conditions e.g {WHERE: 'a is not null', GROUP_BY: 'profileID', HAVING: 'COUNT(a) > 5' }
360 | * @returns {Promise} a promise that completes once the server response is received, returning a response object
361 | */
362 | createTableAs = (tableName, source, selectArray, propertiesObj, conditionsObj) => {
363 | validateInputs([tableName, 'string', 'tableName', true], [source, 'string', 'source', true], [selectArray, 'array', 'selectArray', true], [propertiesObj, 'object', 'propertiesObj'], [conditionsObj, 'object', 'conditionsObj']);
364 |
365 | let selectColStr = selectArray.reduce((result, current) => result + ', ' + current);
366 |
367 | // expect user to input properties object of format {topic: ... , value_format: ..., partitions: ...}
368 | // check for properties object, look for properties, if any are missing assign it a default value, if there's no property
369 | const defaultProps = {
370 | topic: tableName,
371 | value_format: 'json',
372 | partitions: 1
373 | };
374 | Object.assign(defaultProps, propertiesObj);
375 |
376 | // if there's no properties Obj, assign them all default values
377 |
378 | // expect user to input a conditions object of format {WHERE: condition, GROUP_BY: condition, HAVING: condition};
379 | // generate conditions string based on object
380 | // const builder = new queryBuilder();
381 |
382 | let conditionQuery = '';
383 | if (conditionsObj) {
384 | const conditionsArr = ['WHERE', 'GROUP_BY', 'HAVING'];
385 | const sqlClauses = [];
386 |
387 | let i = 0;
388 | while (conditionsArr.length) {
389 | if (conditionsObj[conditionsArr[0]]) {
390 | sqlClauses[i] = [conditionsArr[0].replace('_', ' ')]; // clause values are set as arrays for query builder
391 | sqlClauses[i + 1] = [' ' + conditionsObj[conditionsArr[0]] + ' '];
392 | }
393 | else {
394 | sqlClauses[i] = [''];
395 | sqlClauses[i + 1] = [''];
396 | }
397 | i += 2;
398 | conditionsArr.shift()
399 | }
400 | conditionQuery = builder.build(`${sqlClauses[0][0]}${sqlClauses[1][0]}??${sqlClauses[4][0]}${sqlClauses[5][0]}`, sqlClauses[2], sqlClauses[3]);
401 | }
402 |
403 | // reformat for builder
404 | tableName = [tableName];
405 | selectColStr = [selectColStr];
406 | source = [source];
407 | conditionQuery = [conditionQuery]
408 |
409 |
410 | const query = builder.build(`CREATE TABLE ? WITH (kafka_topic=?, value_format=?, partitions=?) AS SELECT ? FROM ? ${conditionQuery} EMIT CHANGES;`, tableName, defaultProps.topic, defaultProps.value_format, defaultProps.partitions, selectColStr, source)
411 | return axios.post(this.ksqldbURL + '/ksql', { ksql: query }, {
412 | headers:
413 | this.API && this.secret ?
414 | {
415 | "Authorization": `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}`,
416 | }
417 | :
418 | {},
419 | httpsAgent: this.httpsAgentAxios ? this.httpsAgentAxios : null,
420 | })
421 | .catch(error => console.log(error));
422 | }
423 |
424 | /**
425 | * Inserts rows of data into a stream.
426 | *
427 | *
This method may be used to insert new rows of data into a stream.
428 | *
429 | *
This method is sql injection protected with the use of queryBuilder.
430 | *
431 | * @param {string} stream the name of the stream to insert data into.
432 | * @param {object} rows an array that contains data that is being inserted into the stream.
433 | * @return {Promise} this method returns a promise that resolves into an array describing the status of the row inserted.
434 | */
435 | //---------------------Insert Rows Into Existing Streams-----------------
436 | insertStream = (stream, rows) => {
437 | validateInputs([stream, 'string', 'stream', true], [rows, 'array', 'rows', true]);
438 |
439 | return new Promise((resolve, reject) => {
440 | const msgOutput = [];
441 |
442 | const session = http2.connect(
443 | this.ksqldbURL,
444 | this.httpsAgentHttp2 ? this.httpsAgentHttp2 : {}
445 | );
446 |
447 | session.on("error", (err) => reject(err));
448 |
449 | const req = session.request(
450 | this.secret && this.API ?
451 | {
452 | ":path": "/inserts-stream",
453 | ":method": "POST",
454 | "Authorization": this.API && this.secret ? `Basic ${Buffer.from(this.API + ":" + this.secret, 'utf8').toString('base64')}` : '',
455 | }
456 | :
457 | {
458 | ":path": "/inserts-stream",
459 | ":method": "POST",
460 | }
461 | );
462 |
463 | let reqBody = `{ "target": "${stream}" }`;
464 |
465 | for (let row of rows) {
466 | reqBody += `\n${JSON.stringify(row)}`;
467 | }
468 |
469 | req.write(reqBody, "utf8");
470 | req.end();
471 | req.setEncoding("utf8");
472 |
473 | req.on("data", (chunk) => {
474 | // check for chunk containing errors
475 | if (JSON.parse(chunk)['@type']?.includes('error')) throw new ksqlDBError(JSON.parse(chunk));
476 | // continue if chunk indicates a healthy response
477 | msgOutput.push(JSON.parse(chunk));
478 | });
479 |
480 | req.on("end", () => {
481 | session.close();
482 | resolve(msgOutput);
483 | });
484 | })
485 | }
486 |
487 | /**
488 | * Pulls data between two different time points.
489 | *
490 | *
This method may be used to pull data from within two specific points in time. The first three
491 | * parameters are required, with the fourth parameter being optional.
492 | *
493 | *
This method is sql injection protected with the use of queryBuilder.
494 | *
495 | * @param {string} streamName the name of the stream to pull data from.
496 | * @param {string} timeZone desired timezone that the data should conform to.
497 | * @param {array} from array of the format ['2200-01-01', '16', '10', '20'], with the values being
498 | * date, hour, minute, and second respectively.
499 | * @param {array} to array of the format ['2000-01-01', '16', '10', '20'], with the values being
500 | * date, hour, minute, and second respectively. This defaults to ['2200-03-14', '00', '00', '00'].
501 | * @return {array} this method returns an array that contains arrays with the data, along with an extra value at
502 | * the end of the array that includes the time that the data was inserted into the ksqldb.
503 | */
504 | pullFromTo = async (streamName, timezone = 'Greenwich', from = [undefined, '00', '00', '00'], to = ['2200-03-14', '00', '00', '00']) => {
505 | validateInputs([streamName, 'string', 'streamName', true], [timezone, 'string', 'timezone', true], [from, 'array', 'from', true], [to, 'array', 'to', true]);
506 |
507 | const userFrom = `${from[0]}T${from[1]}:${from[2]}:${from[3]}`;
508 | const userTo = `${to[0]}T${to[1]}:${to[2]}:${to[3]}`;
509 | const userFromUnix = new Date(userFrom).getTime();
510 | const userToUnix = new Date(userTo).getTime();
511 | const query = builder.build("SELECT *, CONVERT_TZ(FROM_UNIXTIME(ROWTIME), 'UTC', ?) AS DATE, ROWTIME FROM ?;", timezone, [streamName]);
512 | const data = await this.pull(query);
513 | data.shift();
514 | const filtered = [];
515 | data.map((element) => {
516 | if (element[element.length - 1] >= userFromUnix && element[element.length - 1] <= userToUnix) {
517 | filtered.push(element.slice(0, element.length - 1));
518 | }
519 | })
520 | return filtered;
521 | }
522 |
523 | /**
524 | * Inspects a specific query and returns the results.
525 | *
526 | * @link https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/status-endpoint/
527 | *
528 | *
This method may be used to inspect the status of a query.
529 | *
530 | * @param {string} commandId this id is obtained when using the .ksql method (/ksql endpoint) to run CREATE, DROP, TERMINATE commands.
531 | * @return {Promise} this method returns a promise, that resolves to a JSON object that has the following two properties->
532 | *
533 | * status (string): One of QUEUED, PARSING, EXECUTING, TERMINATED, SUCCESS, or ERROR.
534 | *
535 | * message (string): Detailed message regarding the status of the execution statement.
536 | */
537 | inspectQueryStatus(commandId) {
538 | validateInputs([commandId, 'string', 'commandId', true]);
539 |
540 | return axios.get(this.ksqldbURL + `/status/${commandId}`)
541 | .then(response => response)
542 | .catch(error => {
543 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
544 | });
545 | }
546 |
547 | /**
548 | * Inspects a ksqlDB server and returns the results.
549 | *
550 | * @link https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/info-endpoint/
551 | *
552 | *
This method is mainly used for troubleshooting.
553 | *
554 | * @return {Promise} this method returns a promise that resolves to an object containing the version, clusterId, and ksqlservice id.
555 | */
556 | inspectServerInfo() {
557 | return axios.get(this.ksqldbURL + `/info`)
558 | .then(response => response)
559 | .catch(error => {
560 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
561 | });
562 | }
563 |
564 | /**
565 | * Inspects the health status of a ksqlDB server.
566 | *
567 | * @link https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/info-endpoint/
568 | *
569 | *
This method may be used to give the health status of a ksqlDB server.
570 | *
571 | * @return {Promise} this method returns a promise that resolves to an object containing the metastore, kafka, and commandRunner info.
572 | */
573 | inspectServerHealth() {
574 | return axios.get(this.ksqldbURL + `/healthcheck`)
575 | .then(response => response)
576 | .catch(error => {
577 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
578 | });
579 | }
580 |
581 | /**
582 | * Inspects all servers in a ksqlDB cluster and returns the results.
583 | *
584 | * @link https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/cluster-status-endpoint/
585 | *
586 | *
This method may be used to get information about the status of all ksqlDB servers in a ksqlDB cluster, which can be useful
587 | * for troubleshooting.
588 | *
589 | * @return {Promise} this method returns a promise that resolves to an object containing information about the ksqlDB servers.
590 | */
591 | inspectClusterStatus() {
592 | return axios.get(this.ksqldbURL + `/clusterStatus`)
593 | .then(response => response)
594 | .catch(error => {
595 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
596 | });
597 | }
598 |
599 | /**
600 | * Terminates a ksqlDB cluster.
601 | *
602 | * @link https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/terminate-endpoint/
603 | *
604 | *
This method may be used to terminate a ksqlDB cluster. First, shut down all the servers except one.
605 | *
606 | * @param {string[]} topicsToDelete an array of topic names or regular expressions for topic names to delete.
607 | * @return {Promise} this method returns a promise that returns a response object.
608 | */
609 | terminateCluster(topicsToDelete = []) {
610 | validateInputs([topicsToDelete, 'array', 'topicsToDelete', true]);
611 |
612 | return axios.post(this.ksqldbURL + `/ksql/terminate`, {
613 | "deleteTopicList": topicsToDelete
614 | }, {
615 | headers: {
616 | // 'application/json' is the modern content-type for JSON, but some
617 | // older servers may use 'text/json'.
618 | 'Accept': 'application/vnd.ksql.v1+json',
619 | 'Content-Type': 'application/vnd.ksql.v1+json'
620 | }
621 | })
622 | .then(response => response)
623 | .catch(error => {
624 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
625 | });
626 | }
627 |
628 | /**
629 | * Checks whether a ksqldb server property is allowed to be changed.
630 | *
631 | * @link https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/is_valid_property-endpoint/
632 | *
633 | *
This method may be used to check if a property in a ksqldb server is prohibited from being changed.
634 | *
635 | *
If the property is prohibited from setting, the following object will be returned:
636 | * {
637 | * "@type": "generic_error",
638 | * "error_code": 40000,
639 | * "message": "One or more properties overrides set locally are prohibited by the KSQL server (use UNSET to reset their default value): [ksql.service.id]"
640 | * }
641 | *
642 | * @param {string} propertyName - the name of the property to validate
643 | * @return {Promise} this method returns a promise that resolves to a boolean true if the property is allowed to be changed.
644 | */
645 | isValidProperty(propertyName) {
646 | validateInputs([propertyName, 'string', 'propertyName', true]);
647 |
648 | return axios.get(this.ksqldbURL + `/is_valid_property/${propertyName}`)
649 | .then(response => response)
650 | .catch(error => {
651 | throw error.response?.data['@type'] ? new ksqlDBError(error.response.data) : error;
652 | });
653 | }
654 | };
655 |
656 | module.exports = ksqldb;
657 |
--------------------------------------------------------------------------------
/ksQlient/ksqldb/queryBuilder.js:
--------------------------------------------------------------------------------
1 | const { QueryBuilderError, EmptyQueryError, NumParamsError, InappropriateStringParamError } = require('./customErrors.js');
2 |
3 | class queryBuilder {
4 | constructor() {
5 | }
6 |
7 | build = (query, ...params) => {
8 | // check for empty query
9 | if (this._checkEmptyQuery(query)) throw new EmptyQueryError();
10 |
11 | let output = this._bind(query, ...params);
12 |
13 | return output;
14 | }
15 | _bind = (query, ...params) => {
16 | const numParams = params.length;
17 | // count num of ? in query
18 | const numMark = query.split("?").length - 1;
19 | if (numParams > numMark) { throw new NumParamsError('more params than wildcards in query') };
20 | if (numParams < numMark) { throw new NumParamsError('less params than wildcards in query') };
21 | for (let i = 0; i < numParams; i++) {
22 | const newParam = this._replaceWith(params[i]);
23 | // if (newParam instanceof Error) return newParam;
24 | query = query.replace(/\?/, newParam);
25 | };
26 | return query;
27 | }
28 | _replaceWith = (param) => {
29 | switch (typeof param) {
30 | case "number":
31 | return param;
32 | case "bigint":
33 | return param;
34 | case "boolean":
35 | return param;
36 | case "object":
37 | if (Array.isArray(param)) {
38 | if (param[0]?.includes(";")) {
39 | throw new InappropriateStringParamError("string params not wrapped in quotes should not include semi-colons");
40 | }
41 | return `${param[0]?.replaceAll("'", "''")}`
42 | }
43 | throw new QueryBuilderError("object should not be passed in as query argument");
44 | case "function":
45 | throw new QueryBuilderError("function should not be passed in as query argument");
46 | case "string":
47 | // https://stackoverflow.com/questions/5139770/escape-character-in-sql-server
48 | // example of injection in Go: https://go.dev/play/p/4KoWROjK903
49 | return `'${param.replaceAll("'", "''")}'`;
50 | default:
51 | // code block
52 | }
53 | };
54 | _checkEmptyQuery = (query) => {
55 | if (query === "" || query === undefined || query === null) {
56 | return true;
57 | }
58 | return false;
59 | }
60 | }
61 | module.exports = queryBuilder;
62 |
--------------------------------------------------------------------------------
/ksQlient/ksqldb/validateInputs.js:
--------------------------------------------------------------------------------
1 | const { invalidArgumentTypes } = require("./customErrors");
2 |
3 | const validateInputs = (...args) => {
4 | const invalidArguments = [];
5 |
6 | // iterate through args to verify allowed types are provided
7 | for (let i = 0; i < args.length; i++) {
8 | const [currentArg, intendedType, actualType, required] = [args[i][0], args[i][1], typeof args[i][0], args[i][3] || false];
9 |
10 | if (intendedType === 'array') {
11 | if (!Array.isArray(currentArg)) invalidArguments.push(args[i]);
12 | }
13 | else if (required && actualType !== intendedType) invalidArguments.push(args[i]);
14 | else if (!required && currentArg !== undefined && currentArg !== null && actualType !== intendedType) invalidArguments.push(args[i]);
15 | }
16 |
17 | // craft error message if error needs to be thrown
18 | if (invalidArguments.length) {
19 | let errorMessage = '';
20 |
21 | for (let i = 0; i < invalidArguments.length; i++) {
22 | errorMessage += `argument "${invalidArguments[i][2]}" must be of type ${invalidArguments[i][1]}`;
23 | if (i < invalidArguments.length - 1) errorMessage += ', '
24 | }
25 | throw new invalidArgumentTypes(errorMessage);
26 | }
27 | }
28 |
29 | module.exports = validateInputs;
--------------------------------------------------------------------------------
/ksQlient/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "ksqlient",
3 | "version": "1.0.1",
4 | "description": "Javascript KsqlDB client for Node.js",
5 | "main": "./ksqldb/ksqldb",
6 | "scripts": {
7 | "test": "jest --verbose --forceExit"
8 | },
9 | "repository": {
10 | "type": "git",
11 | "url": "git+https://github.com/oslabs-beta/ksqlSuite.git"
12 | },
13 | "keywords": [],
14 | "author": "Michael Synder, Matthew Xing, Jonathan Luu, Gerry Bong, Javan Ang",
15 | "license": "ISC",
16 | "bugs": {
17 | "url": "https://github.com/oslabs-beta/ksqlSuite/issues"
18 | },
19 | "homepage": "https://github.com/oslabs-beta/ksqlSuite#readme",
20 | "dependencies": {
21 | "axios": "^0.27.2"
22 | },
23 | "devDependencies": {
24 | "jest": "^28.1.0"
25 | }
26 | }
27 |
--------------------------------------------------------------------------------
/ksQlient/static/light.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/oslabs-beta/ksqlSuite/999b83890287cceff44169afd51af7c232cbd554/ksQlient/static/light.png
--------------------------------------------------------------------------------
/ksQlient/static/name.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/oslabs-beta/ksqlSuite/999b83890287cceff44169afd51af7c232cbd554/ksQlient/static/name.png
--------------------------------------------------------------------------------
/ksqLight/.gitignore:
--------------------------------------------------------------------------------
1 | # See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
2 |
3 | # dependencies
4 | /node_modules
5 | /.pnp
6 | .pnp.js
7 |
8 | # testing
9 | /coverage
10 |
11 | # production
12 | /build
13 |
14 | # misc
15 | .DS_Store
16 | .env.local
17 | .env.development.local
18 | .env.test.local
19 | .env.production.local
20 |
21 | npm-debug.log*
22 | yarn-debug.log*
23 | yarn-error.log*
24 |
--------------------------------------------------------------------------------
/ksqLight/README.md:
--------------------------------------------------------------------------------
1 | # ksqLight
2 |
3 |
7 | 
8 | 
5 | 
6 | 
7 | 
8 |
9 | 
5 | ![Homepage]({require("../static/ksqLight2.png")}){kind=small}
24 | 
5 | 
10 | 
11 | 
12 |
13 |