4 |
5 |
6 | ## Overview
7 |
8 | This is a community based IBM SQL Query Dremio connector made using the ARP framework. Check [Dremio Hub](https://github.com/dremio-hub) for more examples and [ARP Docs](https://github.com/dremio-hub/dremio-sqllite-connector#arp-file-format) for documentation.
9 |
10 | ### What is Dremio?
11 |
12 | Dremio delivers lightning fast query speed and a self-service semantic layer operating directly against your data lake storage and other sources. No moving data to proprietary data warehouses or creating cubes, aggregation tables and BI extracts. Just flexibility and control for Data Architects, and self-service for Data Consumers.
13 |
14 | ## Installation
15 |
16 | ### Run Dremio with SQL Query connector locally
17 |
18 | You can launch the ready-to-use Dremio image with the latest release of the IBM SQL Query connector and the IBM SQL Query JDBC driver like follows:
19 | * `docker run -d -p 9047:9047 -p 31010:31010 -p 45678:45678 torsstei/dremio-ibm-sql-query`
20 |
21 | Now you can open the Dremio console at this URL: http://localhost:9047 and proceed as descibed in Usage below.
22 |
23 | ### Run Dremio with SQL Query connector in IBM Cloud
24 |
25 | You can use helm charts to deploy an entire Dremio cluster on Kubernetes in IBM Cloud's managed [Kubernetes Service](https://www.ibm.com/cloud/kubernetes-service).
26 |
27 | * Make sure you have helm installed on your local machine. For instance use `brew install kubernetes-helm` on Mac OS.
28 | * Make sure you have Kubernetes cluster created in your IBM Cloud account.
29 | * Install the IBM Cloud [CLI tool](https://cloud.ibm.com/docs/cli?topic=cli-install-ibmcloud-cli).
30 | * Install and Set up the Kubernetes container service plugin for the IBM Cloud CLI and the kubectl CLI tool for your Kubernetes cluster as described [here](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_install).
31 | * Clone the github repository with the Dremio helm charts for IBM Cloud:
32 | * `git clone https://github.com/IBM-Cloud/dremio-cloud-tools.git`
33 | * Go to the directoy with the desired configuration
34 | * `cd dremio-cloud-tools/charts/dremio_v2`
35 | * If you want to start with a smaller cluster with less resources use `cd dremio-cloud-tools/charts/dremio_v2_small` instead.
36 | * Submit the helm chart deployment `helm install --wait . --generate-name`
37 | * You can list the deployed helm charts with `helm list`
38 | * You can use the Kubernetes dashboard (can be launched from the IBM Cloud dashboard UI for your Kubernetes cluster) or the `kubectl` CLI to observe the status of the individual assets that helm has deployed in your Kubernetes cluster.
39 | * Identify the load balancer ingress host name of the Dremio server in your Kubernetes cluster
40 | * `kubectl describe service dremio-client | grep LoadBalancer`
41 | * Now open URL `http://{hostname}:9047` in your browser to get the Dremio web console.
42 | * Alternatively you can also go to the Kubernetes dashboard and click on the second Endpoint in `Service->Services->"dremio-client"->External Endpoints` (this is the endpoint with port 9047 for the Dremio console).
43 |
44 | ## Usage
45 |
46 | ### Creating a new IBM SQL Query Source
47 |
48 | In the Dremio UI click the `+` option for `External Sources` in the bottom left corner. Then select `IBMSQL` and specify the following parameters:
49 |
50 | * JDBC URL
51 | * Ex: `jdbc:ibmcloudsql:{instance-crn}`
52 | * The `{instance-crn}` is the unique identifier of your instance of SQL Query in your IBM Cloud account.
53 | * Make sure that the URL does not include the paramter `?targetcosurl=...`. This is specified in a separate parameter. See below.
54 | * See also `Connect` menu in your IBM SQL Query web console.
55 | * API Key (IBM Cloud IAM API Key)
56 | * Target COS URL
57 | * Ex: `cos://{IBM cloud region alias}/{bucket name}/{prefix}`
58 | * This is the IBM cloud object storage location where results for each SQL query are written. You can configure automatic [expiration](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry) if you want an automatic cleanup.
59 | * This is the same value as in the field `Targert location` in your IBM SQL Query web console.
60 |
61 | Learn about IBM SQL Query JDBC client properties [here](https://cloud.ibm.com/docs/sql-query?topic=sql-query-jdbc)
62 |
63 | ## Development
64 |
65 | ### Manual Build and Installation of the connector
66 |
67 | 0. Change the pom's dremio.version to suit your Dremio's version.
68 | `{renderHighlightedMarkup(expr, keyPrefix)}
152 | {data}*/}
31 | 32 | Time series 33 |91 |
34 | 35 | - return column named time (in UTC), as a unix time stamp or any sql native date data type. You can use the macros below. 36 |
37 | - any other columns returned will be the time point values. 38 |
39 | Optional: 40 |
41 | - return column named metric to represent the series name. 42 |
43 | - If multiple value columns are returned the metric column is used as prefix. 44 |
45 | - If no column named metric is found the column name of the value column is used as series name 46 |
47 | 48 | Resultsets of time series queries need to be sorted by time. 49 |
50 |
51 |
52 | Table: 53 |
54 | - return any set of columns 55 |
56 | 57 |
58 | Variables: 59 |
60 | - ${variable_name} - > in the dashboard's setting, you select 'Variables' and create a new one with the name `variable_name` (with values can be user-input or retrieved as a result of a CloudSQL Query), then you can reference to the selected value in the query using the above syntax: dollar sign, open curly brance, name and closing curly brace. 61 |
62 | NOTE: For string-value, put that into the double-quote, i.e. "${variable_name}" 63 |
64 | 65 |
66 | Macros: 67 |
68 | - $__source -> the datasource as provided in DataSource setting 69 |
70 | - $__source_test -> the fake time-series datasource, e.g. $__source_test(TS) or $__source_test(MTS) 71 |
72 | - $__dest[(format [,suffix])] -> the location to store data as provided in DataSource setting, e.g. $__dest, $__dest(), $__dest(csv), $__dest(CSV), $__dest(parquet, a/b/c) 73 |
74 | Example you want to save queried data to the TARGET_COS_URL with suffix 'a/b/c' in the format 'PARQUET: 75 |
76 | SELECT * FROM $__source INTO $__dest(parquet, a/b/c) 77 |
78 |
79 | - $__source_prev -> reference to the output from a previous query in the same dashboard/panel, e.g. $__source_prev(A) 80 |
81 | - $__timeFilter() -> time_column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z' 82 | - $__timeFilterColumn(column-name, [type]) -> add time filter using the given column name (1st argument), and its type (2nd argument) 83 |
84 | 85 |
86 | Cloud SQL: 87 |
88 | DISTRIBUTE BY, SORT BY and CLUSTER BY only have an effect during your SQL query execution and do not influence the query result written back to Cloud Object Storage. Use these clauses only in execution of subqueries in order to optimize the outer query execution that works on the intermediate result sets produced by the sub queries. 89 |
90 |