├── tests ├── __init__.py ├── pytest.ini ├── test_config.py ├── conftest.py ├── test_main.py ├── utils.py ├── test_odfesql_cli.py ├── test_plan.md ├── test_esconnection.py └── test_formatter.py ├── src └── odfe_sql_cli │ ├── conf │ ├── __init__.py │ └── clirc │ ├── esliterals │ ├── __init__.py │ └── esliterals.json │ ├── __init__.py │ ├── utils.py │ ├── esbuffer.py │ ├── config.py │ ├── esstyle.py │ ├── main.py │ ├── formatter.py │ ├── esconnection.py │ └── odfesql_cli.py ├── requirements-dev.txt ├── screenshots └── usage.gif ├── NOTICE ├── tox.ini ├── CODE_OF_CONDUCT.md ├── CONTRIBUTORS.md ├── .github ├── PULL_REQUEST_TEMPLATE.md ├── workflows │ ├── draft-release-notes-workflow.yml │ ├── release-workflow.yml │ └── test-and-build-workflow.yml └── draft-release-notes-config.yml ├── release-notes ├── odfe-sql-cli.release-notes-1.8.0.0.md ├── odfe-sql-cli.release-notes-1.9.0.0.md └── odfe-sql-cli.release-notes-1.7.0.0.md ├── .gitignore ├── setup.py ├── development_guide.md ├── CONTRIBUTING.md ├── README.md ├── LICENSE.TXT └── THIRD-PARTY /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/conf/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/esliterals/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | addopts=--capture=sys --showlocals -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | pytest==4.6.3 2 | mock==3.0.5 3 | pexpect==3.3 4 | twine==1.13.0 5 | tox>=1.9.2 -------------------------------------------------------------------------------- /screenshots/usage.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-archives/sql-cli/HEAD/screenshots/usage.gif -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Open Distro for Elasticsearch SQL CLI 2 | Copyright 2019-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py38 3 | [testenv] 4 | deps = pytest==4.6.3 5 | mock==3.0.5 6 | pexpect==3.3 7 | commands = pytest -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted an [Open Source Code of Conduct](https://opendistro.github.io/for-elasticsearch/codeofconduct.html). 3 | -------------------------------------------------------------------------------- /CONTRIBUTORS.md: -------------------------------------------------------------------------------- 1 | Contributors in order of last name: 2 | 3 | Abbas Hussain 4 | 5 | Zhongnan Su 6 | 7 | Chloe Zhang 8 | 9 | Anirudh Jadhav 10 | 11 | Alolita Sharma 12 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /release-notes/odfe-sql-cli.release-notes-1.8.0.0.md: -------------------------------------------------------------------------------- 1 | ## 2020-05-18 Version 1.8.0.0 2 | 3 | ### Features 4 | #### Elasticsearch and ODFE SQL Plugin Compatibility 5 | * Feature [#41](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/41): Elasticsearch 7.7.0 and ODFE SQL Plugin 1.8.0 compatibility (issue: [#40](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/40)) 6 | -------------------------------------------------------------------------------- /.github/workflows/draft-release-notes-workflow.yml: -------------------------------------------------------------------------------- 1 | name: Release Drafter 2 | 3 | on: 4 | push: 5 | branches: 6 | - master 7 | 8 | jobs: 9 | update_release_draft: 10 | name: Update draft release notes 11 | runs-on: ubuntu-latest 12 | steps: 13 | - name: Update draft release notes 14 | uses: release-drafter/release-drafter@v5 15 | with: 16 | config-name: draft-release-notes-config.yml 17 | tag: (None) 18 | version: 1.9.0.0 19 | env: 20 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 21 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | __version__ = "1.9.0.0" 16 | -------------------------------------------------------------------------------- /release-notes/odfe-sql-cli.release-notes-1.9.0.0.md: -------------------------------------------------------------------------------- 1 | ## 2020-06-24 Version 1.9.0.0 (Current) 2 | 3 | ### Features 4 | #### Elasticsearch and ODFE SQL Plugin Compatibility 5 | * Feature [#55](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/55): Elasticsearch 7.8.0 and ODFE SQL Plugin 1.9.0 compatibility 6 | (issue: [#54](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/54)) 7 | 8 | #### Documentation 9 | * Feature [#48](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/48): Added README badges 10 | 11 | ### Enhancements 12 | * Enhancement [#45](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/45): Update project layout for better module import 13 | (issue: [#43](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/43)) 14 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import sys 16 | 17 | from collections import namedtuple 18 | 19 | OutputSettings = namedtuple("OutputSettings", "table_format is_vertical max_width style_output missingval") 20 | 21 | OutputSettings.__new__.__defaults__ = (None, False, sys.maxsize, None, "null") 22 | -------------------------------------------------------------------------------- /.github/draft-release-notes-config.yml: -------------------------------------------------------------------------------- 1 | # The overall template of the release notes 2 | template: | 3 | Open Distro for Elasticsearch Version $RESOLVED_VERSION 4 | $CHANGES 5 | 6 | # Setting the formatting and sorting for the release notes body 7 | name-template: Version $RESOLVED_VERSION 8 | change-template: "- $TITLE (PR [#$NUMBER](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/$NUMBER))" 9 | sort-by: merged_at 10 | sort-direction: ascending 11 | 12 | # Organizing the tagged PRs into categories 13 | categories: 14 | - title: "Version Upgrades" 15 | labels: 16 | - "version compatibility" 17 | - title: "Features" 18 | labels: 19 | - "feature" 20 | - title: "Enhancements" 21 | labels: 22 | - "enhancement" 23 | - title: "Bug Fixes" 24 | labels: 25 | - "bug" 26 | - "bug fix" 27 | - title: "Documentation" 28 | labels: 29 | - "documentation" 30 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/esliterals/esliterals.json: -------------------------------------------------------------------------------- 1 | { 2 | "keywords": [ 3 | "ADD", 4 | "ALIASES", 5 | "ALL", 6 | "AND", 7 | "AS", 8 | "ASC", 9 | "BETWEEN", 10 | "BY", 11 | "DATE", 12 | "DELETE", 13 | "DESC", 14 | "DESCRIBE", 15 | "FROM", 16 | "FULL", 17 | "GROUP BY", 18 | "HAVING", 19 | "IN", 20 | "INTO", 21 | "IS", 22 | "INNER", 23 | "JOIN", 24 | "KEY", 25 | "LEFT", 26 | "LIKE", 27 | "LIMIT", 28 | "MINUS", 29 | "NOT", 30 | "NULLS", 31 | "ON", 32 | "OR", 33 | "ORDER BY", 34 | "SELECT", 35 | "SHOW", 36 | "TABLES", 37 | "UNION", 38 | "WHEN", 39 | "WHERE" 40 | ], 41 | "functions": [ 42 | "AVG", 43 | "CONCAT_WS", 44 | "COUNT", 45 | "DISTINCT", 46 | "FLOOR", 47 | "ISNULL", 48 | "LOG", 49 | "LOG10", 50 | "MAX", 51 | "MID", 52 | "MIN", 53 | "ROUND", 54 | "SUBSTRING", 55 | "SUM", 56 | "SQRT" 57 | ] 58 | } -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | pyvenv/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | *.egg-info/ 23 | .installed.cfg 24 | *.egg 25 | 26 | # PyInstaller 27 | # Usually these files are written by a python script from a template 28 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 29 | *.manifest 30 | *.spec 31 | 32 | # Installer logs 33 | pip-log.txt 34 | pip-delete-this-directory.txt 35 | 36 | # Unit test / coverage reports 37 | htmlcov/ 38 | .tox/ 39 | .coverage 40 | .coverage.* 41 | .cache 42 | nosetests.xml 43 | coverage.xml 44 | .pytest_cache 45 | 46 | # Translations 47 | *.mo 48 | *.pot 49 | 50 | # Django stuff: 51 | *.log 52 | 53 | # Sphinx documentation 54 | docs/_build/ 55 | 56 | # PyBuilder 57 | target/ 58 | 59 | # PyCharm 60 | .idea/ 61 | *.iml 62 | 63 | # Vagrant 64 | .vagrant/ 65 | 66 | # Generated Packages 67 | *.deb 68 | *.rpm 69 | 70 | .vscode/ 71 | venv/ 72 | 73 | .DS_Store 74 | -------------------------------------------------------------------------------- /tests/test_config.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import os 16 | import stat 17 | import pytest 18 | 19 | from src.odfe_sql_cli.config import ensure_dir_exists 20 | 21 | 22 | class TestConfig: 23 | def test_ensure_file_parent(self, tmpdir): 24 | subdir = tmpdir.join("subdir") 25 | rcfile = subdir.join("rcfile") 26 | ensure_dir_exists(str(rcfile)) 27 | 28 | def test_ensure_existing_dir(self, tmpdir): 29 | rcfile = str(tmpdir.mkdir("subdir").join("rcfile")) 30 | 31 | # should just not raise 32 | ensure_dir_exists(rcfile) 33 | 34 | def test_ensure_other_create_error(self, tmpdir): 35 | subdir = tmpdir.join("subdir") 36 | rcfile = subdir.join("rcfile") 37 | 38 | # trigger an oserror that isn't "directory already exists" 39 | os.chmod(str(tmpdir), stat.S_IREAD) 40 | 41 | with pytest.raises(OSError): 42 | ensure_dir_exists(str(rcfile)) 43 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/esbuffer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | from __future__ import unicode_literals 16 | 17 | from prompt_toolkit.enums import DEFAULT_BUFFER 18 | from prompt_toolkit.filters import Condition 19 | from prompt_toolkit.application import get_app 20 | 21 | 22 | def es_is_multiline(odfesql_cli): 23 | """Return function that returns boolean to enable/unable multiline mode.""" 24 | 25 | @Condition 26 | def cond(): 27 | doc = get_app().layout.get_buffer_by_name(DEFAULT_BUFFER).document 28 | 29 | if not odfesql_cli.multi_line: 30 | return False 31 | if odfesql_cli.multiline_mode == "safe": 32 | return True 33 | else: 34 | return not _multiline_exception(doc.text) 35 | 36 | return cond 37 | 38 | 39 | def _is_complete(sql): 40 | # A complete command is an sql statement that ends with a semicolon 41 | return sql.endswith(";") 42 | 43 | 44 | def _multiline_exception(text): 45 | text = text.strip() 46 | return _is_complete(text) 47 | -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | 16 | """ 17 | We can define the fixture functions in this file to make them 18 | accessible across multiple test modules. 19 | """ 20 | import os 21 | import pytest 22 | 23 | from .utils import create_index, delete_index, get_connection 24 | 25 | 26 | @pytest.fixture(scope="function") 27 | def connection(): 28 | test_connection = get_connection() 29 | create_index(test_connection) 30 | 31 | yield test_connection 32 | delete_index(test_connection) 33 | 34 | 35 | @pytest.fixture(scope="function") 36 | def default_config_location(): 37 | from src.odfe_sql_cli.conf import __file__ as package_root 38 | 39 | package_root = os.path.dirname(package_root) 40 | default_config = os.path.join(package_root, "clirc") 41 | 42 | yield default_config 43 | 44 | 45 | @pytest.fixture(scope="session", autouse=True) 46 | def temp_config(tmpdir_factory): 47 | # this function runs on start of test session. 48 | # use temporary directory for conf home so user conf will not be used 49 | os.environ["XDG_CONFIG_HOME"] = str(tmpdir_factory.mktemp("data")) 50 | -------------------------------------------------------------------------------- /.github/workflows/release-workflow.yml: -------------------------------------------------------------------------------- 1 | name: Release SQL CLI Artifacts 2 | # This workflows is triggered on creating tags to master 3 | on: 4 | push: 5 | tags: 6 | - 'v*' 7 | 8 | jobs: 9 | build: 10 | 11 | runs-on: [ubuntu-16.04] 12 | strategy: 13 | matrix: 14 | python-version: [3.8] 15 | 16 | steps: 17 | - name: Checkout SQL CLI 18 | uses: actions/checkout@v2 19 | 20 | - name: Set up Python ${{ matrix.python-version }} 21 | uses: actions/setup-python@v2 22 | with: 23 | python-version: ${{ matrix.python-version }} 24 | 25 | - name: Configure AWS Credentials 26 | uses: aws-actions/configure-aws-credentials@v1 27 | with: 28 | aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} 29 | aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} 30 | aws-region: us-east-1 31 | 32 | - name: Install Dependencies 33 | run: | 34 | python -m pip install --upgrade pip 35 | pip install setuptools wheel twine 36 | # publish to S3 and PyPI 37 | - name: Build and Publish 38 | run: | 39 | python setup.py sdist bdist_wheel 40 | artifact=`ls ./dist/*.tar.gz` 41 | wheel_artifact=`ls ./dist/*.whl` 42 | 43 | aws s3 cp $artifact s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-cli/ 44 | aws s3 cp $wheel_artifact s3://artifacts.opendistroforelasticsearch.amazon.com/downloads/elasticsearch-clients/opendistro-sql-cli/ 45 | 46 | # aws cloudfront create-invalidation --distribution-id ${{ secrets.DISTRIBUTION_ID }} --paths "/downloads/*" 47 | 48 | # TODO: Publish to PyPI 49 | # env: 50 | # TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} 51 | # TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} 52 | # run: twine upload dist/* 53 | -------------------------------------------------------------------------------- /.github/workflows/test-and-build-workflow.yml: -------------------------------------------------------------------------------- 1 | name: Test and Build 2 | 3 | on: [pull_request, push] 4 | 5 | jobs: 6 | build: 7 | 8 | runs-on: [ubuntu-16.04] 9 | strategy: 10 | matrix: 11 | python-version: [3.8] 12 | 13 | steps: 14 | - name: Checkout SQL CLI 15 | uses: actions/checkout@v2 16 | 17 | - name: Set up Python ${{ matrix.python-version }} 18 | uses: actions/setup-python@v2 19 | with: 20 | python-version: ${{ matrix.python-version }} 21 | 22 | - name: Install Python Dependencies 23 | run: | 24 | python -m pip install --upgrade pip 25 | pip install -r requirements-dev.txt 26 | pip install setuptools wheel 27 | 28 | - name: Set up ES and install SQL plugin 29 | run: | 30 | sudo add-apt-repository ppa:openjdk-r/ppa 31 | sudo apt update 32 | sudo apt install openjdk-11-jdk 33 | sudo apt install unzip 34 | wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.6.1-amd64.deb 35 | sudo dpkg -i elasticsearch-oss-7.6.1-amd64.deb 36 | sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install https://d3g5vo6xdbdb9a.cloudfront.net/downloads/elasticsearch-plugins/opendistro-sql/opendistro_sql-1.6.0.0.zip 37 | sudo systemctl start elasticsearch.service 38 | 39 | - name: Run Tox Testing 40 | run: tox 41 | 42 | - name: Stop ES 43 | run: sudo systemctl stop elasticsearch.service 44 | 45 | - name: Build Artifact 46 | run: python setup.py sdist bdist_wheel 47 | 48 | - name: Create Artifact Path 49 | run: | 50 | mkdir -p opendistro-sql-cli-builds 51 | cp -r ./dist/*.tar.gz ./dist/*.whl opendistro-sql-cli-builds/ 52 | 53 | - name: Upload Artifact 54 | uses: actions/upload-artifact@v2 55 | with: 56 | name: opendistro-sql-cli 57 | path: opendistro-sql-cli-builds -------------------------------------------------------------------------------- /tests/test_main.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import mock 16 | from textwrap import dedent 17 | 18 | from click.testing import CliRunner 19 | 20 | from .utils import estest, load_data, TEST_INDEX_NAME 21 | from src.odfe_sql_cli.main import cli 22 | from src.odfe_sql_cli.odfesql_cli import OdfeSqlCli 23 | 24 | INVALID_ENDPOINT = "http://invalid:9200" 25 | ENDPOINT = "http://localhost:9200" 26 | QUERY = "select * from %s" % TEST_INDEX_NAME 27 | 28 | 29 | class TestMain: 30 | @estest 31 | def test_explain(self, connection): 32 | doc = {"a": "aws"} 33 | load_data(connection, doc) 34 | 35 | err_message = "Can not connect to endpoint %s" % INVALID_ENDPOINT 36 | expected_output = {"from": 0, "size": 200} 37 | expected_tabular_output = dedent( 38 | """\ 39 | fetched rows / total rows = 1/1 40 | +-----+ 41 | | a | 42 | |-----| 43 | | aws | 44 | +-----+""" 45 | ) 46 | 47 | with mock.patch("src.odfe_sql_cli.main.click.echo") as mock_echo, mock.patch("src.odfe_sql_cli.main.click.secho") as mock_secho: 48 | runner = CliRunner() 49 | 50 | # test -q -e 51 | result = runner.invoke(cli, [f"-q{QUERY}", "-e"]) 52 | mock_echo.assert_called_with(expected_output) 53 | assert result.exit_code == 0 54 | 55 | # test -q 56 | result = runner.invoke(cli, [f"-q{QUERY}"]) 57 | mock_echo.assert_called_with(expected_tabular_output) 58 | assert result.exit_code == 0 59 | 60 | # test invalid endpoint 61 | runner.invoke(cli, [INVALID_ENDPOINT, f"-q{QUERY}", "-e"]) 62 | mock_secho.assert_called_with(message=err_message, fg="red") 63 | 64 | @estest 65 | def test_cli(self): 66 | with mock.patch.object(OdfeSqlCli, "connect") as mock_connect, mock.patch.object( 67 | OdfeSqlCli, "run_cli" 68 | ) as mock_run_cli: 69 | runner = CliRunner() 70 | result = runner.invoke(cli) 71 | 72 | mock_connect.assert_called_with(ENDPOINT, None) 73 | mock_run_cli.asset_called() 74 | assert result.exit_code == 0 75 | -------------------------------------------------------------------------------- /tests/utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import json 16 | import pytest 17 | from elasticsearch import ConnectionError, helpers, ConnectionPool 18 | 19 | from src.odfe_sql_cli.esconnection import ESConnection 20 | from src.odfe_sql_cli.utils import OutputSettings 21 | from src.odfe_sql_cli.formatter import Formatter 22 | 23 | TEST_INDEX_NAME = "odfesql_cli_test" 24 | ENDPOINT = "http://localhost:9200" 25 | 26 | 27 | def create_index(test_executor): 28 | es = test_executor.client 29 | es.indices.create(index=TEST_INDEX_NAME) 30 | 31 | 32 | def delete_index(test_executor): 33 | es = test_executor.client 34 | es.indices.delete(index=TEST_INDEX_NAME) 35 | 36 | 37 | def close_connection(es): 38 | ConnectionPool.close(es) 39 | 40 | 41 | def load_file(test_executor, filename="accounts.json"): 42 | es = test_executor.client 43 | 44 | filepath = "./test_data/" + filename 45 | 46 | # generate iterable data 47 | def load_json(): 48 | with open(filepath, "r") as f: 49 | for line in f: 50 | yield json.loads(line) 51 | 52 | helpers.bulk(es, load_json(), index=TEST_INDEX_NAME) 53 | 54 | 55 | def load_data(test_executor, doc): 56 | es = test_executor.client 57 | es.index(index=TEST_INDEX_NAME, body=doc) 58 | es.indices.refresh(index=TEST_INDEX_NAME) 59 | 60 | 61 | def get_connection(): 62 | test_es_connection = ESConnection(endpoint=ENDPOINT) 63 | test_es_connection.set_connection() 64 | 65 | return test_es_connection 66 | 67 | 68 | def run(test_executor, query, use_console=True): 69 | data = test_executor.execute_query(query=query, use_console=use_console) 70 | settings = OutputSettings(table_format="psql") 71 | formatter = Formatter(settings) 72 | 73 | if data: 74 | res = formatter.format_output(data) 75 | res = "\n".join(res) 76 | 77 | return res 78 | 79 | 80 | # build client for testing 81 | try: 82 | connection = get_connection() 83 | CAN_CONNECT_TO_ES = True 84 | 85 | except ConnectionError: 86 | CAN_CONNECT_TO_ES = False 87 | 88 | # use @estest annotation to mark test functions 89 | estest = pytest.mark.skipif( 90 | not CAN_CONNECT_TO_ES, reason="Need a Elasticsearch server running at localhost:9200 accessible" 91 | ) -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import re 16 | import ast 17 | 18 | from setuptools import setup, find_packages 19 | 20 | install_requirements = [ 21 | "click == 7.1.1", 22 | "prompt_toolkit == 2.0.6", 23 | "Pygments == 2.6.1", 24 | "cli_helpers[styles] == 1.2.1", 25 | "elasticsearch == 7.5.1", 26 | "pyfiglet == 0.8.post1", 27 | "boto3 == 1.9.181", 28 | "requests-aws4auth == 0.9", 29 | ] 30 | 31 | _version_re = re.compile(r"__version__\s+=\s+(.*)") 32 | 33 | with open("src/odfe_sql_cli/__init__.py", "rb") as f: 34 | version = str( 35 | ast.literal_eval(_version_re.search(f.read().decode("utf-8")).group(1)) 36 | ) 37 | 38 | description = "Open Distro for Elasticsearch SQL CLI with auto-completion and syntax highlighting" 39 | 40 | with open("README.md", "r") as fh: 41 | long_description = fh.read() 42 | 43 | setup( 44 | name="odfe-sql-cli", 45 | author="Open Distro for Elasticsearch", 46 | author_email="odfe-infra@amazon.com", 47 | version=version, 48 | license="Apache 2.0", 49 | url="https://opendistro.github.io/for-elasticsearch-docs/docs/sql/cli/", 50 | packages=find_packages('src'), 51 | package_dir={'': 'src'}, 52 | package_data={"odfe_sql_cli": ["conf/clirc", "esliterals/esliterals.json"]}, 53 | description=description, 54 | long_description=long_description, 55 | long_description_content_type="text/markdown", 56 | install_requires=install_requirements, 57 | entry_points={"console_scripts": ["odfesql=odfe_sql_cli.main:cli"]}, 58 | classifiers=[ 59 | "Intended Audience :: Developers", 60 | "License :: OSI Approved :: Apache Software License", 61 | "Operating System :: Unix", 62 | "Operating System :: POSIX :: Linux", 63 | "Programming Language :: Python", 64 | "Programming Language :: Python :: 3", 65 | "Programming Language :: Python :: 3.4", 66 | "Programming Language :: Python :: 3.5", 67 | "Programming Language :: Python :: 3.6", 68 | "Programming Language :: Python :: 3.7", 69 | "Programming Language :: Python :: 3.8", 70 | "Programming Language :: SQL", 71 | "Topic :: Database", 72 | "Topic :: Database :: Front-Ends", 73 | "Topic :: Software Development", 74 | "Topic :: Software Development :: Libraries :: Python Modules", 75 | ], 76 | python_requires='>=3.0' 77 | ) 78 | -------------------------------------------------------------------------------- /tests/test_odfesql_cli.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import mock 16 | import pytest 17 | from prompt_toolkit.shortcuts import PromptSession 18 | from prompt_toolkit.input.defaults import create_pipe_input 19 | 20 | from src.odfe_sql_cli.esbuffer import es_is_multiline 21 | from .utils import estest, load_data, TEST_INDEX_NAME, ENDPOINT 22 | from src.odfe_sql_cli.odfesql_cli import OdfeSqlCli 23 | from src.odfe_sql_cli.esconnection import ESConnection 24 | from src.odfe_sql_cli.esstyle import style_factory 25 | 26 | AUTH = None 27 | QUERY_WITH_CTRL_D = "select * from %s;\r\x04\r" % TEST_INDEX_NAME 28 | USE_AWS_CREDENTIALS = False 29 | 30 | 31 | @pytest.fixture() 32 | def cli(default_config_location): 33 | return OdfeSqlCli(clirc_file=default_config_location, always_use_pager=False) 34 | 35 | 36 | class TestOdfeSqlCli: 37 | def test_connect(self, cli): 38 | with mock.patch.object(ESConnection, "__init__", return_value=None) as mock_ESConnection, mock.patch.object( 39 | ESConnection, "set_connection" 40 | ) as mock_set_connectiuon: 41 | cli.connect(endpoint=ENDPOINT) 42 | 43 | mock_ESConnection.assert_called_with(ENDPOINT, AUTH, USE_AWS_CREDENTIALS) 44 | mock_set_connectiuon.assert_called() 45 | 46 | @estest 47 | @pytest.mark.skip(reason="due to prompt_toolkit throwing error, no way of currently testing this") 48 | def test_run_cli(self, connection, cli, capsys): 49 | doc = {"a": "aws"} 50 | load_data(connection, doc) 51 | 52 | # the title is colored by formatter 53 | expected = ( 54 | "fetched rows / total rows = 1/1" "\n+-----+\n| \x1b[38;5;47;01ma\x1b[39;00m |\n|-----|\n| aws |\n+-----+" 55 | ) 56 | 57 | with mock.patch.object(OdfeSqlCli, "echo_via_pager") as mock_pager, mock.patch.object( 58 | cli, "build_cli" 59 | ) as mock_prompt: 60 | inp = create_pipe_input() 61 | inp.send_text(QUERY_WITH_CTRL_D) 62 | 63 | mock_prompt.return_value = PromptSession( 64 | input=inp, multiline=es_is_multiline(cli), style=style_factory(cli.syntax_style, cli.cli_style) 65 | ) 66 | 67 | cli.connect(ENDPOINT) 68 | cli.run_cli() 69 | out, err = capsys.readouterr() 70 | inp.close() 71 | 72 | mock_pager.assert_called_with(expected) 73 | assert out.__contains__("Endpoint: %s" % ENDPOINT) 74 | assert out.__contains__("See you next search!") 75 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/config.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import errno 16 | import os 17 | import platform 18 | import shutil 19 | 20 | from os.path import expanduser, exists, dirname 21 | from configobj import ConfigObj 22 | 23 | 24 | def config_location(): 25 | """Return absolute conf file path according to different OS.""" 26 | if "XDG_CONFIG_HOME" in os.environ: 27 | return "%s/odfesql-cli/" % expanduser(os.environ["XDG_CONFIG_HOME"]) 28 | elif platform.system() == "Windows": 29 | # USERPROFILE is typically C:\Users\{username} 30 | return "%s\\AppData\\Local\\dbcli\\odfesql-cli\\" % os.getenv("USERPROFILE") 31 | else: 32 | return expanduser("~/.config/odfesql-cli/") 33 | 34 | 35 | def _load_config(user_config, default_config=None): 36 | config = ConfigObj() 37 | config.merge(ConfigObj(default_config, interpolation=False)) 38 | config.merge(ConfigObj(expanduser(user_config), interpolation=False, encoding="utf-8")) 39 | config.filename = expanduser(user_config) 40 | 41 | return config 42 | 43 | 44 | def ensure_dir_exists(path): 45 | """ 46 | Try to create config file in OS. 47 | 48 | Ignore existing destination. Raise error for other OSError, such as errno.EACCES (Permission denied), 49 | errno.ENOSPC (No space left on device) 50 | """ 51 | parent_dir = expanduser(dirname(path)) 52 | try: 53 | os.makedirs(parent_dir) 54 | except OSError as exc: 55 | if exc.errno != errno.EEXIST: 56 | raise 57 | 58 | 59 | def _write_default_config(source, destination, overwrite=False): 60 | destination = expanduser(destination) 61 | if not overwrite and exists(destination): 62 | return 63 | 64 | ensure_dir_exists(destination) 65 | shutil.copyfile(source, destination) 66 | 67 | 68 | # https://stackoverflow.com/questions/40193112/python-setuptools-distribute-configuration-files-to-os-specific-directories 69 | def get_config(clirc_file=None): 70 | """ 71 | Get config for odfesql cli. 72 | 73 | This config comes from either existing config in the OS, or create a config file in the OS, and write default config 74 | including in the package to it. 75 | """ 76 | from .conf import __file__ as package_root 77 | 78 | package_root = os.path.dirname(package_root) 79 | 80 | clirc_file = clirc_file or "%sconfig" % config_location() 81 | default_config = os.path.join(package_root, "clirc") 82 | 83 | _write_default_config(default_config, clirc_file) 84 | 85 | return _load_config(clirc_file, default_config) 86 | -------------------------------------------------------------------------------- /tests/test_plan.md: -------------------------------------------------------------------------------- 1 | # Test Plan 2 | The purpose of this checklist is to guide you through the basic usage of ODFE SQL CLI, as well as a manual test process. 3 | 4 | 5 | ## Display 6 | 7 | * [ ] Auto-completion 8 | * SQL syntax auto-completion 9 | * index name auto-completion 10 | * [ ] Test pagination with different output length / width. 11 | * query for long results to see the pagination activated automatically. 12 | * [ ] Test table formatted output. 13 | * [ ] Test successful conversion from horizontal to vertical display with confirmation. 14 | * resize the terminal window before launching sql cli, there will be a warning message if your terminal is too narrow for horizontal output. It will ask if you want to convert to vertical display 15 | * [ ] Test warning message when output > 200 rows of data. (Limited by ODFE SQL syntax) 16 | * `SELECT * FROM accounts` 17 | * Run above command, you’ll see the max output is 200, and there will be a message at the top of your results telling you how much data was fetched. 18 | * If you want to query more than 200 rows of data, try add a `LIMIT` with more than 200. 19 | 20 | 21 | ## Connection 22 | 23 | * [ ] Test connection to a local Elasticsearch instance 24 | * [ ] Standard Elastic version, with/without authentication by Elastic X-pack security (https://www.elastic.co/guide/en/elasticsearch/reference/7.6/security-getting-started.html) 25 | * [ ] OSS version, no authentication 26 | * [ ] OSS version, install [ODFE Security plugin](https://opendistro.github.io/for-elasticsearch-docs/docs/install/plugins/) to enable authentication and SSL 27 | * Run command like `odfesql -u -w ` to connect to instance with authentication. 28 | * [ ] Test connection to [Amazon Elasticsearch domain](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) with 29 | [Fine Grained Access Control](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/fgac.html) enabled. 30 | * Have your aws credentials correctly configured by `aws configure` 31 | * `odfesql --aws-auth -u -w ` 32 | * [ ] Test connection fail when connecting to invalid endpoint. 33 | * `odfesql invalidendpoint.com` 34 | 35 | 36 | ## Execution 37 | 38 | * [ ] Test successful execution given a query. e.g. 39 | * `SELECT * FROM bank WHERE age >30 AND gender = 'm'` 40 | * [ ] Test unsuccessful execution with an invalid SQL query will give an error message 41 | * [ ] Test load config file 42 | * `vim .config/odfesql-cli/config` 43 | * change settings such as `table_format = github` 44 | * restart sql cli, check the tabular output change 45 | 46 | 47 | ## Query Options 48 | 49 | * [ ] Test explain option -e 50 | * `odfesql -q "SELECT * FROM accounts LIMIT 5;" -e` 51 | * [ ] Test query and format option -q, -f 52 | * `odfesql -q "SELECT * FROM accounts LIMIT 5;" -f csv` 53 | * [ ] Test vertical output option -v 54 | * `odfesql -q "SELECT * FROM accounts LIMIT 5;" -v` 55 | 56 | ## OS and Python Version compatibility 57 | 58 | * [ ] Manually test on Linux(Ubuntu) and MacOS 59 | * [ ] Test against python 3.X versions (optional) 60 | 61 | -------------------------------------------------------------------------------- /development_guide.md: -------------------------------------------------------------------------------- 1 | ## Development Guide 2 | ### Development Environment Set Up 3 | - `pip install virtualenv` 4 | - `virtualenv venv` to create virtual environment for **Python 3** 5 | - `source ./venv/bin/activate` activate virtual env. 6 | - `cd` into project root folder. 7 | - `pip install --editable .` will install all dependencies from `setup.py`. 8 | 9 | ### Run CLI 10 | - Start an Elasticsearch instance from either local, Docker with Open Distro SQL plugin, or AWS Elasticsearch 11 | - To launch the cli, use 'wake' word `odfesql` followed by endpoint of your running ES instance. If not specifying 12 | any endpoint, it uses http://localhost:9200 by default. If not provided with port number, http endpoint uses 9200 and 13 | https uses 443 by default. 14 | 15 | ### Testing 16 | - Prerequisites 17 | - Build the application 18 | - Start a local Elasticsearch instance (OSS) with 19 | [Open Distro SQL plugin for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/sql/) installed 20 | and listening at http://localhost:9200. 21 | - Pytest 22 | - `pip install -r requirements-dev.txt` Install test frameworks including Pytest and mock. 23 | - `cd` into `tests` and run `pytest` 24 | - Refer to [test_plan](./tests/test_plan.md) for manual test guidance. 25 | 26 | ### Style 27 | - Use [black](https://github.com/psf/black) to format code, with option of `--line-length 120` 28 | 29 | ## Release guide 30 | 31 | - Package Manager: pip 32 | - Repository of software for Python: PyPI 33 | 34 | ### Workflow 35 | 36 | 1. Update version number 37 | 1. Modify the version number in `__init__.py` under `src` package. It will be used by `setup.py` for release. 38 | 2. Create/Update `setup.py` (if needed) 39 | 1. For more details refer to https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py 40 | 3. Update README.md, Legal and copyright files(if needed) 41 | 1. Update README.md when there is a critical feature added. 42 | 2. Update `THIRD-PARTY` files if there is a new dependency added. 43 | 4. Generate distribution archives 44 | 1. Make sure you have the latest versions of `setuptools` and `wheel` installed: `python3 -m pip install --user --upgrade setuptools wheel` 45 | 2. Run this command from the same directory where `setup.py` is located: `python3 setup.py sdist bdist_wheel` 46 | 3. Check artifacts under `sql-cli/dist/`, there should be a `.tar.gz` file and a `.whi` file with correct version. Remove other deprecated artifacts. 47 | 5. Upload the distribution archives to TestPyPI 48 | 1. Register an account on [testPyPI](https://test.pypi.org/) 49 | 2. `python3 -m pip install --user --upgrade twine` 50 | 3. `python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*` 51 | 6. Install your package from TestPyPI and do manual test 52 | 1. `pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple odfe-sql-cli` 53 | 7. Upload to PyPI 54 | 1. Register an account on [PyPI](https://pypi.org/), note that these are two separate servers and the credentials from the test server are not shared with the main server. 55 | 2. Use `twine upload dist/*` to upload your package and enter your credentials for the account you registered on PyPI.You don’t need to specify --repository; the package will upload to https://pypi.org/ by default. 56 | 8. Install your package from PyPI using `pip install [your-package-name]` 57 | 58 | ### Reference 59 | - https://medium.com/@joel.barmettler/how-to-upload-your-python-package-to-pypi-65edc5fe9c56 60 | - https://packaging.python.org/tutorials/packaging-projects/ 61 | - https://packaging.python.org/guides/using-testpypi/ -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/opendistro-for-elasticsearch/sql-cli/issues), or [recently closed](https://github.com/opendistro-for-elasticsearch/sql-cli/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/opendistro-for-elasticsearch/sql-cli/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/opendistro-for-elasticsearch/sql-cli/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/conf/clirc: -------------------------------------------------------------------------------- 1 | # Copyright 2020, Amazon Web Services Inc. 2 | # Licensed under the Apache License, Version 2.0 (the "License"); 3 | # you may not use this file except in compliance with the License. 4 | # You may obtain a copy of the License at 5 | # 6 | # http://www.apache.org/licenses/LICENSE-2.0 7 | # 8 | # Unless required by applicable law or agreed to in writing, software 9 | # distributed under the License is distributed on an "AS IS" BASIS, 10 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11 | # See the License for the specific language governing permissions and 12 | # limitations under the License. 13 | 14 | # vi: ft=dosini 15 | [main] 16 | 17 | # Multi-line mode allows breaking up the sql statements into multiple lines. If 18 | # this is set to True, then the end of the statements must have a semi-colon. 19 | # If this is set to False then sql statements can't be split into multiple 20 | # lines. End of line (return) is considered as the end of the statement. 21 | multi_line = True 22 | 23 | # If multi_line_mode is set to "odfesql_cli", in multi-line mode, [Enter] will execute 24 | # the current input if the input ends in a semicolon. 25 | # If multi_line_mode is set to "safe", in multi-line mode, [Enter] will always 26 | # insert a newline, and [Esc] [Enter] or [Alt]-[Enter] must be used to execute 27 | # a command. 28 | multi_line_mode = odfesql_cli 29 | 30 | # log_file location. 31 | # In Unix/Linux: ~/.conf/odfesql-cli/log 32 | # In Windows: %USERPROFILE%\AppData\Local\dbcli\odfesql-cli\log 33 | # %USERPROFILE% is typically C:\Users\{username} 34 | log_file = default 35 | 36 | # history_file location. 37 | # In Unix/Linux: ~/.conf/odfesql-cli/history 38 | # In Windows: %USERPROFILE%\AppData\Local\dbcli\odfesql-cli\history 39 | # %USERPROFILE% is typically C:\Users\{username} 40 | history_file = default 41 | 42 | # Default log level. Possible values: "CRITICAL", "ERROR", "WARNING", "INFO" 43 | # and "DEBUG". "NONE" disables logging. 44 | log_level = INFO 45 | 46 | # Table format. Possible values: psql, plain, simple, grid, fancy_grid, pipe, 47 | # ascii, double, github, orgtbl, rst, mediawiki, html, latex, latex_booktabs, 48 | # textile, moinmoin, jira, vertical, tsv, csv. 49 | # Recommended: psql, fancy_grid and grid. 50 | table_format = psql 51 | 52 | # Syntax Style. Possible values: manni, igor, xcode, vim, autumn, vs, rrt, 53 | # native, perldoc, borland, tango, emacs, friendly, monokai, paraiso-dark, 54 | # colorful, murphy, bw, pastie, paraiso-light, trac, default, fruity 55 | syntax_style = default 56 | 57 | # Set threshold for row limit prompt. Use 0 to disable prompt. 58 | # maybe not now, since elasticsearch opendisto sql plugin returns 200 rows of data by default if not 59 | # using LIMIT. 60 | row_limit = 1000 61 | 62 | # Character used to left pad multi-line queries to match the prompt size. 63 | multiline_continuation_char = '.' 64 | 65 | # The string used in place of a null value. 66 | null_string = 'null' 67 | 68 | # Custom colors for the completion menu, toolbar, etc. 69 | [colors] 70 | completion-menu.completion.current = 'bg:#ffffff #000000' 71 | completion-menu.completion = 'bg:#008888 #ffffff' 72 | completion-menu.meta.completion.current = 'bg:#44aaaa #000000' 73 | completion-menu.meta.completion = 'bg:#448888 #ffffff' 74 | completion-menu.multi-column-meta = 'bg:#aaffff #000000' 75 | scrollbar.arrow = 'bg:#003333' 76 | scrollbar = 'bg:#00aaaa' 77 | selected = '#ffffff bg:#6666aa' 78 | search = '#ffffff bg:#4444aa' 79 | search.current = '#ffffff bg:#44aa44' 80 | bottom-toolbar = 'bg:#222222 #aaaaaa' 81 | bottom-toolbar.off = 'bg:#222222 #888888' 82 | bottom-toolbar.on = 'bg:#222222 #ffffff' 83 | search-toolbar = 'noinherit bold' 84 | search-toolbar.text = 'nobold' 85 | system-toolbar = 'noinherit bold' 86 | arg-toolbar = 'noinherit bold' 87 | arg-toolbar.text = 'nobold' 88 | bottom-toolbar.transaction.valid = 'bg:#222222 #00ff5f bold' 89 | bottom-toolbar.transaction.failed = 'bg:#222222 #ff005f bold' 90 | 91 | # style classes for colored table output 92 | output.header = "#00ff5f bold" 93 | output.odd-row = "" 94 | output.even-row = "" -------------------------------------------------------------------------------- /src/odfe_sql_cli/esstyle.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | from __future__ import unicode_literals 16 | 17 | import logging 18 | 19 | import pygments.styles 20 | from pygments.token import string_to_tokentype, Token 21 | from pygments.style import Style as PygmentsStyle 22 | from pygments.util import ClassNotFound 23 | from prompt_toolkit.styles.pygments import style_from_pygments_cls 24 | from prompt_toolkit.styles import merge_styles, Style 25 | 26 | logger = logging.getLogger(__name__) 27 | 28 | # map Pygments tokens (ptk 1.0) to class names (ptk 2.0). 29 | TOKEN_TO_PROMPT_STYLE = { 30 | Token.Menu.Completions.Completion.Current: "completion-menu.completion.current", 31 | Token.Menu.Completions.Completion: "completion-menu.completion", 32 | Token.Menu.Completions.Meta.Current: "completion-menu.meta.completion.current", 33 | Token.Menu.Completions.Meta: "completion-menu.meta.completion", 34 | Token.Menu.Completions.MultiColumnMeta: "completion-menu.multi-column-meta", 35 | Token.Menu.Completions.ProgressButton: "scrollbar.arrow", # best guess 36 | Token.Menu.Completions.ProgressBar: "scrollbar", # best guess 37 | Token.SelectedText: "selected", 38 | Token.SearchMatch: "search", 39 | Token.SearchMatch.Current: "search.current", 40 | Token.Toolbar: "bottom-toolbar", 41 | Token.Toolbar.Off: "bottom-toolbar.off", 42 | Token.Toolbar.On: "bottom-toolbar.on", 43 | Token.Toolbar.Search: "search-toolbar", 44 | Token.Toolbar.Search.Text: "search-toolbar.text", 45 | Token.Toolbar.System: "system-toolbar", 46 | Token.Toolbar.Arg: "arg-toolbar", 47 | Token.Toolbar.Arg.Text: "arg-toolbar.text", 48 | Token.Toolbar.Transaction.Valid: "bottom-toolbar.transaction.valid", 49 | Token.Toolbar.Transaction.Failed: "bottom-toolbar.transaction.failed", 50 | Token.Output.Header: "output.header", 51 | Token.Output.OddRow: "output.odd-row", 52 | Token.Output.EvenRow: "output.even-row", 53 | } 54 | 55 | # reverse dict for cli_helpers, because they still expect Pygments tokens. 56 | PROMPT_STYLE_TO_TOKEN = {v: k for k, v in TOKEN_TO_PROMPT_STYLE.items()} 57 | 58 | 59 | def style_factory(name, cli_style): 60 | try: 61 | style = pygments.styles.get_style_by_name(name) 62 | except ClassNotFound: 63 | style = pygments.styles.get_style_by_name("native") 64 | 65 | prompt_styles = [] 66 | 67 | for token in cli_style: 68 | # treat as prompt style name (2.0). See default style names here: 69 | # https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/prompt_toolkit/styles/defaults.py 70 | prompt_styles.append((token, cli_style[token])) 71 | 72 | override_style = Style([("bottom-toolbar", "noreverse")]) 73 | return merge_styles([style_from_pygments_cls(style), override_style, Style(prompt_styles)]) 74 | 75 | 76 | def style_factory_output(name, cli_style): 77 | try: 78 | style = pygments.styles.get_style_by_name(name).styles 79 | except ClassNotFound: 80 | style = pygments.styles.get_style_by_name("native").styles 81 | 82 | for token in cli_style: 83 | 84 | if token in PROMPT_STYLE_TO_TOKEN: 85 | token_type = PROMPT_STYLE_TO_TOKEN[token] 86 | style.update({token_type: cli_style[token]}) 87 | else: 88 | # TODO: cli helpers will have to switch to ptk.Style 89 | logger.error("Unhandled style / class name: %s", token) 90 | 91 | class OutputStyle(PygmentsStyle): 92 | default_style = "" 93 | styles = style 94 | 95 | return OutputStyle 96 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/main.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | from __future__ import unicode_literals 16 | 17 | import click 18 | import sys 19 | 20 | from .config import config_location 21 | from .esconnection import ESConnection 22 | from .utils import OutputSettings 23 | from .odfesql_cli import OdfeSqlCli 24 | from .formatter import Formatter 25 | 26 | click.disable_unicode_literals_warning = True 27 | 28 | 29 | @click.command() 30 | @click.argument("endpoint", default="http://localhost:9200") 31 | @click.option("-q", "--query", "query", type=click.STRING, help="Run single query in non-interactive mode") 32 | @click.option("-e", "--explain", "explain", is_flag=True, help="Explain SQL to ES DSL") 33 | @click.option( 34 | "--clirc", 35 | default=config_location() + "config", 36 | envvar="CLIRC", 37 | help="Location of clirc file.", 38 | type=click.Path(dir_okay=False), 39 | ) 40 | @click.option( 41 | "-f", 42 | "--format", 43 | "result_format", 44 | type=click.STRING, 45 | default="jdbc", 46 | help="Specify format of output, jdbc/csv. By default, it's jdbc", 47 | ) 48 | @click.option( 49 | "-v", 50 | "--vertical", 51 | "is_vertical", 52 | is_flag=True, 53 | default=False, 54 | help="Convert output from horizontal to vertical. Only used for non-interactive mode", 55 | ) 56 | @click.option("-u", "--username", help="Username to connect to the Elasticsearch") 57 | @click.option("-w", "--password", help="password corresponding to username") 58 | @click.option( 59 | "-p", 60 | "--pager", 61 | "always_use_pager", 62 | is_flag=True, 63 | default=False, 64 | help="Always use pager to display output. If not specified, smart pager mode will be used according to the \ 65 | length/width of output", 66 | ) 67 | @click.option( 68 | "--aws-auth", 69 | "use_aws_authentication", 70 | is_flag=True, 71 | default=False, 72 | help="Use AWS sigV4 to connect to AWS ELasticsearch domain", 73 | ) 74 | def cli(endpoint, query, explain, clirc, result_format, is_vertical, username, password, always_use_pager, 75 | use_aws_authentication): 76 | """ 77 | Provide endpoint for Elasticsearch client. 78 | By default, it uses http://localhost:9200 to connect. 79 | """ 80 | 81 | if username and password: 82 | http_auth = (username, password) 83 | else: 84 | http_auth = None 85 | 86 | # TODO add validation for endpoint to avoid the cost of connecting to some obviously invalid endpoint 87 | 88 | # handle single query without more interaction with user 89 | if query: 90 | es_executor = ESConnection(endpoint, http_auth, use_aws_authentication) 91 | es_executor.set_connection() 92 | if explain: 93 | output = es_executor.execute_query(query, explain=True, use_console=False) 94 | else: 95 | output = es_executor.execute_query(query, output_format=result_format, use_console=False) 96 | if output and result_format == "jdbc": 97 | settings = OutputSettings(table_format="psql", is_vertical=is_vertical) 98 | formatter = Formatter(settings) 99 | output = formatter.format_output(output) 100 | output = "\n".join(output) 101 | 102 | click.echo(output) 103 | sys.exit(0) 104 | 105 | # use console to interact with user 106 | odfesql_cli = OdfeSqlCli(clirc_file=clirc, always_use_pager=always_use_pager, use_aws_authentication=use_aws_authentication) 107 | odfesql_cli.connect(endpoint, http_auth) 108 | odfesql_cli.run_cli() 109 | 110 | 111 | if __name__ == "__main__": 112 | cli() 113 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/formatter.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import click 16 | import itertools 17 | 18 | from cli_helpers.tabular_output import TabularOutputFormatter 19 | from cli_helpers.tabular_output.preprocessors import format_numbers 20 | 21 | click.disable_unicode_literals_warning = True 22 | 23 | 24 | class Formatter: 25 | """Formatter instance is used to format the data retrieved from Elasticsearch.""" 26 | 27 | def __init__(self, settings): 28 | """A formatter can be customized by passing settings as a parameter.""" 29 | self.settings = settings 30 | self.table_format = "vertical" if self.settings.is_vertical else self.settings.table_format 31 | self.max_width = self.settings.max_width 32 | 33 | def format_array(val): 34 | if val is None: 35 | return self.settings.missingval 36 | if not isinstance(val, list): 37 | return val 38 | return "[" + ",".join(str(format_array(e)) for e in val) + "]" 39 | 40 | def format_arrays(field_data, headers, **_): 41 | field_data = list(field_data) 42 | for row in field_data: 43 | row[:] = [format_array(val) if isinstance(val, list) else val for val in row] 44 | 45 | return field_data, headers 46 | 47 | self.output_kwargs = { 48 | "sep_title": "RECORD {n}", 49 | "sep_character": "-", 50 | "sep_length": (1, 25), 51 | "missing_value": self.settings.missingval, 52 | "preprocessors": (format_numbers, format_arrays), 53 | "disable_numparse": True, 54 | "preserve_whitespace": True, 55 | "style": self.settings.style_output, 56 | } 57 | 58 | def format_output(self, data): 59 | """Format data. 60 | 61 | :param data: raw data get from ES 62 | :return: formatted output, it's either table or vertical format 63 | """ 64 | formatter = TabularOutputFormatter(format_name=self.table_format) 65 | 66 | # parse response data 67 | datarows = data["datarows"] 68 | schema = data["schema"] 69 | total_hits = data["total"] 70 | cur_size = data["size"] 71 | # unused data for now, 72 | fields = [] 73 | types = [] 74 | 75 | # get header and type as lists, for future usage 76 | for i in schema: 77 | fields.append(i["name"]) 78 | types.append(i["type"]) 79 | 80 | output = formatter.format_output(datarows, fields, **self.output_kwargs) 81 | output_message = "fetched rows / total rows = %d/%d" % (cur_size, total_hits) 82 | 83 | # Open Distro for ES sql has a restriction of retrieving 200 rows of data by default 84 | if total_hits > 200 == cur_size: 85 | output_message += "\n" + "Attention: Use LIMIT keyword when retrieving more than 200 rows of data" 86 | 87 | # check width overflow, change format_name for better visual effect 88 | first_line = next(output) 89 | output = itertools.chain([output_message], [first_line], output) 90 | 91 | if len(first_line) > self.max_width: 92 | click.secho(message="Output longer than terminal width", fg="red") 93 | if click.confirm("Do you want to display data vertically for better visual effect?"): 94 | output = formatter.format_output(datarows, fields, format_name="vertical", **self.output_kwargs) 95 | output = itertools.chain([output_message], output) 96 | 97 | # TODO: if decided to add row_limit. Refer to pgcli -> main -> line 866. 98 | 99 | return output 100 | -------------------------------------------------------------------------------- /release-notes/odfe-sql-cli.release-notes-1.7.0.0.md: -------------------------------------------------------------------------------- 1 | ## 2020-05-04 Version 1.7.0.0 2 | 3 | This is the first official release of Open Distro for Elasticsearch SQL CLI 4 | 5 | ODFE SQL CLI is a stand alone Python application and can be launched by a wake word `odfesql`. It serves as a support only for 6 | [Open Distro SQL plugin for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/sql/). User must have ODFE SQL 7 | plugin installed to the Elasticsearch instance for connection. Usr can run this CLI from MacOS and Linux, and connect to any valid Elasticsearch 8 | endpoint such as AWS Elasticsearch. 9 | 10 | ### Features 11 | #### CLI Features 12 | * Feature [#12](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/12): Initial development for SQL CLI 13 | * prototype launch: app -> check connection -> take input -> query ES -> serve raw results(format=jdbc) 14 | * enrich auto-completion corpus 15 | * Convert to vertical output format if fields length is larger than terminal window 16 | * Add style to output fields name. Add logic to confirm choice from user for vertical output 17 | * Add single query without getting into console. Integrate "_explain" api 18 | * Add config base logic. Add pagination for long output 19 | * Add nice little welcome banner. 20 | * Add params -f for format_output (jdbc/raw/csv), -v for vertical display 21 | * Initial implementation of connection to OD cluster and AES with auth 22 | * Create test module and write first test 23 | * Add fake test data. Add test utils to set up connection 24 | * [Test] Add pagination test and query test 25 | * Add Test plan and dependency list 26 | * [Test] Add test case for ConnectionFailExeption 27 | * [Feature] initial implementation of index suggestion during auto-completion 28 | * [Feature] display (data retrieved / total hits), and tell user to use "limit" to get more than 200 lines of data 29 | * Added legal and copyright files, 30 | * Added THIRD PARTY file 31 | * Added setup.py for packaging and releasing 32 | * Feature [#24](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/24): Provide user option to toggle to use AWS sigV4 authentication 33 | (issue: [#23](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/23)) 34 | 35 | #### Testing 36 | * Feature [#28](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/28) :Added tox scripts for testing automation 37 | 38 | #### Documentation 39 | * Change [#22](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/22): Update documentation and CLI naming 40 | (issues: [#21](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/21), [#7](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/17)) 41 | * Change [#32](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/32): Update copyright to 2020 42 | * Change [#33](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/33): Updated package naming and created folder for release notes 43 | * Change [#34](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/34): Added CONTRIBUTORS.md 44 | * Change [#36](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/36): Polish README.md and test_plan.md 45 | 46 | 47 | ### Enhancements 48 | * Enhancement [#31](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/31): Added github action workflow for CI/CD 49 | (issue: [#20](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/21)) 50 | * Enhancement [#35](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/35): Update github action test and build workflow to spin up ES instance 51 | 52 | 53 | ### BugFixes 54 | * BugFix[#12](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/12): Initial development for SQL CLI 55 | * Fix the logic of passing self-constructed settings 56 | * [Fix] get rid of unicode warning. Fix meta info display 57 | * [fix] Refactor executor code 58 | * [Fix] Fix test cases corresponding to fraction display. 59 | * [Fix] fix code style using Black, update documentation and comments 60 | * BugFix[#18](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/18): Fix typos, remove unused dependencies, add .gitignore and legal file 61 | (issue: [#15](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/15)) 62 | * BugFix[#19](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/19): Fix test failures 63 | (issue: [#16](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/16)) 64 | * BugFix[#26](https://github.com/opendistro-for-elasticsearch/sql-cli/pull/26): Update usage gif, fix http/https issue when connect to AWS Elasticsearch (issue: [#25](https://github.com/opendistro-for-elasticsearch/sql-cli/issues/25)) 65 | 66 | 67 | -------------------------------------------------------------------------------- /tests/test_esconnection.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import pytest 16 | import mock 17 | from textwrap import dedent 18 | 19 | from elasticsearch.exceptions import ConnectionError 20 | from elasticsearch import Elasticsearch, RequestsHttpConnection 21 | 22 | from .utils import estest, load_data, run, TEST_INDEX_NAME 23 | from src.odfe_sql_cli.esconnection import ESConnection 24 | 25 | INVALID_ENDPOINT = "http://invalid:9200" 26 | OPEN_DISTRO_ENDPOINT = "https://opedistro:9200" 27 | AES_ENDPOINT = "https://fake.es.amazonaws.com" 28 | AUTH = ("username", "password") 29 | 30 | 31 | class TestExecutor: 32 | def load_data_to_es(self, connection): 33 | doc = {"a": "aws"} 34 | load_data(connection, doc) 35 | 36 | @estest 37 | def test_query(self, connection): 38 | self.load_data_to_es(connection) 39 | 40 | assert run(connection, "select * from %s" % TEST_INDEX_NAME) == dedent( 41 | """\ 42 | fetched rows / total rows = 1/1 43 | +-----+ 44 | | a | 45 | |-----| 46 | | aws | 47 | +-----+""" 48 | ) 49 | 50 | @estest 51 | def test_query_nonexistent_index(self, connection): 52 | self.load_data_to_es(connection) 53 | 54 | expected = { 55 | "reason": "Error occurred in Elasticsearch engine: no such index [non-existed]", 56 | "details": "org.elasticsearch.index.IndexNotFoundException: no such index [non-existed]\nFor more " 57 | "details, please send request for Json format to see the raw response from elasticsearch " 58 | "engine.", 59 | "type": "IndexNotFoundException", 60 | } 61 | 62 | with mock.patch("src.odfe_sql_cli.esconnection.click.secho") as mock_secho: 63 | run(connection, "select * from non-existed") 64 | 65 | mock_secho.assert_called_with(message=str(expected), fg="red") 66 | 67 | def test_connection_fail(self): 68 | test_executor = ESConnection(endpoint=INVALID_ENDPOINT) 69 | err_message = "Can not connect to endpoint %s" % INVALID_ENDPOINT 70 | 71 | with mock.patch("sys.exit") as mock_sys_exit, mock.patch("src.odfe_sql_cli.esconnection.click.secho") as mock_secho: 72 | test_executor.set_connection() 73 | 74 | mock_sys_exit.assert_called() 75 | mock_secho.assert_called_with(message=err_message, fg="red") 76 | 77 | def test_lost_connection(self): 78 | test_esexecutor = ESConnection(endpoint=INVALID_ENDPOINT) 79 | 80 | def side_effect_set_connection(is_reconnected): 81 | if is_reconnected: 82 | pass 83 | else: 84 | return ConnectionError() 85 | 86 | with mock.patch("src.odfe_sql_cli.esconnection.click.secho") as mock_secho, mock.patch.object( 87 | test_esexecutor, "set_connection" 88 | ) as mock_set_connection: 89 | # Assume reconnection success 90 | mock_set_connection.side_effect = side_effect_set_connection(is_reconnected=True) 91 | test_esexecutor.handle_server_close_connection() 92 | 93 | mock_secho.assert_any_call(message="Reconnecting...", fg="green") 94 | mock_secho.assert_any_call(message="Reconnected! Please run query again", fg="green") 95 | # Assume reconnection fail 96 | mock_set_connection.side_effect = side_effect_set_connection(is_reconnected=False) 97 | test_esexecutor.handle_server_close_connection() 98 | 99 | mock_secho.assert_any_call(message="Reconnecting...", fg="green") 100 | mock_secho.assert_any_call( 101 | message="Connection Failed. Check your ES is running and then come back", fg="red" 102 | ) 103 | 104 | def test_reconnection_exception(self): 105 | test_executor = ESConnection(endpoint=INVALID_ENDPOINT) 106 | 107 | with pytest.raises(ConnectionError) as error: 108 | assert test_executor.set_connection(True) 109 | 110 | def test_select_client(self): 111 | od_test_executor = ESConnection(endpoint=OPEN_DISTRO_ENDPOINT, http_auth=AUTH) 112 | aes_test_executor = ESConnection(endpoint=AES_ENDPOINT, use_aws_authentication=True) 113 | 114 | with mock.patch.object(od_test_executor, "get_open_distro_client") as mock_od_client, mock.patch.object( 115 | ESConnection, "is_sql_plugin_installed", return_value=True 116 | ): 117 | od_test_executor.set_connection() 118 | mock_od_client.assert_called() 119 | 120 | with mock.patch.object(aes_test_executor, "get_aes_client") as mock_aes_client, mock.patch.object( 121 | ESConnection, "is_sql_plugin_installed", return_value=True 122 | ): 123 | aes_test_executor.set_connection() 124 | mock_aes_client.assert_called() 125 | 126 | def test_get_od_client(self): 127 | od_test_executor = ESConnection(endpoint=OPEN_DISTRO_ENDPOINT, http_auth=AUTH) 128 | 129 | with mock.patch.object(Elasticsearch, "__init__", return_value=None) as mock_es: 130 | od_test_executor.get_open_distro_client() 131 | 132 | mock_es.assert_called_with( 133 | [OPEN_DISTRO_ENDPOINT], http_auth=AUTH, verify_certs=False, ssl_context=od_test_executor.ssl_context 134 | ) 135 | 136 | def test_get_aes_client(self): 137 | aes_test_executor = ESConnection(endpoint=AES_ENDPOINT, use_aws_authentication=True) 138 | 139 | with mock.patch.object(Elasticsearch, "__init__", return_value=None) as mock_es: 140 | aes_test_executor.get_aes_client() 141 | 142 | mock_es.assert_called_with( 143 | hosts=[AES_ENDPOINT], 144 | http_auth=aes_test_executor.aws_auth, 145 | use_ssl=True, 146 | verify_certs=True, 147 | connection_class=RequestsHttpConnection, 148 | ) 149 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Test and Build Workflow](https://github.com/opendistro-for-elasticsearch/sql-cli/workflows/Test%20and%20Build/badge.svg)](https://github.com/opendistro-for-elasticsearch/sql-cli/actions) 2 | [![Latest Version](https://img.shields.io/pypi/v/odfe-sql-cli.svg)](https://pypi.python.org/pypi/odfe-sql-cli/) 3 | [![Documentation](https://img.shields.io/badge/documentation-blue.svg)](https://opendistro.github.io/for-elasticsearch-docs/docs/sql/cli/) 4 | [![Chat](https://img.shields.io/badge/chat-on%20forums-blue)](https://discuss.opendistrocommunity.dev/c/sql/) 5 | ![PyPi Downloads](https://img.shields.io/pypi/dm/odfe-sql-cli.svg) 6 | ![PRs welcome!](https://img.shields.io/badge/PRs-welcome!-success) 7 | 8 | --- 9 | __NOTE:__ We have merged this repository into [Open Distro for Elasticsearch SQL](https://github.com/opendistro-for-elasticsearch/sql) as of 7/9/20. Please visit [here](https://github.com/opendistro-for-elasticsearch/sql/tree/master/sql-cli) for latest updates on SQL CLI. Thanks. 10 | --- 11 | 12 | # Open Distro for Elasticsearch SQL CLI 13 | 14 | The SQL CLI component in Open Distro for Elasticsearch (ODFE) is a stand-alone Python application and can be launched by a 'wake' word `odfesql`. 15 | 16 | It only supports [Open Distro for Elasticsearch (ODFE) SQL Plugin](https://opendistro.github.io/for-elasticsearch-docs/docs/sql/) 17 | You must have the ODFE SQL plugin installed to your Elasticsearch instance to connect. 18 | Users can run this CLI from MacOS and Linux, and connect to any valid Elasticsearch end-point such as Amazon Elasticsearch Service (AES). 19 | 20 | ![](./screenshots/usage.gif) 21 | 22 | 23 | 24 | ## Features 25 | 26 | * Multi-line input 27 | * Autocomplete for SQL syntax and index names 28 | * Syntax highlighting 29 | * Formatted output: 30 | * Tabular format 31 | * Field names with color 32 | * Enabled horizontal display (by default) and vertical display when output is too wide for your terminal, for better visualization 33 | * Pagination for large output 34 | * Connect to Elasticsearch with/without security enabled on either **Elasticsearch OSS or Amazon Elasticsearch Service domains**. 35 | * Supports loading configuration files 36 | * Supports all SQL plugin queries 37 | 38 | ## Install 39 | 40 | Launch your local Elasticsearch instance and make sure you have the Open Distro for Elasticsearch SQL plugin installed. 41 | 42 | To install the SQL CLI: 43 | 44 | 45 | 1. We suggest you install and activate a python3 virtual environment to avoid changing your local environment: 46 | 47 | ``` 48 | pip install virtualenv 49 | virtualenv venv 50 | cd venv 51 | source ./bin/activate 52 | ``` 53 | 54 | 55 | 1. Install the CLI: 56 | 57 | ``` 58 | pip3 install odfe-sql-cli 59 | ``` 60 | 61 | The SQL CLI only works with Python 3, since Python 2 is no longer maintained since 01/01/2020. See https://pythonclock.org/ 62 | 63 | 64 | 1. To launch the CLI, run: 65 | 66 | ``` 67 | odfesql https://localhost:9200 --username admin —password admin 68 | ``` 69 | By default, the `odfesql` command connects to [http://localhost:9200](http://localhost:9200/). 70 | 71 | 72 | 73 | ## Configure 74 | 75 | When you first launch the SQL CLI, a configuration file is automatically created at `~/.config/odfesql-cli/config` (for MacOS and Linux), the configuration is auto-loaded thereafter. 76 | 77 | You can also configure the following connection properties: 78 | 79 | 80 | * `endpoint`: You do not need to specify an option, anything that follows the launch command `odfesql` is considered as the endpoint. If you do not provide an endpoint, by default, the SQL CLI connects to [http://localhost:9200](http://localhost:9200/). 81 | * `-u/-w`: Supports username and password for HTTP basic authentication, such as: 82 | * Elasticsearch OSS with [Open Distro for Elasticsearch Security Plugin](https://opendistro.github.io/for-elasticsearch-docs/docs/install/plugins/) installed 83 | * Amazon Elasticsearch Service domain with [Fine Grained Access Control](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/fgac.html) enabled 84 | * Elasticsearch with X-pack security enabled 85 | * `--aws-auth`: Turns on AWS sigV4 authentication to connect to an Amazon Elasticsearch Service endpoint. Use with the AWS CLI (`aws configure`) to retrieve the local AWS configuration to authenticate and connect. 86 | 87 | For a list of all available configurations, see [clirc](https://github.com/opendistro-for-elasticsearch/sql-cli/blob/master/src/conf/clirc). 88 | 89 | 90 | 91 | ## Using the CLI 92 | 93 | 1. Save the sample [accounts test data](https://github.com/opendistro-for-elasticsearch/sql/blob/master/src/test/resources/doctest/testdata/accounts.json) file. 94 | 2. Index the sample data. 95 | 96 | ``` 97 | curl -H "Content-Type: application/x-ndjson" -POST https://localhost:9200/data/_bulk -u admin:admin --insecure —data-binary "@accounts.json" 98 | ``` 99 | 100 | 101 | 1. Run a simple SQL command in ODFE SQL CLI: 102 | 103 | ``` 104 | SELECT * FROM accounts; 105 | ``` 106 | 107 | By default, you see a maximum output of 200 rows. To show more results, add a `LIMIT` clause with the desired value. 108 | 109 | The CLI supports all types of query that ODFE SQL supports. Refer to [ODFE SQL basic usage documentation.](https://github.com/opendistro-for-elasticsearch/sql#basic-usage) 110 | 111 | 112 | ## Query options 113 | 114 | Run single query from command line with options 115 | 116 | 117 | * `--help`: help page for options 118 | * `-q`: follow by a single query 119 | * `-f`: support *jdbc/raw* format output 120 | * `-v`: display data vertically 121 | * `-e`: translate sql to DSL 122 | 123 | ## CLI Options 124 | 125 | * `-p`: always use pager to display output 126 | * `--clirc`: provide path of config file to load 127 | 128 | ## Code of Conduct 129 | 130 | This project has adopted an [Open Source Code of Conduct](https://opendistro.github.io/for-elasticsearch/codeofconduct.html). 131 | 132 | 133 | 134 | ## Security issue notifications 135 | 136 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue for security bugs you report. 137 | 138 | ## Licensing 139 | 140 | See the [LICENSE](https://github.com/opendistro-for-elasticsearch/sql-cli/blob/master/LICENSE.TXT) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 141 | 142 | 143 | 144 | ## Copyright 145 | 146 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 147 | 148 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/esconnection.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | import boto3 16 | import click 17 | import logging 18 | import ssl 19 | import sys 20 | import urllib3 21 | 22 | from elasticsearch import Elasticsearch, RequestsHttpConnection 23 | from elasticsearch.exceptions import ConnectionError, RequestError 24 | from elasticsearch.connection import create_ssl_context 25 | from requests_aws4auth import AWS4Auth 26 | 27 | 28 | class ESConnection: 29 | """ESConnection instances are used to set up and maintain client to Elasticsearch cluster, 30 | as well as send user's SQL query to Elasticsearch. 31 | """ 32 | 33 | def __init__(self, endpoint=None, http_auth=None, use_aws_authentication=False): 34 | """Initialize an ESConnection instance. 35 | 36 | Set up client and get indices list. 37 | 38 | :param endpoint: an url in the format of "http:localhost:9200" 39 | :param http_auth: a tuple in the format of (username, password) 40 | """ 41 | self.client = None 42 | self.ssl_context = None 43 | self.es_version = None 44 | self.plugins = None 45 | self.aws_auth = None 46 | self.indices_list = [] 47 | self.endpoint = endpoint 48 | self.http_auth = http_auth 49 | self.use_aws_authentication = use_aws_authentication 50 | 51 | def get_indices(self): 52 | if self.client: 53 | res = self.client.indices.get_alias().keys() 54 | self.indices_list = list(res) 55 | 56 | def get_aes_client(self): 57 | service = "es" 58 | session = boto3.Session() 59 | credentials = session.get_credentials() 60 | region = session.region_name 61 | 62 | if credentials is not None: 63 | self.aws_auth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service) 64 | else: 65 | click.secho(message="Can not retrieve your AWS credentials, check your AWS config", fg="red") 66 | 67 | aes_client = Elasticsearch( 68 | hosts=[self.endpoint], 69 | http_auth=self.aws_auth, 70 | use_ssl=True, 71 | verify_certs=True, 72 | connection_class=RequestsHttpConnection, 73 | ) 74 | 75 | return aes_client 76 | 77 | def get_open_distro_client(self): 78 | ssl_context = self.ssl_context = create_ssl_context() 79 | ssl_context.check_hostname = False 80 | ssl_context.verify_mode = ssl.CERT_NONE 81 | 82 | open_distro_client = Elasticsearch( 83 | [self.endpoint], http_auth=self.http_auth, verify_certs=False, ssl_context=ssl_context 84 | ) 85 | 86 | return open_distro_client 87 | 88 | def is_sql_plugin_installed(self, es_client): 89 | self.plugins = es_client.cat.plugins(params={"s": "component", "v": "true"}) 90 | sql_plugin_name_list = ["opendistro-sql", "opendistro_sql"] 91 | return any(x in self.plugins for x in sql_plugin_name_list) 92 | 93 | def set_connection(self, is_reconnect=False): 94 | urllib3.disable_warnings() 95 | logging.captureWarnings(True) 96 | 97 | if self.http_auth: 98 | es_client = self.get_open_distro_client() 99 | 100 | elif self.use_aws_authentication: 101 | es_client = self.get_aes_client() 102 | else: 103 | es_client = Elasticsearch([self.endpoint], verify_certs=True) 104 | 105 | # check connection. check Open Distro Elasticsearch SQL plugin availability. 106 | try: 107 | if not self.is_sql_plugin_installed(es_client): 108 | click.secho( 109 | message="Must have Open Distro SQL plugin installed in your Elasticsearch " 110 | "instance!\nCheck this out: https://github.com/opendistro-for-elasticsearch/sql", 111 | fg="red", 112 | ) 113 | click.echo(self.plugins) 114 | sys.exit() 115 | 116 | # info() may throw ConnectionError, if connection fails to establish 117 | info = es_client.info() 118 | self.es_version = info["version"]["number"] 119 | self.client = es_client 120 | self.get_indices() 121 | 122 | except ConnectionError as error: 123 | if is_reconnect: 124 | # re-throw error 125 | raise error 126 | else: 127 | click.secho(message="Can not connect to endpoint %s" % self.endpoint, fg="red") 128 | click.echo(repr(error)) 129 | sys.exit(0) 130 | 131 | def handle_server_close_connection(self): 132 | """Used during CLI execution.""" 133 | try: 134 | click.secho(message="Reconnecting...", fg="green") 135 | self.set_connection(is_reconnect=True) 136 | click.secho(message="Reconnected! Please run query again", fg="green") 137 | except ConnectionError as reconnection_err: 138 | click.secho(message="Connection Failed. Check your ES is running and then come back", fg="red") 139 | click.secho(repr(reconnection_err), err=True, fg="red") 140 | 141 | def execute_query(self, query, output_format="jdbc", explain=False, use_console=True): 142 | """ 143 | Handle user input, send SQL query and get response. 144 | 145 | :param use_console: use console to interact with user, otherwise it's single query 146 | :param query: SQL query 147 | :param output_format: jdbc/csv 148 | :param explain: if True, use _explain API. 149 | :return: raw http response 150 | """ 151 | 152 | # TODO: consider add evaluator/handler to filter obviously-invalid input, 153 | # to save cost of http client. 154 | # deal with input 155 | final_query = query.strip().strip(";") 156 | 157 | try: 158 | data = self.client.transport.perform_request( 159 | url="/_opendistro/_sql/_explain" if explain else "/_opendistro/_sql/", 160 | method="POST", 161 | params=None if explain else {"format": output_format}, 162 | body={"query": final_query}, 163 | ) 164 | return data 165 | 166 | # handle client lost during execution 167 | except ConnectionError: 168 | if use_console: 169 | self.handle_server_close_connection() 170 | except RequestError as error: 171 | click.secho(message=str(error.info["error"]), fg="red") 172 | -------------------------------------------------------------------------------- /tests/test_formatter.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | from __future__ import unicode_literals, print_function 16 | 17 | import mock 18 | import pytest 19 | from collections import namedtuple 20 | 21 | from src.odfe_sql_cli.odfesql_cli import OdfeSqlCli, COLOR_CODE_REGEX 22 | from src.odfe_sql_cli.formatter import Formatter 23 | from src.odfe_sql_cli.utils import OutputSettings 24 | 25 | 26 | class TestFormatter: 27 | @pytest.fixture 28 | def pset_pager_mocks(self): 29 | cli = OdfeSqlCli() 30 | with mock.patch("src.odfe_sql_cli.main.click.echo") as mock_echo, mock.patch( 31 | "src.odfe_sql_cli.main.click.echo_via_pager" 32 | ) as mock_echo_via_pager, mock.patch.object(cli, "prompt_app") as mock_app: 33 | yield cli, mock_echo, mock_echo_via_pager, mock_app 34 | 35 | termsize = namedtuple("termsize", ["rows", "columns"]) 36 | test_line = "-" * 10 37 | test_data = [ 38 | (10, 10, "\n".join([test_line] * 7)), 39 | (10, 10, "\n".join([test_line] * 6)), 40 | (10, 10, "\n".join([test_line] * 5)), 41 | (10, 10, "-" * 11), 42 | (10, 10, "-" * 10), 43 | (10, 10, "-" * 9), 44 | ] 45 | 46 | use_pager_when_on = [True, True, False, True, False, False] 47 | 48 | test_ids = [ 49 | "Output longer than terminal height", 50 | "Output equal to terminal height", 51 | "Output shorter than terminal height", 52 | "Output longer than terminal width", 53 | "Output equal to terminal width", 54 | "Output shorter than terminal width", 55 | ] 56 | 57 | pager_test_data = [l + (r,) for l, r in zip(test_data, use_pager_when_on)] 58 | 59 | def test_format_output(self): 60 | settings = OutputSettings(table_format="psql") 61 | formatter = Formatter(settings) 62 | data = { 63 | "schema": [{"name": "name", "type": "text"}, {"name": "age", "type": "long"}], 64 | "total": 1, 65 | "datarows": [["Tim", 24]], 66 | "size": 1, 67 | "status": 200, 68 | } 69 | 70 | results = formatter.format_output(data) 71 | 72 | expected = [ 73 | "fetched rows / total rows = 1/1", 74 | "+--------+-------+", 75 | "| name | age |", 76 | "|--------+-------|", 77 | "| Tim | 24 |", 78 | "+--------+-------+", 79 | ] 80 | assert list(results) == expected 81 | 82 | def test_format_array_output(self): 83 | settings = OutputSettings(table_format="psql") 84 | formatter = Formatter(settings) 85 | data = { 86 | "schema": [{"name": "name", "type": "text"}, {"name": "age", "type": "long"}], 87 | "total": 1, 88 | "datarows": [["Tim", [24, 25]]], 89 | "size": 1, 90 | "status": 200, 91 | } 92 | 93 | results = formatter.format_output(data) 94 | 95 | expected = [ 96 | "fetched rows / total rows = 1/1", 97 | "+--------+---------+", 98 | "| name | age |", 99 | "|--------+---------|", 100 | "| Tim | [24,25] |", 101 | "+--------+---------+", 102 | ] 103 | assert list(results) == expected 104 | 105 | def test_format_output_vertical(self): 106 | settings = OutputSettings(table_format="psql", max_width=1) 107 | formatter = Formatter(settings) 108 | data = { 109 | "schema": [{"name": "name", "type": "text"}, {"name": "age", "type": "long"}], 110 | "total": 1, 111 | "datarows": [["Tim", 24]], 112 | "size": 1, 113 | "status": 200, 114 | } 115 | 116 | expanded = [ 117 | "fetched rows / total rows = 1/1", 118 | "-[ RECORD 1 ]-------------------------", 119 | "name | Tim", 120 | "age | 24", 121 | ] 122 | 123 | with mock.patch("src.odfe_sql_cli.main.click.secho") as mock_secho, mock.patch("src.odfe_sql_cli.main.click.confirm") as mock_confirm: 124 | expanded_results = formatter.format_output(data) 125 | 126 | mock_secho.assert_called_with(message="Output longer than terminal width", fg="red") 127 | mock_confirm.assert_called_with("Do you want to display data vertically for better visual effect?") 128 | 129 | assert "\n".join(expanded_results) == "\n".join(expanded) 130 | 131 | def test_fake_large_output(self): 132 | settings = OutputSettings(table_format="psql") 133 | formatter = Formatter(settings) 134 | fake_large_data = { 135 | "schema": [{"name": "name", "type": "text"}, {"name": "age", "type": "long"}], 136 | "total": 1000, 137 | "datarows": [["Tim", [24, 25]]], 138 | "size": 200, 139 | "status": 200, 140 | } 141 | 142 | results = formatter.format_output(fake_large_data) 143 | 144 | expected = [ 145 | "fetched rows / total rows = 200/1000\n" 146 | "Attention: Use LIMIT keyword when retrieving more than 200 rows of data", 147 | "+--------+---------+", 148 | "| name | age |", 149 | "|--------+---------|", 150 | "| Tim | [24,25] |", 151 | "+--------+---------+", 152 | ] 153 | assert list(results) == expected 154 | 155 | @pytest.mark.parametrize("term_height,term_width,text,use_pager", pager_test_data, ids=test_ids) 156 | def test_pager(self, term_height, term_width, text, use_pager, pset_pager_mocks): 157 | cli, mock_echo, mock_echo_via_pager, mock_cli = pset_pager_mocks 158 | mock_cli.output.get_size.return_value = self.termsize(rows=term_height, columns=term_width) 159 | 160 | cli.echo_via_pager(text) 161 | 162 | if use_pager: 163 | mock_echo.assert_not_called() 164 | mock_echo_via_pager.assert_called() 165 | else: 166 | mock_echo_via_pager.assert_not_called() 167 | mock_echo.assert_called() 168 | 169 | @pytest.mark.parametrize( 170 | "text,expected_length", 171 | [ 172 | ( 173 | "22200K .......\u001b[0m\u001b[91m... .......... ...\u001b[0m\u001b[91m.\u001b[0m\u001b[91m...... " 174 | ".........\u001b[0m\u001b[91m.\u001b[0m\u001b[91m \u001b[0m\u001b[91m.\u001b[0m\u001b[91m.\u001b[" 175 | "0m\u001b[91m.\u001b[0m\u001b[91m.\u001b[0m\u001b[91m...... 50% 28.6K 12m55s", 176 | 78, 177 | ), 178 | ("=\u001b[m=", 2), 179 | ("-\u001b]23\u0007-", 2), 180 | ], 181 | ) 182 | def test_color_pattern(self, text, expected_length): 183 | assert len(COLOR_CODE_REGEX.sub("", text)) == expected_length 184 | -------------------------------------------------------------------------------- /src/odfe_sql_cli/odfesql_cli.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"). 5 | You may not use this file except in compliance with the License. 6 | A copy of the License is located at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | or in the "license" file accompanying this file. This file is distributed 11 | on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 12 | express or implied. See the License for the specific language governing 13 | permissions and limitations under the License. 14 | """ 15 | from __future__ import unicode_literals 16 | 17 | import click 18 | import re 19 | import pyfiglet 20 | import os 21 | import json 22 | 23 | from prompt_toolkit.completion import WordCompleter 24 | from prompt_toolkit.enums import DEFAULT_BUFFER 25 | from prompt_toolkit.shortcuts import PromptSession 26 | from prompt_toolkit.filters import HasFocus, IsDone 27 | from prompt_toolkit.lexers import PygmentsLexer 28 | from prompt_toolkit.layout.processors import ConditionalProcessor, HighlightMatchingBracketProcessor 29 | from prompt_toolkit.auto_suggest import AutoSuggestFromHistory 30 | from pygments.lexers.sql import SqlLexer 31 | 32 | from .config import get_config 33 | from .esconnection import ESConnection 34 | from .esbuffer import es_is_multiline 35 | from .esstyle import style_factory, style_factory_output 36 | from .formatter import Formatter 37 | from .utils import OutputSettings 38 | from . import __version__ 39 | 40 | 41 | # Ref: https://stackoverflow.com/questions/30425105/filter-special-chars-such-as-color-codes-from-shell-output 42 | COLOR_CODE_REGEX = re.compile(r"\x1b(\[.*?[@-~]|\].*?(\x07|\x1b\\))") 43 | 44 | click.disable_unicode_literals_warning = True 45 | 46 | 47 | class OdfeSqlCli: 48 | """OdfeSqlCli instance is used to build and run the ODFE SQL CLI.""" 49 | 50 | def __init__(self, clirc_file=None, always_use_pager=False, use_aws_authentication=False): 51 | # Load conf file 52 | config = self.config = get_config(clirc_file) 53 | literal = self.literal = self._get_literals() 54 | 55 | self.prompt_app = None 56 | self.es_executor = None 57 | self.always_use_pager = always_use_pager 58 | self.use_aws_authentication = use_aws_authentication 59 | self.keywords_list = literal["keywords"] 60 | self.functions_list = literal["functions"] 61 | self.syntax_style = config["main"]["syntax_style"] 62 | self.cli_style = config["colors"] 63 | self.table_format = config["main"]["table_format"] 64 | self.multiline_continuation_char = config["main"]["multiline_continuation_char"] 65 | self.multi_line = config["main"].as_bool("multi_line") 66 | self.multiline_mode = config["main"].get("multi_line_mode", "src") 67 | self.null_string = config["main"].get("null_string", "null") 68 | self.style_output = style_factory_output(self.syntax_style, self.cli_style) 69 | 70 | def build_cli(self): 71 | # TODO: Optimize index suggestion to serve indices options only at the needed position, such as 'from' 72 | indices_list = self.es_executor.indices_list 73 | sql_completer = WordCompleter(self.keywords_list + self.functions_list + indices_list, ignore_case=True) 74 | 75 | # https://stackoverflow.com/a/13726418 denote multiple unused arguments of callback in Python 76 | def get_continuation(width, *_): 77 | continuation = self.multiline_continuation_char * (width - 1) + " " 78 | return [("class:continuation", continuation)] 79 | 80 | prompt_app = PromptSession( 81 | lexer=PygmentsLexer(SqlLexer), 82 | completer=sql_completer, 83 | complete_while_typing=True, 84 | # TODO: add history, refer to pgcli approach 85 | # history=history, 86 | style=style_factory(self.syntax_style, self.cli_style), 87 | prompt_continuation=get_continuation, 88 | multiline=es_is_multiline(self), 89 | auto_suggest=AutoSuggestFromHistory(), 90 | input_processors=[ 91 | ConditionalProcessor( 92 | processor=HighlightMatchingBracketProcessor(chars="[](){}"), 93 | filter=HasFocus(DEFAULT_BUFFER) & ~IsDone(), 94 | ) 95 | ], 96 | tempfile_suffix=".sql", 97 | ) 98 | 99 | return prompt_app 100 | 101 | def run_cli(self): 102 | """ 103 | Print welcome page, goodbye message. 104 | 105 | Run the CLI and keep listening to user's input. 106 | """ 107 | self.prompt_app = self.build_cli() 108 | 109 | settings = OutputSettings( 110 | max_width=self.prompt_app.output.get_size().columns, 111 | style_output=self.style_output, 112 | table_format=self.table_format, 113 | missingval=self.null_string, 114 | ) 115 | 116 | # print Banner 117 | banner = pyfiglet.figlet_format("Open Distro", font="slant") 118 | print(banner) 119 | 120 | # print info on the welcome page 121 | print("Server: Open Distro for ES %s" % self.es_executor.es_version) 122 | print("CLI Version: %s" % __version__) 123 | print("Endpoint: %s" % self.es_executor.endpoint) 124 | 125 | while True: 126 | try: 127 | text = self.prompt_app.prompt(message="odfesql> ") 128 | except KeyboardInterrupt: 129 | continue # Control-C pressed. Try again. 130 | except EOFError: 131 | break # Control-D pressed. 132 | 133 | try: 134 | output = self.es_executor.execute_query(text) 135 | if output: 136 | formatter = Formatter(settings) 137 | formatted_output = formatter.format_output(output) 138 | self.echo_via_pager("\n".join(formatted_output)) 139 | 140 | except Exception as e: 141 | print(repr(e)) 142 | 143 | print("See you next search!") 144 | 145 | def is_too_wide(self, line): 146 | """Will this line be too wide to fit into terminal?""" 147 | if not self.prompt_app: 148 | return False 149 | return len(COLOR_CODE_REGEX.sub("", line)) > self.prompt_app.output.get_size().columns 150 | 151 | def is_too_tall(self, lines): 152 | """Are there too many lines to fit into terminal?""" 153 | if not self.prompt_app: 154 | return False 155 | return len(lines) >= (self.prompt_app.output.get_size().rows - 4) 156 | 157 | def echo_via_pager(self, text, color=None): 158 | lines = text.split("\n") 159 | if self.always_use_pager: 160 | click.echo_via_pager(text, color=color) 161 | 162 | elif self.is_too_tall(lines) or any(self.is_too_wide(l) for l in lines): 163 | click.echo_via_pager(text, color=color) 164 | else: 165 | click.echo(text, color=color) 166 | 167 | def connect(self, endpoint, http_auth=None): 168 | self.es_executor = ESConnection(endpoint, http_auth, self.use_aws_authentication) 169 | self.es_executor.set_connection() 170 | 171 | def _get_literals(self): 172 | """Parse "esliterals.json" with literal type of SQL "keywords" and "functions", which 173 | are SQL keywords and functions supported by Open Distro SQL Plugin. 174 | 175 | :return: a dict that is parsed from esliterals.json 176 | """ 177 | from .esliterals import __file__ as package_root 178 | 179 | package_root = os.path.dirname(package_root) 180 | 181 | literal_file = os.path.join(package_root, "esliterals.json") 182 | with open(literal_file) as f: 183 | literals = json.load(f) 184 | return literals 185 | -------------------------------------------------------------------------------- /LICENSE.TXT: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. -------------------------------------------------------------------------------- /THIRD-PARTY: -------------------------------------------------------------------------------- 1 | ** Boto3; version 1.9.187 -- https://github.com/boto/boto3/ 2 | Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | ** coverage; version 4.5.3 -- https://github.com/nedbat/coveragepy 4 | Copyright 2001 Gareth Rees. All rights reserved. 5 | Copyright 2004-2019 Ned Batchelder. All rights reserved. 6 | 7 | Except where noted otherwise, this software is licensed under the Apache 8 | License, Version 2.0 (the "License"); you may not use this work except in 9 | compliance with the License. You may obtain a copy of the License at 10 | 11 | http://www.apache.org/licenses/LICENSE-2.0 12 | 13 | Unless required by applicable law or agreed to in writing, software 14 | distributed under the License is distributed on an "AS IS" BASIS, 15 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | See the License for the specific language governing permissions and 17 | limitations under the License. 18 | ** elasticsearch 6.5.4; version 6.5.4 -- 19 | https://github.com/elastic/elasticsearch/tree/v6.5.4 20 | Elasticsearch 21 | Copyright 2009-2018 Elasticsearch 22 | 23 | This product includes software developed by The Apache Software 24 | Foundation (http://www.apache.org/). 25 | ** twine; version 1.13.0 -- https://github.com/pypa/twine 26 | none 27 | 28 | Apache License 29 | 30 | Version 2.0, January 2004 31 | 32 | http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND 33 | DISTRIBUTION 34 | 35 | 1. Definitions. 36 | 37 | "License" shall mean the terms and conditions for use, reproduction, and 38 | distribution as defined by Sections 1 through 9 of this document. 39 | 40 | "Licensor" shall mean the copyright owner or entity authorized by the 41 | copyright owner that is granting the License. 42 | 43 | "Legal Entity" shall mean the union of the acting entity and all other 44 | entities that control, are controlled by, or are under common control 45 | with that entity. For the purposes of this definition, "control" means 46 | (i) the power, direct or indirect, to cause the direction or management 47 | of such entity, whether by contract or otherwise, or (ii) ownership of 48 | fifty percent (50%) or more of the outstanding shares, or (iii) 49 | beneficial ownership of such entity. 50 | 51 | "You" (or "Your") shall mean an individual or Legal Entity exercising 52 | permissions granted by this License. 53 | 54 | "Source" form shall mean the preferred form for making modifications, 55 | including but not limited to software source code, documentation source, 56 | and configuration files. 57 | 58 | "Object" form shall mean any form resulting from mechanical 59 | transformation or translation of a Source form, including but not limited 60 | to compiled object code, generated documentation, and conversions to 61 | other media types. 62 | 63 | "Work" shall mean the work of authorship, whether in Source or Object 64 | form, made available under the License, as indicated by a copyright 65 | notice that is included in or attached to the work (an example is 66 | provided in the Appendix below). 67 | 68 | "Derivative Works" shall mean any work, whether in Source or Object form, 69 | that is based on (or derived from) the Work and for which the editorial 70 | revisions, annotations, elaborations, or other modifications represent, 71 | as a whole, an original work of authorship. For the purposes of this 72 | License, Derivative Works shall not include works that remain separable 73 | from, or merely link (or bind by name) to the interfaces of, the Work and 74 | Derivative Works thereof. 75 | 76 | "Contribution" shall mean any work of authorship, including the original 77 | version of the Work and any modifications or additions to that Work or 78 | Derivative Works thereof, that is intentionally submitted to Licensor for 79 | inclusion in the Work by the copyright owner or by an individual or Legal 80 | Entity authorized to submit on behalf of the copyright owner. For the 81 | purposes of this definition, "submitted" means any form of electronic, 82 | verbal, or written communication sent to the Licensor or its 83 | representatives, including but not limited to communication on electronic 84 | mailing lists, source code control systems, and issue tracking systems 85 | that are managed by, or on behalf of, the Licensor for the purpose of 86 | discussing and improving the Work, but excluding communication that is 87 | conspicuously marked or otherwise designated in writing by the copyright 88 | owner as "Not a Contribution." 89 | 90 | "Contributor" shall mean Licensor and any individual or Legal Entity on 91 | behalf of whom a Contribution has been received by Licensor and 92 | subsequently incorporated within the Work. 93 | 94 | 2. Grant of Copyright License. Subject to the terms and conditions of this 95 | License, each Contributor hereby grants to You a perpetual, worldwide, 96 | non-exclusive, no-charge, royalty-free, irrevocable copyright license to 97 | reproduce, prepare Derivative Works of, publicly display, publicly perform, 98 | sublicense, and distribute the Work and such Derivative Works in Source or 99 | Object form. 100 | 101 | 3. Grant of Patent License. Subject to the terms and conditions of this 102 | License, each Contributor hereby grants to You a perpetual, worldwide, 103 | non-exclusive, no-charge, royalty-free, irrevocable (except as stated in 104 | this section) patent license to make, have made, use, offer to sell, sell, 105 | import, and otherwise transfer the Work, where such license applies only to 106 | those patent claims licensable by such Contributor that are necessarily 107 | infringed by their Contribution(s) alone or by combination of their 108 | Contribution(s) with the Work to which such Contribution(s) was submitted. 109 | If You institute patent litigation against any entity (including a 110 | cross-claim or counterclaim in a lawsuit) alleging that the Work or a 111 | Contribution incorporated within the Work constitutes direct or contributory 112 | patent infringement, then any patent licenses granted to You under this 113 | License for that Work shall terminate as of the date such litigation is 114 | filed. 115 | 116 | 4. Redistribution. You may reproduce and distribute copies of the Work or 117 | Derivative Works thereof in any medium, with or without modifications, and 118 | in Source or Object form, provided that You meet the following conditions: 119 | 120 | (a) You must give any other recipients of the Work or Derivative Works a 121 | copy of this License; and 122 | 123 | (b) You must cause any modified files to carry prominent notices stating 124 | that You changed the files; and 125 | 126 | (c) You must retain, in the Source form of any Derivative Works that You 127 | distribute, all copyright, patent, trademark, and attribution notices 128 | from the Source form of the Work, excluding those notices that do not 129 | pertain to any part of the Derivative Works; and 130 | 131 | (d) If the Work includes a "NOTICE" text file as part of its 132 | distribution, then any Derivative Works that You distribute must include 133 | a readable copy of the attribution notices contained within such NOTICE 134 | file, excluding those notices that do not pertain to any part of the 135 | Derivative Works, in at least one of the following places: within a 136 | NOTICE text file distributed as part of the Derivative Works; within the 137 | Source form or documentation, if provided along with the Derivative 138 | Works; or, within a display generated by the Derivative Works, if and 139 | wherever such third-party notices normally appear. The contents of the 140 | NOTICE file are for informational purposes only and do not modify the 141 | License. You may add Your own attribution notices within Derivative Works 142 | that You distribute, alongside or as an addendum to the NOTICE text from 143 | the Work, provided that such additional attribution notices cannot be 144 | construed as modifying the License. 145 | 146 | You may add Your own copyright statement to Your modifications and may 147 | provide additional or different license terms and conditions for use, 148 | reproduction, or distribution of Your modifications, or for any such 149 | Derivative Works as a whole, provided Your use, reproduction, and 150 | distribution of the Work otherwise complies with the conditions stated in 151 | this License. 152 | 153 | 5. Submission of Contributions. Unless You explicitly state otherwise, any 154 | Contribution intentionally submitted for inclusion in the Work by You to the 155 | Licensor shall be under the terms and conditions of this License, without 156 | any additional terms or conditions. Notwithstanding the above, nothing 157 | herein shall supersede or modify the terms of any separate license agreement 158 | you may have executed with Licensor regarding such Contributions. 159 | 160 | 6. Trademarks. This License does not grant permission to use the trade 161 | names, trademarks, service marks, or product names of the Licensor, except 162 | as required for reasonable and customary use in describing the origin of the 163 | Work and reproducing the content of the NOTICE file. 164 | 165 | 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in 166 | writing, Licensor provides the Work (and each Contributor provides its 167 | Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 168 | KIND, either express or implied, including, without limitation, any 169 | warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or 170 | FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining 171 | the appropriateness of using or redistributing the Work and assume any risks 172 | associated with Your exercise of permissions under this License. 173 | 174 | 8. Limitation of Liability. In no event and under no legal theory, whether 175 | in tort (including negligence), contract, or otherwise, unless required by 176 | applicable law (such as deliberate and grossly negligent acts) or agreed to 177 | in writing, shall any Contributor be liable to You for damages, including 178 | any direct, indirect, special, incidental, or consequential damages of any 179 | character arising as a result of this License or out of the use or inability 180 | to use the Work (including but not limited to damages for loss of goodwill, 181 | work stoppage, computer failure or malfunction, or any and all other 182 | commercial damages or losses), even if such Contributor has been advised of 183 | the possibility of such damages. 184 | 185 | 9. Accepting Warranty or Additional Liability. While redistributing the Work 186 | or Derivative Works thereof, You may choose to offer, and charge a fee for, 187 | acceptance of support, warranty, indemnity, or other liability obligations 188 | and/or rights consistent with this License. However, in accepting such 189 | obligations, You may act only on Your own behalf and on Your sole 190 | responsibility, not on behalf of any other Contributor, and only if You 191 | agree to indemnify, defend, and hold each Contributor harmless for any 192 | liability incurred by, or claims asserted against, such Contributor by 193 | reason of your accepting any such warranty or additional liability. END OF 194 | TERMS AND CONDITIONS 195 | 196 | APPENDIX: How to apply the Apache License to your work. 197 | 198 | To apply the Apache License to your work, attach the following boilerplate 199 | notice, with the fields enclosed by brackets "[]" replaced with your own 200 | identifying information. (Don't include the brackets!) The text should be 201 | enclosed in the appropriate comment syntax for the file format. We also 202 | recommend that a file or class name and description of purpose be included on 203 | the same "printed page" as the copyright notice for easier identification 204 | within third-party archives. 205 | 206 | Copyright [yyyy] [name of copyright owner] 207 | 208 | Licensed under the Apache License, Version 2.0 (the "License"); 209 | 210 | you may not use this file except in compliance with the License. 211 | 212 | You may obtain a copy of the License at 213 | 214 | http://www.apache.org/licenses/LICENSE-2.0 215 | 216 | Unless required by applicable law or agreed to in writing, software 217 | 218 | distributed under the License is distributed on an "AS IS" BASIS, 219 | 220 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 221 | 222 | See the License for the specific language governing permissions and 223 | 224 | limitations under the License. 225 | 226 | * For Boto3 see also this required NOTICE: 227 | Copyright 2013-2017 Amazon.com, Inc. or its affiliates. All Rights 228 | Reserved. 229 | * For coverage see also this required NOTICE: 230 | Copyright 2001 Gareth Rees. All rights reserved. 231 | Copyright 2004-2019 Ned Batchelder. All rights reserved. 232 | 233 | Except where noted otherwise, this software is licensed under the Apache 234 | License, Version 2.0 (the "License"); you may not use this work except in 235 | compliance with the License. You may obtain a copy of the License at 236 | 237 | http://www.apache.org/licenses/LICENSE-2.0 238 | 239 | Unless required by applicable law or agreed to in writing, software 240 | distributed under the License is distributed on an "AS IS" BASIS, 241 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 242 | See the License for the specific language governing permissions and 243 | limitations under the License. 244 | * For elasticsearch 6.5.4 see also this required NOTICE: 245 | Elasticsearch 246 | Copyright 2009-2018 Elasticsearch 247 | 248 | This product includes software developed by The Apache Software 249 | Foundation (http://www.apache.org/). 250 | * For twine see also this required NOTICE: 251 | none 252 | 253 | ------ 254 | 255 | ** mock; version 3.0.5 -- https://github.com/testing-cabal/mock 256 | Copyright (c) 2003-2013, Michael Foord & the mock team 257 | All rights reserved. 258 | 259 | Copyright (c) 2003-2013, Michael Foord & the mock team 260 | All rights reserved. 261 | 262 | Redistribution and use in source and binary forms, with or without 263 | modification, are permitted provided that the following conditions are 264 | met: 265 | 266 | * Redistributions of source code must retain the above copyright 267 | notice, this list of conditions and the following disclaimer. 268 | 269 | * Redistributions in binary form must reproduce the above 270 | copyright notice, this list of conditions and the following 271 | disclaimer in the documentation and/or other materials provided 272 | with the distribution. 273 | 274 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 275 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 276 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 277 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 278 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 279 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 280 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 281 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 282 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 283 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 284 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 285 | 286 | ------ 287 | 288 | ** cli-helpers; version 1.2.1 -- https://github.com/dbcli/cli_helpers 289 | Copyright (c) 2017, dbcli 290 | All rights reserved. 291 | 292 | Copyright (c) 2017, dbcli 293 | All rights reserved. 294 | 295 | Redistribution and use in source and binary forms, with or without 296 | modification, are permitted provided that the following conditions are met: 297 | 298 | * Redistributions of source code must retain the above copyright notice, this 299 | list of conditions and the following disclaimer. 300 | 301 | * Redistributions in binary form must reproduce the above copyright notice, 302 | this list of conditions and the following disclaimer in the documentation 303 | and/or other materials provided with the distribution. 304 | 305 | * Neither the name of dbcli nor the names of its 306 | contributors may be used to endorse or promote products derived from 307 | this software without specific prior written permission. 308 | 309 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 310 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 311 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 312 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 313 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 314 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 315 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 316 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 317 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 318 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 319 | 320 | ------ 321 | 322 | ** Pygments; version 2.4.2 -- 323 | https://bitbucket.org/birkenfeld/pygments-main/src/default/ 324 | Copyright (c) 2006-2019 by the respective authors (see AUTHORS file). 325 | All rights reserved. 326 | 327 | Copyright (c) 2006-2019 by the respective authors (see AUTHORS file). 328 | All rights reserved. 329 | 330 | Redistribution and use in source and binary forms, with or without 331 | modification, are permitted provided that the following conditions are 332 | met: 333 | 334 | * Redistributions of source code must retain the above copyright 335 | notice, this list of conditions and the following disclaimer. 336 | 337 | * Redistributions in binary form must reproduce the above copyright 338 | notice, this list of conditions and the following disclaimer in the 339 | documentation and/or other materials provided with the distribution. 340 | 341 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 342 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 343 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 344 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 345 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 346 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 347 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 348 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 349 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 350 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 351 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 352 | 353 | ------ 354 | 355 | ** prompt-toolkit; version 2 -- 356 | https://github.com/prompt-toolkit/python-prompt-toolkit 357 | Copyright (c) 2014, Jonathan Slenders 358 | All rights reserved. 359 | 360 | Copyright (c) 2014, Jonathan Slenders 361 | All rights reserved. 362 | 363 | Redistribution and use in source and binary forms, with or without 364 | modification, 365 | are permitted provided that the following conditions are met: 366 | 367 | * Redistributions of source code must retain the above copyright notice, this 368 | list of conditions and the following disclaimer. 369 | 370 | * Redistributions in binary form must reproduce the above copyright notice, 371 | this 372 | list of conditions and the following disclaimer in the documentation and/or 373 | other materials provided with the distribution. 374 | 375 | * Neither the name of the {organization} nor the names of its 376 | contributors may be used to endorse or promote products derived from 377 | this software without specific prior written permission. 378 | 379 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 380 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 381 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 382 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 383 | FOR 384 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 385 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 386 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 387 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 388 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 389 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 390 | 391 | ------ 392 | 393 | ** click; version 7.0 -- https://click.palletsprojects.com/en/7.x/ 394 | Copyright © 2014 by the Pallets team. 395 | 396 | Copyright © 2014 by the Pallets team. 397 | 398 | Some rights reserved. 399 | 400 | Redistribution and use in source and binary forms of the software as well as 401 | documentation, with or without modification, are permitted provided that the 402 | following conditions are met: 403 | 404 | Redistributions of source code must retain the above copyright notice, this 405 | list of conditions and the following disclaimer. 406 | Redistributions in binary form must reproduce the above copyright notice, this 407 | list of conditions and the following disclaimer in the documentation and/or 408 | other materials provided with the distribution. 409 | Neither the name of the copyright holder nor the names of its contributors may 410 | be used to endorse or promote products derived from this software without 411 | specific prior written permission. 412 | THIS SOFTWARE AND DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND 413 | CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 414 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 415 | PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR 416 | CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 417 | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT 418 | OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 419 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 420 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 421 | IN ANY WAY OUT OF THE USE OF THIS SOFTWARE AND DOCUMENTATION, EVEN IF ADVISED 422 | OF THE POSSIBILITY OF SUCH DAMAGE. 423 | 424 | Click uses parts of optparse written by Gregory P. Ward and maintained by the 425 | Python Software Foundation. This is limited to code in parser.py. 426 | 427 | Copyright © 2001-2006 Gregory P. Ward. All rights reserved. Copyright © 428 | 2002-2006 Python Software Foundation. All rights reserved. 429 | 430 | ------ 431 | 432 | ** pexpect; version 3.3 -- https://github.com/pexpect/pexpect 433 | http://opensource.org/licenses/isc-license.txt 434 | 435 | Copyright (c) 2013-2016, Pexpect development team 436 | Copyright (c) 2012, Noah Spurrier 437 | 438 | ISC LICENSE 439 | 440 | This license is approved by the OSI and FSF as GPL-compatible. 441 | http://opensource.org/licenses/isc-license.txt 442 | 443 | Copyright (c) 2013-2014, Pexpect development team 444 | Copyright (c) 2012, Noah Spurrier 445 | 446 | Permission to use, copy, modify, and/or distribute this software for any 447 | purpose with or without fee is hereby granted, provided that the above 448 | copyright notice and this permission notice appear in all copies. 449 | 450 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 451 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 452 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 453 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 454 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 455 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 456 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 457 | 458 | ------ 459 | 460 | ** pyfiglet; version 0.8.post1 -- https://github.com/pwaller/pyfiglet 461 | Copyright © 2007-2018 462 | Christopher Jones 463 | Stefano Rivera 464 | Peter Waller 465 | And various contributors (see git history) 466 | 467 | PyFiglet: An implementation of figlet written in Python 468 | 469 | The MIT License (MIT) 470 | 471 | Copyright © 2007-2018 472 | Christopher Jones 473 | Stefano Rivera 474 | Peter Waller 475 | And various contributors (see git history). 476 | 477 | Permission is hereby granted, free of charge, to any person obtaining a copy of 478 | this software and associated documentation files (the “Software”), to deal in 479 | the Software without restriction, including without limitation the rights to 480 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 481 | of the Software, and to permit persons to whom the Software is furnished to do 482 | so, subject to the following conditions: 483 | 484 | The above copyright notice and this permission notice shall be included in all 485 | copies or substantial portions of the Software. 486 | 487 | THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 488 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 489 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 490 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 491 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 492 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 493 | DEALINGS IN THE SOFTWARE. 494 | 495 | ------ 496 | 497 | ** pytest; version 4.6.3 -- https://docs.pytest.org/en/latest/ 498 | Copyright (c) 2004-2017 Holger Krekel and others 499 | 500 | The MIT License (MIT) 501 | 502 | Copyright (c) 2004-2017 Holger Krekel and others 503 | 504 | Permission is hereby granted, free of charge, to any person obtaining a copy of 505 | this software and associated documentation files (the "Software"), to deal in 506 | the Software without restriction, including without limitation the rights to 507 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 508 | of the Software, and to permit persons to whom the Software is furnished to do 509 | so, subject to the following conditions: 510 | 511 | The above copyright notice and this permission notice shall be included in all 512 | copies or substantial portions of the Software. 513 | 514 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 515 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 516 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 517 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 518 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 519 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 520 | SOFTWARE. 521 | 522 | ------ 523 | 524 | ** setuptools; version v40.0.0 -- https://github.com/pypa/setuptools 525 | Copyright (C) 2016 Jason R Coombs 526 | 527 | Copyright (C) 2016 Jason R Coombs 528 | 529 | Permission is hereby granted, free of charge, to any person obtaining a copy of 530 | this software and associated documentation files (the "Software"), to deal in 531 | the Software without restriction, including without limitation the rights to 532 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 533 | of the Software, and to permit persons to whom the Software is furnished to do 534 | so, subject to the following conditions: 535 | 536 | The above copyright notice and this permission notice shall be included in all 537 | copies or substantial portions of the Software. 538 | 539 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 540 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 541 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 542 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 543 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 544 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 545 | SOFTWARE. 546 | 547 | ------ 548 | 549 | ** requests-aws4auth; version 0.9 -- 550 | https://github.com/sam-washington/requests-aws4auth 551 | requests-aws4auth includes the six library. 552 | 553 | six License 554 | =========== 555 | 556 | This is the MIT license: http://www.opensource.org/licenses/mit-license.php 557 | 558 | Copyright (c) 2010-2015 Benjamin Peterson 559 | 560 | Permission is hereby granted, free of charge, to any person obtaining a copy 561 | of this software and associated documentation files (the "Software"), to deal 562 | in the Software without restriction, including without limitation the rights 563 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 564 | copies of the Software, and to permit persons to whom the Software is 565 | furnished to do so, subject to the following conditions: 566 | 567 | The above copyright notice and this permission notice shall be included in all 568 | copies or substantial portions of the Software. 569 | 570 | The MIT License (MIT) 571 | 572 | Copyright (c) 2015 Sam Washington 573 | 574 | Permission is hereby granted, free of charge, to any person obtaining a copy 575 | of this software and associated documentation files (the "Software"), to deal 576 | in the Software without restriction, including without limitation the rights 577 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 578 | copies of the Software, and to permit persons to whom the Software is 579 | furnished to do so, subject to the following conditions: 580 | 581 | The above copyright notice and this permission notice shall be included in all 582 | copies or substantial portions of the Software. 583 | 584 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 585 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 586 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 587 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 588 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 589 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 590 | SOFTWARE. --------------------------------------------------------------------------------