├── .github ├── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── pull_request_template.md └── workflows │ ├── build.yml │ ├── codeql-analysis.yml │ └── release.yml ├── .gitignore ├── .pylintrc ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── SECURITY.md ├── azure-kusto-data ├── MANIFEST.in ├── README.rst ├── azure-kusto-data.iml ├── azure │ ├── __init__.py │ └── kusto │ │ ├── __init__.py │ │ └── data │ │ ├── __init__.py │ │ ├── _cloud_settings.py │ │ ├── _converters.py │ │ ├── _decorators.py │ │ ├── _models.py │ │ ├── _string_utils.py │ │ ├── _telemetry.py │ │ ├── _token_providers.py │ │ ├── _version.py │ │ ├── aio │ │ ├── __init__.py │ │ ├── _models.py │ │ ├── client.py │ │ ├── response.py │ │ └── streaming_response.py │ │ ├── client.py │ │ ├── client_base.py │ │ ├── client_details.py │ │ ├── client_request_properties.py │ │ ├── data_format.py │ │ ├── env_utils.py │ │ ├── exceptions.py │ │ ├── helpers.py │ │ ├── kcsb.json │ │ ├── kcsb.py │ │ ├── kusto_trusted_endpoints.py │ │ ├── py.typed │ │ ├── response.py │ │ ├── security.py │ │ ├── streaming_response.py │ │ └── wellKnownKustoEndpoints.json ├── setup.cfg ├── setup.py ├── tests │ ├── __init__.py │ ├── aio │ │ ├── __init__.py │ │ ├── test_async_token_providers.py │ │ └── test_kusto_client.py │ ├── input │ │ ├── adminthenquery.json │ │ ├── alternative_order.json │ │ ├── big_source.csv │ │ ├── dataframe.json │ │ ├── deft.json │ │ ├── deft_with_progressive_result.json │ │ ├── dynamic.json │ │ ├── null_values.json │ │ ├── pandas_bool.json │ │ ├── progressive_result.json │ │ ├── query_partial_results_defer_is_false.json │ │ ├── query_partial_results_defer_is_true.json │ │ ├── versionshowcommandresult.json │ │ └── zero_results.json │ ├── kusto_client_common.py │ ├── sample.py │ ├── test_client_request_properties.py │ ├── test_cloud_settings.py │ ├── test_converter.py │ ├── test_e2e_data.py │ ├── test_exceptions.py │ ├── test_functional.py │ ├── test_helpers.py │ ├── test_http_adapter_with_socket_options.py │ ├── test_kusto_client.py │ ├── test_kusto_connection_string_builder.py │ ├── test_models.py │ ├── test_security.py │ ├── test_streaming_query.py │ ├── test_telemetry.py │ ├── test_token_providers.py │ └── test_well_known_kusto_endpoints.py └── tox.ini ├── azure-kusto-ingest ├── MANIFEST.in ├── README.rst ├── azure-kusto-ingest.iml ├── azure │ ├── __init__.py │ └── kusto │ │ ├── __init__.py │ │ └── ingest │ │ ├── __init__.py │ │ ├── _ingest_telemetry.py │ │ ├── _ranked_storage_account.py │ │ ├── _resource_manager.py │ │ ├── _status_q.py │ │ ├── _storage_account_set.py │ │ ├── _stream_extensions.py │ │ ├── _version.py │ │ ├── base_ingest_client.py │ │ ├── descriptors.py │ │ ├── exceptions.py │ │ ├── ingest_client.py │ │ ├── ingestion_blob_info.py │ │ ├── ingestion_properties.py │ │ ├── managed_streaming_ingest_client.py │ │ ├── py.typed │ │ ├── status.py │ │ └── streaming_ingest_client.py ├── setup.cfg ├── setup.py ├── tests │ ├── input │ │ ├── __init__.py │ │ ├── dataset.csv │ │ ├── dataset.csv.gz │ │ ├── dataset.csv.zip │ │ ├── dataset.json │ │ ├── dataset.jsonz.gz │ │ └── dataset.tsv │ ├── sample.py │ ├── test_connection_string.py │ ├── test_descriptors.py │ ├── test_e2e_ingest.py │ ├── test_ingest_telemetry.py │ ├── test_ingestion_blob_info.py │ ├── test_ingestion_properties.py │ ├── test_kusto_ingest_client.py │ ├── test_kusto_streaming_ingest_client.py │ ├── test_managed_streaming_ingest.py │ ├── test_status_q.py │ └── test_storage_account_set.py └── tox.ini ├── back_to_black.bat ├── build_packages.py ├── codecov.yml ├── dev_requirements.txt ├── pyproject.toml ├── quick_start ├── README.md ├── dataset.csv ├── dataset.json ├── kusto_sample_config.json ├── oneclick_instruction_box.md ├── quick_start.iml ├── requirements.txt ├── sample_app.py └── utils.py ├── setup.cfg ├── setup.py └── test.bat /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | #### Code Sample, a copy-pastable example if possible 11 | 12 | ```python 13 | # Your code here 14 | 15 | ``` 16 | #### Problem description 17 | 18 | [this should explain **why** the current behavior is a problem and why the expected output is a better solution.] 19 | 20 | #### If query related, does it happen on other platforms (Kusto Web UI, Kusto Explorer)? 21 | 22 | [this step is to help pin point problems that are only specific to this platform.] 23 | 24 | 25 | #### Output of ``pip freeze`` 26 | 27 |
28 | 29 | [paste the output of ``pip freeze`` here below this line] 30 | 31 |
32 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | ### Added 2 | ### Changed 3 | ### Fixed 4 | -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | # This workflow will install the dependencies, run tests and lint every push 2 | 3 | name: Python package 4 | permissions: 5 | checks: write 6 | pull-requests: write 7 | id-token: write 8 | contents: read 9 | 10 | on: 11 | push: 12 | branches: [ master ] 13 | pull_request: 14 | jobs: 15 | build: 16 | runs-on: ubuntu-latest 17 | environment: build 18 | 19 | strategy: 20 | fail-fast: false 21 | matrix: 22 | python-version: [ '3.8', '3.9', '3.10', '3.11', '3.12', '3.13' ] 23 | env: 24 | PYTHON: ${{ matrix.python-version }} 25 | 26 | steps: 27 | - name: Azure login 28 | uses: azure/login@v2 29 | with: 30 | client-id: ${{ secrets.AZURE_CLIENT_ID }} 31 | tenant-id: ${{ secrets.AZURE_TENANT_ID }} 32 | subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} 33 | - uses: actions/checkout@v4 34 | - name: Set up Python ${{ matrix.python-version }} 35 | uses: actions/setup-python@v4 36 | with: 37 | python-version: ${{ matrix.python-version }} 38 | - name: Install dependencies 39 | run: | 40 | python -m pip install --upgrade pip --user 41 | pip install -r dev_requirements.txt --user 42 | pip install pytest-cov --user 43 | pip install ./azure-kusto-data[aio,pandas] ./azure-kusto-ingest[aio,pandas] --user 44 | pip freeze 45 | # We have to use an old version of this plugin, as the new one assumes python 3.8 46 | - uses: psf/black@c8f1a5542c257491e1e093b1404481ece7f7e02c 47 | with: 48 | options: "--check --diff --line-length 160" 49 | version: "23.3.0" 50 | - name: EtoE Test with pytest 51 | env: 52 | TEST_DATABASE: ${{ secrets.TEST_DATABASE }} 53 | TEST_BLOB: ${{ secrets.TEST_BLOB }} 54 | ENGINE_CONNECTION_STRING: ${{ secrets.ENGINE_CONNECTION_STRING }} 55 | APPLICATION_INSIGHTS_ENGINE_CONNECTION_STRING: https://ade.applicationinsights.io/subscriptions/12534eb3-8109-4d84-83ad-576c0d5e1d06/resourcegroups/clients_e2e_test/providers/microsoft.insights/components/kusto-e2e-app-insights 56 | APPLICATION_INSIGHTS_TEST_DATABASE: kusto-e2e-app-insights 57 | run: | 58 | pytest -v . --junit-xml pytest.xml --cov=/home/runner/.local/lib/python${{ matrix.python-version }}/site-packages/azure/kusto --cov-report=xml:coverage2.xml 59 | - name: Upload Unit Test Results 60 | if: always() 61 | uses: actions/upload-artifact@v4 62 | with: 63 | name: Unit Test Results (Python ${{ matrix.python-version }}) 64 | path: pytest*.xml 65 | 66 | - name: Upload coverage to Codecov 67 | uses: codecov/codecov-action@v2 68 | with: 69 | env_vars: PYTHON 70 | 71 | 72 | publish-test-results: 73 | name: "Publish Unit Tests Results" 74 | needs: build 75 | runs-on: ubuntu-latest 76 | if: always() 77 | permissions: 78 | checks: write 79 | pull-requests: write 80 | 81 | steps: 82 | - name: Download Artifacts 83 | uses: actions/download-artifact@v4 84 | with: 85 | path: artifacts 86 | 87 | - name: Publish Unit Test Results 88 | uses: EnricoMi/publish-unit-test-result-action@v2 89 | with: 90 | files: artifacts/**/*.xml 91 | -------------------------------------------------------------------------------- /.github/workflows/codeql-analysis.yml: -------------------------------------------------------------------------------- 1 | # For most projects, this workflow file will not need changing; you simply need 2 | # to commit it to your repository. 3 | # 4 | # You may wish to alter this file to override the set of languages analyzed, 5 | # or to provide custom queries or build logic. 6 | # 7 | # ******** NOTE ******** 8 | # We have attempted to detect the languages in your repository. Please check 9 | # the `language` matrix defined below to confirm you have the correct set of 10 | # supported CodeQL languages. 11 | # 12 | name: "CodeQL" 13 | 14 | on: 15 | push: 16 | branches: [ "master" ] 17 | pull_request: 18 | # The branches below must be a subset of the branches above 19 | branches: [ "master" ] 20 | schedule: 21 | - cron: '19 11 * * 4' 22 | 23 | jobs: 24 | analyze: 25 | name: Analyze 26 | runs-on: ubuntu-latest 27 | permissions: 28 | actions: read 29 | contents: read 30 | security-events: write 31 | 32 | strategy: 33 | fail-fast: false 34 | matrix: 35 | language: [ 'python' ] 36 | # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ] 37 | # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support 38 | 39 | steps: 40 | - name: Checkout repository 41 | uses: actions/checkout@v3 42 | 43 | # Initializes the CodeQL tools for scanning. 44 | - name: Initialize CodeQL 45 | uses: github/codeql-action/init@v2 46 | with: 47 | languages: ${{ matrix.language }} 48 | # If you wish to specify custom queries, you can do so here or in a config file. 49 | # By default, queries listed here will override any specified in a config file. 50 | # Prefix the list here with "+" to use these queries and those in the config file. 51 | 52 | # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs 53 | # queries: security-extended,security-and-quality 54 | 55 | 56 | # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). 57 | # If this step fails, then you should remove it and run the build manually (see below) 58 | - name: Autobuild 59 | uses: github/codeql-action/autobuild@v2 60 | 61 | # ℹ️ Command-line programs to run using the OS shell. 62 | # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun 63 | 64 | # If the Autobuild fails above, remove it and uncomment the following three lines. 65 | # modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance. 66 | 67 | # - run: | 68 | # echo "Run, Build Application using script" 69 | # ./location_of_script_within_repo/buildscript.sh 70 | 71 | - name: Perform CodeQL Analysis 72 | uses: github/codeql-action/analyze@v2 73 | -------------------------------------------------------------------------------- /.github/workflows/release.yml: -------------------------------------------------------------------------------- 1 | # This workflows will upload a Python Package using Twine when a release is created 2 | # For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries 3 | 4 | name: release 5 | permissions: 6 | checks: write 7 | pull-requests: write 8 | packages: write 9 | deployments: write 10 | contents: write 11 | id-token: write 12 | 13 | on: 14 | push: 15 | tags: 16 | - "v[0-9]+.[0-9]+.[0-9]+" 17 | 18 | jobs: 19 | deploy: 20 | runs-on: ubuntu-latest 21 | environment: release 22 | steps: 23 | - uses: actions/checkout@v4 24 | - name: Set up Python 25 | uses: actions/setup-python@v5 26 | with: 27 | python-version: '3.8' 28 | - name: Install dependencies 29 | run: | 30 | python -m pip install --upgrade pip 31 | pip install setuptools wheel twine 32 | - name: Build azure-kusto-data 33 | working-directory: ./azure-kusto-data 34 | run: | 35 | python setup.py sdist bdist_wheel 36 | - name: Build azure-kusto-ingest 37 | working-directory: ./azure-kusto-ingest 38 | run: | 39 | python setup.py sdist bdist_wheel 40 | - name: Publish azure-kusto-data 41 | uses: pypa/gh-action-pypi-publish@release/v1 42 | with: 43 | packages-dir: azure-kusto-data/dist 44 | - name: Publish azure-kusto-ingest 45 | uses: pypa/gh-action-pypi-publish@release/v1 46 | with: 47 | packages-dir: azure-kusto-ingest/dist 48 | - name: Release 49 | uses: docker://antonyurchenko/git-release:latest 50 | env: 51 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 52 | with: 53 | args: azure-kusto-data/dist/*.tar.gz azure-kusto-ingest/dist/*.tar.gz azure-kusto-data/dist/*.whl azure-kusto-ingest/dist/*.whl 54 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.suo 8 | *.user 9 | *.userosscache 10 | *.sln.docstates 11 | 12 | # User-specific files (MonoDevelop/Xamarin Studio) 13 | *.userprefs 14 | 15 | # Build results 16 | [Dd]ebug/ 17 | [Dd]ebugPublic/ 18 | [Rr]elease/ 19 | [Rr]eleases/ 20 | x64/ 21 | x86/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | [Ll]og/ 26 | dist/ 27 | 28 | # Visual Studio 2015 cache/options directory 29 | .vs/ 30 | # Uncomment if you have tasks that create the project's static files in wwwroot 31 | #wwwroot/ 32 | 33 | # MSTest test Results 34 | [Tt]est[Rr]esult*/ 35 | [Bb]uild[Ll]og.* 36 | 37 | # NUNIT 38 | *.VisualState.xml 39 | TestResult.xml 40 | 41 | # Build Results of an ATL Project 42 | [Dd]ebugPS/ 43 | [Rr]eleasePS/ 44 | dlldata.c 45 | 46 | # .NET Core 47 | project.lock.json 48 | project.fragment.lock.json 49 | artifacts/ 50 | **/Properties/launchSettings.json 51 | 52 | *_i.c 53 | *_p.c 54 | *_i.h 55 | *.ilk 56 | *.meta 57 | *.obj 58 | *.pch 59 | *.pdb 60 | *.pgc 61 | *.pgd 62 | *.rsp 63 | *.sbr 64 | *.tlb 65 | *.tli 66 | *.tlh 67 | *.tmp 68 | *.tmp_proj 69 | *.log 70 | *.vspscc 71 | *.vssscc 72 | .builds 73 | *.pidb 74 | *.svclog 75 | *.scc 76 | 77 | # Chutzpah Test files 78 | _Chutzpah* 79 | 80 | # Visual C++ cache files 81 | ipch/ 82 | *.aps 83 | *.ncb 84 | *.opendb 85 | *.opensdf 86 | *.sdf 87 | *.cachefile 88 | *.VC.db 89 | *.VC.VC.opendb 90 | 91 | # Visual Studio profiler 92 | *.psess 93 | *.vsp 94 | *.vspx 95 | *.sap 96 | 97 | # TFS 2012 Local Workspace 98 | $tf/ 99 | 100 | # Guidance Automation Toolkit 101 | *.gpState 102 | 103 | # ReSharper is a .NET coding add-in 104 | _ReSharper*/ 105 | *.[Rr]e[Ss]harper 106 | *.DotSettings.user 107 | 108 | # JustCode is a .NET coding add-in 109 | .JustCode 110 | 111 | # TeamCity is a build add-in 112 | _TeamCity* 113 | 114 | # DotCover is a Code Coverage Tool 115 | *.dotCover 116 | 117 | # Visual Studio code coverage results 118 | *.coverage 119 | *.coveragexml 120 | 121 | # NCrunch 122 | _NCrunch_* 123 | .*crunch*.local.xml 124 | nCrunchTemp_* 125 | 126 | # MightyMoose 127 | *.mm.* 128 | AutoTest.Net/ 129 | 130 | # Web workbench (sass) 131 | .sass-cache/ 132 | 133 | # Installshield output folder 134 | [Ee]xpress/ 135 | 136 | # DocProject is a documentation generator add-in 137 | DocProject/buildhelp/ 138 | DocProject/Help/*.HxT 139 | DocProject/Help/*.HxC 140 | DocProject/Help/*.hhc 141 | DocProject/Help/*.hhk 142 | DocProject/Help/*.hhp 143 | DocProject/Help/Html2 144 | DocProject/Help/html 145 | 146 | # Click-Once directory 147 | publish/ 148 | 149 | # Publish Web Output 150 | *.[Pp]ublish.xml 151 | *.azurePubxml 152 | # TODO: Comment the next line if you want to checkin your web deploy settings 153 | # but database connection strings (with potential passwords) will be unencrypted 154 | *.pubxml 155 | *.publishproj 156 | 157 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 158 | # checkin your Azure Web App publish settings, but sensitive information contained 159 | # in these scripts will be unencrypted 160 | PublishScripts/ 161 | 162 | # NuGet Packages 163 | *.nupkg 164 | # The packages folder can be ignored because of Package Restore 165 | **/packages/* 166 | # except build/, which is used as an MSBuild target. 167 | !**/packages/build/ 168 | # Uncomment if necessary however generally it will be regenerated when needed 169 | #!**/packages/repositories.config 170 | # NuGet v3's project.json files produces more ignorable files 171 | *.nuget.props 172 | *.nuget.targets 173 | 174 | # Microsoft Azure Build Output 175 | csx/ 176 | *.build.csdef 177 | 178 | # Microsoft Azure Emulator 179 | ecf/ 180 | rcf/ 181 | 182 | # Windows Store app package directories and files 183 | AppPackages/ 184 | BundleArtifacts/ 185 | Package.StoreAssociation.xml 186 | _pkginfo.txt 187 | 188 | # Visual Studio cache files 189 | # files ending in .cache can be ignored 190 | *.[Cc]ache 191 | # but keep track of directories ending in .cache 192 | !*.[Cc]ache/ 193 | 194 | # Others 195 | ClientBin/ 196 | ~$* 197 | *~ 198 | *.dbmdl 199 | *.dbproj.schemaview 200 | *.jfm 201 | *.pfx 202 | *.publishsettings 203 | orleans.codegen.cs 204 | 205 | # Since there are multiple workflows, uncomment next line to ignore bower_components 206 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 207 | #bower_components/ 208 | 209 | # RIA/Silverlight projects 210 | Generated_Code/ 211 | 212 | # Backup & report files from converting an old project file 213 | # to a newer Visual Studio version. Backup files are not needed, 214 | # because we have git ;-) 215 | _UpgradeReport_Files/ 216 | Backup*/ 217 | UpgradeLog*.XML 218 | UpgradeLog*.htm 219 | 220 | # SQL Server files 221 | *.mdf 222 | *.ldf 223 | *.ndf 224 | 225 | # Business Intelligence projects 226 | *.rdl.data 227 | *.bim.layout 228 | *.bim_*.settings 229 | 230 | # Microsoft Fakes 231 | FakesAssemblies/ 232 | 233 | # GhostDoc plugin setting file 234 | *.GhostDoc.xml 235 | 236 | # Node.js Tools for Visual Studio 237 | .ntvs_analysis.dat 238 | node_modules/ 239 | 240 | # Typescript v1 declaration files 241 | typings/ 242 | 243 | # Visual Studio 6 build log 244 | *.plg 245 | 246 | # Visual Studio 6 workspace options file 247 | *.opt 248 | 249 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 250 | *.vbw 251 | 252 | # Visual Studio LightSwitch build output 253 | **/*.HTMLClient/GeneratedArtifacts 254 | **/*.DesktopClient/GeneratedArtifacts 255 | **/*.DesktopClient/ModelManifest.xml 256 | **/*.Server/GeneratedArtifacts 257 | **/*.Server/ModelManifest.xml 258 | _Pvt_Extensions 259 | 260 | # Paket dependency manager 261 | .paket/paket.exe 262 | paket-files/ 263 | 264 | # FAKE - F# Make 265 | .fake/ 266 | 267 | # JetBrains Rider 268 | .idea/ 269 | .run/ 270 | *.sln.iml 271 | 272 | # CodeRush 273 | .cr/ 274 | 275 | # Python Tools for Visual Studio (PTVS) 276 | __pycache__/ 277 | *.pyc 278 | 279 | # Cake - Uncomment if you are using it 280 | # tools/** 281 | # !tools/packages.config 282 | 283 | # Telerik's JustMock configuration file 284 | *.jmconfig 285 | 286 | # BizTalk build output 287 | *.btp.cs 288 | *.btm.cs 289 | *.odx.cs 290 | *.xsd.cs 291 | 292 | # Result of running python setup.py install/pip install -e 293 | RECORD.txt 294 | build/ 295 | *.egg-info/ 296 | 297 | # Wheel files 298 | *.whl 299 | *.tar.gz 300 | 301 | # pytest temp files 302 | .pytest_cache* 303 | 304 | # VS code files 305 | .vscode* 306 | 307 | #mypy files 308 | .mypy* 309 | 310 | 311 | */.tox/ -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- 1 | # https://github.com/getsentry/responses/issues/74 2 | [TYPECHECK] 3 | ignored-classes= responses 4 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to Azure Python SDK 2 | 3 | If you would like to become an active contributor to this project please 4 | follow the instructions provided in [Microsoft Azure Projects Contribution Guidelines](https://azure.github.io/azure-sdk/python_documentation.html). 5 | 6 | ## Prerequisites 7 | In order to run E2E tests, you need a Kusto database you have admin rights on. 8 | A Kusto free cluster is the easiest way to acquire one. 9 | You can cerate a free Kusto cluster here: https://dataexplorer.azure.com/home 10 | 11 | Make sure you set streaming ingestion to enabled in the cluster's configuration: 12 | https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Ccsharp 13 | 14 | You should set then following environment vars where you run E2Es (in IDE run config, shell window, computer, etc). 15 | ```shell 16 | ENGINE_CONNECTION_STRING= 17 | DM_CONNECTION_STRING= # Optional - if not set, will infer from ENGINE_CONNECTION_STRING 18 | TEST_DATABASE= 19 | ``` 20 | 21 | The E2E tests authenticate with DefaultAzureCredential, and will fall back to interactive login if needed. 22 | 23 | 24 | For more information on DefaultAzureCredential, see: 25 | https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python 26 | 27 | To run the optional `token_provider` tests, you will need to set the booleans at the top of the file `test_token_providers.py` and the following environment variables in addition to the previous ones: 28 | ```shell 29 | # app key provider: 30 | AZURE_CLIENT_ID= 31 | AZURE_CLIENT_SECRET= 32 | AZURE_TENANT_ID= 33 | # certificate provider: 34 | CERT_THUMBPRINT= 35 | CERT_PUBLIC_CERT_PATH= 36 | CERT_PEM_KEY_PATH= 37 | # managed identity provider: 38 | MSI_OBJECT_ID= 39 | MSI_CLIENT_ID= 40 | # user password provider: 41 | USER_NAME= 42 | USER_PASS= 43 | USER_AUTH_ID= # optional 44 | ``` 45 | 46 | ## Requirements 47 | 48 | In order to work on this project, we recommend using the dev requirements: 49 | 50 | ```bash 51 | pip install -r dev_requirements.txt 52 | ``` 53 | 54 | These including testing related packages as well as styling ([black](https://black.readthedocs.io/en/stable/)) 55 | 56 | ## Building and Testing 57 | 58 | This project uses [pytest](https://docs.pytest.org/en/latest/). 59 | 60 | 61 | Since the tests use the package as a third-party, the easiest way to set it up is installing it in edit mode: 62 | 63 | ```bash 64 | pip install -e ./azure-kusto-data ./azure-kusto-ingest 65 | ``` 66 | 67 | After which, running tests is simple. 68 | 69 | Just run: 70 | 71 | ```bash 72 | pytest ./azure-kusto-data ./azure-kusto-ingest 73 | ``` 74 | 75 | ## Style 76 | 77 | We use black, and allow for line-length of 160, so please run: 78 | 79 | ```bash 80 | black --line-length=160 ./azure-kusto-data ./azure-kusto-ingest 81 | ``` 82 | 83 | ## PRs 84 | We welcome contributions. In order to make the PR process efficient, please follow the below checklist: 85 | 86 | * **There is an issue open concerning the code added** - (either bug or enhancement). 87 | Preferably there is an agreed upon approach in the issue. 88 | * **PR comment explains the changes done** - (This should be a TL;DR; as the rest of it should be documented in the related issue). 89 | * **PR is concise** - try and avoid make drastic changes in a single PR. Split it into multiple changes if possible. If you feel a major change is needed, it is ok, but make sure commit history is clear and one of the maintainers can comfortably review both the code and the logic behind the change. 90 | * **Please provide any related information needed to understand the change** - docs, guidelines, use-case, best practices and so on. Opinions are accepted, but have to be backed up. 91 | * **Checks should pass** - these including linting with black and running tests. 92 | 93 | ## Code of Conduct 94 | This project's code of conduct can be found in the 95 | [CODE_OF_CONDUCT.md file](https://github.com/Azure/azure-sdk-for-python/blob/master/CODE_OF_CONDUCT.md) 96 | (v1.4.0 of the http://contributor-covenant.org/ CoC). 97 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. All rights reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Microsoft Azure Kusto (Azure Data Explorer) SDK for Python 2 | 3 | [*azure-kusto-data*]("https://github.com/Azure/azure-kusto-python/tree/master/azure-kusto-data") Package provides the capability to query Kusto clusters with Python.
4 | [![PyPI version](https://badge.fury.io/py/azure-kusto-data.svg)](https://badge.fury.io/py/azure-kusto-data) 5 | [![Downloads](https://pepy.tech/badge/azure-kusto-data)](https://pepy.tech/project/azure-kusto-data)
6 | [*azure-kusto-ingest*]("https://github.com/Azure/azure-kusto-python/tree/master/azure-kusto-ingest") Package allows sending data to Kusto service - i.e. ingest data.
7 | [![PyPI version](https://badge.fury.io/py/azure-kusto-ingest.svg)](https://badge.fury.io/py/azure-kusto-ingest) 8 | [![Downloads](https://pepy.tech/badge/azure-kusto-ingest)](https://pepy.tech/project/azure-kusto-ingest)
9 | 10 | 11 | ## Install 12 | ### Option 1: Via PyPi 13 | To install via the Python Package Index (PyPI), type: 14 | 15 | * `pip install azure-kusto-data` 16 | * `pip install azure-kusto-ingest` 17 | 18 | ### Option 2: Source Via Git 19 | To get the source code of the SDK via git just type: 20 | 21 | ```python 22 | git clone https://github.com/Azure/azure-kusto-python 23 | cd ./azure-kusto-python/azure-kusto-data 24 | python3 setup.py install 25 | cd ../azure-kusto-ingest 26 | python3 setup.py install 27 | ``` 28 | 29 | ### Option 3: Source Zip 30 | Download a zip of the code via GitHub or PyPi. Then follow the same instructions in option 2. 31 | 32 | ### Optionals: 33 | * [_Pandas_](http://pandas.pydata.org/) - Package provides extra functionality for use with pandas. Since these are optional dependencies, install with pandas: 34 | * `pip install azure-kusto-data[pandas]` 35 | * `pip install azure-kusto-ingest[pandas]` 36 | 37 | ## Minimum Requirements 38 | * Python 3.5 and above 39 | * See setup.py for dependencies 40 | 41 | ## Authentication methods: 42 | 43 | * AAD Username/password - Provide your AAD username and password to Kusto client (**check the notice below**). 44 | * AAD application - Provide app ID and app secret to Kusto client. 45 | * AAD code - Provide only your AAD username, and authenticate yourself using a code, generated by ADAL. 46 | * AZ CLI - For those already using [azure-cli](https://github.com/Azure/azure-cli), provide access token for the logged in user`. 47 | 48 | ** IMPORTANT NOTICE **: 49 | User authentication (using username and password) has a major caveat: 50 | Sometimes users are required to use Multi-Factor Authentication. In such a case, this flow won't work for them. 51 | It is a limitation of the AAD library we are using under the hood. There are [several bugs reported](https://github.com/AzureAD/azure-activedirectory-library-for-python/issues?utf8=%E2%9C%93&q=is%3Aissue+mfa). 52 | 53 | There is also a feature request for the adal team to work on implementing IWA (Intergrated Windows Auth) so that signed in users won't have to authenticate. Feel free to [upvote](https://github.com/AzureAD/microsoft-authentication-library-for-python/issues/31) if it is relevant in your case. 54 | 55 | ## Samples: 56 | 57 | * [Kusto Quick Start Sample App](https://github.com/Azure/azure-kusto-python/tree/master/quick_start) 58 | 59 | * [Kusto query sample snippets](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-data/tests/sample.py) 60 | 61 | * [Data ingest sample snippets](https://github.com/Azure/azure-kusto-python/blob/master/azure-kusto-ingest/tests/sample.py) 62 | 63 | ## Best Practices 64 | See the SDK [best practices guide](https://docs.microsoft.com/azure/data-explorer/kusto/api/netfx/kusto-ingest-best-practices), which though written for the .NET SDK, applies similarly here. 65 | 66 | ## Need Support? 67 | - **Have a feature request for SDKs?** Please post it on [User Voice](https://feedback.azure.com/forums/915733-azure-data-explorer) to help us prioritize 68 | - **Have a technical question?** Ask on [Stack Overflow with tag "azure-data-explorer"](https://stackoverflow.com/questions/tagged/azure-data-explorer) 69 | - **Need Support?** Every customer with an active Azure subscription has access to [support](https://docs.microsoft.com/en-us/azure/azure-supportability/how-to-create-azure-support-request) with guaranteed response time. Consider submitting a ticket and get assistance from Microsoft support team 70 | - **Found a bug?** Please help us fix it by thoroughly documenting it and [filing an issue](https://github.com/Azure/azure-kusto-python/issues/new). 71 | 72 | ## Looking for SDKs for other languages/platforms? 73 | - [Node](https://github.com/azure/azure-kusto-node) 74 | - [Java](https://github.com/azure/azure-kusto-java) 75 | - [.NET](https://docs.microsoft.com/en-us/azure/kusto/api/netfx/about-the-sdk) 76 | - [Go](https://github.com/Azure/azure-kusto-go) 77 | 78 | # Contribute 79 | 80 | We gladly accept community contributions. 81 | 82 | - Issues: Please report bugs using the Issues section of GitHub 83 | - Forums: Interact with the development teams on StackOverflow or the Microsoft Azure Forums 84 | - Source Code Contributions: If you would like to become an active contributor to this project please follow the instructions provided in [Contributing.md](CONTRIBUTING.md). 85 | 86 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 87 | 88 | For general suggestions about Microsoft Azure please use our [UserVoice forum](http://feedback.azure.com/forums/34192--general-feedback). 89 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /azure-kusto-data/MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.rst 2 | include azure/__init__.py 3 | include azure/kusto/data/wellKnownKustoEndpoints.json 4 | include azure/kusto/data/kcsb.json 5 | recursive-exclude tests * 6 | -------------------------------------------------------------------------------- /azure-kusto-data/README.rst: -------------------------------------------------------------------------------- 1 | Microsoft Azure Kusto Library for Python 2 | ======================================== 3 | 4 | Overview 5 | -------- 6 | 7 | .. code-block:: python 8 | 9 | from azure.kusto.data import KustoClient, KustoConnectionStringBuilder 10 | 11 | cluster = "" 12 | client_id = "" 13 | client_secret = "" 14 | authority_id = "" 15 | 16 | kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(cluster, client_id, client_secret, authority_id) 17 | # It is a good practice to re-use the KustoClient instance, as it maintains a pool of connections to the Kusto service. 18 | # This sample shows how to create a client and close it in the same scope, for demonstration purposes. 19 | with KustoClient(kcsb) as client: 20 | db = "Samples" 21 | query = "StormEvents | take 10" 22 | 23 | response = client.execute(db, query) 24 | for row in response.primary_results[0]: 25 | print(row[0], " ", row["EventType"]) 26 | 27 | 28 | 29 | *Kusto Python Client* Library provides the capability to query Kusto clusters using Python. 30 | It is Python 3.x compatible and supports 31 | all data types through familiar Python DB API interface. 32 | 33 | It's possible to use the library, for instance, from `Jupyter Notebooks 34 | `_. 35 | which are attached to Spark clusters, 36 | including, but not exclusively, `Azure Databricks 37 | `_. instances. 38 | 39 | Async Client 40 | ~~~~~~~~~~~~ 41 | Kusto now provides an asynchronous client for queries. 42 | 43 | To use the client, first install the package with the "aio" extra: 44 | 45 | .. code:: bash 46 | 47 | pip install azure-kusto-data[aio] 48 | 49 | The async client uses exact same interface as the regular client, except 50 | that it lives in the ``azure.kusto.data.aio`` namespace, and it returns 51 | ``Futures`` you will need to ``await`` its 52 | 53 | .. code:: python 54 | 55 | from azure.kusto.data import KustoConnectionStringBuilder 56 | from azure.kusto.data.aio import KustoClient 57 | 58 | cluster = "" 59 | client_id = "" 60 | client_secret = "" 61 | authority_id = "" 62 | 63 | 64 | async def sample(): 65 | kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(cluster, client_id, client_secret, authority_id) 66 | async with KustoClient(kcsb) as client: 67 | db = "Samples" 68 | query = "StormEvents | take 10" 69 | 70 | response = await client.execute(db, query) 71 | for row in response.primary_results[0]: 72 | print(row[0], " ", row["EventType"]) 73 | 74 | Links 75 | ~~~~~ 76 | 77 | * `How to install the package `_. 78 | 79 | * `Kusto query sample `_. 80 | 81 | * `GitHub Repository `_. 82 | -------------------------------------------------------------------------------- /azure-kusto-data/azure-kusto-data.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 16 | 17 | 19 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | """Init file for azure namespace. https://github.com/Azure/azure-sdk-for-python/wiki/Azure-packaging""" 4 | __path__ = __import__("pkgutil").extend_path(__path__, __name__) 5 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | """Init file for azure namespace. https://github.com/Azure/azure-sdk-for-python/wiki/Azure-packaging""" 4 | __path__ = __import__("pkgutil").extend_path(__path__, __name__) 5 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | from ._version import VERSION as __version__ 5 | from .client import KustoClient 6 | from .client_request_properties import ClientRequestProperties 7 | from .kcsb import KustoConnectionStringBuilder 8 | from .data_format import DataFormat 9 | 10 | __all__ = [ 11 | "KustoClient", 12 | "ClientRequestProperties", 13 | "KustoConnectionStringBuilder", 14 | "DataFormat", 15 | ] 16 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/_cloud_settings.py: -------------------------------------------------------------------------------- 1 | import dataclasses 2 | from threading import Lock 3 | from typing import Optional, Dict 4 | from urllib.parse import urlparse 5 | 6 | import requests 7 | 8 | from azure.core.tracing.decorator import distributed_trace 9 | from azure.core.tracing import SpanKind 10 | 11 | from .env_utils import get_env 12 | from ._telemetry import Span, MonitoredActivity 13 | from .exceptions import KustoServiceError, KustoNetworkError 14 | 15 | METADATA_ENDPOINT = "v1/rest/auth/metadata" 16 | 17 | DEFAULT_AUTH_ENV_VAR_NAME = "AadAuthorityUri" 18 | DEFAULT_KUSTO_CLIENT_APP_ID = "db662dc1-0cfe-4e1c-a843-19a68e65be58" 19 | DEFAULT_PUBLIC_LOGIN_URL = "https://login.microsoftonline.com" 20 | DEFAULT_REDIRECT_URI = "http://localhost" 21 | DEFAULT_KUSTO_SERVICE_RESOURCE_ID = "https://kusto.kusto.windows.net" 22 | DEFAULT_DEV_KUSTO_SERVICE_RESOURCE_ID = "https://kusto.dev.kusto.windows.net" 23 | DEFAULT_FIRST_PARTY_AUTHORITY_URL = "https://login.microsoftonline.com/f8cdef31-a31e-4b4a-93e4-5f571e91255a" 24 | 25 | 26 | @dataclasses.dataclass 27 | class CloudInfo: 28 | """This class holds the data for a specific cloud instance.""" 29 | 30 | login_endpoint: str 31 | login_mfa_required: bool 32 | kusto_client_app_id: str 33 | kusto_client_redirect_uri: str 34 | kusto_service_resource_id: str 35 | first_party_authority_url: str 36 | 37 | def authority_uri(self, authority_id: Optional[str]): 38 | return self.login_endpoint + "/" + (authority_id or "organizations") 39 | 40 | 41 | class CloudSettings: 42 | """This class holds data for all cloud instances, and returns the specific data instance by parsing the dns suffix from a URL""" 43 | 44 | _cloud_info = None 45 | _cloud_cache = {} 46 | _cloud_cache_lock = Lock() 47 | 48 | DEFAULT_CLOUD = CloudInfo( 49 | login_endpoint=get_env(DEFAULT_AUTH_ENV_VAR_NAME, default=DEFAULT_PUBLIC_LOGIN_URL), 50 | login_mfa_required=False, 51 | kusto_client_app_id=DEFAULT_KUSTO_CLIENT_APP_ID, 52 | kusto_client_redirect_uri=DEFAULT_REDIRECT_URI, 53 | kusto_service_resource_id=DEFAULT_KUSTO_SERVICE_RESOURCE_ID, 54 | first_party_authority_url=DEFAULT_FIRST_PARTY_AUTHORITY_URL, 55 | ) 56 | 57 | @classmethod 58 | @distributed_trace(name_of_span="CloudSettings.get_cloud_info", kind=SpanKind.CLIENT) 59 | def get_cloud_info_for_cluster(cls, kusto_uri: str, proxies: Optional[Dict[str, str]] = None) -> CloudInfo: 60 | normalized_authority = cls._normalize_uri(kusto_uri) 61 | 62 | # tracing attributes for cloud info 63 | Span.set_cloud_info_attributes(kusto_uri) 64 | 65 | if normalized_authority in cls._cloud_cache: # Double-checked locking to avoid unnecessary lock access 66 | return cls._cloud_cache[normalized_authority] 67 | 68 | with cls._cloud_cache_lock: 69 | if normalized_authority in cls._cloud_cache: 70 | return cls._cloud_cache[normalized_authority] 71 | 72 | url_parts = urlparse(kusto_uri) 73 | url = f"{url_parts.scheme}://{url_parts.netloc}/{METADATA_ENDPOINT}" 74 | 75 | try: 76 | # trace http get call for result 77 | result = MonitoredActivity.invoke( 78 | lambda: requests.get(url, proxies=proxies, allow_redirects=False), 79 | name_of_span="CloudSettings.http_get", 80 | tracing_attributes=Span.create_http_attributes(url=url, method="GET"), 81 | ) 82 | except Exception as e: 83 | raise KustoNetworkError(url) from e 84 | 85 | if result.status_code == 200: 86 | content = result.json() 87 | if content is None or content == {}: 88 | raise KustoServiceError("Kusto returned an invalid cloud metadata response", result) 89 | root = content["AzureAD"] 90 | if root is not None: 91 | cls._cloud_cache[normalized_authority] = CloudInfo( 92 | login_endpoint=root["LoginEndpoint"], 93 | login_mfa_required=root["LoginMfaRequired"], 94 | kusto_client_app_id=root["KustoClientAppId"], 95 | kusto_client_redirect_uri=root["KustoClientRedirectUri"], 96 | kusto_service_resource_id=root["KustoServiceResourceId"], 97 | first_party_authority_url=root["FirstPartyAuthorityUrl"], 98 | ) 99 | else: 100 | cls._cloud_cache[normalized_authority] = cls.DEFAULT_CLOUD 101 | elif result.status_code == 404: 102 | # For now as long not all proxies implement the metadata endpoint, if no endpoint exists return public cloud data 103 | cls._cloud_cache[normalized_authority] = cls.DEFAULT_CLOUD 104 | else: 105 | raise KustoServiceError("Kusto returned an invalid cloud metadata response", result) 106 | return cls._cloud_cache[normalized_authority] 107 | 108 | @classmethod 109 | def add_to_cache(cls, url: str, cloud_info: CloudInfo): 110 | with cls._cloud_cache_lock: 111 | cls._cloud_cache[cls._normalize_uri(url)] = cloud_info 112 | 113 | @classmethod 114 | def _normalize_uri(cls, kusto_uri): 115 | """Extracts and returns the authority part of the URI (schema, host, port)""" 116 | url_parts = urlparse(kusto_uri) 117 | # Return only the scheme and netloc (which contains host and port if present) 118 | return f"{url_parts.scheme}://{url_parts.netloc}" 119 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/_converters.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License. 3 | 4 | import re 5 | from datetime import timedelta 6 | 7 | from dateutil import parser 8 | 9 | # Regex for TimeSpan 10 | _TIMESPAN_PATTERN = re.compile(r"(-?)((?P[0-9]*).)?(?P[0-9]{2}):(?P[0-9]{2}):(?P[0-9]{2}(\.[0-9]+)?$)") 11 | 12 | 13 | def to_datetime(value): 14 | """Converts a string to a datetime.""" 15 | if isinstance(value, int): 16 | return parser.parse(value) 17 | return parser.isoparse(value) 18 | 19 | 20 | def to_timedelta(value): 21 | """Converts a string to a timedelta.""" 22 | if isinstance(value, (int, float)): 23 | return timedelta(microseconds=(float(value) / 10)) 24 | match = _TIMESPAN_PATTERN.match(value) 25 | if match: 26 | if match.group(1) == "-": 27 | factor = -1 28 | else: 29 | factor = 1 30 | return factor * timedelta(days=int(match.group("d") or 0), hours=int(match.group("h")), minutes=int(match.group("m")), seconds=float(match.group("s"))) 31 | else: 32 | raise ValueError("Timespan value '{}' cannot be decoded".format(value)) 33 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/_decorators.py: -------------------------------------------------------------------------------- 1 | def aio_documented_by(original): 2 | def wrapper(target): 3 | target.__doc__ = "Aio function: {original_doc}".format(original_doc=original.__doc__) 4 | return target 5 | 6 | return wrapper 7 | 8 | 9 | def documented_by(original): 10 | def wrapper(target): 11 | target.__doc__ = original.__doc__ 12 | return target 13 | 14 | return wrapper 15 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/_string_utils.py: -------------------------------------------------------------------------------- 1 | def assert_string_is_not_empty(value: str): 2 | if not value or not value.strip(): 3 | raise ValueError("Should not be empty") 4 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/_telemetry.py: -------------------------------------------------------------------------------- 1 | from typing import Callable, Optional, TypeVar 2 | 3 | from azure.core.settings import settings 4 | from azure.core.tracing.decorator import distributed_trace 5 | from azure.core.tracing.decorator_async import distributed_trace_async 6 | from azure.core.tracing import SpanKind 7 | 8 | from .client_request_properties import ClientRequestProperties 9 | 10 | 11 | class Span: 12 | """ 13 | Additional ADX attributes for telemetry spans 14 | """ 15 | 16 | _KUSTO_CLUSTER = "kusto_cluster" 17 | _DATABASE = "database" 18 | _TABLE = "table" 19 | 20 | _AUTH_METHOD = "authentication_method" 21 | _CLIENT_ACTIVITY_ID = "client_activity_id" 22 | 23 | _SPAN_COMPONENT = "component" 24 | _HTTP = "http" 25 | _HTTP_USER_AGENT = "http.user_agent" 26 | _HTTP_METHOD = "http.method" 27 | _HTTP_URL = "http.url" 28 | 29 | @classmethod 30 | def add_attributes(cls, **kwargs) -> None: 31 | """ 32 | Add ADX attributes to the current span 33 | :key dict tracing_attributes: key, val ADX attributes to include in span of trace 34 | """ 35 | tracing_attributes: dict = kwargs.pop("tracing_attributes", {}) 36 | span_impl_type = settings.tracing_implementation() 37 | if span_impl_type is None: 38 | return 39 | current_span = span_impl_type.get_current_span() 40 | span = span_impl_type(span=current_span) 41 | for key, val in tracing_attributes.items(): 42 | span.add_attribute(key, val) 43 | 44 | @classmethod 45 | def set_query_attributes(cls, cluster: str, database: str, properties: Optional[ClientRequestProperties] = None) -> None: 46 | query_attributes: dict = cls.create_query_attributes(cluster, database, properties) 47 | cls.add_attributes(tracing_attributes=query_attributes) 48 | 49 | @classmethod 50 | def set_streaming_ingest_attributes(cls, cluster: str, database: str, table: str, properties: Optional[ClientRequestProperties] = None) -> None: 51 | ingest_attributes: dict = cls.create_streaming_ingest_attributes(cluster, database, table, properties) 52 | cls.add_attributes(tracing_attributes=ingest_attributes) 53 | 54 | @classmethod 55 | def set_cloud_info_attributes(cls, url: str) -> None: 56 | cloud_info_attributes: dict = cls.create_cloud_info_attributes(url) 57 | cls.add_attributes(tracing_attributes=cloud_info_attributes) 58 | 59 | @classmethod 60 | def create_query_attributes(cls, cluster: str, database: str, properties: Optional[ClientRequestProperties] = None) -> dict: 61 | query_attributes: dict = {cls._KUSTO_CLUSTER: cluster, cls._DATABASE: database} 62 | if properties: 63 | query_attributes.update(properties.get_tracing_attributes()) 64 | 65 | return query_attributes 66 | 67 | @classmethod 68 | def create_streaming_ingest_attributes(cls, cluster: str, database: str, table: str, properties: Optional[ClientRequestProperties] = None) -> dict: 69 | ingest_attributes: dict = {cls._KUSTO_CLUSTER: cluster, cls._DATABASE: database, cls._TABLE: table} 70 | if properties: 71 | ingest_attributes.update(properties.get_tracing_attributes()) 72 | 73 | return ingest_attributes 74 | 75 | @classmethod 76 | def create_http_attributes(cls, method: str, url: str, headers: dict = None) -> dict: 77 | if headers is None: 78 | headers = {} 79 | http_tracing_attributes: dict = { 80 | cls._SPAN_COMPONENT: cls._HTTP, 81 | cls._HTTP_METHOD: method, 82 | cls._HTTP_URL: url, 83 | } 84 | user_agent = headers.get("User-Agent") 85 | if user_agent: 86 | http_tracing_attributes[cls._HTTP_USER_AGENT] = user_agent 87 | return http_tracing_attributes 88 | 89 | @classmethod 90 | def create_cloud_info_attributes(cls, url: str) -> dict: 91 | ingest_attributes: dict = {cls._HTTP_URL: url} 92 | return ingest_attributes 93 | 94 | @classmethod 95 | def create_cluster_attributes(cls, cluster_uri: str) -> dict: 96 | cluster_attributes = {cls._KUSTO_CLUSTER: cluster_uri} 97 | return cluster_attributes 98 | 99 | 100 | class MonitoredActivity: 101 | """ 102 | Invoker class for telemetry 103 | """ 104 | 105 | T = TypeVar("T") 106 | 107 | @staticmethod 108 | def invoke(invoker: Callable[[], T], name_of_span: str = None, tracing_attributes=None, kind: str = SpanKind.INTERNAL) -> T: 109 | """ 110 | Runs the span on given function 111 | """ 112 | if tracing_attributes is None: 113 | tracing_attributes = {} 114 | span_shell: Callable = distributed_trace(name_of_span=name_of_span, tracing_attributes=tracing_attributes, kind=kind) 115 | span = span_shell(invoker) 116 | return span() 117 | 118 | @staticmethod 119 | async def invoke_async(invoker: Callable[[], T], name_of_span: str = None, tracing_attributes=None, kind: str = SpanKind.INTERNAL) -> T: 120 | """ 121 | Runs a span on given function 122 | """ 123 | if tracing_attributes is None: 124 | tracing_attributes = {} 125 | span_shell: Callable = distributed_trace_async(name_of_span=name_of_span, tracing_attributes=tracing_attributes, kind=kind) 126 | span = span_shell(invoker) 127 | return await span() 128 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/_version.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | VERSION = "5.0.3" 4 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/aio/__init__.py: -------------------------------------------------------------------------------- 1 | from .client import KustoClient 2 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/aio/_models.py: -------------------------------------------------------------------------------- 1 | from typing import AsyncIterator 2 | 3 | from azure.kusto.data._models import KustoResultRow, BaseStreamingKustoResultTable 4 | 5 | 6 | class KustoStreamingResultTable(BaseStreamingKustoResultTable): 7 | """Async Iterator over a Kusto result table.""" 8 | 9 | async def __anext__(self) -> KustoResultRow: 10 | try: 11 | row = await self.raw_rows.__anext__() 12 | except StopAsyncIteration: 13 | self.finished = True 14 | raise 15 | self.row_count += 1 16 | return KustoResultRow(self.columns, row) 17 | 18 | def __aiter__(self) -> AsyncIterator[KustoResultRow]: 19 | return self 20 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/aio/response.py: -------------------------------------------------------------------------------- 1 | from typing import List, AsyncIterator, Union 2 | 3 | from azure.kusto.data._models import WellKnownDataSet, KustoResultTable, BaseKustoResultTable 4 | from azure.kusto.data.aio._models import KustoStreamingResultTable 5 | from azure.kusto.data.aio.streaming_response import StreamingDataSetEnumerator 6 | from azure.kusto.data.exceptions import KustoStreamingQueryError 7 | from azure.kusto.data.response import BaseKustoResponseDataSet 8 | from azure.kusto.data.streaming_response import FrameType 9 | 10 | 11 | class KustoStreamingResponseDataSet(BaseKustoResponseDataSet): 12 | _status_column = "Payload" 13 | _error_column = "Level" 14 | _crid_column = "ClientRequestId" 15 | 16 | def __init__(self, streamed_data: StreamingDataSetEnumerator): 17 | self._current_table = None 18 | self._skip_incomplete_tables = False 19 | self.tables = [] 20 | self.streamed_data = streamed_data 21 | self.finished = False 22 | 23 | def iter_primary_results(self) -> "PrimaryResultsIterator": 24 | return PrimaryResultsIterator(self) 25 | 26 | def __aiter__(self) -> AsyncIterator[BaseKustoResultTable]: 27 | return self 28 | 29 | async def __anext__(self) -> BaseKustoResultTable: 30 | if self.finished: 31 | raise StopAsyncIteration() 32 | 33 | if type(self._current_table) == KustoStreamingResultTable and not self._current_table.finished and not self._skip_incomplete_tables: 34 | raise KustoStreamingQueryError( 35 | "Tried retrieving a new primary_result table before the old one was finished. To override call `set_skip_incomplete_tables(True)`" 36 | ) 37 | 38 | while True: 39 | try: 40 | table = await self.streamed_data.__anext__() 41 | except StopAsyncIteration: 42 | self.finished = True 43 | return 44 | if table["FrameType"] == FrameType.DataTable: 45 | break 46 | 47 | if table["TableKind"] == WellKnownDataSet.PrimaryResult.value: 48 | self._current_table = KustoStreamingResultTable(table) 49 | else: 50 | self._current_table = KustoResultTable(table) 51 | 52 | self.tables.append(self._current_table) 53 | return self._current_table 54 | 55 | def set_skip_incomplete_tables(self, value: bool): 56 | self._skip_incomplete_tables = value 57 | 58 | @property 59 | def errors_count(self) -> int: 60 | if not self.finished: 61 | raise KustoStreamingQueryError("Unable to get errors count before reading all of the tables.") 62 | return super().errors_count 63 | 64 | def get_exceptions(self) -> List[str]: 65 | if not self.finished: 66 | raise KustoStreamingQueryError("Unable to get errors count before reading all of the tables.") 67 | return super().get_exceptions() 68 | 69 | def __getitem__(self, key: Union[int, str]) -> KustoResultTable: 70 | if isinstance(key, int): 71 | return self.tables[key] 72 | try: 73 | return next(t for t in self.tables if t.table_name == key) 74 | except StopIteration: 75 | raise LookupError(key) 76 | 77 | def __len__(self) -> int: 78 | return len(self.tables) 79 | 80 | 81 | class PrimaryResultsIterator: 82 | # This class exists because you can't raise exception from an generator and keep working 83 | def __init__(self, dataset: KustoStreamingResponseDataSet): 84 | self.dataset = dataset 85 | 86 | def __aiter__(self) -> AsyncIterator[KustoStreamingResultTable]: 87 | return self 88 | 89 | async def __anext__(self) -> KustoStreamingResultTable: 90 | while True: 91 | table = await self.dataset.__anext__() 92 | if type(table) is KustoStreamingResultTable: 93 | return table 94 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/client_details.py: -------------------------------------------------------------------------------- 1 | import functools 2 | import os 3 | import re 4 | import sys 5 | from dataclasses import dataclass 6 | from typing import List, Tuple, Optional 7 | 8 | from .env_utils import get_env 9 | from azure.kusto.data._version import VERSION 10 | 11 | NONE = "[none]" 12 | 13 | REPLACE_REGEX = re.compile(r"[\r\n\s{}|]+") 14 | 15 | 16 | @functools.lru_cache(maxsize=1) 17 | def default_script() -> str: 18 | """Returns the name of the script that is currently running""" 19 | try: 20 | return os.path.basename(sys.argv[0]) or NONE 21 | except Exception: 22 | return NONE 23 | 24 | 25 | @functools.lru_cache(maxsize=1) 26 | def get_user_from_env() -> str: 27 | user = get_env("USERNAME", optional=True) 28 | domain = get_env("USERDOMAIN", optional=True) 29 | if domain and user: 30 | user = domain + "\\" + user 31 | if user: 32 | return user 33 | return NONE 34 | 35 | 36 | @functools.lru_cache(maxsize=1) 37 | def default_user(): 38 | """Returns the name of the user that is currently logged in""" 39 | try: 40 | return os.getlogin() or get_user_from_env() 41 | except Exception: 42 | return get_user_from_env() 43 | 44 | 45 | @functools.lru_cache(maxsize=1) 46 | def format_version(): 47 | return format_header( 48 | [ 49 | ("Kusto.Python.Client", VERSION), 50 | (f"Runtime.{escape_field(sys.implementation.name)}", sys.version), 51 | ] 52 | ) 53 | 54 | 55 | def format_header(args: List[Tuple[str, str]]) -> str: 56 | return "|".join(f"{key}:{escape_field(val)}" for (key, val) in args if key and val) 57 | 58 | 59 | def escape_field(field: str): 60 | return f"{{{REPLACE_REGEX.sub('_', field)}}}" 61 | 62 | 63 | @dataclass 64 | class ClientDetails: 65 | application_for_tracing: str 66 | user_name_for_tracing: str 67 | version_for_tracing: str = format_version() 68 | 69 | def __post_init__(self): 70 | self.application_for_tracing = self.application_for_tracing or default_script() 71 | self.user_name_for_tracing = self.user_name_for_tracing or default_user() 72 | 73 | @staticmethod 74 | def set_connector_details( 75 | name: str, 76 | version: str, 77 | app_name: Optional[str] = None, 78 | app_version: Optional[str] = None, 79 | send_user: bool = False, 80 | override_user: Optional[str] = None, 81 | additional_fields: Optional[List[Tuple[str, str]]] = None, 82 | ) -> "ClientDetails": 83 | params = [("Kusto." + name, version)] 84 | 85 | app_name = app_name or default_script() 86 | app_version = app_version or NONE 87 | 88 | params.append(("App." + escape_field(app_name), app_version)) 89 | params.extend(additional_fields or []) 90 | 91 | user = NONE 92 | 93 | if send_user: 94 | user = override_user or default_user() 95 | 96 | return ClientDetails(application_for_tracing=format_header(params), user_name_for_tracing=user) 97 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/client_request_properties.py: -------------------------------------------------------------------------------- 1 | import json 2 | from typing import Any 3 | 4 | from ._string_utils import assert_string_is_not_empty 5 | 6 | 7 | class ClientRequestProperties: 8 | """This class is a POD used by client making requests to describe specific needs from the service executing the requests. 9 | For more information please look at: https://docs.microsoft.com/en-us/azure/kusto/api/netfx/request-properties 10 | """ 11 | 12 | client_request_id: str 13 | application: str 14 | user: str 15 | _CLIENT_REQUEST_ID = "client_request_id" 16 | 17 | results_defer_partial_query_failures_option_name = "deferpartialqueryfailures" 18 | request_timeout_option_name = "servertimeout" 19 | no_request_timeout_option_name = "norequesttimeout" 20 | 21 | def __init__(self): 22 | self._options = {} 23 | self._parameters = {} 24 | self.client_request_id = None 25 | self.application = None 26 | self.user = None 27 | 28 | def set_parameter(self, name: str, value: str): 29 | """Sets a parameter's value""" 30 | assert_string_is_not_empty(name) 31 | self._parameters[name] = value 32 | 33 | def has_parameter(self, name: str) -> bool: 34 | """Checks if a parameter is specified.""" 35 | return name in self._parameters 36 | 37 | def get_parameter(self, name: str, default_value: str) -> str: 38 | """Gets a parameter's value.""" 39 | return self._parameters.get(name, default_value) 40 | 41 | def set_option(self, name: str, value: Any): 42 | """Sets an option's value""" 43 | assert_string_is_not_empty(name) 44 | self._options[name] = value 45 | 46 | def has_option(self, name: str) -> bool: 47 | """Checks if an option is specified.""" 48 | return name in self._options 49 | 50 | def get_option(self, name: str, default_value: Any) -> str: 51 | """Gets an option's value.""" 52 | return self._options.get(name, default_value) 53 | 54 | def to_json(self) -> str: 55 | """Safe serialization to a JSON string.""" 56 | return json.dumps({"Options": self._options, "Parameters": self._parameters}, default=str) 57 | 58 | def get_tracing_attributes(self) -> dict: 59 | """Gets dictionary of attributes to be documented during tracing""" 60 | return {self._CLIENT_REQUEST_ID: str(self.client_request_id)} 61 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/data_format.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | from enum import Enum 4 | 5 | 6 | class IngestionMappingKind(Enum): 7 | CSV = "Csv" 8 | JSON = "Json" 9 | AVRO = "Avro" 10 | APACHEAVRO = "ApacheAvro" 11 | PARQUET = "Parquet" 12 | SSTREAM = "SStream" 13 | ORC = "Orc" 14 | W3CLOGFILE = "W3CLogFile" 15 | UNKNOWN = "Unknown" 16 | 17 | 18 | class DataFormat(Enum): 19 | """All data formats supported by Kusto.""" 20 | 21 | CSV = ("csv", IngestionMappingKind.CSV, True) 22 | TSV = ("tsv", IngestionMappingKind.CSV, True) 23 | SCSV = ("scsv", IngestionMappingKind.CSV, True) 24 | SOHSV = ("sohsv", IngestionMappingKind.CSV, True) 25 | PSV = ("psv", IngestionMappingKind.CSV, True) 26 | TXT = ("txt", IngestionMappingKind.CSV, True) 27 | TSVE = ("tsve", IngestionMappingKind.CSV, True) 28 | JSON = ("json", IngestionMappingKind.JSON, True) 29 | SINGLEJSON = ("singlejson", IngestionMappingKind.JSON, True) 30 | MULTIJSON = ("multijson", IngestionMappingKind.JSON, True) 31 | AVRO = ("avro", IngestionMappingKind.AVRO, False) 32 | APACHEAVRO = ("apacheavro", IngestionMappingKind.APACHEAVRO, False) 33 | PARQUET = ("parquet", IngestionMappingKind.PARQUET, False) 34 | SSTREAM = ("sstream", IngestionMappingKind.SSTREAM, False) 35 | ORC = ("orc", IngestionMappingKind.ORC, False) 36 | RAW = ("raw", IngestionMappingKind.CSV, True) 37 | W3CLOGFILE = ("w3clogfile", IngestionMappingKind.W3CLOGFILE, True) 38 | 39 | def __init__(self, kusto_value: str, ingestion_mapping_kind: IngestionMappingKind, compressible: bool): 40 | self.kusto_value = kusto_value # Formatted how Kusto Service expects it 41 | self.ingestion_mapping_kind = ingestion_mapping_kind 42 | self.compressible = compressible # Binary formats should not be compressed 43 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/env_utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | from dataclasses import dataclass, astuple 3 | from typing import Optional 4 | 5 | 6 | def get_env(*args, optional=False, default=None): 7 | """Return the first environment variable that is defined.""" 8 | for arg in args: 9 | if arg in os.environ: 10 | return os.environ[arg] 11 | if optional or default: 12 | return default 13 | raise ValueError("No environment variables found: {}".format(args)) 14 | 15 | 16 | def set_env(key, value): 17 | """Set the environment variable.""" 18 | os.environ[key] = value 19 | 20 | 21 | def get_app_id(optional=False): 22 | """Return the app id.""" 23 | result = get_env("APP_ID", "AZURE_CLIENT_ID", optional=optional) 24 | if result: 25 | set_env("AZURE_CLIENT_ID", result) 26 | return result 27 | 28 | 29 | def get_auth_id(optional=False): 30 | """Return the auth id.""" 31 | result = get_env("AUTH_ID", "APP_AUTH_ID", "AZURE_TENANT_ID", optional=optional) 32 | if result: 33 | set_env("AZURE_TENANT_ID", result) 34 | return result 35 | 36 | 37 | def get_app_key(optional=False): 38 | """Return the app key.""" 39 | result = get_env("APP_KEY", "AZURE_CLIENT_SECRET", optional=optional) 40 | if result: 41 | set_env("AZURE_CLIENT_SECRET", result) 42 | return result 43 | 44 | 45 | @dataclass(frozen=True) 46 | class AppKeyAuth: 47 | app_id: str 48 | app_key: str 49 | auth_id: str 50 | 51 | def __iter__(self): 52 | return iter(astuple(self)) 53 | 54 | 55 | def prepare_app_key_auth(optional=False) -> Optional[AppKeyAuth]: 56 | """Gets app key auth information from the env, sets the correct values for azidentity, and returns the AppKeyAuth object.""" 57 | app_id = get_app_id(optional=optional) 58 | app_key = get_app_key(optional=optional) 59 | auth_id = get_auth_id(optional=optional) 60 | if app_id and app_key and auth_id: 61 | return AppKeyAuth(app_id, app_key, auth_id) 62 | return None 63 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/kcsb.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "2025-01-15", 3 | "keywords": [ 4 | { 5 | "name": "Data Source", 6 | "aliases": [ 7 | "server", 8 | "addr", 9 | "address", 10 | "networkaddress" 11 | ], 12 | "type": "string", 13 | "secret": false 14 | }, 15 | { 16 | "name": "dSTS Federated Security", 17 | "aliases": [ 18 | "dstsfed", 19 | "dststokentype" 20 | ], 21 | "type": "bool", 22 | "secret": false 23 | }, 24 | { 25 | "name": "Streaming", 26 | "aliases": [], 27 | "type": "bool", 28 | "secret": false 29 | }, 30 | { 31 | "name": "Uncompressed", 32 | "aliases": [], 33 | "type": "bool", 34 | "secret": false 35 | }, 36 | { 37 | "name": "EnforceMfa", 38 | "aliases": [], 39 | "type": "bool", 40 | "secret": false 41 | }, 42 | { 43 | "name": "Accept", 44 | "aliases": [], 45 | "type": "bool", 46 | "secret": false 47 | }, 48 | { 49 | "name": "Query Consistency", 50 | "aliases": [], 51 | "type": "bool", 52 | "secret": false 53 | }, 54 | { 55 | "name": "Password", 56 | "aliases": [ 57 | "pwd" 58 | ], 59 | "type": "string", 60 | "secret": true 61 | }, 62 | { 63 | "name": "Data Source Uri", 64 | "aliases": [ 65 | "serveruri", 66 | "clusteruri" 67 | ], 68 | "type": "string", 69 | "secret": false 70 | }, 71 | { 72 | "name": "Azure Region", 73 | "aliases": [ 74 | "region" 75 | ], 76 | "type": "string", 77 | "secret": false 78 | }, 79 | { 80 | "name": "Namespace", 81 | "aliases": [ 82 | "ns" 83 | ], 84 | "type": "string", 85 | "secret": false 86 | }, 87 | { 88 | "name": "Application Certificate Thumbprint", 89 | "aliases": [ 90 | "appcert" 91 | ], 92 | "type": "string", 93 | "secret": true 94 | }, 95 | { 96 | "name": "Application Certificate Issuer Distinguished Name", 97 | "aliases": [ 98 | "applicationcertificateissuer" 99 | ], 100 | "type": "string", 101 | "secret": true 102 | }, 103 | { 104 | "name": "Application Certificate Subject Distinguished Name", 105 | "aliases": [ 106 | "applicationcertificatesubject" 107 | ], 108 | "type": "string", 109 | "secret": true 110 | }, 111 | { 112 | "name": "Application Token", 113 | "aliases": [ 114 | "apptoken" 115 | ], 116 | "type": "string", 117 | "secret": true 118 | }, 119 | { 120 | "name": "User Token", 121 | "aliases": [ 122 | "usrtoken" 123 | ], 124 | "type": "string", 125 | "secret": true 126 | }, 127 | { 128 | "name": "Application Key", 129 | "aliases": [ 130 | "appkey" 131 | ], 132 | "type": "string", 133 | "secret": true 134 | }, 135 | { 136 | "name": "Application Certificate Blob", 137 | "aliases": [], 138 | "type": "string", 139 | "secret": true 140 | }, 141 | { 142 | "name": "Application Certificate SendX5c", 143 | "aliases": [ 144 | "applicationcertificatex5c", 145 | "sendx5c", 146 | "applicationcertificatesendpubliccertificate", 147 | "sendcertificatechain" 148 | ], 149 | "type": "bool", 150 | "secret": false 151 | }, 152 | { 153 | "name": "User ID", 154 | "aliases": [ 155 | "user", 156 | "uid", 157 | "loginhint", 158 | "aaduserid" 159 | ], 160 | "type": "string", 161 | "secret": false 162 | }, 163 | { 164 | "name": "Initial Catalog", 165 | "aliases": [ 166 | "database" 167 | ], 168 | "type": "string", 169 | "secret": false 170 | }, 171 | { 172 | "name": "AAD Federated Security", 173 | "aliases": [ 174 | "federatedsecurity", 175 | "aadfed", 176 | "fed", 177 | "federated" 178 | ], 179 | "type": "bool", 180 | "secret": false 181 | }, 182 | { 183 | "name": "Authority Id", 184 | "aliases": [ 185 | "tenantid", 186 | "tid", 187 | "tenant", 188 | "authority", 189 | "domainhint" 190 | ], 191 | "type": "string", 192 | "secret": false 193 | }, 194 | { 195 | "name": "Application Name for Tracing", 196 | "aliases": [ 197 | "traceappname" 198 | ], 199 | "type": "string", 200 | "secret": false 201 | }, 202 | { 203 | "name": "User Name for Tracing", 204 | "aliases": [ 205 | "traceusername" 206 | ], 207 | "type": "string", 208 | "secret": false 209 | }, 210 | { 211 | "name": "Application Client Id", 212 | "aliases": [ 213 | "appclientid" 214 | ], 215 | "type": "string", 216 | "secret": false 217 | } 218 | ] 219 | } 220 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/kusto_trusted_endpoints.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import json 3 | from pathlib import Path 4 | from typing import List, Dict 5 | from urllib.parse import urlparse 6 | 7 | from azure.kusto.data.helpers import get_string_tail_lower_case 8 | from azure.kusto.data.security import _is_local_address 9 | from .exceptions import KustoClientInvalidConnectionStringException 10 | from .helpers import load_bundled_json 11 | 12 | 13 | class MatchRule: 14 | def __init__(self, suffix, exact): 15 | self.suffix = suffix.lower() 16 | self.exact = exact 17 | 18 | 19 | class FastSuffixMatcher: 20 | def __init__(self, rules: List[MatchRule]): 21 | self._suffix_length = min(len(rule.suffix) for rule in rules) 22 | _processed_rules: Dict[str, List] = {} 23 | for rule in rules: 24 | suffix = get_string_tail_lower_case(rule.suffix, self._suffix_length) 25 | if suffix not in _processed_rules: 26 | _processed_rules[suffix] = [] 27 | _processed_rules[suffix].append(rule) 28 | 29 | self.rules = _processed_rules 30 | 31 | def is_match(self, candidate): 32 | if len(candidate) < self._suffix_length: 33 | return False 34 | 35 | _match_rules = self.rules.get(get_string_tail_lower_case(candidate, self._suffix_length)) 36 | if _match_rules: 37 | for rule in _match_rules: 38 | if candidate.lower().endswith(rule.suffix): 39 | if len(candidate) == len(rule.suffix) or not rule.exact: 40 | return True 41 | 42 | return False 43 | 44 | 45 | def create_fast_suffix_matcher_from_existing(rules: List[MatchRule], existing: FastSuffixMatcher) -> FastSuffixMatcher: 46 | if existing is None or len(existing.rules) == 0: 47 | return FastSuffixMatcher(rules) 48 | 49 | if not rules: 50 | return existing 51 | 52 | return FastSuffixMatcher([*copy.deepcopy(rules), *(v for item in existing.rules.values() for v in item)]) 53 | 54 | 55 | class KustoTrustedEndpoints: 56 | def __init__(self): 57 | self._matchers = { 58 | k: FastSuffixMatcher( 59 | [*(MatchRule(suffix, False) for suffix in v["AllowedKustoSuffixes"]), *(MatchRule(hostname, True) for hostname in v["AllowedKustoHostnames"])] 60 | ) 61 | for (k, v) in _well_known_kusto_endpoints_data["AllowedEndpointsByLogin"].items() 62 | } 63 | 64 | self._additional_matcher = None 65 | self._override_matcher = None 66 | 67 | def set_override_matcher(self, matcher): 68 | self._override_matcher = matcher 69 | 70 | def add_trusted_hosts(self, rules, replace): 71 | if rules is None or not rules: 72 | if replace: 73 | self._additional_matcher = None 74 | return 75 | 76 | self._additional_matcher = create_fast_suffix_matcher_from_existing(rules, None if replace else self._additional_matcher) 77 | 78 | def validate_trusted_endpoint(self, endpoint: str, login_endpoint: str): 79 | hostname = urlparse(endpoint).hostname 80 | self.validate_hostname_is_trusted(hostname if hostname is not None else endpoint, login_endpoint) 81 | 82 | def validate_hostname_is_trusted(self, hostname: str, login_endpoint: str): 83 | if _is_local_address(hostname): 84 | return 85 | if self._override_matcher is not None: 86 | if self._override_matcher(hostname): 87 | return 88 | else: 89 | matcher = self._matchers.get(login_endpoint.lower()) 90 | if matcher is not None and matcher.is_match(hostname): 91 | return 92 | 93 | matcher = self._additional_matcher 94 | if matcher is not None and matcher.is_match(hostname): 95 | return 96 | 97 | raise KustoClientInvalidConnectionStringException( 98 | f"Can't communicate with '{hostname}' as this hostname is currently not trusted; please see " f"https://aka.ms/kustotrustedendpoints" 99 | ) 100 | 101 | def set_override_policy(self, matcher): 102 | self._override_matcher = matcher 103 | 104 | 105 | _well_known_kusto_endpoints_data = load_bundled_json("wellKnownKustoEndpoints.json") 106 | well_known_kusto_endpoints = KustoTrustedEndpoints() 107 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/py.typed: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-data/azure/kusto/data/py.typed -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/security.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | from typing import TYPE_CHECKING 4 | from urllib.parse import urlparse 5 | 6 | from ._token_providers import ( 7 | TokenProviderBase, 8 | BasicTokenProvider, 9 | CallbackTokenProvider, 10 | MsiTokenProvider, 11 | AzCliTokenProvider, 12 | UserPassTokenProvider, 13 | DeviceLoginTokenProvider, 14 | InteractiveLoginTokenProvider, 15 | ApplicationKeyTokenProvider, 16 | ApplicationCertificateTokenProvider, 17 | TokenConstants, 18 | AzureIdentityTokenCredentialProvider, 19 | ) 20 | from .exceptions import KustoAuthenticationError, KustoClientError 21 | 22 | if TYPE_CHECKING: 23 | from . import KustoConnectionStringBuilder 24 | 25 | 26 | class _AadHelper: 27 | kusto_uri = None # type: str 28 | authority_uri = None # type: str 29 | token_provider = None # type: TokenProviderBase 30 | 31 | def __init__(self, kcsb: "KustoConnectionStringBuilder", is_async: bool): 32 | parsed_url = urlparse(kcsb.data_source) 33 | self.kusto_uri = f"{parsed_url.scheme}://{parsed_url.hostname}" 34 | if parsed_url.port is not None: 35 | self.kusto_uri += f":{parsed_url.port}" 36 | 37 | self.username = None 38 | 39 | if kcsb.interactive_login: 40 | self.token_provider = InteractiveLoginTokenProvider(self.kusto_uri, kcsb.authority_id, kcsb.login_hint, kcsb.domain_hint, is_async=is_async) 41 | elif all([kcsb.aad_user_id, kcsb.password]): 42 | self.token_provider = UserPassTokenProvider(self.kusto_uri, kcsb.authority_id, kcsb.aad_user_id, kcsb.password, is_async=is_async) 43 | elif all([kcsb.application_client_id, kcsb.application_key]): 44 | self.token_provider = ApplicationKeyTokenProvider( 45 | self.kusto_uri, kcsb.authority_id, kcsb.application_client_id, kcsb.application_key, is_async=is_async 46 | ) 47 | elif all([kcsb.application_client_id, kcsb.application_certificate, kcsb.application_certificate_thumbprint]): 48 | # kcsb.application_public_certificate can be None if SNI is not used 49 | self.token_provider = ApplicationCertificateTokenProvider( 50 | self.kusto_uri, 51 | kcsb.application_client_id, 52 | kcsb.authority_id, 53 | kcsb.application_certificate, 54 | kcsb.application_certificate_thumbprint, 55 | kcsb.application_public_certificate, 56 | is_async=is_async, 57 | ) 58 | elif kcsb.msi_authentication: 59 | self.token_provider = MsiTokenProvider(self.kusto_uri, kcsb.msi_parameters, is_async=is_async) 60 | elif kcsb.user_token: 61 | self.token_provider = BasicTokenProvider(kcsb.user_token, is_async=is_async) 62 | elif kcsb.application_token: 63 | self.token_provider = BasicTokenProvider(kcsb.application_token, is_async=is_async) 64 | elif kcsb.az_cli_login: 65 | self.token_provider = AzCliTokenProvider(self.kusto_uri, is_async=is_async) 66 | elif kcsb.token_provider or kcsb.async_token_provider: 67 | self.token_provider = CallbackTokenProvider(token_callback=kcsb.token_provider, async_token_callback=kcsb.async_token_provider, is_async=is_async) 68 | elif kcsb.token_credential_login: 69 | self.token_provider = AzureIdentityTokenCredentialProvider( 70 | self.kusto_uri, 71 | is_async=is_async, 72 | credential=kcsb.azure_credential, 73 | credential_from_login_endpoint=kcsb.azure_credential_from_login_endpoint, 74 | ) 75 | elif kcsb.device_login: 76 | self.token_provider = DeviceLoginTokenProvider(self.kusto_uri, kcsb.authority_id, kcsb.device_callback, is_async=is_async) 77 | else: 78 | self.token_provider = InteractiveLoginTokenProvider(self.kusto_uri, kcsb.authority_id, kcsb.login_hint, kcsb.domain_hint, is_async=is_async) 79 | 80 | def acquire_authorization_header(self): 81 | try: 82 | return _get_header_from_dict(self.token_provider.get_token()) 83 | except Exception as error: 84 | kwargs = self.token_provider.context() 85 | kwargs["kusto_uri"] = self.kusto_uri 86 | raise KustoAuthenticationError(self.token_provider.name(), error, **kwargs) 87 | 88 | async def acquire_authorization_header_async(self): 89 | try: 90 | return _get_header_from_dict(await self.token_provider.get_token_async()) 91 | except Exception as error: 92 | kwargs = await self.token_provider.context_async() 93 | kwargs["resource"] = self.kusto_uri 94 | raise KustoAuthenticationError(self.token_provider.name(), error, **kwargs) 95 | 96 | def close(self): 97 | self.token_provider.close() 98 | 99 | async def close_async(self): 100 | await self.token_provider.close_async() 101 | 102 | 103 | def _get_header_from_dict(token: dict): 104 | if TokenConstants.MSAL_ACCESS_TOKEN in token: 105 | return _get_header(token[TokenConstants.MSAL_TOKEN_TYPE], token[TokenConstants.MSAL_ACCESS_TOKEN]) 106 | elif TokenConstants.AZ_ACCESS_TOKEN in token: 107 | return _get_header(token[TokenConstants.AZ_TOKEN_TYPE], token[TokenConstants.AZ_ACCESS_TOKEN]) 108 | else: 109 | raise KustoClientError("Unable to determine the token type. Neither 'tokenType' nor 'token_type' property is present.") 110 | 111 | 112 | def _get_header(token_type: str, access_token: str) -> str: 113 | return "{0} {1}".format(token_type, access_token) 114 | 115 | 116 | def _is_local_address(host): 117 | if host == "localhost" or host == "127.0.0.1" or host == "::1" or host == "[::1]": 118 | return True 119 | 120 | if host.startswith("127.") and 15 >= len(host) >= 9: 121 | for i in range(len(host)): 122 | c = host[i] 123 | if c != "." and (c < "0" or c > "9"): 124 | return False 125 | i += 1 126 | return True 127 | 128 | return False 129 | -------------------------------------------------------------------------------- /azure-kusto-data/azure/kusto/data/wellKnownKustoEndpoints.json: -------------------------------------------------------------------------------- 1 | { 2 | "_Comments": [ 3 | "KWE .linkedin.com suffix is excluded from list pending a more specific suffix", 4 | "LogAnalytics, AppInsigts & AzureMonitor are taken from https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/138095/Calling-the-ADX-Proxy following talk with Raz Ronen", 5 | "The two DXP suffixes come from KWE code", 6 | "The PlayFab suffixes are a superset of KWE and PowerBI Kusto Connector code", 7 | "Aria hostname is fixed" 8 | ], 9 | "AllowedEndpointsByLogin": { 10 | "https://login.microsoftonline.com": { 11 | "AllowedKustoSuffixes": [ 12 | ".dxp.aad.azure.com", 13 | ".dxp-dev.aad.azure.com", 14 | ".kusto.azuresynapse.net", 15 | ".kusto.windows.net", 16 | ".kustodev.azuresynapse-dogfood.net", 17 | ".kustodev.windows.net", 18 | ".kustomfa.windows.net", 19 | ".playfabapi.com", 20 | ".playfab.com", 21 | ".azureplayfab.com", 22 | ".kusto.data.microsoft.com", 23 | ".kusto.fabric.microsoft.com", 24 | ".api.securityplatform.microsoft.com", 25 | ".securitycenter.windows.com" 26 | ], 27 | "AllowedKustoHostnames": [ 28 | "ade.applicationinsights.io", 29 | "ade.loganalytics.io", 30 | "adx.aimon.applicationinsights.azure.com", 31 | "adx.applicationinsights.azure.com", 32 | "adx.int.applicationinsights.azure.com", 33 | "adx.int.loganalytics.azure.com", 34 | "adx.int.monitor.azure.com", 35 | "adx.loganalytics.azure.com", 36 | "adx.monitor.azure.com", 37 | "kusto.aria.microsoft.com", 38 | "eu.kusto.aria.microsoft.com" 39 | ] 40 | }, 41 | "https://login.microsoftonline.us": { 42 | "AllowedKustoSuffixes": [ 43 | ".kusto.usgovcloudapi.net", 44 | ".kustomfa.usgovcloudapi.net" 45 | 46 | ], 47 | "AllowedKustoHostnames": [ 48 | "adx.applicationinsights.azure.us", 49 | "adx.loganalytics.azure.us", 50 | "adx.monitor.azure.us" 51 | ] 52 | }, 53 | "https://login.partner.microsoftonline.cn": { 54 | "AllowedKustoSuffixes": [ 55 | ".kusto.azuresynapse.azure.cn", 56 | ".kusto.chinacloudapi.cn", 57 | ".kustomfa.chinacloudapi.cn", 58 | ".playfab.cn" 59 | ], 60 | "AllowedKustoHostnames": [ 61 | "adx.applicationinsights.azure.cn", 62 | "adx.loganalytics.azure.cn", 63 | "adx.monitor.azure.cn" 64 | ] 65 | }, 66 | "https://login.microsoftonline.eaglex.ic.gov": { 67 | "AllowedKustoSuffixes": [ 68 | ".kusto.core.eaglex.ic.gov", 69 | ".kustomfa.core.eaglex.ic.gov" 70 | 71 | ], 72 | "AllowedKustoHostnames": [ 73 | "adx.applicationinsights.azure.eaglex.ic.gov", 74 | "adx.loganalytics.azure.eaglex.ic.gov", 75 | "adx.monitor.azure.eaglex.ic.gov" 76 | ] 77 | }, 78 | "https://login.microsoftonline.microsoft.scloud": { 79 | "AllowedKustoSuffixes": [ 80 | ".kusto.core.microsoft.scloud", 81 | ".kustomfa.core.microsoft.scloud" 82 | ], 83 | "AllowedKustoHostnames": [ 84 | "adx.applicationinsights.azure.microsoft.scloud", 85 | "adx.loganalytics.azure.microsoft.scloud", 86 | "adx.monitor.azure.microsoft.scloud" 87 | ] 88 | } 89 | } 90 | } 91 | -------------------------------------------------------------------------------- /azure-kusto-data/setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal=1 3 | 4 | [flake8] 5 | ignore = E226,E302,E41 6 | max-line-length = 160 7 | exclude = tests/* 8 | max-complexity = 10 9 | 10 | 11 | [pylint] 12 | max-line-length = 160 -------------------------------------------------------------------------------- /azure-kusto-data/setup.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import re 4 | from os import path 5 | 6 | # Always prefer setuptools over distutils 7 | from setuptools import setup, find_packages 8 | 9 | PACKAGE_NAME = "azure-kusto-data" 10 | 11 | # a-b-c => a/b/c 12 | PACKAGE_FOLDER_PATH = PACKAGE_NAME.replace("-", path.sep) 13 | # a-b-c => a.b.c 14 | NAMESPACE_NAME = PACKAGE_NAME.replace("-", ".") 15 | 16 | with open(path.join(PACKAGE_FOLDER_PATH, "_version.py"), "r") as fd: 17 | VERSION = re.search(r'^VERSION\s*=\s*[\'"]([^\'"]*)[\'"]', fd.read(), re.MULTILINE).group(1) 18 | 19 | if not VERSION: 20 | raise RuntimeError("Cannot find version information") 21 | 22 | 23 | setup( 24 | name=PACKAGE_NAME, 25 | version=VERSION, 26 | description="Kusto Data Client", 27 | long_description_content_type="text/markdown", 28 | long_description=open("README.rst", "r").read(), 29 | url="https://github.com/Azure/azure-kusto-python", 30 | author="Microsoft Corporation", 31 | author_email="kustalk@microsoft.com", 32 | license="MIT", 33 | classifiers=[ 34 | # 5 - Production/Stable depends on multi-threading / aio / perf 35 | "Development Status :: 4 - Beta", 36 | "Intended Audience :: Developers", 37 | "Topic :: Software Development", 38 | "Programming Language :: Python :: 3.8", 39 | "Programming Language :: Python :: 3.8", 40 | "Programming Language :: Python :: 3.9", 41 | "Programming Language :: Python :: 3.10", 42 | "Programming Language :: Python :: 3.11", 43 | "Programming Language :: Python :: 3.12", 44 | "Programming Language :: Python :: 3.13", 45 | "License :: OSI Approved :: MIT License", 46 | ], 47 | namespace_packages=["azure"], 48 | keywords="kusto wrapper client library", 49 | packages=find_packages(exclude=["azure", "*tests*", "*tests.*"]), 50 | package_data={"": ["wellKnownKustoEndpoints.json", "py.typed", "kcsb.json"]}, 51 | include_package_data=True, 52 | install_requires=["python-dateutil>=2.8.0", "requests>=2.32.3", "azure-identity>=1.21.0,<2", "msal>=1.9.0,<2", "ijson~=3.1", "azure-core>=1.33.0,<2"], 53 | extras_require={"pandas": ["pandas"], "aio": ["aiohttp>=3.8.0,<4", "asgiref>=3.2.3,<4"]}, 54 | ) 55 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-data/tests/__init__.py -------------------------------------------------------------------------------- /azure-kusto-data/tests/aio/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-data/tests/aio/__init__.py -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/adminthenquery.json: -------------------------------------------------------------------------------- 1 | { 2 | "Tables": [{ 3 | "TableName": "Table_0", 4 | "Columns": [{ 5 | "ColumnName": "DatabaseName", 6 | "DataType": "String" 7 | }, 8 | { 9 | "ColumnName": "TableName", 10 | "DataType": "String" 11 | }], 12 | "Rows": [["Kuskus", 13 | "KustoLogs"], 14 | ["Kuskus", 15 | "LiorTmp"]] 16 | }, 17 | { 18 | "TableName": "Table_1", 19 | "Columns": [{ 20 | "ColumnName": "Value", 21 | "DataType": "String" 22 | }], 23 | "Rows": [["{\"Visualization\": null,\"Title\": null,\"XColumn\": null,\"Series\": null,\"YColumns\": null,\"XTitle\": null,\"YTitle\": null,\"XAxis\": null,\"YAxis\": null,\"Legend\": null,\"YSplit\": null,\"Accumulate\": false,\"IsQuerySorted\": false,\"Kind\": null}"]] 24 | }, 25 | { 26 | "TableName": "Table_2", 27 | "Columns": [{ 28 | "ColumnName": "Timestamp", 29 | "DataType": "DateTime" 30 | }, 31 | { 32 | "ColumnName": "Severity", 33 | "DataType": "Int32" 34 | }, 35 | { 36 | "ColumnName": "SeverityName", 37 | "DataType": "String" 38 | }, 39 | { 40 | "ColumnName": "StatusCode", 41 | "DataType": "Int32" 42 | }, 43 | { 44 | "ColumnName": "StatusDescription", 45 | "DataType": "String" 46 | }, 47 | { 48 | "ColumnName": "Count", 49 | "DataType": "Int32" 50 | }, 51 | { 52 | "ColumnName": "RequestId", 53 | "DataType": "Guid" 54 | }, 55 | { 56 | "ColumnName": "ActivityId", 57 | "DataType": "Guid" 58 | }, 59 | { 60 | "ColumnName": "SubActivityId", 61 | "DataType": "Guid" 62 | }, 63 | { 64 | "ColumnName": "ClientActivityId", 65 | "DataType": "String" 66 | }], 67 | "Rows": [["2018-08-12T09:13:19.5200972Z", 68 | 4, 69 | "Info", 70 | 0, 71 | "Querycompletedsuccessfully", 72 | 1, 73 | "b6651693-9325-41c8-a5bf-dcc21202cdf2", 74 | "b6651693-9325-41c8-a5bf-dcc21202cdf2", 75 | "1cfa59c2-7f29-4e58-aef8-590d4609818f", 76 | "KPC.execute;721add30-f61a-44d0-ad89-812d06736260"], 77 | ["2018-08-12T09:13:19.5200972Z", 78 | 6, 79 | "Stats", 80 | 0, 81 | { 82 | "ExecutionTime": 0.0, 83 | "resource_usage": { 84 | "cache": { 85 | "memory": { 86 | "hits": 0, 87 | "misses": 0, 88 | "total": 0 89 | }, 90 | "disk": { 91 | "hits": 0, 92 | "misses": 0, 93 | "total": 0 94 | } 95 | }, 96 | "cpu": { 97 | "user": "00:00:00", 98 | "kernel": "00:00:00", 99 | "total cpu": "00:00:00" 100 | }, 101 | "memory": { 102 | "peak_per_node": 0 103 | } 104 | }, 105 | "input_dataset_statistics": { 106 | "extents": { 107 | "total": 0, 108 | "scanned": 0 109 | }, 110 | "rows": { 111 | "total": 0, 112 | "scanned": 0 113 | } 114 | }, 115 | "dataset_statistics": [{ 116 | "table_row_count": 2, 117 | "table_size": 46 118 | }] 119 | }, 120 | 1, 121 | "b6651693-9325-41c8-a5bf-dcc21202cdf2", 122 | "b6651693-9325-41c8-a5bf-dcc21202cdf2", 123 | "1cfa59c2-7f29-4e58-aef8-590d4609818f", 124 | "KPC.execute;721add30-f61a-44d0-ad89-812d06736260"]] 125 | }, 126 | { 127 | "TableName": "Table_3", 128 | "Columns": [{ 129 | "ColumnName": "Ordinal", 130 | "DataType": "Int64" 131 | }, 132 | { 133 | "ColumnName": "Kind", 134 | "DataType": "String" 135 | }, 136 | { 137 | "ColumnName": "Name", 138 | "DataType": "String" 139 | }, 140 | { 141 | "ColumnName": "Id", 142 | "DataType": "String" 143 | }, 144 | { 145 | "ColumnName": "PrettyName", 146 | "DataType": "String" 147 | }], 148 | "Rows": [[0, 149 | "QueryResult", 150 | "PrimaryResult", 151 | "d6331ef2-d9f7-4d8c-8268-99f574babc82", 152 | ""], 153 | [1, 154 | "QueryProperties", 155 | "@ExtendedProperties", 156 | "876ccb1a-818e-431f-9147-6b72547cca3d", 157 | ""], 158 | [2, 159 | "QueryStatus", 160 | "QueryStatus", 161 | "00000000-0000-0000-0000-000000000000", 162 | ""]] 163 | }] 164 | } 165 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/dynamic.json: -------------------------------------------------------------------------------- 1 | [{ 2 | "FrameType": "DataSetHeader", 3 | "IsProgressive": false, 4 | "Version": "v2.0" 5 | }, 6 | { 7 | "FrameType": "DataTable", 8 | "TableId": 0, 9 | "TableKind": "QueryProperties", 10 | "TableName": "@ExtendedProperties", 11 | "Columns": [{ 12 | "ColumnName": "TableId", 13 | "ColumnType": "int" 14 | }, 15 | { 16 | "ColumnName": "Key", 17 | "ColumnType": "string" 18 | }, 19 | { 20 | "ColumnName": "Value", 21 | "ColumnType": "dynamic" 22 | }], 23 | "Rows": [[1, 24 | "Visualization", 25 | "{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"AnomalyColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null,\"Ymin\":\"NaN\",\"Ymax\":\"NaN\"}"]] 26 | }, 27 | { 28 | "FrameType": "DataTable", 29 | "TableId": 1, 30 | "TableKind": "PrimaryResult", 31 | "TableName": "PrimaryResult", 32 | "Columns": [{ 33 | "ColumnName": "print_0", 34 | "ColumnType": "dynamic" 35 | }, 36 | { 37 | "ColumnName": "print_1", 38 | "ColumnType": "dynamic" 39 | }, 40 | { 41 | "ColumnName": "print_2", 42 | "ColumnType": "dynamic" 43 | }, 44 | { 45 | "ColumnName": "print_3", 46 | "ColumnType": "dynamic" 47 | }, 48 | { 49 | "ColumnName": "print_4", 50 | "ColumnType": "dynamic" 51 | }, 52 | { 53 | "ColumnName": "print_5", 54 | "ColumnType": "dynamic" 55 | }], 56 | "Rows": [[123, 57 | "123", 58 | "test bad json", 59 | null, 60 | "{\"rowId\":2,\"arr\":[0,2]}", 61 | { 62 | "rowId": 2, 63 | "arr": [0, 64 | 2] 65 | }]] 66 | }, 67 | { 68 | "FrameType": "DataTable", 69 | "TableId": 2, 70 | "TableKind": "QueryCompletionInformation", 71 | "TableName": "QueryCompletionInformation", 72 | "Columns": [{ 73 | "ColumnName": "Timestamp", 74 | "ColumnType": "datetime" 75 | }, 76 | { 77 | "ColumnName": "ClientRequestId", 78 | "ColumnType": "string" 79 | }, 80 | { 81 | "ColumnName": "ActivityId", 82 | "ColumnType": "guid" 83 | }, 84 | { 85 | "ColumnName": "SubActivityId", 86 | "ColumnType": "guid" 87 | }, 88 | { 89 | "ColumnName": "ParentActivityId", 90 | "ColumnType": "guid" 91 | }, 92 | { 93 | "ColumnName": "Level", 94 | "ColumnType": "int" 95 | }, 96 | { 97 | "ColumnName": "LevelName", 98 | "ColumnType": "string" 99 | }, 100 | { 101 | "ColumnName": "StatusCode", 102 | "ColumnType": "int" 103 | }, 104 | { 105 | "ColumnName": "StatusCodeName", 106 | "ColumnType": "string" 107 | }, 108 | { 109 | "ColumnName": "EventType", 110 | "ColumnType": "int" 111 | }, 112 | { 113 | "ColumnName": "EventTypeName", 114 | "ColumnType": "string" 115 | }, 116 | { 117 | "ColumnName": "Payload", 118 | "ColumnType": "string" 119 | }], 120 | "Rows": [["2019-02-10T12:07:01.0562684Z", 121 | "KPC.execute;f0131f65-d1ed-4c9d-9110-ddf9879f7ff6", 122 | "4400a720-f11d-43b6-9097-63462d35bcd4", 123 | "bf1b857c-e857-44f6-8d8c-156081bfb92c", 124 | "57e0f58d-398c-490b-b900-9f308c531af7", 125 | 4, 126 | "Info", 127 | 0, 128 | "S_OK (0)", 129 | 4, 130 | "QueryInfo", 131 | "{\"Count\":1,\"Text\":\"Query completed successfully\"}"], 132 | ["2019-02-10T12:07:01.0562684Z", 133 | "KPC.execute;f0131f65-d1ed-4c9d-9110-ddf9879f7ff6", 134 | "4400a720-f11d-43b6-9097-63462d35bcd4", 135 | "bf1b857c-e857-44f6-8d8c-156081bfb92c", 136 | "57e0f58d-398c-490b-b900-9f308c531af7", 137 | 6, 138 | "Stats", 139 | 0, 140 | "S_OK (0)", 141 | 0, 142 | "QueryResourceConsumption", 143 | "{\"ExecutionTime\":0.1875076,\"resource_usage\":{\"cache\":{\"memory\":{\"hits\":0,\"misses\":0,\"total\":0},\"disk\":{\"hits\":0,\"misses\":0,\"total\":0},\"shards\":{\"hitbytes\":0,\"missbytes\":0,\"bypassbytes\":0}},\"cpu\":{\"user\":\"00:00:00\",\"kernel\":\"00:00:00\",\"total cpu\":\"00:00:00\"},\"memory\":{\"peak_per_node\":0}},\"input_dataset_statistics\":{\"extents\":{\"total\":0,\"scanned\":0},\"rows\":{\"total\":0,\"scanned\":0},\"rowstores\":{\"scanned_rows\":0,\"scanned_values_size\":0}},\"dataset_statistics\":[{\"table_row_count\":1,\"table_size\":141}]}"]] 144 | }, 145 | { 146 | "FrameType": "DataSetCompletion", 147 | "HasErrors": false, 148 | "Cancelled": false 149 | }] -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/null_values.json: -------------------------------------------------------------------------------- 1 | [{ 2 | "FrameType": "DataSetHeader", 3 | "IsProgressive": false, 4 | "Version": "v2.0" 5 | }, 6 | { 7 | "FrameType": "DataTable", 8 | "TableId": 0, 9 | "TableKind": "QueryProperties", 10 | "TableName": "@ExtendedProperties", 11 | "Columns": [{ 12 | "ColumnName": "TableId", 13 | "ColumnType": "int" 14 | }, 15 | { 16 | "ColumnName": "Key", 17 | "ColumnType": "string" 18 | }, 19 | { 20 | "ColumnName": "Value", 21 | "ColumnType": "dynamic" 22 | }], 23 | "Rows": [[null, 24 | null, 25 | null]] 26 | }, 27 | { 28 | "FrameType": "DataTable", 29 | "TableId": 1, 30 | "TableKind": "PrimaryResult", 31 | "TableName": "PrimaryResultName", 32 | "Columns": [{ 33 | "ColumnName": "String", 34 | "ColumnType": "string" 35 | }, 36 | { 37 | "ColumnName": "Int", 38 | "ColumnType": "int" 39 | }, 40 | { 41 | "ColumnName": "Bool", 42 | "ColumnType": "bool" 43 | }, 44 | { 45 | "ColumnName": "Datetime", 46 | "ColumnType": "datetime" 47 | }, 48 | { 49 | "ColumnName": "Dynamic", 50 | "ColumnType": "dynamic" 51 | }, 52 | { 53 | "ColumnName": "Guid", 54 | "ColumnType": "guid" 55 | }, 56 | { 57 | "ColumnName": "Long", 58 | "ColumnType": "long" 59 | }, 60 | { 61 | "ColumnName": "Real", 62 | "ColumnType": "real" 63 | }, 64 | { 65 | "ColumnName": "Timespan", 66 | "ColumnType": "timespan" 67 | }, 68 | { 69 | "ColumnName": "Decimal", 70 | "ColumnType": "decimal" 71 | }], 72 | "Rows": [[null,null,null,null,null,null,null,null,null,null]] 73 | }, 74 | { 75 | "FrameType": "DataTable", 76 | "TableId": 2, 77 | "TableKind": "QueryCompletionInformation", 78 | "TableName": "QueryCompletionInformation", 79 | "Columns": [{ 80 | "ColumnName": "Timestamp", 81 | "ColumnType": "datetime" 82 | }, 83 | { 84 | "ColumnName": "ClientRequestId", 85 | "ColumnType": "string" 86 | }, 87 | { 88 | "ColumnName": "ActivityId", 89 | "ColumnType": "guid" 90 | }, 91 | { 92 | "ColumnName": "SubActivityId", 93 | "ColumnType": "guid" 94 | }, 95 | { 96 | "ColumnName": "ParentActivityId", 97 | "ColumnType": "guid" 98 | }, 99 | { 100 | "ColumnName": "Level", 101 | "ColumnType": "int" 102 | }, 103 | { 104 | "ColumnName": "LevelName", 105 | "ColumnType": "string" 106 | }, 107 | { 108 | "ColumnName": "StatusCode", 109 | "ColumnType": "int" 110 | }, 111 | { 112 | "ColumnName": "StatusCodeName", 113 | "ColumnType": "string" 114 | }, 115 | { 116 | "ColumnName": "EventType", 117 | "ColumnType": "int" 118 | }, 119 | { 120 | "ColumnName": "EventTypeName", 121 | "ColumnType": "string" 122 | }, 123 | { 124 | "ColumnName": "Payload", 125 | "ColumnType": "string" 126 | }], 127 | "Rows": [["2019-02-12T10:23:02.0413963Z", 128 | "KPC.execute;57050c90-8a7d-4b29-b8d0-a40688a8185c", 129 | "dfbaa865-e29d-46e0-af17-be22c6c113ac", 130 | "e2bf7a6c-adf1-48da-b117-667a92874d38", 131 | "81409d63-718b-4d06-9711-7eab476b7ceb", 132 | 4, 133 | "Info", 134 | 0, 135 | "S_OK (0)", 136 | 4, 137 | "QueryInfo", 138 | "{\"Count\":1,\"Text\":\"Querycompletedsuccessfully\"}"], 139 | ["2019-02-12T10:23:02.0413963Z", 140 | "KPC.execute;57050c90-8a7d-4b29-b8d0-a40688a8185c", 141 | "dfbaa865-e29d-46e0-af17-be22c6c113ac", 142 | "e2bf7a6c-adf1-48da-b117-667a92874d38", 143 | "81409d63-718b-4d06-9711-7eab476b7ceb", 144 | 6, 145 | "Stats", 146 | 0, 147 | "S_OK (0)", 148 | 0, 149 | "QueryResourceConsumption", 150 | "{\"ExecutionTime\":0.0156475,\"resource_usage\":{\"cache\":{\"memory\":{\"hits\":0,\"misses\":0,\"total\":0},\"disk\":{\"hits\":0,\"misses\":0,\"total\":0},\"shards\":{\"hitbytes\":0,\"missbytes\":0,\"bypassbytes\":0}},\"cpu\":{\"user\":\"00: 00: 00\",\"kernel\":\"00: 00: 00\",\"totalcpu\":\"00: 00: 00\"},\"memory\":{\"peak_per_node\":0}},\"input_dataset_statistics\":{\"extents\":{\"total\":0,\"scanned\":0},\"rows\":{\"total\":0,\"scanned\":0},\"rowstores\":{\"scanned_rows\":0,\"scanned_values_size\":0}},\"dataset_statistics\":[{\"table_row_count\":0,\"table_size\":0}]}"]] 151 | }, 152 | { 153 | "FrameType": "DataSetCompletion", 154 | "HasErrors": false, 155 | "Cancelled": false 156 | }] -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/pandas_bool.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "FrameType": "DataSetHeader", 4 | "IsProgressive": false, 5 | "Version": "v2.0" 6 | }, 7 | { 8 | "FrameType": "DataTable", 9 | "TableId": 0, 10 | "TableName": "@ExtendedProperties", 11 | "TableKind": "QueryProperties", 12 | "Columns": [ 13 | { 14 | "ColumnName": "TableId", 15 | "ColumnType": "int" 16 | }, 17 | { 18 | "ColumnName": "Key", 19 | "ColumnType": "string" 20 | }, 21 | { 22 | "ColumnName": "Value", 23 | "ColumnType": "dynamic" 24 | } 25 | ], 26 | "Rows": [ 27 | [ 28 | 1, 29 | "Visualization", 30 | "{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null}" 31 | ] 32 | ] 33 | }, 34 | { 35 | "FrameType": "DataTable", 36 | "TableId": 1, 37 | "TableName": "Deft", 38 | "TableKind": "PrimaryResult", 39 | "Columns": [ 40 | { 41 | "ColumnName": "xbool", 42 | "ColumnType": "bool" 43 | } 44 | ], 45 | "Rows": [ 46 | [ 47 | null 48 | ], 49 | [ 50 | true 51 | ], 52 | [ 53 | false 54 | ] 55 | ] 56 | }, 57 | { 58 | "FrameType": "DataTable", 59 | "TableId": 2, 60 | "TableName": "QueryCompletionInformation", 61 | "TableKind": "QueryCompletionInformation", 62 | "Columns": [ 63 | { 64 | "ColumnName": "Timestamp", 65 | "ColumnType": "datetime" 66 | }, 67 | { 68 | "ColumnName": "ClientRequestId", 69 | "ColumnType": "string" 70 | }, 71 | { 72 | "ColumnName": "ActivityId", 73 | "ColumnType": "guid" 74 | }, 75 | { 76 | "ColumnName": "SubActivityId", 77 | "ColumnType": "guid" 78 | }, 79 | { 80 | "ColumnName": "ParentActivityId", 81 | "ColumnType": "guid" 82 | }, 83 | { 84 | "ColumnName": "Level", 85 | "ColumnType": "int" 86 | }, 87 | { 88 | "ColumnName": "LevelName", 89 | "ColumnType": "string" 90 | }, 91 | { 92 | "ColumnName": "StatusCode", 93 | "ColumnType": "int" 94 | }, 95 | { 96 | "ColumnName": "StatusCodeName", 97 | "ColumnType": "string" 98 | }, 99 | { 100 | "ColumnName": "EventType", 101 | "ColumnType": "int" 102 | }, 103 | { 104 | "ColumnName": "EventTypeName", 105 | "ColumnType": "string" 106 | }, 107 | { 108 | "ColumnName": "Payload", 109 | "ColumnType": "string" 110 | } 111 | ], 112 | "Rows": [ 113 | [ 114 | "2018-04-30T12:25:11.0778067Z", 115 | "unspecified;cc5ee2b9-9b77-4509-9a61-84ec8f0159c2", 116 | "eeac049e-8a7d-4188-b797-6b5f2c9f9526", 117 | "c6fb9714-5183-4092-8a02-825bd7aa1aee", 118 | "df0aaeb0-4c8a-4d77-bc77-f714a4484a6b", 119 | 4, 120 | "Info", 121 | 0, 122 | "S_OK (0)", 123 | 4, 124 | "QueryInfo", 125 | "{\"Count\":1,\"Text\":\"Query completed successfully\"}" 126 | ], 127 | [ 128 | "2018-04-30T12:25:11.0778067Z", 129 | "unspecified;cc5ee2b9-9b77-4509-9a61-84ec8f0159c2", 130 | "eeac049e-8a7d-4188-b797-6b5f2c9f9526", 131 | "c6fb9714-5183-4092-8a02-825bd7aa1aee", 132 | "df0aaeb0-4c8a-4d77-bc77-f714a4484a6b", 133 | 6, 134 | "Stats", 135 | 0, 136 | "S_OK (0)", 137 | 0, 138 | "QueryResourceConsumption", 139 | "{\"ExecutionTime\":0.0156154,\"resource_usage\":{\"cache\":{\"memory\":{\"hits\":40,\"misses\":0,\"total\":40},\"disk\":{\"hits\":0,\"misses\":0,\"total\":0}},\"cpu\":{\"user\":\"00:00:00\",\"kernel\":\"00:00:00\",\"total cpu\":\"00:00:00\"},\"memory\":{\"peak_per_node\":0}},\"dataset_statistics\":[{\"table_row_count\":11,\"table_size\":2444}]}" 140 | ] 141 | ] 142 | }, 143 | { 144 | "FrameType": "DataSetCompletion", 145 | "HasErrors": false, 146 | "Cancelled": false 147 | } 148 | ] 149 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/query_partial_results_defer_is_false.json: -------------------------------------------------------------------------------- 1 | [{ 2 | "FrameType": "DataSetHeader", 3 | "IsProgressive": false, 4 | "Version": "v2.0" 5 | }, 6 | { 7 | "FrameType": "DataTable", 8 | "TableId": 0, 9 | "TableKind": "QueryProperties", 10 | "TableName": "@ExtendedProperties", 11 | "Columns": [{ 12 | "ColumnName": "TableId", 13 | "ColumnType": "int" 14 | }, 15 | { 16 | "ColumnName": "Key", 17 | "ColumnType": "string" 18 | }, 19 | { 20 | "ColumnName": "Value", 21 | "ColumnType": "dynamic" 22 | }], 23 | "Rows": [[1, 24 | "Visualization", 25 | "{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"AnomalyColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null}"]] 26 | }, 27 | { 28 | "FrameType": "DataTable", 29 | "TableId": 1, 30 | "TableKind": "PrimaryResult", 31 | "TableName": "PrimaryResult", 32 | "Columns": [{ 33 | "ColumnName": "x", 34 | "ColumnType": "long" 35 | }], 36 | "Rows": [[1], 37 | [2], 38 | [3], 39 | [4], 40 | [5], 41 | { 42 | "OneApiErrors": [{ 43 | "error": { 44 | "code": "LimitsExceeded", 45 | "message": "Request is invalid and cannot be executed.", 46 | "@type": "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", 47 | "@message": "Query execution has exceeded the allowed limits (80DA0003): .", 48 | "@context": { 49 | "timestamp": "2018-12-10T15:10:48.8352222Z", 50 | "machineName": "RD0003FFBEDEB9", 51 | "processName": "Kusto.Azure.Svc", 52 | "processId": 4328, 53 | "threadId": 7284, 54 | "appDomainName": "RdRuntime", 55 | "clientRequestId": "KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3", 56 | "activityId": "a57ec272-8846-49e6-b458-460b841ed47d", 57 | "subActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 58 | "activityType": "PO-OWIN-CallContext", 59 | "parentActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 60 | "activityStack": "(Activity stack: CRID=KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3 ARID=a57ec272-8846-49e6-b458-460b841ed47d > PO-OWIN-CallContext/a57ec272-8846-49e6-b458-460b841ed47d)" 61 | }, 62 | "@permanent": false 63 | } 64 | }] 65 | }] 66 | }, 67 | { 68 | "FrameType": "DataSetCompletion", 69 | "HasErrors": true, 70 | "Cancelled": false, 71 | "OneApiErrors": [{ 72 | "error": { 73 | "code": "LimitsExceeded", 74 | "message": "Request is invalid and cannot be executed.", 75 | "@type": "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", 76 | "@message": "Query execution has exceeded the allowed limits (80DA0003): .", 77 | "@context": { 78 | "timestamp": "2018-12-10T15:10:48.8352222Z", 79 | "machineName": "RD0003FFBEDEB9", 80 | "processName": "Kusto.Azure.Svc", 81 | "processId": 4328, 82 | "threadId": 7284, 83 | "appDomainName": "RdRuntime", 84 | "clientRequestId": "KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3", 85 | "activityId": "a57ec272-8846-49e6-b458-460b841ed47d", 86 | "subActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 87 | "activityType": "PO-OWIN-CallContext", 88 | "parentActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 89 | "activityStack": "(Activity stack: CRID=KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3 ARID=a57ec272-8846-49e6-b458-460b841ed47d > PO-OWIN-CallContext/a57ec272-8846-49e6-b458-460b841ed47d)" 90 | }, 91 | "@permanent": false 92 | } 93 | }] 94 | }] 95 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/query_partial_results_defer_is_true.json: -------------------------------------------------------------------------------- 1 | [{ 2 | "FrameType": "DataSetHeader", 3 | "IsProgressive": false, 4 | "Version": "v2.0" 5 | }, 6 | { 7 | "FrameType": "DataTable", 8 | "TableId": 0, 9 | "TableKind": "QueryProperties", 10 | "TableName": "@ExtendedProperties", 11 | "Columns": [{ 12 | "ColumnName": "TableId", 13 | "ColumnType": "int" 14 | }, 15 | { 16 | "ColumnName": "Key", 17 | "ColumnType": "string" 18 | }, 19 | { 20 | "ColumnName": "Value", 21 | "ColumnType": "dynamic" 22 | }], 23 | "Rows": [[1, 24 | "Visualization", 25 | "{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"AnomalyColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null}"]] 26 | }, 27 | { 28 | "FrameType": "DataTable", 29 | "TableId": 1, 30 | "TableKind": "PrimaryResult", 31 | "TableName": "PrimaryResult", 32 | "Columns": [{ 33 | "ColumnName": "x", 34 | "ColumnType": "long" 35 | }], 36 | "Rows": [[1], 37 | [2], 38 | [3], 39 | [4], 40 | [5]] 41 | }, 42 | { 43 | "FrameType": "DataTable", 44 | "TableId": 2, 45 | "TableKind": "QueryCompletionInformation", 46 | "TableName": "QueryCompletionInformation", 47 | "Columns": [{ 48 | "ColumnName": "Timestamp", 49 | "ColumnType": "datetime" 50 | }, 51 | { 52 | "ColumnName": "ClientRequestId", 53 | "ColumnType": "string" 54 | }, 55 | { 56 | "ColumnName": "ActivityId", 57 | "ColumnType": "guid" 58 | }, 59 | { 60 | "ColumnName": "SubActivityId", 61 | "ColumnType": "guid" 62 | }, 63 | { 64 | "ColumnName": "ParentActivityId", 65 | "ColumnType": "guid" 66 | }, 67 | { 68 | "ColumnName": "Level", 69 | "ColumnType": "int" 70 | }, 71 | { 72 | "ColumnName": "LevelName", 73 | "ColumnType": "string" 74 | }, 75 | { 76 | "ColumnName": "StatusCode", 77 | "ColumnType": "int" 78 | }, 79 | { 80 | "ColumnName": "StatusCodeName", 81 | "ColumnType": "string" 82 | }, 83 | { 84 | "ColumnName": "EventType", 85 | "ColumnType": "int" 86 | }, 87 | { 88 | "ColumnName": "EventTypeName", 89 | "ColumnType": "string" 90 | }, 91 | { 92 | "ColumnName": "Payload", 93 | "ColumnType": "string" 94 | }], 95 | "Rows": [["2018-12-10T15:12:14.2991851Z", 96 | "KPC.execute;eba30dbd-9963-4c70-902d-60169b8b5a9c", 97 | "031555a6-3a4f-4766-b3c3-fd28bf5d8ad6", 98 | "a1e0d6fd-1def-46eb-b630-12efa5f88041", 99 | "8421d4a0-d5b8-4e04-b345-019b2b2edf85", 100 | 2, 101 | "Error", 102 | -2133196797, 103 | "Query result set too large (E_QUERY_RESULT_SET_TOO_LARGE). (-2133196797)", 104 | 1, 105 | "QueryLimitsExceeded", 106 | "{\"Count\":1,\"Text\":\"Queryresultsethasexceededtheinternalrecordcountlimit5(E_QUERY_RESULT_SET_TOO_LARGE;seehttp: //aka.ms/kustoquerylimits)\"}"], 107 | ["2018-12-10T15:12:14.2991851Z", 108 | "KPC.execute;eba30dbd-9963-4c70-902d-60169b8b5a9c", 109 | "031555a6-3a4f-4766-b3c3-fd28bf5d8ad6", 110 | "a1e0d6fd-1def-46eb-b630-12efa5f88041", 111 | "8421d4a0-d5b8-4e04-b345-019b2b2edf85", 112 | 6, 113 | "Stats", 114 | 0, 115 | "S_OK (0)", 116 | 0, 117 | "QueryResourceConsumption", 118 | "{\"ExecutionTime\":0.0,\"resource_usage\":{\"cache\":{\"memory\":{\"hits\":0,\"misses\":0,\"total\":0},\"disk\":{\"hits\":0,\"misses\":0,\"total\":0}},\"cpu\":{\"user\":\"00: 00: 00\",\"kernel\":\"00: 00: 00\",\"totalcpu\":\"00: 00: 00\"},\"memory\":{\"peak_per_node\":16777312}},\"input_dataset_statistics\":{\"extents\":{\"total\":0,\"scanned\":0},\"rows\":{\"total\":0,\"scanned\":0},\"rowstores\":{\"scanned_rows\":0,\"scanned_values_size\":0}},\"dataset_statistics\":[{\"table_row_count\":5,\"table_size\":40}]}"]] 119 | }, 120 | { 121 | "FrameType": "DataSetCompletion", 122 | "HasErrors": false, 123 | "Cancelled": false 124 | }] -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/versionshowcommandresult.json: -------------------------------------------------------------------------------- 1 | {"Tables":[{"TableName":"Table_0","Columns":[{"ColumnName":"BuildVersion","DataType":"String"},{"ColumnName":"BuildTime","DataType":"DateTime"},{"ColumnName":"ServiceType","DataType":"String"},{"ColumnName":"ProductVersion","DataType":"String"}],"Rows":[["1.0.6693.14577","2018-04-29T08:05:54Z","Engine","KustoMain_2018.04.29.5"]]}]} -------------------------------------------------------------------------------- /azure-kusto-data/tests/input/zero_results.json: -------------------------------------------------------------------------------- 1 | [{ 2 | "FrameType": "DataSetHeader", 3 | "IsProgressive": false, 4 | "Version": "v2.0" 5 | }, 6 | { 7 | "FrameType": "DataTable", 8 | "TableId": 0, 9 | "TableKind": "QueryProperties", 10 | "TableName": "@ExtendedProperties", 11 | "Columns": [{ 12 | "ColumnName": "TableId", 13 | "ColumnType": "int" 14 | }, 15 | { 16 | "ColumnName": "Key", 17 | "ColumnType": "string" 18 | }, 19 | { 20 | "ColumnName": "Value", 21 | "ColumnType": "dynamic" 22 | }], 23 | "Rows": [[1, 24 | "Visualization", 25 | "{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"AnomalyColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null,\"Ymin\":\"NaN\",\"Ymax\":\"NaN\"}"]] 26 | }, 27 | { 28 | "FrameType": "DataTable", 29 | "TableId": 1, 30 | "TableKind": "PrimaryResult", 31 | "TableName": "PrimaryResult", 32 | "Columns": [{ 33 | "ColumnName": "print_0", 34 | "ColumnType": "string" 35 | }], 36 | "Rows": [] 37 | }, 38 | { 39 | "FrameType": "DataTable", 40 | "TableId": 2, 41 | "TableKind": "QueryCompletionInformation", 42 | "TableName": "QueryCompletionInformation", 43 | "Columns": [{ 44 | "ColumnName": "Timestamp", 45 | "ColumnType": "datetime" 46 | }, 47 | { 48 | "ColumnName": "ClientRequestId", 49 | "ColumnType": "string" 50 | }, 51 | { 52 | "ColumnName": "ActivityId", 53 | "ColumnType": "guid" 54 | }, 55 | { 56 | "ColumnName": "SubActivityId", 57 | "ColumnType": "guid" 58 | }, 59 | { 60 | "ColumnName": "ParentActivityId", 61 | "ColumnType": "guid" 62 | }, 63 | { 64 | "ColumnName": "Level", 65 | "ColumnType": "int" 66 | }, 67 | { 68 | "ColumnName": "LevelName", 69 | "ColumnType": "string" 70 | }, 71 | { 72 | "ColumnName": "StatusCode", 73 | "ColumnType": "int" 74 | }, 75 | { 76 | "ColumnName": "StatusCodeName", 77 | "ColumnType": "string" 78 | }, 79 | { 80 | "ColumnName": "EventType", 81 | "ColumnType": "int" 82 | }, 83 | { 84 | "ColumnName": "EventTypeName", 85 | "ColumnType": "string" 86 | }, 87 | { 88 | "ColumnName": "Payload", 89 | "ColumnType": "string" 90 | }], 91 | "Rows": [["2019-02-12T10:23:02.0413963Z", 92 | "KPC.execute;57050c90-8a7d-4b29-b8d0-a40688a8185c", 93 | "dfbaa865-e29d-46e0-af17-be22c6c113ac", 94 | "e2bf7a6c-adf1-48da-b117-667a92874d38", 95 | "81409d63-718b-4d06-9711-7eab476b7ceb", 96 | 4, 97 | "Info", 98 | 0, 99 | "S_OK (0)", 100 | 4, 101 | "QueryInfo", 102 | "{\"Count\":1,\"Text\":\"Querycompletedsuccessfully\"}"], 103 | ["2019-02-12T10:23:02.0413963Z", 104 | "KPC.execute;57050c90-8a7d-4b29-b8d0-a40688a8185c", 105 | "dfbaa865-e29d-46e0-af17-be22c6c113ac", 106 | "e2bf7a6c-adf1-48da-b117-667a92874d38", 107 | "81409d63-718b-4d06-9711-7eab476b7ceb", 108 | 6, 109 | "Stats", 110 | 0, 111 | "S_OK (0)", 112 | 0, 113 | "QueryResourceConsumption", 114 | "{\"ExecutionTime\":0.0156475,\"resource_usage\":{\"cache\":{\"memory\":{\"hits\":0,\"misses\":0,\"total\":0},\"disk\":{\"hits\":0,\"misses\":0,\"total\":0},\"shards\":{\"hitbytes\":0,\"missbytes\":0,\"bypassbytes\":0}},\"cpu\":{\"user\":\"00: 00: 00\",\"kernel\":\"00: 00: 00\",\"totalcpu\":\"00: 00: 00\"},\"memory\":{\"peak_per_node\":0}},\"input_dataset_statistics\":{\"extents\":{\"total\":0,\"scanned\":0},\"rows\":{\"total\":0,\"scanned\":0},\"rowstores\":{\"scanned_rows\":0,\"scanned_values_size\":0}},\"dataset_statistics\":[{\"table_row_count\":0,\"table_size\":0}]}"]] 115 | }, 116 | { 117 | "FrameType": "DataSetCompletion", 118 | "HasErrors": false, 119 | "Cancelled": false 120 | }] -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_cloud_settings.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import pytest 4 | 5 | from azure.kusto.data._cloud_settings import CloudSettings, CloudInfo 6 | 7 | 8 | @pytest.fixture 9 | def clear_cache(): 10 | """Fixture to clear the CloudSettings cache before each test""" 11 | with CloudSettings._cloud_cache_lock: 12 | CloudSettings._cloud_cache.clear() 13 | yield 14 | # Clean up after test if needed 15 | with CloudSettings._cloud_cache_lock: 16 | CloudSettings._cloud_cache.clear() 17 | 18 | 19 | def test_normalize_uri_extracts_authority(): 20 | """Test that _normalize_uri extracts only the authority part (schema, host, port) from a URI.""" 21 | # Test with various URI formats 22 | test_cases = [ 23 | ("https://cluster.kusto.windows.net", "https://cluster.kusto.windows.net"), 24 | ("https://cluster.kusto.windows.net/", "https://cluster.kusto.windows.net"), 25 | ("https://cluster.kusto.windows.net/v1/rest", "https://cluster.kusto.windows.net"), 26 | ("https://cluster.kusto.windows.net:443/v1/rest", "https://cluster.kusto.windows.net:443"), 27 | ("http://localhost:8080/v1/rest/query", "http://localhost:8080"), 28 | ("https://cluster.kusto.windows.net/database", "https://cluster.kusto.windows.net"), 29 | ] 30 | 31 | for input_uri, expected_authority in test_cases: 32 | assert CloudSettings._normalize_uri(input_uri) == expected_authority 33 | 34 | 35 | def test_cloud_info_cached_by_authority(clear_cache): 36 | """Test that CloudInfo is cached by authority part of the URI (schema, host, port).""" 37 | # Create a test CloudInfo object 38 | test_cloud_info = CloudInfo( 39 | login_endpoint="https://login.test.com", 40 | login_mfa_required=False, 41 | kusto_client_app_id="test-app-id", 42 | kusto_client_redirect_uri="http://localhost/redirect", 43 | kusto_service_resource_id="https://test.kusto.windows.net", 44 | first_party_authority_url="https://login.test.com/tenant-id", 45 | ) 46 | 47 | # Add to cache with a specific URL 48 | base_url = "https://cluster.kusto.windows.net" 49 | CloudSettings.add_to_cache(base_url, test_cloud_info) 50 | 51 | # Test that it can be retrieved with different path variations but same authority 52 | variations = [ 53 | base_url + "/", 54 | base_url + "/database", 55 | base_url + "/v1/rest/query", 56 | base_url + "/some/other/path", 57 | ] 58 | 59 | for url in variations: 60 | # Use the internal _normalize_uri to get the cache key 61 | normalized_url = CloudSettings._normalize_uri(url) 62 | assert normalized_url == "https://cluster.kusto.windows.net" 63 | assert normalized_url in CloudSettings._cloud_cache 64 | 65 | # Verify the retrieved CloudInfo is the same instance 66 | retrieved_info = CloudSettings._cloud_cache[normalized_url] 67 | assert retrieved_info is test_cloud_info 68 | 69 | 70 | def test_cloud_info_cached_with_port(clear_cache): 71 | """Test that URIs with ports are cached separately from those without.""" 72 | # Create two different CloudInfo objects 73 | cloud_info_default = CloudInfo( 74 | login_endpoint="https://login.default.com", 75 | login_mfa_required=False, 76 | kusto_client_app_id="default-app-id", 77 | kusto_client_redirect_uri="http://localhost/redirect", 78 | kusto_service_resource_id="https://default.kusto.windows.net", 79 | first_party_authority_url="https://login.default.com/tenant-id", 80 | ) 81 | 82 | cloud_info_with_port = CloudInfo( 83 | login_endpoint="https://login.withport.com", 84 | login_mfa_required=True, 85 | kusto_client_app_id="port-app-id", 86 | kusto_client_redirect_uri="http://localhost/redirect", 87 | kusto_service_resource_id="https://port.kusto.windows.net", 88 | first_party_authority_url="https://login.withport.com/tenant-id", 89 | ) 90 | 91 | # Add both to cache with different authorities 92 | CloudSettings.add_to_cache("https://cluster.kusto.windows.net", cloud_info_default) 93 | CloudSettings.add_to_cache("https://cluster.kusto.windows.net:443", cloud_info_with_port) 94 | 95 | # Verify they are cached separately 96 | assert "https://cluster.kusto.windows.net" in CloudSettings._cloud_cache 97 | assert "https://cluster.kusto.windows.net:443" in CloudSettings._cloud_cache 98 | 99 | # Verify each URI gets the correct CloudInfo 100 | assert CloudSettings._cloud_cache["https://cluster.kusto.windows.net"] is cloud_info_default 101 | assert CloudSettings._cloud_cache["https://cluster.kusto.windows.net:443"] is cloud_info_with_port 102 | 103 | # Additional verification with variations 104 | assert CloudSettings._cloud_cache[CloudSettings._normalize_uri("https://cluster.kusto.windows.net/database")] is cloud_info_default 105 | assert CloudSettings._cloud_cache[CloudSettings._normalize_uri("https://cluster.kusto.windows.net:443/database")] is cloud_info_with_port 106 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_converter.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import unittest 4 | from datetime import timedelta 5 | 6 | from azure.kusto.data._converters import to_datetime, to_timedelta 7 | 8 | 9 | class ConverterTests(unittest.TestCase): 10 | """These are unit tests that should test custom converters used in.""" 11 | 12 | def test_to_timestamp(self): 13 | """Happy path to test converter from TimeSpan to timedelta.""" 14 | # Test hours, minutes and seconds 15 | assert to_timedelta("00:00:00") == timedelta(seconds=0) 16 | assert to_timedelta("00:00:03") == timedelta(seconds=3) 17 | assert to_timedelta("00:04:03") == timedelta(minutes=4, seconds=3) 18 | assert to_timedelta("02:04:03") == timedelta(hours=2, minutes=4, seconds=3) 19 | # Test milliseconds 20 | assert to_timedelta("00:00:00.099") == timedelta(milliseconds=99) 21 | assert to_timedelta("02:04:03.0123") == timedelta(hours=2, minutes=4, seconds=3, microseconds=12300) 22 | # Test days 23 | assert to_timedelta("01.00:00:00") == timedelta(days=1) 24 | assert to_timedelta("02.04:05:07") == timedelta(days=2, hours=4, minutes=5, seconds=7) 25 | # Test negative 26 | assert to_timedelta("-01.00:00:00") == -timedelta(days=1) 27 | assert to_timedelta("-02.04:05:07") == -timedelta(days=2, hours=4, minutes=5, seconds=7) 28 | # Test all together 29 | assert to_timedelta("00.00:00:00.000") == timedelta(seconds=0) 30 | assert to_timedelta("02.04:05:07.789") == timedelta(days=2, hours=4, minutes=5, seconds=7, milliseconds=789) 31 | assert to_timedelta("03.00:00:00.111") == timedelta(days=3, milliseconds=111) 32 | # Test from Ticks 33 | assert to_timedelta(-80080008) == timedelta(microseconds=-8008001) 34 | assert to_timedelta(10010001) == timedelta(microseconds=1001000) 35 | 36 | def test_to_timestamp_fail(self): 37 | """ 38 | Sad path to test TimeSpan to timedelta converter 39 | """ 40 | self.assertRaises(ValueError, to_timedelta, "") 41 | self.assertRaises(ValueError, to_timedelta, "foo") 42 | self.assertRaises(ValueError, to_timedelta, "00") 43 | self.assertRaises(ValueError, to_timedelta, "00:00") 44 | self.assertRaises(ValueError, to_timedelta, "03.00:00:00.") 45 | self.assertRaises(ValueError, to_timedelta, "03.00:00:00.111a") 46 | 47 | def test_to_datetime(self): 48 | """Tests datetime read by KustoResultIter""" 49 | assert to_datetime("2016-06-07T16:00:00Z") is not None 50 | 51 | def test_to_datetime_fail(self): 52 | """Tests that invalid strings fails to convert to datetime""" 53 | self.assertRaises(ValueError, to_datetime, "invalid") 54 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_exceptions.py: -------------------------------------------------------------------------------- 1 | from azure.kusto.data.exceptions import OneApiError 2 | 3 | 4 | def test_parse_one_api_error(): 5 | result = OneApiError.from_dict( 6 | { 7 | "code": "LimitsExceeded", 8 | "message": "Request is invalid and cannot be executed.", 9 | "@type": "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", 10 | "@message": "Query execution has exceeded the allowed limits (80DA0003): .", 11 | "@context": { 12 | "timestamp": "2018-12-10T15:10:48.8352222Z", 13 | "machineName": "RD0003FFBEDEB9", 14 | "processName": "Kusto.Azure.Svc", 15 | "processId": 4328, 16 | "threadId": 7284, 17 | "appDomainName": "RdRuntime", 18 | "clientRequestId": "KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3", 19 | "activityId": "a57ec272-8846-49e6-b458-460b841ed47d", 20 | "subActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 21 | "activityType": "PO-OWIN-CallContext", 22 | "parentActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 23 | "activityStack": "(Activity stack: CRID=KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3 ARID=a57ec272-8846-49e6-b458-460b841ed47d > PO-OWIN-CallContext/a57ec272-8846-49e6-b458-460b841ed47d)", 24 | }, 25 | "@permanent": False, 26 | } 27 | ) 28 | 29 | assert result.code == "LimitsExceeded" 30 | assert result.type == "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException" 31 | assert result.message == "Request is invalid and cannot be executed." 32 | assert result.description == "Query execution has exceeded the allowed limits (80DA0003): ." 33 | assert not result.permanent 34 | assert result.context["timestamp"] == "2018-12-10T15:10:48.8352222Z" 35 | assert result.context["machineName"] == "RD0003FFBEDEB9" 36 | assert result.context["processName"] == "Kusto.Azure.Svc" 37 | assert result.context["processId"] == 4328 38 | assert result.context["threadId"] == 7284 39 | assert result.context["appDomainName"] == "RdRuntime" 40 | assert result.context["clientRequestId"] == "KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3" 41 | assert result.context["activityId"] == "a57ec272-8846-49e6-b458-460b841ed47d" 42 | assert result.context["subActivityId"] == "a57ec272-8846-49e6-b458-460b841ed47d" 43 | assert result.context["activityType"] == "PO-OWIN-CallContext" 44 | assert result.context["parentActivityId"] == "a57ec272-8846-49e6-b458-460b841ed47d" 45 | assert ( 46 | result.context["activityStack"] == "(Activity stack: CRID=KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3 ARID=a57ec272-8846-49e6-b458-460b841ed47d " 47 | "> PO-OWIN-CallContext/a57ec272-8846-49e6-b458-460b841ed47d)" 48 | ) 49 | 50 | 51 | def test_one_api_error_no_type_success(): 52 | result = OneApiError.from_dict( 53 | { 54 | "code": "LimitsExceeded", 55 | "message": "Request is invalid and cannot be executed.", 56 | "@message": "Query execution has exceeded the allowed limits (80DA0003): .", 57 | "@context": { 58 | "timestamp": "2018-12-10T15:10:48.8352222Z", 59 | "machineName": "RD0003FFBEDEB9", 60 | "processName": "Kusto.Azure.Svc", 61 | "processId": 4328, 62 | "threadId": 7284, 63 | "appDomainName": "RdRuntime", 64 | "clientRequestId": "KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3", 65 | "activityId": "a57ec272-8846-49e6-b458-460b841ed47d", 66 | "subActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 67 | "activityType": "PO-OWIN-CallContext", 68 | "parentActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 69 | "activityStack": "(Activity stack: CRID=KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3 ARID=a57ec272-8846-49e6-b458-460b841ed47d > PO-OWIN-CallContext/a57ec272-8846-49e6-b458-460b841ed47d)", 70 | }, 71 | } 72 | ) 73 | 74 | assert result.code == "LimitsExceeded" 75 | assert not result.permanent 76 | 77 | 78 | def test_one_api_error_no_code_fail(): 79 | result = OneApiError.from_dict( 80 | { 81 | "message": "Request is invalid and cannot be executed.", 82 | "@message": "Query execution has exceeded the allowed limits (80DA0003): .", 83 | "@context": { 84 | "timestamp": "2018-12-10T15:10:48.8352222Z", 85 | "machineName": "RD0003FFBEDEB9", 86 | "processName": "Kusto.Azure.Svc", 87 | "processId": 4328, 88 | "threadId": 7284, 89 | "appDomainName": "RdRuntime", 90 | "clientRequestId": "KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3", 91 | "activityId": "a57ec272-8846-49e6-b458-460b841ed47d", 92 | "subActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 93 | "activityType": "PO-OWIN-CallContext", 94 | "parentActivityId": "a57ec272-8846-49e6-b458-460b841ed47d", 95 | "activityStack": "(Activity stack: CRID=KPC.execute;d3a43e37-0d7f-47a9-b6cd-a889b2aee3d3 ARID=a57ec272-8846-49e6-b458-460b841ed47d > PO-OWIN-CallContext/a57ec272-8846-49e6-b458-460b841ed47d)", 96 | }, 97 | "@permanent": False, 98 | } 99 | ) 100 | 101 | assert result.code == "FailedToParse" 102 | assert result.type == "FailedToParseOneApiError" 103 | assert not result.permanent 104 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_helpers.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import datetime 4 | import json 5 | import os 6 | 7 | import pytest 8 | 9 | from azure.kusto.data._models import KustoResultTable 10 | from azure.kusto.data.helpers import dataframe_from_result_table 11 | from azure.kusto.data.response import KustoResponseDataSetV2 12 | import pandas 13 | import numpy 14 | 15 | 16 | def test_dataframe_from_result_table(): 17 | """Test conversion of KustoResultTable to pandas.DataFrame, including fixes for certain column types""" 18 | 19 | 20 | with open(os.path.join(os.path.dirname(__file__), "input", "dataframe.json"), "r") as response_file: 21 | data = response_file.read() 22 | 23 | response = KustoResponseDataSetV2(json.loads(data)) 24 | # Test when given both types of dictionary parameters that type conversion doesn't override column name conversion 25 | test_dict_by_name = { 26 | "RecordName": lambda col, frame: frame[col].astype("str"), 27 | "RecordInt64": lambda col, frame: frame[col].astype("int64"), 28 | "MissingType": lambda col, frame: frame[col].astype("str"), 29 | } 30 | test_dict_by_type = {"int": lambda col, frame: frame[col].astype("int32")} 31 | df = dataframe_from_result_table(response.primary_results[0], converters_by_type=test_dict_by_type, converters_by_column_name=test_dict_by_name) 32 | 33 | if hasattr(pandas, "StringDType"): 34 | assert df["RecordName"].dtype == pandas.StringDtype() 35 | assert str(df.iloc[0].RecordName) == "now" 36 | assert df["RecordGUID"].dtype == pandas.StringDtype() 37 | assert str(df.iloc[0].RecordGUID) == "6f3c1072-2739-461c-8aa7-3cfc8ff528a8" 38 | assert df["RecordDynamic"].dtype == pandas.StringDtype() 39 | assert ( 40 | str(df.iloc[0].RecordDynamic) 41 | == '{"Visualization":null,"Title":null,"XColumn":null,"Series":null,"YColumns":null,"XTitle":null,"YTitle":null,"XAxis":null,"YAxis":null,"Legend":null,"YSplit":null,"Accumulate":false,"IsQuerySorted":false,"Kind":null}' 42 | ) 43 | else: 44 | assert df.iloc[0].RecordName == "now" 45 | assert df.iloc[0].RecordGUID == "6f3c1072-2739-461c-8aa7-3cfc8ff528a8" 46 | 47 | assert type(df.iloc[0].RecordTime) is pandas._libs.tslibs.timestamps.Timestamp 48 | 49 | for k, v in {"year": 2021, "month": 12, "day": 22, "hour": 11, "minute": 43, "second": 00}.items(): 50 | assert getattr(df.iloc[0].RecordTime, k) == v 51 | assert type(df.iloc[0].RecordBool) is numpy.bool_ 52 | assert df.iloc[0].RecordBool == True 53 | assert type(df.iloc[0].RecordInt) is numpy.int32 54 | assert df.iloc[0].RecordInt == 5678 55 | assert type(df.iloc[0].RecordInt64) is numpy.int64 56 | assert df.iloc[0].RecordInt64 == 222 57 | assert type(df.iloc[0].RecordLong) is numpy.int64 58 | assert df.iloc[0].RecordLong == 92233720368 59 | assert type(df.iloc[0].RecordReal) is numpy.float64 60 | assert df.iloc[0].RecordReal == 3.14159 61 | assert type(df.iloc[0].RecordDouble) is numpy.float64 62 | assert df.iloc[0].RecordDouble == 7.89 63 | assert type(df.iloc[0].RecordDecimal) is numpy.float64 64 | assert df.iloc[0].RecordDecimal == 1.2 65 | 66 | # Kusto datetime(0000-01-01T00:00:00Z), which Pandas can't represent. 67 | assert df.iloc[1].RecordName == "earliest datetime" 68 | assert type(df.iloc[1].RecordTime) is pandas._libs.tslibs.nattype.NaTType 69 | assert pandas.isnull(df.iloc[1].RecordReal) 70 | 71 | # Kusto datetime(9999-12-31T23:59:59Z), which Pandas can't represent. 72 | assert df.iloc[2].RecordName == "latest datetime" 73 | assert type(df.iloc[2].RecordTime) is pandas._libs.tslibs.nattype.NaTType 74 | assert type(df.iloc[2].RecordReal) is numpy.float64 75 | assert df.iloc[2].RecordReal == numpy.inf 76 | 77 | # Pandas earliest datetime 78 | assert df.iloc[3].RecordName == "earliest pandas datetime" 79 | assert type(df.iloc[3].RecordTime) is pandas._libs.tslibs.timestamps.Timestamp 80 | assert type(df.iloc[3].RecordReal) is numpy.float64 81 | assert df.iloc[3].RecordReal == -numpy.inf 82 | 83 | # Pandas latest datetime 84 | assert df.iloc[4].RecordName == "latest pandas datetime" 85 | assert type(df.iloc[4].RecordTime) is pandas._libs.tslibs.timestamps.Timestamp 86 | 87 | # Kusto 600000000 ticks timedelta 88 | assert df.iloc[5].RecordName == "timedelta ticks" 89 | assert type(df.iloc[5].RecordTime) is pandas._libs.tslibs.timestamps.Timestamp 90 | assert type(df.iloc[5].RecordOffset) is pandas._libs.tslibs.timestamps.Timedelta 91 | assert df.iloc[5].RecordOffset == pandas.to_timedelta("00:01:00") 92 | 93 | # Kusto timedelta(1.01:01:01.0) == 94 | assert df.iloc[6].RecordName == "timedelta string" 95 | assert type(df.iloc[6].RecordTime) is pandas._libs.tslibs.timestamps.Timestamp 96 | assert type(df.iloc[6].RecordOffset) is pandas._libs.tslibs.timestamps.Timedelta 97 | assert df.iloc[6].RecordOffset == pandas.to_timedelta("1 days 01:01:01") 98 | 99 | # Testing int to float conversion 100 | test_int_to_float = {"int": "float64"} 101 | ignore_missing_type = { 102 | "MissingType": lambda col, frame: frame[col].astype("str"), 103 | } 104 | df_int_to_float = dataframe_from_result_table(response.primary_results[0], converters_by_type=test_int_to_float, converters_by_column_name=ignore_missing_type) 105 | assert type(df_int_to_float.iloc[0].RecordInt) is numpy.float64 106 | assert df.iloc[0].RecordInt == 5678 107 | 108 | # Testing missing type conversion 109 | with pytest.raises(Exception): 110 | df_missing_type = dataframe_from_result_table(response.primary_results[0]) 111 | 112 | 113 | def test_pandas_mixed_date(): 114 | df = dataframe_from_result_table( 115 | KustoResultTable( 116 | { 117 | "TableName": "Table_0", 118 | "Columns": [ 119 | {"ColumnName": "Date", "ColumnType": "datetime"}, 120 | ], 121 | "Rows": [ 122 | ["2023-12-12T01:59:59.352Z"], 123 | ["2023-12-12T01:54:44Z"], 124 | ], 125 | } 126 | ) 127 | ) 128 | 129 | assert df["Date"][0] == pandas.Timestamp(year=2023, month=12, day=12, hour=1, minute=59, second=59, microsecond=352000, tzinfo=datetime.timezone.utc) 130 | assert df["Date"][1] == pandas.Timestamp(year=2023, month=12, day=12, hour=1, minute=54, second=44, tzinfo=datetime.timezone.utc) 131 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_http_adapter_with_socket_options.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import pickle 4 | import socket 5 | 6 | from azure.kusto.data.client import HTTPAdapterWithSocketOptions 7 | 8 | 9 | def test_pickle(): 10 | socket_options = [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)] 11 | original = HTTPAdapterWithSocketOptions(socket_options=socket_options) 12 | unpickled = pickle.loads(pickle.dumps(original)) 13 | assert unpickled.socket_options == socket_options 14 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_models.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import json 4 | import os 5 | 6 | from azure.kusto.data._models import KustoResultTable 7 | 8 | 9 | def test_str_and_dates_smoke(): 10 | with open(os.path.join(os.path.dirname(__file__), "input", "deft.json"), "r") as f: 11 | data = f.read() 12 | json_table = json.loads(data)[2] 13 | 14 | result_table = KustoResultTable(json_table) 15 | assert len(str(result_table)) == 4537 16 | 17 | 18 | def test_to_dict_json(): 19 | with open(os.path.join(os.path.dirname(__file__), "input", "deft.json"), "r") as f: 20 | data = f.read() 21 | json_table = json.loads(data)[2] 22 | 23 | result_table = KustoResultTable(json_table) 24 | assert ( 25 | json.dumps(result_table.to_dict(), default=str) 26 | == """{"name": "Deft", "kind": "PrimaryResult", "data": [{"rownumber": null, "rowguid": "", "xdouble": null, "xfloat": null, "xbool": null, "xint16": null, "xint32": null, "xint64": null, "xuint8": null, "xuint16": null, "xuint32": null, "xuint64": null, "xdate": null, "xsmalltext": "", "xtext": "", "xnumberAsText": "", "xtime": null, "xtextWithNulls": "", "xdynamicWithNulls": ""}, {"rownumber": 0, "rowguid": "00000000-0000-0000-0001-020304050607", "xdouble": 0.0, "xfloat": 0.0, "xbool": false, "xint16": 0, "xint32": 0, "xint64": 0, "xuint8": 0, "xuint16": 0, "xuint32": 0, "xuint64": 0, "xdate": "2014-01-01 01:01:01+00:00", "xsmalltext": "Zero", "xtext": "Zero", "xnumberAsText": "0", "xtime": "0:00:00", "xtextWithNulls": "", "xdynamicWithNulls": ""}, {"rownumber": 1, "rowguid": "00000001-0000-0000-0001-020304050607", "xdouble": 1.0001, "xfloat": 1.01, "xbool": true, "xint16": 1, "xint32": 1, "xint64": 1, "xuint8": 1, "xuint16": 1, "xuint32": 1, "xuint64": 1, "xdate": "2015-01-01 01:01:01+00:00", "xsmalltext": "One", "xtext": "One", "xnumberAsText": "1", "xtime": "1 day, 0:00:01.001000", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 1, "arr": [0, 1]}}, {"rownumber": 2, "rowguid": "00000002-0000-0000-0001-020304050607", "xdouble": 2.0002, "xfloat": 2.02, "xbool": false, "xint16": 2, "xint32": 2, "xint64": 2, "xuint8": 2, "xuint16": 2, "xuint32": 2, "xuint64": 2, "xdate": "2016-01-01 01:01:01+00:00", "xsmalltext": "Two", "xtext": "Two", "xnumberAsText": "2", "xtime": "-3 days, 23:59:57.998000", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 2, "arr": [0, 2]}}, {"rownumber": 3, "rowguid": "00000003-0000-0000-0001-020304050607", "xdouble": 3.0003, "xfloat": 3.03, "xbool": true, "xint16": 3, "xint32": 3, "xint64": 3, "xuint8": 3, "xuint16": 3, "xuint32": 3, "xuint64": 3, "xdate": "2017-01-01 01:01:01+00:00", "xsmalltext": "Three", "xtext": "Three", "xnumberAsText": "3", "xtime": "3 days, 0:00:03.003000", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 3, "arr": [0, 3]}}, {"rownumber": 4, "rowguid": "00000004-0000-0000-0001-020304050607", "xdouble": 4.0004, "xfloat": 4.04, "xbool": false, "xint16": 4, "xint32": 4, "xint64": 4, "xuint8": 4, "xuint16": 4, "xuint32": 4, "xuint64": 4, "xdate": "2018-01-01 01:01:01+00:00", "xsmalltext": "Four", "xtext": "Four", "xnumberAsText": "4", "xtime": "-5 days, 23:59:55.996000", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 4, "arr": [0, 4]}}, {"rownumber": 5, "rowguid": "00000005-0000-0000-0001-020304050607", "xdouble": 5.0005, "xfloat": 5.05, "xbool": true, "xint16": 5, "xint32": 5, "xint64": 5, "xuint8": 5, "xuint16": 5, "xuint32": 5, "xuint64": 5, "xdate": "2019-01-01 01:01:01+00:00", "xsmalltext": "Five", "xtext": "Five", "xnumberAsText": "5", "xtime": "5 days, 0:00:05.005001", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 5, "arr": [0, 5]}}, {"rownumber": 6, "rowguid": "00000006-0000-0000-0001-020304050607", "xdouble": 6.0006, "xfloat": 6.06, "xbool": false, "xint16": 6, "xint32": 6, "xint64": 6, "xuint8": 6, "xuint16": 6, "xuint32": 6, "xuint64": 6, "xdate": "2020-01-01 01:01:01+00:00", "xsmalltext": "Six", "xtext": "Six", "xnumberAsText": "6", "xtime": "-7 days, 23:59:53.993999", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 6, "arr": [0, 6]}}, {"rownumber": 7, "rowguid": "00000007-0000-0000-0001-020304050607", "xdouble": 7.0007, "xfloat": 7.07, "xbool": true, "xint16": 7, "xint32": 7, "xint64": 7, "xuint8": 7, "xuint16": 7, "xuint32": 7, "xuint64": 7, "xdate": "2021-01-01 01:01:01+00:00", "xsmalltext": "Seven", "xtext": "Seven", "xnumberAsText": "7", "xtime": "7 days, 0:00:07.007001", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 7, "arr": [0, 7]}}, {"rownumber": 8, "rowguid": "00000008-0000-0000-0001-020304050607", "xdouble": 8.0008, "xfloat": 8.08, "xbool": false, "xint16": 8, "xint32": 8, "xint64": 8, "xuint8": 8, "xuint16": 8, "xuint32": 8, "xuint64": 8, "xdate": "2022-01-01 01:01:01+00:00", "xsmalltext": "Eight", "xtext": "Eight", "xnumberAsText": "8", "xtime": "-9 days, 23:59:51.991999", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 8, "arr": [0, 8]}}, {"rownumber": 9, "rowguid": "00000009-0000-0000-0001-020304050607", "xdouble": 9.0009, "xfloat": 9.09, "xbool": true, "xint16": 9, "xint32": 9, "xint64": 9, "xuint8": 9, "xuint16": 9, "xuint32": 9, "xuint64": 9, "xdate": "2023-01-01 01:01:01+00:00", "xsmalltext": "Nine", "xtext": "Nine", "xnumberAsText": "9", "xtime": "9 days, 0:00:09.009001", "xtextWithNulls": "", "xdynamicWithNulls": {"rowId": 9, "arr": [0, 9]}}]}""" 27 | ) 28 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_security.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import pytest 4 | 5 | from azure.kusto.data import KustoConnectionStringBuilder 6 | from azure.kusto.data._token_providers import * 7 | from azure.kusto.data.exceptions import KustoAuthenticationError 8 | from azure.kusto.data.security import _AadHelper 9 | 10 | KUSTO_TEST_URI = "https://thisclusterdoesnotexist.kusto.windows.net" 11 | TEST_INTERACTIVE_AUTH = False # User interaction required, enable this when running test manually 12 | 13 | CloudSettings.add_to_cache(KUSTO_TEST_URI, CloudSettings.DEFAULT_CLOUD) 14 | CloudSettings.add_to_cache("https://somecluster.kusto.windows.net", CloudSettings.DEFAULT_CLOUD) 15 | 16 | 17 | def test_unauthorized_exception(): 18 | """Test the exception thrown when authorization fails.""" 19 | cluster = "https://somecluster.kusto.windows.net" 20 | username = "username@microsoft.com" 21 | kcsb = KustoConnectionStringBuilder.with_aad_user_password_authentication(cluster, username, "StrongestPasswordEver", "authorityName") 22 | aad_helper = _AadHelper(kcsb, False) 23 | aad_helper.token_provider._init_resources() 24 | 25 | try: 26 | aad_helper.acquire_authorization_header() 27 | except KustoAuthenticationError as error: 28 | assert error.authentication_method == UserPassTokenProvider.name() 29 | assert error.authority == "https://login.microsoftonline.com/authorityName" 30 | assert error.kusto_cluster == cluster 31 | assert error.kwargs["username"] == username 32 | assert error.kwargs["client_id"] == CloudSettings.DEFAULT_CLOUD.kusto_client_app_id 33 | 34 | 35 | # TODO: remove this once we can control the timeout 36 | @pytest.mark.skip() 37 | def test_msi_auth(): 38 | """ 39 | * * * Note * * * 40 | Each connection test takes about 15-20 seconds which is the time it takes TCP to fail connecting to the nonexistent MSI endpoint 41 | The timeout option does not seem to affect this behavior. Could be it only affects the waiting time fora response in successful connections. 42 | Please be prudent in adding any future tests! 43 | """ 44 | client_guid = "kjhjk" 45 | object_guid = "87687687" 46 | res_guid = "kajsdghdijewhag" 47 | 48 | """ 49 | Use of object_id and msi_res_id is disabled pending support of azure-identity 50 | When version 1.4.1 is released and these parameters are supported enable the functionality and tests back 51 | """ 52 | kcsb = [ 53 | KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication(KUSTO_TEST_URI, timeout=1), 54 | KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication(KUSTO_TEST_URI, client_id=client_guid, timeout=1), 55 | # KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication(KUSTO_TEST_URI, object_id=object_guid, timeout=1), 56 | # KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication(KUSTO_TEST_URI, msi_res_id=res_guid, timeout=1), 57 | ] 58 | 59 | helpers = [_AadHelper(i, False) for i in kcsb] 60 | 61 | for h in helpers: 62 | h.token_provider._init_resources() 63 | 64 | try: 65 | helpers[0].acquire_authorization_header() 66 | except KustoAuthenticationError as e: 67 | assert e.authentication_method == MsiTokenProvider.name() 68 | assert "client_id" not in e.kwargs 69 | assert "object_id" not in e.kwargs 70 | assert "msi_res_id" not in e.kwargs 71 | 72 | try: 73 | helpers[1].acquire_authorization_header() 74 | except KustoAuthenticationError as e: 75 | assert e.authentication_method == MsiTokenProvider.name() 76 | assert e.kwargs["client_id"] == client_guid 77 | assert "object_id" not in e.kwargs 78 | assert "msi_res_id" not in e.kwargs 79 | assert str(e.exception).index("client_id") > -1 80 | assert str(e.exception).index(client_guid) > -1 81 | 82 | 83 | def test_token_provider_auth(): 84 | valid_token_provider = lambda: "caller token" 85 | invalid_token_provider = lambda: 12345678 86 | 87 | valid_kcsb = KustoConnectionStringBuilder.with_token_provider(KUSTO_TEST_URI, valid_token_provider) 88 | invalid_kcsb = KustoConnectionStringBuilder.with_token_provider(KUSTO_TEST_URI, invalid_token_provider) 89 | 90 | valid_helper = _AadHelper(valid_kcsb, False) 91 | valid_helper.token_provider._init_resources() 92 | invalid_helper = _AadHelper(invalid_kcsb, False) 93 | invalid_helper.token_provider._init_resources() 94 | 95 | auth_header = valid_helper.acquire_authorization_header() 96 | assert auth_header.index(valid_token_provider()) > -1 97 | 98 | try: 99 | invalid_helper.acquire_authorization_header() 100 | except KustoAuthenticationError as e: 101 | assert e.authentication_method == CallbackTokenProvider.name() 102 | assert str(e.exception).index(str(type(invalid_token_provider()))) > -1 103 | 104 | 105 | def test_user_app_token_auth(): 106 | token = "123456446" 107 | user_kcsb = KustoConnectionStringBuilder.with_aad_user_token_authentication(KUSTO_TEST_URI, token) 108 | app_kcsb = KustoConnectionStringBuilder.with_aad_application_token_authentication(KUSTO_TEST_URI, token) 109 | 110 | user_helper = _AadHelper(user_kcsb, False) 111 | app_helper = _AadHelper(app_kcsb, False) 112 | user_helper.token_provider._init_resources() 113 | app_helper.token_provider._init_resources() 114 | 115 | auth_header = user_helper.acquire_authorization_header() 116 | assert auth_header.index(token) > -1 117 | 118 | auth_header = app_helper.acquire_authorization_header() 119 | assert auth_header.index(token) > -1 120 | 121 | 122 | def test_interactive_login(): 123 | if not TEST_INTERACTIVE_AUTH: 124 | pytest.skip(" *** Skipped interactive login Test ***") 125 | 126 | kcsb = KustoConnectionStringBuilder.with_interactive_login(KUSTO_TEST_URI) 127 | aad_helper = _AadHelper(kcsb, False) 128 | 129 | # should prompt 130 | header = aad_helper.acquire_authorization_header() 131 | assert header is not None 132 | 133 | # should not prompt 134 | header = aad_helper.acquire_authorization_header() 135 | assert header is not None 136 | -------------------------------------------------------------------------------- /azure-kusto-data/tests/test_telemetry.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from azure.kusto.data._telemetry import MonitoredActivity, Span 4 | from azure.kusto.data.client_request_properties import ClientRequestProperties 5 | 6 | 7 | def test_run_none_invoker(): 8 | # Edge case test for invoke method with None invoker function 9 | with pytest.raises(TypeError): 10 | MonitoredActivity.invoke(None, "test_span") 11 | 12 | 13 | @pytest.mark.asyncio 14 | async def test_run_async_valid_invoker(): 15 | # Happy path test for invoke_async method with valid invoker function and name of span 16 | async def invoker(): 17 | return "Hello World" 18 | 19 | span = await MonitoredActivity.invoke_async(invoker, "test_span") 20 | assert span == "Hello World" 21 | 22 | 23 | def test_run_valid_invoker(): 24 | # Happy path test for invoke method with valid invoker function and name of span 25 | 26 | def invoker(): 27 | return "Hello World" 28 | 29 | span = MonitoredActivity.invoke(invoker, "test_span") 30 | assert span == "Hello World" 31 | 32 | 33 | @pytest.mark.asyncio 34 | async def test_run_async_none_invoker(): 35 | # Edge case test for invoke_async method with None invoker function 36 | with pytest.raises(TypeError): 37 | await MonitoredActivity.invoke_async(None, "test_span") 38 | 39 | 40 | def test_run_sync_behavior(): 41 | # General behavior test for invoke method running the span synchronously 42 | def invoker(): 43 | return "Hello World" 44 | 45 | span = MonitoredActivity.invoke(invoker, "test_span") 46 | assert span == "Hello World" 47 | 48 | 49 | @pytest.mark.asyncio 50 | async def test_run_async_behavior(): 51 | # General behavior test for invoke_async method running the span asynchronously 52 | async def invoker(): 53 | return "Hello World" 54 | 55 | span = await MonitoredActivity.invoke_async(invoker, "test_span") 56 | assert span == "Hello World" 57 | 58 | 59 | def test_tracing_attributes_parameter(): 60 | def invoker(): 61 | return "Hello World" 62 | 63 | tracing_attributes = {"key": "value"} 64 | result = MonitoredActivity.invoke(invoker, tracing_attributes=tracing_attributes) 65 | assert result == "Hello World" 66 | 67 | 68 | def test_get_client_request_properties_attributes(): 69 | attributes = ClientRequestProperties().get_tracing_attributes() 70 | keynames = {"client_request_id"} 71 | assert isinstance(attributes, dict) 72 | for key, val in attributes.items(): 73 | assert key in keynames 74 | assert isinstance(val, str) 75 | for key in keynames: 76 | assert key in attributes.keys() 77 | 78 | 79 | def test_create_query_attributes(): 80 | attributes = Span.create_query_attributes("cluster_test", "database_test", ClientRequestProperties()) 81 | keynames = {"kusto_cluster", "database", "client_request_id"} 82 | assert isinstance(attributes, dict) 83 | for key, val in attributes.items(): 84 | assert isinstance(val, str) 85 | for key in keynames: 86 | assert key in attributes.keys() 87 | attributes = Span.create_query_attributes("cluster_test", "database_test") 88 | keynames = {"kusto_cluster", "database"} 89 | assert isinstance(attributes, dict) 90 | for key, val in attributes.items(): 91 | assert isinstance(val, str) 92 | for key in keynames: 93 | assert key in attributes.keys() 94 | 95 | 96 | def test_create_ingest_attributes(): 97 | attributes = Span.create_streaming_ingest_attributes("cluster_test", "database_test", "table", ClientRequestProperties()) 98 | keynames = {"kusto_cluster", "database", "table", "client_request_id"} 99 | assert isinstance(attributes, dict) 100 | for key, val in attributes.items(): 101 | assert isinstance(val, str) 102 | for key in keynames: 103 | assert key in attributes.keys() 104 | attributes = Span.create_streaming_ingest_attributes("cluster_test", "database_test", "table") 105 | keynames = {"kusto_cluster", "database", "table"} 106 | assert isinstance(attributes, dict) 107 | for key, val in attributes.items(): 108 | assert isinstance(val, str) 109 | for key in keynames: 110 | assert key in attributes.keys() 111 | 112 | 113 | def test_create_http_attributes(): 114 | attributes = Span.create_http_attributes("method_test", "url_test") 115 | assert attributes == {"component": "http", "http.method": "method_test", "http.url": "url_test"} 116 | headers = {"User-Agent": "user_agent_test"} 117 | attributes = Span.create_http_attributes("method_test", "url_test", headers) 118 | assert attributes == {"component": "http", "http.method": "method_test", "http.url": "url_test", "http.user_agent": "user_agent_test"} 119 | -------------------------------------------------------------------------------- /azure-kusto-data/tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27,py35,py37 3 | 4 | [testenv] 5 | deps= 6 | pytest 7 | pandas 8 | commands = pytest 9 | -------------------------------------------------------------------------------- /azure-kusto-ingest/MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.rst 2 | recursive-exclude tests * 3 | -------------------------------------------------------------------------------- /azure-kusto-ingest/README.rst: -------------------------------------------------------------------------------- 1 | Microsoft Azure Kusto Ingest Library for Python 2 | =============================================== 3 | 4 | .. code-block:: python 5 | 6 | from azure.kusto.data import KustoConnectionStringBuilder, DataFormat 7 | from azure.kusto.ingest import QueuedIngestClient, IngestionProperties, FileDescriptor, BlobDescriptor 8 | 9 | ingestion_props = IngestionProperties(database="{database_name}", table="{table_name}", data_format=DataFormat.CSV) 10 | client = QueuedIngestClient(KustoConnectionStringBuilder.with_interactive_login("https://ingest-{cluster_name}.kusto.windows.net")) 11 | 12 | file_descriptor = FileDescriptor("{filename}.csv", 15360) # in this example, the raw (uncompressed) size of the data is 15KB (15360 bytes) 13 | client.ingest_from_file(file_descriptor, ingestion_properties=ingestion_props) 14 | client.ingest_from_file("{filename}.csv", ingestion_properties=ingestion_props) 15 | 16 | blob_descriptor = BlobDescriptor("https://{path_to_blob}.csv.gz?sas", 51200) # in this example, the raw (uncompressed) size of the data is 50KB (52100 bytes) 17 | client.ingest_from_blob(blob_descriptor, ingestion_properties=ingestion_props) 18 | 19 | 20 | Overview 21 | -------- 22 | 23 | *Kusto Python Ingest Client* Library provides the capability to ingest data into Kusto clusters using Python. 24 | It is Python 3.x compatible and supports data types through familiar Python DB API interface. 25 | 26 | It's possible to use the library, for instance, from `Jupyter Notebooks `_ which are attached to Spark clusters, 27 | including, but not exclusively, `Azure Databricks `_ instances. 28 | 29 | * `How to install the package `_. 30 | 31 | * `Data ingest sample `_. 32 | 33 | * `GitHub Repository `_. 34 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure-kusto-ingest.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 17 | 18 | 20 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | """Init file for azure namespace. https://github.com/Azure/azure-sdk-for-python/wiki/Azure-packaging""" 4 | __path__ = __import__("pkgutil").extend_path(__path__, __name__) 5 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | """Init file for azure namespace. https://github.com/Azure/azure-sdk-for-python/wiki/Azure-packaging""" 4 | __path__ = __import__("pkgutil").extend_path(__path__, __name__) 5 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | from ._version import VERSION as __version__ 4 | from .base_ingest_client import IngestionResult, IngestionStatus 5 | from .descriptors import BlobDescriptor, FileDescriptor, StreamDescriptor 6 | from .exceptions import KustoMissingMappingError, KustoMappingError, KustoQueueError, KustoDuplicateMappingError, KustoInvalidEndpointError, KustoClientError 7 | from .ingest_client import QueuedIngestClient 8 | from .ingestion_properties import ( 9 | ValidationPolicy, 10 | ValidationImplications, 11 | ValidationOptions, 12 | ReportLevel, 13 | ReportMethod, 14 | IngestionProperties, 15 | IngestionMappingKind, 16 | ColumnMapping, 17 | TransformationMethod, 18 | ) 19 | from .managed_streaming_ingest_client import ManagedStreamingIngestClient 20 | from .streaming_ingest_client import KustoStreamingIngestClient 21 | from .base_ingest_client import BaseIngestClient 22 | 23 | __all__ = [ 24 | "IngestionResult", 25 | "IngestionStatus", 26 | "BlobDescriptor", 27 | "FileDescriptor", 28 | "StreamDescriptor", 29 | "KustoMissingMappingError", 30 | "KustoMappingError", 31 | "KustoQueueError", 32 | "KustoDuplicateMappingError", 33 | "KustoInvalidEndpointError", 34 | "KustoClientError", 35 | "QueuedIngestClient", 36 | "ValidationPolicy", 37 | "ValidationImplications", 38 | "ValidationOptions", 39 | "ReportLevel", 40 | "ReportMethod", 41 | "IngestionProperties", 42 | "IngestionMappingKind", 43 | "ColumnMapping", 44 | "TransformationMethod", 45 | "ManagedStreamingIngestClient", 46 | "KustoStreamingIngestClient", 47 | "BaseIngestClient", 48 | ] 49 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/_ingest_telemetry.py: -------------------------------------------------------------------------------- 1 | import uuid 2 | 3 | from azure.kusto.data._telemetry import Span 4 | 5 | from .descriptors import DescriptorBase 6 | from .ingestion_properties import IngestionProperties 7 | 8 | 9 | class IngestTracingAttributes: 10 | """ 11 | Additional ADX attributes for telemetry spans 12 | """ 13 | 14 | _BLOB_QUEUE_NAME = "blob_queue_name" 15 | _SOURCE_ID = "source_id" 16 | 17 | @classmethod 18 | def set_ingest_descriptor_attributes(cls, descriptor: DescriptorBase, ingestion_properties: IngestionProperties) -> None: 19 | Span.add_attributes(tracing_attributes={**ingestion_properties.get_tracing_attributes(), **descriptor.get_tracing_attributes()}) 20 | 21 | @classmethod 22 | def create_enqueue_request_attributes(cls, queue_name: str, source_id: uuid.UUID) -> dict: 23 | enqueue_request_attributes = {cls._BLOB_QUEUE_NAME: queue_name, cls._SOURCE_ID: str(source_id)} 24 | return enqueue_request_attributes 25 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/_ranked_storage_account.py: -------------------------------------------------------------------------------- 1 | from typing import Callable 2 | 3 | 4 | class _StorageAccountStats: 5 | def __init__(self): 6 | self.success_count = 0 7 | self.total_count = 0 8 | 9 | def log_result(self, success: bool): 10 | self.total_count += 1 11 | if success: 12 | self.success_count += 1 13 | 14 | def reset(self): 15 | self.success_count = 0 16 | self.total_count = 0 17 | 18 | 19 | class _RankedStorageAccount: 20 | """_RankedStorageAccount is a class that represents a storage account with a rank. 21 | The rank is used to determine the order in which the storage accounts are used for ingestion. 22 | """ 23 | 24 | def __init__(self, account_name: str, number_of_buckets: int, bucket_duration: float, time_provider: Callable[[], float]): 25 | self.account_name = account_name 26 | self.number_of_buckets = number_of_buckets 27 | self.bucket_duration = bucket_duration 28 | self.time_provider = time_provider 29 | self.buckets = [_StorageAccountStats() for _ in range(number_of_buckets)] 30 | self.last_update_time = self.time_provider() 31 | self.current_bucket_index = 0 32 | 33 | def log_result(self, success: bool): 34 | self.current_bucket_index = self._adjust_for_time_passed() 35 | self.buckets[self.current_bucket_index].log_result(success) 36 | 37 | def get_account_name(self) -> str: 38 | return self.account_name 39 | 40 | def get_rank(self) -> float: 41 | rank = 0 42 | total_weight = 0 43 | 44 | # For each bucket, calculate the success rate ( success / total ) and multiply it by the bucket weight. 45 | # The older the bucket, the less weight it has. For example, if there are 3 buckets, the oldest bucket will have 46 | # a weight of 1, the middle bucket will have a weight of 2 and the newest bucket will have a weight of 3. 47 | 48 | for i in range(1, self.number_of_buckets + 1): 49 | bucket_index = (self.current_bucket_index + i) % self.number_of_buckets 50 | bucket = self.buckets[bucket_index] 51 | if bucket.total_count == 0: 52 | continue 53 | success_rate = bucket.success_count / bucket.total_count 54 | rank += success_rate * i 55 | total_weight += i 56 | 57 | # If there are no buckets with data, return 1 (highest rank) 58 | if total_weight == 0: 59 | return 1 60 | 61 | return rank / total_weight 62 | 63 | def _adjust_for_time_passed(self) -> int: 64 | # Get the current window (bucket) index and reset old windows. 65 | # This is part of the moving avarge calculation. 66 | current_time = self.time_provider() 67 | time_delta = current_time - self.last_update_time 68 | window_size = 0 69 | 70 | if time_delta >= self.bucket_duration: 71 | self.last_update_time = current_time 72 | window_size = min(int(time_delta / self.bucket_duration), self.number_of_buckets) 73 | for i in range(1, window_size + 1): 74 | index_to_reset = (self.current_bucket_index + i) % self.number_of_buckets 75 | self.buckets[index_to_reset].reset() 76 | 77 | return (self.current_bucket_index + window_size) % self.number_of_buckets 78 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/_status_q.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import random 4 | 5 | from typing import List, Callable, TYPE_CHECKING 6 | 7 | from azure.kusto.ingest._resource_manager import _ResourceUri 8 | from azure.storage.queue import QueueServiceClient, QueueClient, QueueMessage, TextBase64EncodePolicy, TextBase64DecodePolicy 9 | 10 | if TYPE_CHECKING: 11 | from azure.kusto.ingest.status import StatusMessage 12 | 13 | 14 | class QueueDetails: 15 | def __init__(self, name, service): 16 | self.name = name 17 | self.service = service 18 | 19 | def __str__(self): 20 | return "QueueDetails({0.name})".format(self) 21 | 22 | 23 | class StatusQueue: 24 | """StatusQueue is a class to simplify access to Kusto status queues (backed by azure storage queues).""" 25 | 26 | def __init__(self, get_queues_func: Callable[[], List[_ResourceUri]], message_cls): 27 | self.get_queues_func = get_queues_func 28 | self.message_cls = message_cls 29 | 30 | def _get_queues(self) -> List[QueueClient]: 31 | return [ 32 | QueueServiceClient(q.account_uri).get_queue_client(queue=q.object_name, message_decode_policy=TextBase64DecodePolicy()) 33 | for q in self.get_queues_func() 34 | ] 35 | 36 | def is_empty(self) -> bool: 37 | """Checks if Status queue has any messages""" 38 | return len(self.peek(1, raw=True)) == 0 39 | 40 | def _deserialize_message(self, m: QueueMessage) -> "StatusMessage": 41 | """Deserialize a message and return at as `message_cls` 42 | :param m: original message m. 43 | """ 44 | return self.message_cls(m.content) 45 | 46 | # TODO: current implementation takes a union top n / len(queues), which is not ideal, 47 | # because the user is not supposed to know that there can be multiple underlying queues 48 | def peek(self, n=1, raw=False) -> List["StatusMessage"]: 49 | """Peek status queue 50 | :param int n: number of messages to return as part of peek. 51 | :param bool raw: should message content be returned as is (no parsing). 52 | """ 53 | 54 | def _peek_specific_q(_q: QueueClient, _n: int) -> bool: 55 | has_messages = False 56 | for m in _q.peek_messages(max_messages=_n): 57 | if m: 58 | has_messages = True 59 | result.append(m if raw else self._deserialize_message(m)) 60 | 61 | # short circuit to prevent unneeded work 62 | if len(result) == n: 63 | return True 64 | return has_messages 65 | 66 | queues = self._get_queues() 67 | random.shuffle(queues) 68 | 69 | per_q = int(n / len(queues)) + 1 70 | 71 | result = [] 72 | 73 | non_empty_qs = [] 74 | 75 | for q in queues: 76 | if _peek_specific_q(q, per_q): 77 | non_empty_qs.append(q) 78 | 79 | if len(result) == n: 80 | return result 81 | 82 | # in-case queues aren't balanced, and we didn't get enough messages, iterate again and this time get all that we can 83 | for q in non_empty_qs: 84 | _peek_specific_q(q, n) 85 | if len(result) == n: 86 | return result 87 | 88 | # because we ask for n / len(qs) + 1, we might get more message then requests 89 | return result 90 | 91 | # TODO: current implementation takes a union top n / len(queues), which is not ideal, 92 | # because the user is not supposed to know that there can be multiple underlying queues 93 | def pop(self, n: int = 1, raw: bool = False, delete: bool = True) -> List["StatusMessage"]: 94 | """Pop status queue 95 | :param int n: number of messages to return as part of peek. 96 | :param bool raw: should message content be returned as is (no parsing). 97 | :param bool delete: should message be deleted after pop. default is True as this is expected of a q. 98 | """ 99 | 100 | def _pop_specific_q(_q: QueueClient, _n: int) -> bool: 101 | has_messages = False 102 | for m in _q.receive_messages(messages_per_page=_n): 103 | if m: 104 | has_messages = True 105 | result.append(m if raw else self._deserialize_message(m)) 106 | if delete: 107 | _q.delete_message(m.id, m.pop_receipt) 108 | 109 | # short circuit to prevent unneeded work 110 | if len(result) == n: 111 | return True 112 | return has_messages 113 | 114 | queues = self._get_queues() 115 | random.shuffle(queues) 116 | 117 | per_q = int(n / len(queues)) + 1 118 | 119 | result = [] 120 | 121 | non_empty_qs = [] 122 | 123 | for q in queues: 124 | if _pop_specific_q(q, per_q): 125 | non_empty_qs.append(q) 126 | 127 | if len(result) == n: 128 | return result 129 | 130 | # in-case queues aren't balanced, and we didn't get enough messages, iterate again and this time get all that we can 131 | for q in non_empty_qs: 132 | _pop_specific_q(q, n) 133 | if len(result) == n: 134 | return result 135 | 136 | # because we ask for n / len(qs) + 1, we might get more message then requests 137 | return result 138 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/_storage_account_set.py: -------------------------------------------------------------------------------- 1 | import random 2 | from typing import Callable, Dict, List, Tuple 3 | import time 4 | 5 | from azure.kusto.data.exceptions import KustoClientError 6 | from azure.kusto.ingest._ranked_storage_account import _RankedStorageAccount 7 | 8 | 9 | class _RankedStorageAccountSet: 10 | DEFAULT_NUMBER_OF_BUCKETS: int = 6 11 | DEFAULT_BUCKET_DURATION_IN_SECONDS: int = 10 12 | DEFAULT_TIERS: Tuple[int, int, int, int] = (90, 70, 30, 0) 13 | DEFAULT_TIME_PROVIDER_IN_SECONDS: Callable[[], float] = time.time 14 | 15 | def __init__( 16 | self, 17 | number_of_buckets: int = DEFAULT_NUMBER_OF_BUCKETS, 18 | bucket_duration: float = DEFAULT_BUCKET_DURATION_IN_SECONDS, 19 | tiers: Tuple[int, int, int, int] = DEFAULT_TIERS, 20 | time_provider: Callable[[], float] = DEFAULT_TIME_PROVIDER_IN_SECONDS, 21 | ): 22 | self.accounts: Dict[str, _RankedStorageAccount] = dict() 23 | self.number_of_buckets = number_of_buckets 24 | self.bucket_duration = bucket_duration 25 | self.tiers = tiers 26 | self.time_provider = time_provider 27 | 28 | def add_account_result(self, account_name: str, success: bool): 29 | if self.accounts.get(account_name) is None: 30 | raise KustoClientError(f"Account {account_name} does not exist in the set") 31 | self.accounts[account_name].log_result(success) 32 | 33 | def add_storage_account(self, account_name: str): 34 | if self.accounts.get(account_name) is None: 35 | self.accounts[account_name] = _RankedStorageAccount(account_name, self.number_of_buckets, self.bucket_duration, self.time_provider) 36 | 37 | def get_storage_account(self, account_name: str) -> _RankedStorageAccount: 38 | return self.accounts.get(account_name) 39 | 40 | def get_ranked_shuffled_accounts(self) -> List[_RankedStorageAccount]: 41 | accounts_by_tier: List[List[_RankedStorageAccount]] = [[] for _ in range(len(self.tiers))] 42 | 43 | for account in self.accounts.values(): 44 | rank_percentage = account.get_rank() * 100.0 45 | for i in range(len(self.tiers)): 46 | if rank_percentage >= self.tiers[i]: 47 | accounts_by_tier[i].append(account) 48 | break 49 | 50 | # Shuffle accounts in each tier 51 | for tier in accounts_by_tier: 52 | random.shuffle(tier) 53 | 54 | # Flatten the list 55 | return [item for sublist in accounts_by_tier for item in sublist] 56 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/_stream_extensions.py: -------------------------------------------------------------------------------- 1 | import io 2 | 3 | from typing import IO, AnyStr 4 | 5 | 6 | def read_until_size_or_end(stream: IO[AnyStr], size: int) -> io.BytesIO: 7 | pos = 0 8 | result = io.BytesIO() 9 | while True: 10 | try: 11 | returned = stream.read(size - pos) 12 | pos += len(returned) 13 | result.write(returned) 14 | 15 | if len(returned) == 0 or pos == size: 16 | result.seek(0, io.SEEK_SET) 17 | return result 18 | 19 | except BlockingIOError: 20 | continue 21 | 22 | 23 | class ChainStream(io.RawIOBase): 24 | """ 25 | https://stackoverflow.com/questions/24528278/stream-multiple-files-into-a-readable-object-in-python 26 | """ 27 | 28 | def __init__(self, streams): 29 | self.leftover = b"" 30 | self.stream_iter = iter(streams) 31 | try: 32 | self.stream = next(self.stream_iter) 33 | except StopIteration: 34 | self.stream = None 35 | 36 | def readable(self): 37 | return True 38 | 39 | def _read_next_chunk(self, max_length): 40 | # Return 0 or more bytes from the current stream, first returning all 41 | # leftover bytes. If the stream is closed returns b'' 42 | if self.leftover: 43 | return self.leftover 44 | elif self.stream is not None: 45 | return self.stream.read(max_length) 46 | else: 47 | return b"" 48 | 49 | def readinto(self, b): 50 | buffer_length = len(b) 51 | chunk = self._read_next_chunk(buffer_length) 52 | while len(chunk) == 0: 53 | # move to next stream 54 | if self.stream is not None: 55 | self.stream.close() 56 | try: 57 | self.stream = next(self.stream_iter) 58 | chunk = self._read_next_chunk(buffer_length) 59 | except StopIteration: 60 | # No more streams to chain together 61 | self.stream = None 62 | return 0 # indicate EOF 63 | output, self.leftover = chunk[:buffer_length], chunk[buffer_length:] 64 | b[: len(output)] = output 65 | return len(output) 66 | 67 | 68 | def chain_streams(streams, buffer_size=io.DEFAULT_BUFFER_SIZE): 69 | """ 70 | Chain an iterable of streams together into a single buffered stream. 71 | Usage: 72 | def generate_open_file_streams(): 73 | for file in filenames: 74 | yield open(file, 'rb') 75 | f = chain_streams(generate_open_file_streams()) 76 | f.read() 77 | """ 78 | return io.BufferedReader(ChainStream(streams), buffer_size=buffer_size) 79 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/_version.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | VERSION = "5.0.3" 4 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/exceptions.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | from azure.kusto.data.exceptions import KustoClientError 4 | 5 | 6 | class KustoMappingError(KustoClientError): 7 | """ 8 | Raised when the provided mapping arguments are invalid. 9 | """ 10 | 11 | 12 | class KustoDuplicateMappingError(KustoClientError): 13 | """ 14 | Raised when ingestion properties include both 15 | column mappings and a mapping reference 16 | """ 17 | 18 | def __init__(self): 19 | message = "Ingestion properties can't contain both an explicit mapping and a mapping reference." 20 | super(KustoDuplicateMappingError, self).__init__(message) 21 | 22 | 23 | class KustoMissingMappingError(KustoClientError): 24 | """ 25 | Raised when provided a mapping kind without a mapping reference or column mapping. 26 | """ 27 | 28 | 29 | class KustoInvalidEndpointError(KustoClientError): 30 | """Raised when trying to ingest to invalid cluster type.""" 31 | 32 | def __init__(self, expected_service_type, actual_service_type, suggested_endpoint_url=None): 33 | message = f"You are using '{expected_service_type}' client type, but the provided endpoint is of ServiceType '{actual_service_type}'. Initialize the client with the appropriate endpoint URI" 34 | if suggested_endpoint_url: 35 | message = message + ": '" + suggested_endpoint_url + "'" 36 | super(KustoInvalidEndpointError, self).__init__(message) 37 | 38 | 39 | class KustoQueueError(KustoClientError): 40 | 41 | """Raised when not succeeding to upload message to queue in all retries""" 42 | 43 | def __init__(self): 44 | message = "Failed to upload message to queues in all reties." 45 | super(KustoQueueError, self).__init__(message) 46 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/ingestion_blob_info.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import json 4 | import uuid 5 | from datetime import datetime 6 | from typing import TYPE_CHECKING 7 | 8 | if TYPE_CHECKING: 9 | from azure.kusto.ingest import BlobDescriptor, IngestionProperties 10 | 11 | 12 | class IngestionBlobInfo: 13 | def __init__( 14 | self, 15 | blob_descriptor: "BlobDescriptor", 16 | ingestion_properties: "IngestionProperties", 17 | auth_context=None, 18 | application_for_tracing=None, 19 | client_version_for_tracing=None, 20 | ): 21 | self.properties = dict() 22 | self.properties["BlobPath"] = blob_descriptor.path 23 | if blob_descriptor.size: 24 | self.properties["RawDataSize"] = blob_descriptor.size 25 | self.properties["DatabaseName"] = ingestion_properties.database 26 | self.properties["TableName"] = ingestion_properties.table 27 | self.properties["RetainBlobOnSuccess"] = True 28 | self.properties["FlushImmediately"] = ingestion_properties.flush_immediately 29 | self.properties["IgnoreSizeLimit"] = False 30 | self.properties["ReportLevel"] = ingestion_properties.report_level.value 31 | self.properties["ReportMethod"] = ingestion_properties.report_method.value 32 | self.properties["SourceMessageCreationTime"] = datetime.utcnow().isoformat() 33 | self.properties["Id"] = str(blob_descriptor.source_id) 34 | self.properties["ApplicationForTracing"] = application_for_tracing 35 | self.properties["ClientVersionForTracing"] = client_version_for_tracing 36 | 37 | additional_properties = ingestion_properties.additional_properties or {} 38 | additional_properties["authorizationContext"] = auth_context 39 | 40 | tags = [] 41 | if ingestion_properties.additional_tags: 42 | tags.extend(ingestion_properties.additional_tags) 43 | if ingestion_properties.drop_by_tags: 44 | tags.extend(["drop-by:" + drop for drop in ingestion_properties.drop_by_tags]) 45 | if ingestion_properties.ingest_by_tags: 46 | tags.extend(["ingest-by:" + ingest for ingest in ingestion_properties.ingest_by_tags]) 47 | if tags: 48 | additional_properties["tags"] = _convert_list_to_json(tags) 49 | if ingestion_properties.ingest_if_not_exists: 50 | additional_properties["ingestIfNotExists"] = _convert_list_to_json(ingestion_properties.ingest_if_not_exists) 51 | if ingestion_properties.ingestion_mapping: 52 | json_string = _convert_dict_to_json(ingestion_properties.ingestion_mapping) 53 | additional_properties["ingestionMapping"] = json_string 54 | 55 | if ingestion_properties.ingestion_mapping_reference: 56 | additional_properties["ingestionMappingReference"] = ingestion_properties.ingestion_mapping_reference 57 | if ingestion_properties.ingestion_mapping_type: 58 | additional_properties["ingestionMappingType"] = ingestion_properties.ingestion_mapping_type.value 59 | if ingestion_properties.validation_policy: 60 | additional_properties["ValidationPolicy"] = _convert_dict_to_json(ingestion_properties.validation_policy) 61 | if ingestion_properties.format: 62 | additional_properties["format"] = ingestion_properties.format.kusto_value 63 | if ingestion_properties.ignore_first_record: 64 | additional_properties["ignoreFirstRecord"] = ingestion_properties.ignore_first_record 65 | 66 | if additional_properties: 67 | self.properties["AdditionalProperties"] = additional_properties 68 | 69 | def to_json(self): 70 | """Converts this object to a json string""" 71 | return _convert_list_to_json(self.properties) 72 | 73 | 74 | def _convert_list_to_json(array): 75 | """Converts array to a json string""" 76 | return json.dumps(array, skipkeys=False, allow_nan=False, indent=None, separators=(",", ":")) 77 | 78 | 79 | def _convert_dict_to_json(array): 80 | """Converts array to a json string""" 81 | return json.dumps(array, skipkeys=False, allow_nan=False, indent=None, separators=(",", ":"), sort_keys=True, default=lambda o: o.__dict__) 82 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/py.typed: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-ingest/azure/kusto/ingest/py.typed -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/status.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import json 4 | 5 | from ._status_q import StatusQueue 6 | 7 | 8 | class StatusMessage: 9 | OperationId = None 10 | Database = None 11 | Table = None 12 | IngestionSourceId = None 13 | IngestionSourcePath = None 14 | RootActivityId = None 15 | 16 | _raw = None 17 | 18 | def __init__(self, s): 19 | self._raw = s 20 | 21 | o = json.loads(s) 22 | for key, value in o.items(): 23 | if hasattr(self, key): 24 | try: 25 | setattr(self, key, value) 26 | except: 27 | # TODO: should we set up a logger? 28 | pass 29 | 30 | def __str__(self): 31 | return "{}".format(self._raw) 32 | 33 | def __repr__(self): 34 | return "{0.__class__.__name__}({0._raw})".format(self) 35 | 36 | 37 | class SuccessMessage(StatusMessage): 38 | SucceededOn = None 39 | 40 | 41 | class FailureMessage(StatusMessage): 42 | FailedOn = None 43 | Details = None 44 | ErrorCode = None 45 | FailureStatus = None 46 | OriginatesFromUpdatePolicy = None 47 | ShouldRetry = None 48 | 49 | 50 | class KustoIngestStatusQueues: 51 | """Kusto ingest Status Queue. 52 | Use this class to get status messages from Kusto status queues. 53 | Currently there are two queues exposed: `failure` and `success` queues. 54 | """ 55 | 56 | def __init__(self, kusto_ingest_client): 57 | self.success = StatusQueue(kusto_ingest_client._resource_manager.get_successful_ingestions_queues, message_cls=SuccessMessage) 58 | self.failure = StatusQueue(kusto_ingest_client._resource_manager.get_failed_ingestions_queues, message_cls=FailureMessage) 59 | -------------------------------------------------------------------------------- /azure-kusto-ingest/azure/kusto/ingest/streaming_ingest_client.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | from typing import Union, AnyStr, Optional 4 | from typing import IO 5 | 6 | from azure.core.tracing.decorator import distributed_trace 7 | from azure.core.tracing import SpanKind 8 | 9 | from azure.kusto.data import KustoClient, KustoConnectionStringBuilder, ClientRequestProperties 10 | 11 | from ._ingest_telemetry import IngestTracingAttributes 12 | from .base_ingest_client import BaseIngestClient, IngestionResult, IngestionStatus 13 | from .descriptors import FileDescriptor, StreamDescriptor, BlobDescriptor 14 | from .ingestion_properties import IngestionProperties 15 | 16 | 17 | class KustoStreamingIngestClient(BaseIngestClient): 18 | """Kusto streaming ingest client for Python. 19 | KustoStreamingIngestClient works with both 2.x and 3.x flavors of Python. 20 | All primitive types are supported. 21 | Tests are run using pytest. 22 | """ 23 | 24 | def __init__(self, kcsb: Union[KustoConnectionStringBuilder, str], auto_correct_endpoint: bool = True): 25 | """Kusto Streaming Ingest Client constructor. 26 | :param KustoConnectionStringBuilder kcsb: The connection string to initialize KustoClient. 27 | """ 28 | super().__init__() 29 | 30 | if isinstance(kcsb, str): 31 | kcsb = KustoConnectionStringBuilder(kcsb) 32 | 33 | if auto_correct_endpoint: 34 | kcsb["Data Source"] = BaseIngestClient.get_query_endpoint(kcsb.data_source) 35 | self._kusto_client = KustoClient(kcsb) 36 | 37 | def close(self): 38 | if not self._is_closed: 39 | self._kusto_client.close() 40 | super().close() 41 | 42 | def set_proxy(self, proxy_url: str): 43 | self._kusto_client.set_proxy(proxy_url) 44 | 45 | @distributed_trace(kind=SpanKind.CLIENT) 46 | def ingest_from_file(self, file_descriptor: Union[FileDescriptor, str], ingestion_properties: IngestionProperties) -> IngestionResult: 47 | """Ingest from local files. 48 | :param file_descriptor: a FileDescriptor to be ingested. 49 | :param azure.kusto.ingest.IngestionProperties ingestion_properties: Ingestion properties. 50 | """ 51 | file_descriptor = FileDescriptor.get_instance(file_descriptor) 52 | IngestTracingAttributes.set_ingest_descriptor_attributes(file_descriptor, ingestion_properties) 53 | 54 | super().ingest_from_file(file_descriptor, ingestion_properties) 55 | 56 | stream_descriptor = StreamDescriptor.from_file_descriptor(file_descriptor) 57 | 58 | with stream_descriptor.stream: 59 | return self.ingest_from_stream(stream_descriptor, ingestion_properties) 60 | 61 | @distributed_trace(kind=SpanKind.CLIENT) 62 | def ingest_from_stream(self, stream_descriptor: Union[StreamDescriptor, IO[AnyStr]], ingestion_properties: IngestionProperties) -> IngestionResult: 63 | """Ingest from io streams. 64 | :param azure.kusto.ingest.StreamDescriptor stream_descriptor: An object that contains a description of the stream to 65 | be ingested. 66 | :param azure.kusto.ingest.IngestionProperties ingestion_properties: Ingestion properties. 67 | """ 68 | stream_descriptor = StreamDescriptor.get_instance(stream_descriptor) 69 | IngestTracingAttributes.set_ingest_descriptor_attributes(stream_descriptor, ingestion_properties) 70 | 71 | super().ingest_from_stream(stream_descriptor, ingestion_properties) 72 | 73 | return self._ingest_from_stream_with_client_request_id(stream_descriptor, ingestion_properties, None) 74 | 75 | def _ingest_from_stream_with_client_request_id( 76 | self, stream_descriptor: Union[StreamDescriptor, IO[AnyStr]], ingestion_properties: IngestionProperties, client_request_id: Optional[str] 77 | ) -> IngestionResult: 78 | stream_descriptor = BaseIngestClient._prepare_stream(stream_descriptor, ingestion_properties) 79 | additional_properties = None 80 | if client_request_id: 81 | additional_properties = ClientRequestProperties() 82 | additional_properties.client_request_id = client_request_id 83 | 84 | self._kusto_client.execute_streaming_ingest( 85 | ingestion_properties.database, 86 | ingestion_properties.table, 87 | stream_descriptor.stream, 88 | None, 89 | ingestion_properties.format.name, 90 | additional_properties, 91 | mapping_name=ingestion_properties.ingestion_mapping_reference, 92 | ) 93 | 94 | return IngestionResult(IngestionStatus.SUCCESS, ingestion_properties.database, ingestion_properties.table, stream_descriptor.source_id) 95 | 96 | def ingest_from_blob( 97 | self, blob_descriptor: BlobDescriptor, ingestion_properties: IngestionProperties, client_request_id: Optional[str] = None 98 | ) -> IngestionResult: 99 | IngestTracingAttributes.set_ingest_descriptor_attributes(blob_descriptor, ingestion_properties) 100 | additional_properties = None 101 | if client_request_id: 102 | additional_properties = ClientRequestProperties() 103 | additional_properties.client_request_id = client_request_id 104 | 105 | self._kusto_client.execute_streaming_ingest( 106 | ingestion_properties.database, 107 | ingestion_properties.table, 108 | None, 109 | blob_descriptor.path, 110 | ingestion_properties.format.name, 111 | additional_properties, 112 | mapping_name=ingestion_properties.ingestion_mapping_reference, 113 | ) 114 | return IngestionResult(IngestionStatus.SUCCESS, ingestion_properties.database, ingestion_properties.table, blob_descriptor.source_id) 115 | -------------------------------------------------------------------------------- /azure-kusto-ingest/setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal=1 3 | 4 | [flake8] 5 | ignore = E226,E302,E41 6 | max-line-length = 160 7 | exclude = tests/* 8 | max-complexity = 10 9 | 10 | [pylint] 11 | max-line-length = 160 12 | -------------------------------------------------------------------------------- /azure-kusto-ingest/setup.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | 4 | import re 5 | from os import path 6 | 7 | from setuptools import setup, find_packages 8 | 9 | PACKAGE_NAME = "azure-kusto-ingest" 10 | 11 | # a-b-c => a/b/c 12 | PACKAGE_FOLDER_PATH = PACKAGE_NAME.replace("-", path.sep) 13 | # a-b-c => a.b.c 14 | NAMESPACE_NAME = PACKAGE_NAME.replace("-", ".") 15 | 16 | with open(path.join(PACKAGE_FOLDER_PATH, "_version.py"), "r") as fd: 17 | VERSION = re.search(r'^VERSION\s*=\s*[\'"]([^\'"]*)[\'"]', fd.read(), re.MULTILINE).group(1) 18 | 19 | if not VERSION: 20 | raise RuntimeError("Cannot find version information") 21 | 22 | setup( 23 | name=PACKAGE_NAME, 24 | version=VERSION, 25 | description="Kusto Ingest Client", 26 | long_description_content_type="text/markdown", 27 | long_description=open("README.rst", "r").read(), 28 | license="MIT", 29 | author="Microsoft Corporation", 30 | author_email="kustalk@microsoft.com", 31 | url="https://github.com/Azure/azure-kusto-python", 32 | namespace_packages=["azure"], 33 | classifiers=[ 34 | # 5 - Production/Stable depends on multi-threading / aio / perf 35 | "Development Status :: 4 - Beta", 36 | "Programming Language :: Python", 37 | "Programming Language :: Python :: 3.8", 38 | "Programming Language :: Python :: 3.9", 39 | "Programming Language :: Python :: 3.10", 40 | "Programming Language :: Python :: 3.11", 41 | "Programming Language :: Python :: 3.12", 42 | "Programming Language :: Python :: 3.13", 43 | "License :: OSI Approved :: MIT License", 44 | ], 45 | packages=find_packages(exclude=["azure", "*tests*", "*tests.*"]), 46 | package_data={"": ["py.typed"]}, 47 | install_requires=[ 48 | "azure-kusto-data=={}".format(VERSION), 49 | # TODO - this has to be locked to this version due to https://github.com/Azure/azure-sdk-for-python/issues/40041 50 | "azure-storage-blob==12.23.0", 51 | "azure-storage-queue==12.12.0", 52 | "tenacity>=8.0.0", 53 | ], 54 | extras_require={"pandas": ["pandas"], "aio": []}, 55 | ) 56 | -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-ingest/tests/input/__init__.py -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/dataset.csv: -------------------------------------------------------------------------------- 1 | 0,00000000-0000-0000-0001-020304050607,0,0,0,0,0,0,0,0,0,0,2014-01-01T01:01:01.0000000Z,Zero,"Zero",0,00:00:00,,null 2 | 1,00000001-0000-0000-0001-020304050607,1.0001,1.01,1,1,1,1,1,1,1,1,2015-01-01T01:01:01.0000000Z,One,"One",1,00:00:01.0010001,,"{""rowId"": 1, ""arr"": [0,1]}" 3 | 2,00000002-0000-0000-0001-020304050607,2.0002,2.02,0,2,2,2,2,2,2,2,2016-01-01T01:01:01.0000000Z,Two,"Two",2,-00:00:02.0020002,,"{""rowId"": 2, ""arr"": [0,2]}" 4 | 3,00000003-0000-0000-0001-020304050607,3.0003,3.03,1,3,3,3,3,3,3,3,2017-01-01T01:01:01.0000000Z,Three,"Three",3,00:00:03.0030003,,"{""rowId"": 3, ""arr"": [0,3]}" 5 | 4,00000004-0000-0000-0001-020304050607,4.0004,4.04,0,4,4,4,4,4,4,4,2018-01-01T01:01:01.0000000Z,Four,"Four",4,-00:00:04.0040004,,"{""rowId"": 4, ""arr"": [0,4]}" 6 | 5,00000005-0000-0000-0001-020304050607,5.0005,5.05,1,5,5,5,5,5,5,5,2019-01-01T01:01:01.0000000Z,Five,"Five",5,00:00:05.0050005,,"{""rowId"": 5, ""arr"": [0,5]}" 7 | 6,00000006-0000-0000-0001-020304050607,6.0006,6.06,0,6,6,6,6,6,6,6,2020-01-01T01:01:01.0000000Z,Six,"Six",6,-00:00:06.0060006,,"{""rowId"": 6, ""arr"": [0,6]}" 8 | 7,00000007-0000-0000-0001-020304050607,7.0007,7.07,1,7,7,7,7,7,7,7,2021-01-01T01:01:01.0000000Z,Seven,"Seven",7,00:00:07.0070007,,"{""rowId"": 7, ""arr"": [0,7]}" 9 | 8,00000008-0000-0000-0001-020304050607,8.0008,8.08,0,8,8,8,8,8,8,8,2022-01-01T01:01:01.0000000Z,Eight,"Eight",8,-00:00:08.0080008,,"{""rowId"": 8, ""arr"": [0,8]}" 10 | 9,00000009-0000-0000-0001-020304050607,9.0009,9.09,1,9,9,9,9,9,9,9,2023-01-01T01:01:01.0000000Z,Nine,"Nine",9,00:00:09.0090009,,"{""rowId"": 9, ""arr"": [0,9]}" -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/dataset.csv.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-ingest/tests/input/dataset.csv.gz -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/dataset.csv.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-ingest/tests/input/dataset.csv.zip -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/dataset.json: -------------------------------------------------------------------------------- 1 | {"rownumber": 0, "rowguid": "00000000-0000-0000-0001-020304050607", "xdouble": 0.0, "xfloat": 0.0, "xbool": 0, "xint16": 0, "xint32": 0, "xint64": 0, "xunit8": 0, "xuint16": 0, "xunit32": 0, "xunit64": 0, "xdate": "2014-01-01T01:01:01Z", "xsmalltext": "Zero", "xtext": "Zero", "xnumberAsText": "0", "xtime": "00:00:00", "xtextWithNulls": null, "xdynamicWithNulls": ""} 2 | {"rownumber": 1, "rowguid": "00000001-0000-0000-0001-020304050607", "xdouble": 1.00001, "xfloat": 1.01, "xbool": 1, "xint16": 1, "xint32": 1, "xint64": 1, "xuint8": 1, "xuint16": 1, "xuint32": 1, "xuint64": 1, "xdate": "2015-01-01T01:01:01Z", "xsmalltext": "One", "xtext": "One", "xnumberAsText": "1", "xtime": "00:00:01.0010001", "xtextWithNulls": null, "xdynamicWithNulls": "{\"rowId\":1,\"arr\":[0,1]}"} -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/dataset.jsonz.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-kusto-python/d5fcedfdf160f1fb3f267e5ca7732277dde59d11/azure-kusto-ingest/tests/input/dataset.jsonz.gz -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/input/dataset.tsv: -------------------------------------------------------------------------------- 1 | 0 00000000-0000-0000-0001-020304050607 0 0 0 0 0 0 0 0 0 0 2014-01-01T01:01:01.0000000Z Zero "Zero" 0 00:00:00 null 2 | 1 00000001-0000-0000-0001-020304050607 1.0001 1.01 1 1 1 1 1 1 1 1 2015-01-01T01:01:01.0000000Z One "One" 1 00:00:01.0010001 "{""rowId"": 1, ""arr"": [0,1]}" 3 | 2 00000002-0000-0000-0001-020304050607 2.0002 2.02 0 22 2 2 2 2 2 2 2016-01-01T01:01:01.0000000Z Two "Two" 2 -00:00:02.0020002 "{""rowId"": 2, ""arr"": [0,2]}" 4 | 3 00000003-0000-0000-0001-020304050607 3.0003 3.03 1 3 3 3 3 3 3 3 2017-01-01T01:01:01.0000000Z Three "Three" 3 00:00:03.0030003 "{""rowId"": 3, ""arr"": [0,3]}" 5 | 4 00000004-0000-0000-0001-020304050607 4.0004 4.04 0 4 4 4 4 4 4 4 2018-01-01T01:01:01.0000000Z Four "Four" 4 -00:00:04.0040004 "{""rowId"": 4, ""arr"": [0,4]}" 6 | 5 00000005-0000-0000-0001-020304050607 5.0005 5.05 1 5 5 5 5 5 5 5 2019-01-01T01:01:01.0000000Z Five "Five" 5 00:00:05.0050005 "{""rowId"": 5, ""arr"": [0,5]}" 7 | 6 00000006-0000-0000-0001-020304050607 6.0006 6.06 0 6 6 6 6 6 6 6 2020-01-01T01:01:01.0000000Z Six "Six" 6 -00:00:06.0060006 "{""rowId"": 6, ""arr"": [0,6]}" 8 | 7 00000007-0000-0000-0001-020304050607 7.0007 7.07 1 7 7 7 7 7 7 7 2021-01-01T01:01:01.0000000Z Seven "Seven" 7 00:00:07.0070007 "{""rowId"": 7, ""arr"": [0,7]}" 9 | 8 00000008-0000-0000-0001-020304050607 8.0008 8.08 0 8 8 8 8 8 8 8 2022-01-01T01:01:01.0000000Z Eight "Eight" 8 -00:00:08.0080008 "{""rowId"": 8, ""arr"": [0,8]}" 10 | 9 00000009-0000-0000-0001-020304050607 9.0009 9.09 1 9 9 9 9 9 9 9 2023-01-01T01:01:01.0000000Z Nine "Nine" 9 00:00:09.0090009 "{""rowId"": 9, ""arr"": [0,9]}" -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/test_connection_string.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | 4 | from azure.kusto.ingest._resource_manager import _ResourceUri 5 | 6 | 7 | def test_blob_uri(): 8 | """Tests parsing blob uris.""" 9 | storage_name = "storageaccountname" 10 | container_name = "containername" 11 | endpoint_suffix = "core.windows.net" 12 | container_sas = "somesas" 13 | 14 | uri = "https://{}.blob.{}/{}?{}".format(storage_name, endpoint_suffix, container_name, container_sas) 15 | connection_string = _ResourceUri(uri) 16 | assert connection_string.account_uri == "https://storageaccountname.blob.core.windows.net/?somesas" 17 | assert connection_string.object_name == container_name 18 | 19 | 20 | def test_queue_uri(): 21 | """Tests parsing queues uris.""" 22 | storage_name = "storageaccountname" 23 | queue_name = "queuename" 24 | endpoint_suffix = "core.windows.net" 25 | queue_sas = "somesas" 26 | 27 | uri = "https://{}.queue.{}/{}?{}".format(storage_name, endpoint_suffix, queue_name, queue_sas) 28 | connection_string = _ResourceUri(uri) 29 | assert connection_string.account_uri == "https://storageaccountname.queue.core.windows.net/?somesas" 30 | assert connection_string.object_name == queue_name 31 | 32 | 33 | def test_gov_cloud_uri(): 34 | """Tests parsing queues uris.""" 35 | storage_name = "storageaccountname" 36 | queue_name = "queuename" 37 | endpoint_suffix = "core.eaglex.ic.gov" 38 | queue_sas = "somesas" 39 | 40 | uri = "https://{}.queue.{}/{}?{}".format(storage_name, endpoint_suffix, queue_name, queue_sas) 41 | connection_string = _ResourceUri(uri) 42 | assert connection_string.account_uri == "https://storageaccountname.queue.core.eaglex.ic.gov/?somesas" 43 | assert connection_string.object_name == queue_name 44 | -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/test_ingest_telemetry.py: -------------------------------------------------------------------------------- 1 | from io import BytesIO 2 | 3 | from azure.kusto.ingest import FileDescriptor, BlobDescriptor, StreamDescriptor 4 | from azure.kusto.ingest.ingestion_properties import IngestionProperties 5 | 6 | 7 | def test_get_tracing_attributes(): 8 | ingestion_properties = IngestionProperties("database_test", "table_test") 9 | assert {"database": "database_test", "table": "table_test"} == ingestion_properties.get_tracing_attributes() 10 | 11 | dummy_stream = BytesIO(b"dummy") 12 | stream = StreamDescriptor(dummy_stream) 13 | dummy_path = "dummy" 14 | blob = BlobDescriptor(dummy_path) 15 | file = FileDescriptor(dummy_path) 16 | 17 | descriptors = [stream, blob, file] 18 | keynames = [{"stream_name", "source_id"}, {"blob_uri", "source_id"}, {"file_path", "source_id"}] 19 | for i in range(len(descriptors)): 20 | attributes = descriptors[i].get_tracing_attributes() 21 | assert isinstance(attributes, dict) 22 | for key, val in attributes.items(): 23 | assert key in keynames[i] 24 | assert isinstance(val, str) 25 | for key in keynames[i]: 26 | assert key in attributes.keys() 27 | -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/test_ingestion_blob_info.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Microsoft Corporation. 2 | # Licensed under the MIT License 3 | import json 4 | import re 5 | import unittest 6 | from uuid import UUID 7 | 8 | from azure.kusto.data.data_format import DataFormat 9 | from azure.kusto.ingest import ( 10 | BlobDescriptor, 11 | IngestionProperties, 12 | ColumnMapping, 13 | ReportLevel, 14 | ReportMethod, 15 | ValidationPolicy, 16 | ValidationOptions, 17 | ValidationImplications, 18 | ) 19 | from azure.kusto.ingest.ingestion_blob_info import IngestionBlobInfo 20 | 21 | TIMESTAMP_REGEX = "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{6}" 22 | 23 | 24 | class IngestionBlobInfoTest(unittest.TestCase): 25 | """Tests serialization of ingestion blob info. This serialization will be queued to the DM.""" 26 | 27 | def test_blob_info_csv_mapping(self): 28 | """Tests serialization of csv ingestion blob info.""" 29 | validation_policy = ValidationPolicy(ValidationOptions.ValidateCsvInputConstantColumns, ValidationImplications.BestEffort) 30 | column_mapping = ColumnMapping("ColumnName", "cslDataType", ordinal=1) 31 | 32 | properties = IngestionProperties( 33 | database="database", 34 | table="table", 35 | data_format=DataFormat.CSV, 36 | column_mappings=[column_mapping], 37 | additional_tags=["tag"], 38 | ingest_if_not_exists=["ingestIfNotExistTags"], 39 | ingest_by_tags=["ingestByTags"], 40 | drop_by_tags=["dropByTags"], 41 | flush_immediately=True, 42 | report_level=ReportLevel.DoNotReport, 43 | report_method=ReportMethod.Queue, 44 | validation_policy=validation_policy, 45 | ) 46 | blob = BlobDescriptor("somepath", 10) 47 | blob_info = IngestionBlobInfo(blob, properties, auth_context="authorizationContextText") 48 | self._verify_ingestion_blob_info_result(blob_info.to_json()) 49 | 50 | def test_blob_csv_mapping_reference(self): 51 | """Tests serialization of ingestion blob info with csv mapping reference.""" 52 | validation_policy = ValidationPolicy(ValidationOptions.ValidateCsvInputConstantColumns, ValidationImplications.BestEffort) 53 | properties = IngestionProperties( 54 | database="database", 55 | table="table", 56 | data_format=DataFormat.CSV, 57 | ingestion_mapping_reference="csvMappingReference", 58 | additional_tags=["tag"], 59 | ingest_if_not_exists=["ingestIfNotExistTags"], 60 | ingest_by_tags=["ingestByTags"], 61 | drop_by_tags=["dropByTags"], 62 | flush_immediately=True, 63 | report_level=ReportLevel.DoNotReport, 64 | report_method=ReportMethod.Queue, 65 | validation_policy=validation_policy, 66 | ) 67 | blob = BlobDescriptor("somepath", 10) 68 | blob_info = IngestionBlobInfo(blob, properties, auth_context="authorizationContextText") 69 | self._verify_ingestion_blob_info_result(blob_info.to_json()) 70 | 71 | def test_blob_info_json_mapping(self): 72 | """Tests serialization of json ingestion blob info.""" 73 | validation_policy = ValidationPolicy(ValidationOptions.ValidateCsvInputConstantColumns, ValidationImplications.BestEffort) 74 | properties = IngestionProperties( 75 | database="database", 76 | table="table", 77 | data_format=DataFormat.JSON, 78 | column_mappings=[ColumnMapping("ColumnName", "datatype", path="jsonpath")], 79 | additional_tags=["tag"], 80 | ingest_if_not_exists=["ingestIfNotExistTags"], 81 | ingest_by_tags=["ingestByTags"], 82 | drop_by_tags=["dropByTags"], 83 | flush_immediately=True, 84 | report_level=ReportLevel.DoNotReport, 85 | report_method=ReportMethod.Queue, 86 | validation_policy=validation_policy, 87 | ) 88 | blob = BlobDescriptor("somepath", 10) 89 | blob_info = IngestionBlobInfo(blob, properties, auth_context="authorizationContextText") 90 | self._verify_ingestion_blob_info_result(blob_info.to_json()) 91 | 92 | def test_blob_json_mapping_reference(self): 93 | """Tests serialization of ingestion blob info with json mapping reference.""" 94 | validation_policy = ValidationPolicy(ValidationOptions.ValidateCsvInputConstantColumns, ValidationImplications.BestEffort) 95 | properties = IngestionProperties( 96 | database="database", 97 | table="table", 98 | data_format=DataFormat.JSON, 99 | ingestion_mapping_reference="jsonMappingReference", 100 | additional_tags=["tag"], 101 | ingest_if_not_exists=["ingestIfNotExistTags"], 102 | ingest_by_tags=["ingestByTags"], 103 | drop_by_tags=["dropByTags"], 104 | flush_immediately=True, 105 | report_level=ReportLevel.DoNotReport, 106 | report_method=ReportMethod.Queue, 107 | validation_policy=validation_policy, 108 | ) 109 | blob = BlobDescriptor("somepath", 10) 110 | blob_info = IngestionBlobInfo(blob, properties, auth_context="authorizationContextText") 111 | self._verify_ingestion_blob_info_result(blob_info.to_json()) 112 | 113 | def _verify_ingestion_blob_info_result(self, ingestion_blob_info): 114 | result = json.loads(ingestion_blob_info) 115 | assert result is not None 116 | assert isinstance(result, dict) 117 | assert result["BlobPath"] == "somepath" 118 | assert result["DatabaseName"] == "database" 119 | assert result["TableName"] == "table" 120 | assert isinstance(result["RawDataSize"], int) 121 | assert isinstance(result["IgnoreSizeLimit"], bool) 122 | assert isinstance(result["FlushImmediately"], bool) 123 | assert isinstance(result["RetainBlobOnSuccess"], bool) 124 | assert isinstance(result["ReportMethod"], int) 125 | assert isinstance(result["ReportLevel"], int) 126 | assert isinstance(UUID(result["Id"]), UUID) 127 | assert re.match(TIMESTAMP_REGEX, result["SourceMessageCreationTime"]) 128 | assert result["AdditionalProperties"]["authorizationContext"] == "authorizationContextText" 129 | assert result["AdditionalProperties"]["ingestIfNotExists"] == '["ingestIfNotExistTags"]' 130 | assert result["AdditionalProperties"]["ValidationPolicy"] in ( 131 | '{"ValidationImplications":1,"ValidationOptions":1}', 132 | '{"ValidationImplications":ValidationImplications.BestEffort,"ValidationOptions":ValidationOptions.ValidateCsvInputConstantColumns}', 133 | ) 134 | 135 | assert result["AdditionalProperties"]["tags"] == '["tag","drop-by:dropByTags","ingest-by:ingestByTags"]' 136 | -------------------------------------------------------------------------------- /azure-kusto-ingest/tests/test_ingestion_properties.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from azure.kusto.data.data_format import IngestionMappingKind, DataFormat 4 | from azure.kusto.ingest import IngestionProperties, ColumnMapping, TransformationMethod 5 | from azure.kusto.ingest.exceptions import KustoDuplicateMappingError, KustoMissingMappingError, KustoMappingError 6 | 7 | 8 | def test_duplicate_reference_and_column_mappings_raises(): 9 | """Tests invalid ingestion properties.""" 10 | with pytest.raises(KustoDuplicateMappingError): 11 | IngestionProperties( 12 | database="database", table="table", column_mappings=[ColumnMapping("test", "int")], ingestion_mapping_reference="ingestionMappingReference" 13 | ) 14 | 15 | 16 | def test_mapping_kind_without_mapping_raises(): 17 | with pytest.raises(KustoMissingMappingError): 18 | IngestionProperties(database="database", table="table", ingestion_mapping_kind=IngestionMappingKind.CSV) 19 | 20 | 21 | def test_mapping_kind_data_format_mismatch(): 22 | with pytest.raises(KustoMappingError): 23 | IngestionProperties( 24 | database="database", 25 | table="table", 26 | ingestion_mapping_reference="ingestionMappingReference", 27 | data_format=DataFormat.JSON, 28 | ingestion_mapping_kind=IngestionMappingKind.CSV, 29 | ) 30 | 31 | 32 | def test_mapping_kind_data_format_invalid_no_name(): 33 | with pytest.raises(KustoMappingError): 34 | IngestionProperties( 35 | database="database", 36 | table="table", 37 | column_mappings=[ColumnMapping("", "int")], 38 | data_format=DataFormat.JSON, 39 | ingestion_mapping_kind=IngestionMappingKind.JSON, 40 | ) 41 | 42 | 43 | def test_mapping_kind_data_format_invalid_no_path(): 44 | with pytest.raises(KustoMappingError): 45 | IngestionProperties( 46 | database="database", 47 | table="table", 48 | column_mappings=[ColumnMapping("test", "int")], 49 | data_format=DataFormat.JSON, 50 | ingestion_mapping_kind=IngestionMappingKind.JSON, 51 | ) 52 | 53 | 54 | def test_mapping_kind_data_format_with_path(): 55 | IngestionProperties( 56 | database="database", 57 | table="table", 58 | column_mappings=[ColumnMapping("test", "int", "path")], 59 | data_format=DataFormat.JSON, 60 | ingestion_mapping_kind=IngestionMappingKind.JSON, 61 | ) 62 | 63 | 64 | def test_mapping_kind_data_format_with_transform(): 65 | IngestionProperties( 66 | database="database", 67 | table="table", 68 | column_mappings=[ColumnMapping("test", "int", transform=TransformationMethod.SOURCE_LINE_NUMBER)], 69 | data_format=DataFormat.JSON, 70 | ingestion_mapping_kind=IngestionMappingKind.JSON, 71 | ) 72 | 73 | 74 | def test_mapping_kind_data_format_with_no_properties(): 75 | with pytest.raises(KustoMappingError): 76 | IngestionProperties( 77 | database="database", 78 | table="table", 79 | column_mappings=[ColumnMapping("test", "int")], 80 | data_format=DataFormat.AVRO, 81 | ingestion_mapping_kind=IngestionMappingKind.AVRO, 82 | ) 83 | 84 | 85 | def test_with_constant_value(): 86 | IngestionProperties( 87 | database="database", 88 | table="table", 89 | column_mappings=[ColumnMapping("test", "int", const_value="1")], 90 | data_format=DataFormat.PARQUET, 91 | ingestion_mapping_kind=IngestionMappingKind.PARQUET, 92 | ) 93 | -------------------------------------------------------------------------------- /azure-kusto-ingest/tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27,py35,py37 3 | 4 | [testenv] 5 | deps= 6 | pytest 7 | pandas 8 | commands=pytest 9 | -------------------------------------------------------------------------------- /back_to_black.bat: -------------------------------------------------------------------------------- 1 | pip install --upgrade black==23.3.0 2 | black . --line-length 160 3 | -------------------------------------------------------------------------------- /build_packages.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | from subprocess import check_call 4 | from pathlib import Path 5 | 6 | 7 | try: 8 | from packaging.version import parse as Version, InvalidVersion 9 | except ImportError: # Should not happen, but at worst in most case this is the same 10 | from pip._vendor.packaging.version import parse as Version, InvalidVersion 11 | 12 | DEFAULT_DESTINATION_FOLDER = os.path.join("..", "dist") 13 | package_list = ["azure-kusto-data", "azure-kusto-ingest"] 14 | 15 | 16 | def create_package(name, dest_folder=DEFAULT_DESTINATION_FOLDER): 17 | absdirpath = os.path.abspath(name) 18 | check_call(["python", "setup.py", "bdist_wheel", "-d", dest_folder], cwd=absdirpath) 19 | check_call(["python", "setup.py", "sdist", "-d", dest_folder], cwd=absdirpath) 20 | 21 | 22 | def travis_build_package(): 23 | """Assumed called on Travis, to prepare a package to be deployed 24 | This method prints on stdout for Travis. 25 | Return is obj to pass to sys.exit() directly 26 | """ 27 | travis_tag = os.environ.get("TRAVIS_TAG") 28 | if not travis_tag: 29 | print("TRAVIS_TAG environment variable is not present") 30 | return "TRAVIS_TAG environment variable is not present" 31 | 32 | try: 33 | version = Version(travis_tag) 34 | except InvalidVersion: 35 | failure = "Version must be a valid PEP440 version (version is: {})".format(version) 36 | print(failure) 37 | return failure 38 | 39 | abs_dist_path = Path(os.environ["TRAVIS_BUILD_DIR"], "dist") 40 | [create_package(package, str(abs_dist_path)) for package in package_list] 41 | 42 | print("Produced:\n{}".format(list(abs_dist_path.glob("*")))) 43 | 44 | pattern = "*{}*".format(version) 45 | packages = list(abs_dist_path.glob(pattern)) 46 | if not packages: 47 | return "Package version does not match tag {}, abort".format(version) 48 | pypi_server = os.environ.get("PYPI_SERVER", "default PyPI server") 49 | print("Package created as expected and will be pushed to {}".format(pypi_server)) 50 | 51 | 52 | if __name__ == "__main__": 53 | parser = argparse.ArgumentParser(description="Build Azure package.") 54 | parser.add_argument("name", help="The package name") 55 | parser.add_argument("--dest", "-d", default=DEFAULT_DESTINATION_FOLDER, help="Destination folder. Relative to the package dir. [default: %(default)s]") 56 | 57 | args = parser.parse_args() 58 | if args.name == "all": 59 | for package in package_list: 60 | create_package(package, args.dest) 61 | else: 62 | create_package(args.name, args.dest) 63 | -------------------------------------------------------------------------------- /codecov.yml: -------------------------------------------------------------------------------- 1 | coverage: 2 | status: 3 | project: 4 | default: 5 | informational: true 6 | patch: 7 | default: 8 | informational: true 9 | github_checks: 10 | annotations: false 11 | -------------------------------------------------------------------------------- /dev_requirements.txt: -------------------------------------------------------------------------------- 1 | pytest>=3.2.0 2 | responses>=0.9.0 3 | pandas>=0.24.0 4 | black;python_version >= '3.6' 5 | aioresponses>=0.7.6 6 | pytest-asyncio>=0.12.0 7 | asgiref>=3.2.3 8 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.black] 2 | line-length = 160 -------------------------------------------------------------------------------- /quick_start/README.md: -------------------------------------------------------------------------------- 1 | # Quickstart App 2 | 3 | The quick start application is a **self-contained and runnable** example script that demonstrates authenticating, connecting to, administering, ingesting data into and querying Azure Data Explorer using the azure-kusto-python SDK. 4 | You can use it as a baseline to write your own first kusto client application, altering the code as you go, or copy code sections out of it into your app. 5 | 6 | **Tip:** The app includes comments with tips on recommendations, coding best practices, links to reference materials and recommended TODO changes when adapting the code to your needs. 7 | 8 | 9 | ## Using the App for the first time 10 | 11 | ### Prerequisites 12 | 13 | 1. Set up Python version 3.8 or higher on your machine. For instructions, consult a Python environment setup tutorial, like [this](https://www.tutorialspoint.com/python/python_environment.htm). 14 | 15 | ### Retrieving the app from GitHub 16 | 17 | 1. Download the app files from this GitHub repo. 18 | 1. Modify the `kusto_sample_config.json` file, changing `KustoUri`, `IngestUri` and `DatabaseName` appropriately for your cluster. 19 | 20 | ### Retrieving the app from OneClick 21 | 22 | 1. Open a browser and type your cluster's URL (e.g. https://mycluster.westeurope.kusto.windows.net/). You will be redirected to the _Azure Data Explorer_ Web UI. 23 | 1. On the left menu, select **Data**. 24 | 1. Click **Generate Sample App Code** and then follow the instructions in the wizard. 25 | 1. Download the app as a ZIP file. 26 | 1. Extract the app source code. 27 | **Note**: The configuration parameters defined in the `kusto_sample_config.json` file are preconfigured with the appropriate values for your cluster. Verify that these are correct. 28 | 29 | ### Run the app 30 | 31 | 1. Open a command line window and navigate to the folder where you extracted the script. 32 | 1. Run `python -m pip install -r requirements.txt`. If upgrading, append ` --upgrade`. 33 | 1. Run `python sample_app.py`. 34 | 35 | #### Troubleshooting 36 | 37 | * If you are having trouble running the script from your IDE, first check if the script runs from the command line, then consult the troubleshooting references of your IDE. 38 | 39 | ### Optional changes 40 | 41 | You can make the following changes from within the script itself: 42 | 43 | - Change the location of the configuration file by editing `CONFIG_FILE_NAME`. 44 | -------------------------------------------------------------------------------- /quick_start/dataset.csv: -------------------------------------------------------------------------------- 1 | 0,00000000-0000-0000-0001-020304050607,0,0,0,0,0,0,0,0,0,0,2014-01-01T01:01:01.0000000Z,Zero,Zero,0,00:00:00,,null 2 | 1,00000001-0000-0000-0001-020304050607,1.0001,1.01,1,1,1,1,1,1,1,1,2015-01-01T01:01:01.0000000Z,One,One,1,00:01.0,,"{""rowId"": 1, ""arr"": [0,1]}" 3 | 2,00000002-0000-0000-0001-020304050607,2.0002,2.02,0,2,2,2,2,2,2,2,2016-01-01T01:01:01.0000000Z,Two,Two,2,-00:00:02.0020002,,"{""rowId"": 2, ""arr"": [0,2]}" 4 | 3,00000003-0000-0000-0001-020304050607,3.0003,3.03,1,3,3,3,3,3,3,3,2017-01-01T01:01:01.0000000Z,Three,Three,3,00:03.0,,"{""rowId"": 3, ""arr"": [0,3]}" 5 | 4,00000004-0000-0000-0001-020304050607,4.0004,4.04,0,4,4,4,4,4,4,4,2018-01-01T01:01:01.0000000Z,Four,Four,4,-00:00:04.0040004,,"{""rowId"": 4, ""arr"": [0,4]}" 6 | 5,00000005-0000-0000-0001-020304050607,5.0005,5.05,1,5,5,5,5,5,5,5,2019-01-01T01:01:01.0000000Z,Five,Five,5,00:05.0,,"{""rowId"": 5, ""arr"": [0,5]}" 7 | 6,00000006-0000-0000-0001-020304050607,6.0006,6.06,0,6,6,6,6,6,6,6,2020-01-01T01:01:01.0000000Z,Six,Six,6,-00:00:06.0060006,,"{""rowId"": 6, ""arr"": [0,6]}" 8 | 7,00000007-0000-0000-0001-020304050607,7.0007,7.07,1,7,7,7,7,7,7,7,2021-01-01T01:01:01.0000000Z,Seven,Seven,7,00:07.0,,"{""rowId"": 7, ""arr"": [0,7]}" 9 | 8,00000008-0000-0000-0001-020304050607,8.0008,8.08,0,8,8,8,8,8,8,8,2022-01-01T01:01:01.0000000Z,Eight,Eight,8,-00:00:08.0080008,,"{""rowId"": 8, ""arr"": [0,8]}" 10 | 9,00000009-0000-0000-0001-020304050607,9.0009,9.09,1,9,9,9,9,9,9,9,2023-01-01T01:01:01.0000000Z,Nine,Nine,9,00:09.0,,"{""rowId"": 9, ""arr"": [0,9]}" 11 | -------------------------------------------------------------------------------- /quick_start/dataset.json: -------------------------------------------------------------------------------- 1 | {"rownumber": 0, "rowguid": "00000000-0000-0000-0001-020304050607", "xdouble": 0.0, "xfloat": 0.0, "xbool": 0, "xint16": 0, "xint32": 0, "xint64": 0, "xunit8": 0, "xuint16": 0, "xunit32": 0, "xunit64": 0, "xdate": "2014-01-01T01:01:01Z", "xsmalltext": "Zero", "xtext": "Zero", "xnumberAsText": "0", "xtime": "00:00:00", "xtextWithNulls": null, "xdynamicWithNulls": ""} 2 | {"rownumber": 1, "rowguid": "00000001-0000-0000-0001-020304050607", "xdouble": 1.00001, "xfloat": 1.01, "xbool": 1, "xint16": 1, "xint32": 1, "xint64": 1, "xuint8": 1, "xuint16": 1, "xuint32": 1, "xuint64": 1, "xdate": "2015-01-01T01:01:01Z", "xsmalltext": "One", "xtext": "One", "xnumberAsText": "1", "xtime": "00:00:01.0010001", "xtextWithNulls": null, "xdynamicWithNulls": "{\"rowId\":1,\"arr\":[0,1]}"} -------------------------------------------------------------------------------- /quick_start/kusto_sample_config.json: -------------------------------------------------------------------------------- 1 | { 2 | "kustoUri": "https://help.kusto.windows.net", 3 | "ingestUri": "https://ingest-help.kusto.windows.net", 4 | "databaseName": "MyDatabase", 5 | "tableName": "SampleTable", 6 | "useExistingTable": true, 7 | "alterTable": true, 8 | "queryData": true, 9 | "ingestData": true, 10 | "authenticationMode": "UserPrompt", 11 | "waitForUser": true, 12 | "ignoreFirstRecord": false, 13 | "waitForIngestSeconds": 20, 14 | "batchingPolicy": "{ 'MaximumBatchingTimeSpan': '00:00:10', 'MaximumNumberOfItems': 500, 'MaximumRawDataSizeMB': 1024 }", 15 | "tableSchema": "(rownumber:int, rowguid:string, xdouble:real, xfloat:real, xbool:bool, xint16:int, xint32:int, xint64:long, xuint8:long, xuint16:long, xuint32:long, xuint64:long, xdate:datetime, xsmalltext:string, xtext:string, xnumberAsText:string, xtime:timespan, xtextWithNulls:string, xdynamicWithNulls:dynamic)", 16 | "data": [ 17 | { 18 | "sourceType": "localFileSource", 19 | "dataSourceUri": "dataset.csv", 20 | "format": "CSV", 21 | "useExistingMapping": true, 22 | "mappingName": "", 23 | "mappingValue": "" 24 | }, 25 | { 26 | "sourceType": "localFileSource", 27 | "dataSourceUri": "dataset.json", 28 | "format": "MULTIJSON", 29 | "useExistingMapping": false, 30 | "mappingName": "SampleTableMapping", 31 | "mappingValue": "[{\"Properties\":{\"Path\":\"$.rownumber\"},\"column\":\"rownumber\",\"datatype\":\"int\"}, {\"Properties\":{\"Path\":\"$.rowguid\"},\"column\":\"rowguid\",\"datatype\":\"string\"}, {\"Properties\":{\"Path\":\"$.xdouble\"},\"column\":\"xdouble\",\"datatype\":\"real\"}, {\"Properties\":{\"Path\":\"$.xfloat\"},\"column\":\"xfloat\",\"datatype\":\"real\"}, {\"Properties\":{\"Path\":\"$.xbool\"},\"column\":\"xbool\",\"datatype\":\"bool\"}, {\"Properties\":{\"Path\":\"$.xint16\"},\"column\":\"xint16\",\"datatype\":\"int\"}, {\"Properties\":{\"Path\":\"$.xint32\"},\"column\":\"xint32\",\"datatype\":\"int\"}, {\"Properties\":{\"Path\":\"$.xint64\"},\"column\":\"xint64\",\"datatype\":\"long\"}, {\"Properties\":{\"Path\":\"$.xuint8\"},\"column\":\"xuint8\",\"datatype\":\"long\"}, {\"Properties\":{\"Path\":\"$.xuint16\"},\"column\":\"xuint16\",\"datatype\":\"long\"}, {\"Properties\":{\"Path\":\"$.xuint32\"},\"column\":\"xuint32\",\"datatype\":\"long\"}, {\"Properties\":{\"Path\":\"$.xuint64\"},\"column\":\"xuint64\",\"datatype\":\"long\"}, {\"Properties\":{\"Path\":\"$.xdate\"},\"column\":\"xdate\",\"datatype\":\"datetime\"}, {\"Properties\":{\"Path\":\"$.xsmalltext\"},\"column\":\"xsmalltext\",\"datatype\":\"string\"}, {\"Properties\":{\"Path\":\"$.xtext\"},\"column\":\"xtext\",\"datatype\":\"string\"}, {\"Properties\":{\"Path\":\"$.rowguid\"},\"column\":\"xnumberAsText\",\"datatype\":\"string\"}, {\"Properties\":{\"Path\":\"$.xtime\"},\"column\":\"xtime\",\"datatype\":\"timespan\"}, {\"Properties\":{\"Path\":\"$.xtextWithNulls\"},\"column\":\"xtextWithNulls\",\"datatype\":\"string\"}, {\"Properties\":{\"Path\":\"$.xdynamicWithNulls\"},\"column\":\"xdynamicWithNulls\",\"datatype\":\"dynamic\"}]" 32 | } 33 | ] 34 | } -------------------------------------------------------------------------------- /quick_start/oneclick_instruction_box.md: -------------------------------------------------------------------------------- 1 | ### Prerequisites 2 | 3 | 1. Set up Python version 3.8 or higher on your machine. For instructions, consult a Python environment setup tutorial, like [this](https://www.tutorialspoint.com/python/python_environment.htm). 4 | 5 | ### Instructions 6 | 7 | 1. Download the **DOWNLOAD_LINK** as a ZIP file. 8 | 1. Extract the app source code. 9 | 1. Open a command line window and navigate to the folder where you extracted the script. 10 | 1. Run `python -m pip install -r requirements.txt`. If upgrading, append ` --upgrade`. 11 | 1. Run `python sample_app.py`. It will already be configured to your cluster and source. 12 | 13 | ### Troubleshooting 14 | 15 | * If you are having trouble running the script from your IDE, first check if the script runs from command line, then consult the troubleshooting references of your IDE. 16 | -------------------------------------------------------------------------------- /quick_start/quick_start.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 17 | -------------------------------------------------------------------------------- /quick_start/requirements.txt: -------------------------------------------------------------------------------- 1 | azure-core-tracing-opentelemetry~=1.0.0b9 2 | azure-kusto-data>=4.0.0,<5.0.0 3 | azure-kusto-ingest>=4.0.0,<5.0.0 4 | azure-monitor-opentelemetry-exporter~=1.0.0b7 5 | 6 | inflection~=0.5.1 7 | opentelemetry-sdk~=1.12.0 8 | tqdm~=4.64.0 9 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal=1 3 | 4 | [flake8] 5 | ignore = E226,E302,E41 6 | max-line-length = 160 7 | exclude = tests/* 8 | max-complexity = 10 9 | 10 | 11 | [pylint] 12 | max-line-length = 160 -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | if "travis_deploy" in sys.argv: 4 | import build_packages 5 | 6 | sys.exit(build_packages.travis_build_package()) 7 | else: 8 | raise ValueError("Setup file is written to support travis publish.") 9 | -------------------------------------------------------------------------------- /test.bat: -------------------------------------------------------------------------------- 1 | cd %PROJECTS_HOME%\azure-kusto-python 2 | call workon kusto 3 | call pip uninstall azure-kusto-data azure-kusto-ingest -y 4 | call pip install ./azure-kusto-data[pandas] ./azure-kusto-ingest[pandas] 5 | call pip install --force-reinstall azure-nspkg==2.0.0 6 | call pytest 7 | pause --------------------------------------------------------------------------------