├── .github ├── ISSUE_TEMPLATE │ ├── bug_report.md │ ├── config.yml │ └── feature_request.md ├── pull_request_template.md └── workflows │ ├── build.yaml │ └── release.yaml ├── .gitignore ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── Cargo.toml ├── LICENSE ├── README.md ├── examples ├── historical.rs ├── live.rs ├── live_smoke_test.rs └── split_symbols.rs ├── scripts ├── build.sh ├── format.sh ├── get_version.sh ├── lint.sh └── test.sh ├── src ├── error.rs ├── historical.rs ├── historical │ ├── batch.rs │ ├── client.rs │ ├── deserialize.rs │ ├── metadata.rs │ ├── symbology.rs │ └── timeseries.rs ├── lib.rs ├── live.rs └── live │ ├── client.rs │ └── protocol.rs └── tests └── data ├── test_data.definition.dbn ├── test_data.definition.dbn.zst ├── test_data.imbalance.dbn ├── test_data.imbalance.dbn.zst ├── test_data.mbo.dbn ├── test_data.mbo.dbn.zst ├── test_data.mbp-1.dbn ├── test_data.mbp-1.dbn.zst ├── test_data.mbp-10.dbn ├── test_data.mbp-10.dbn.zst ├── test_data.ohlcv-1d.dbn ├── test_data.ohlcv-1d.dbn.zst ├── test_data.ohlcv-1h.dbn ├── test_data.ohlcv-1h.dbn.zst ├── test_data.ohlcv-1m.dbn ├── test_data.ohlcv-1m.dbn.zst ├── test_data.ohlcv-1s.dbn ├── test_data.ohlcv-1s.dbn.zst ├── test_data.statistics.dbn ├── test_data.statistics.dbn.zst ├── test_data.tbbo.dbn ├── test_data.tbbo.dbn.zst ├── test_data.trades.dbn └── test_data.trades.dbn.zst /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug Report 3 | about: Report a bug here 4 | labels: 5 | - bug 6 | --- 7 | 8 | # Bug Report 9 | 10 | ### Expected Behavior 11 | Add here... 12 | 13 | ### Actual Behavior 14 | Add here... 15 | 16 | ### Steps to Reproduce the Problem 17 | 18 | 1. 19 | 2. 20 | 3. 21 | 22 | ### Specifications 23 | 24 | - OS platform: 25 | - Rust version: 26 | - `databento` version: 27 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/config.yml: -------------------------------------------------------------------------------- 1 | blank_issues_enabled: false 2 | contact_links: 3 | - name: General Questions 4 | url: https://github.com/databento/databento/discussions 5 | about: Please ask questions like "How do I achieve x?" here. 6 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature Request 3 | about: Request a new feature to be added here 4 | labels: 5 | - enhancement 6 | --- 7 | 8 | # Feature Request 9 | 10 | Please provide a detailed description of your proposal, with some examples. 11 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | # Pull request 2 | 3 | Please include a summary of the changes. 4 | Please also include relevant motivation and context. 5 | List any dependencies that are required for this change. 6 | 7 | Fixes # (issue) 8 | 9 | ## Type of change 10 | 11 | Please delete options that are not relevant. 12 | 13 | - [ ] Bug fix (non-breaking change which fixes an issue) 14 | - [ ] New feature (non-breaking change which adds functionality) 15 | - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) 16 | - [ ] This change requires a documentation update 17 | 18 | ## How has this change been tested? 19 | 20 | Please describe the tests that you ran to verify your changes. 21 | Provide instructions so we can reproduce. 22 | Please also list any relevant details for your test configuration. 23 | 24 | - [ ] Test A 25 | - [ ] Test B 26 | 27 | ## Checklist 28 | 29 | - [ ] My code builds locally with no new warnings (`scripts/build.sh`) 30 | - [ ] My code follows the style guidelines (`scripts/lint.sh` and `scripts/format.sh`) 31 | - [ ] New and existing unit tests pass locally with my changes (`scripts/test.sh`) 32 | - [ ] I have made corresponding changes to the documentation 33 | - [ ] I have added tests that prove my fix is effective or that my feature works 34 | 35 | ## Declaration 36 | 37 | I confirm this contribution is made under an Apache 2.0 license and that I have the authority 38 | necessary to make this contribution on behalf of its copyright owner. 39 | -------------------------------------------------------------------------------- /.github/workflows/build.yaml: -------------------------------------------------------------------------------- 1 | name: build 2 | on: 3 | pull_request: 4 | push: 5 | 6 | jobs: 7 | build: 8 | strategy: 9 | fail-fast: false 10 | matrix: 11 | os: [macos-latest, ubuntu-latest, windows-latest] 12 | name: build (${{ matrix.os }}) 13 | runs-on: ${{ matrix.os }} 14 | 15 | steps: 16 | - name: Checkout repository 17 | uses: actions/checkout@v4 18 | - name: Set up Rust 19 | run: rustup toolchain add --profile minimal stable --component clippy,rustfmt 20 | # Cargo setup 21 | - name: Set up Cargo cache 22 | uses: actions/cache@v4 23 | with: 24 | path: | 25 | ~/.cargo/registry 26 | ~/.cargo/git 27 | target 28 | key: ${{ runner.os }}-cargo-${{ hashFiles('Cargo.lock') }} 29 | 30 | - name: Format 31 | run: scripts/format.sh 32 | shell: bash 33 | - name: Build 34 | run: scripts/build.sh 35 | shell: bash 36 | - name: Lint 37 | run: scripts/lint.sh 38 | shell: bash 39 | - name: Test 40 | run: scripts/test.sh 41 | shell: bash 42 | -------------------------------------------------------------------------------- /.github/workflows/release.yaml: -------------------------------------------------------------------------------- 1 | name: release 2 | 3 | on: 4 | workflow_run: 5 | workflows: [build] 6 | branches: [main] 7 | types: 8 | - completed 9 | workflow_dispatch: 10 | branches: [main] 11 | 12 | jobs: 13 | tag-release: 14 | if: ${{ github.event.workflow_run.conclusion == 'success' }} || ${{ github.event.workflow_dispatch }} 15 | name: tag-release (ubuntu-latest) 16 | runs-on: ubuntu-latest 17 | outputs: 18 | upload_url: ${{ steps.create-release.outputs.upload_url }} 19 | 20 | steps: 21 | - name: Checkout repository 22 | uses: actions/checkout@v4 23 | with: 24 | fetch-depth: 2 25 | - name: Set up Rust 26 | run: rustup toolchain add --profile minimal stable --component clippy,rustfmt 27 | 28 | # Cargo setup 29 | - name: Set up Cargo cache 30 | uses: actions/cache@v4 31 | with: 32 | path: | 33 | ~/.cargo/registry 34 | ~/.cargo/git 35 | target 36 | key: ${{ runner.os }}-cargo-${{ hashFiles('Cargo.lock') }} 37 | 38 | # Tag the commit with the library version 39 | - name: Create git tag 40 | uses: salsify/action-detect-and-tag-new-version@v2 41 | with: 42 | version-command: scripts/get_version.sh 43 | 44 | # Set release output variables 45 | - name: Set output 46 | id: vars 47 | run: | 48 | echo "TAG_NAME=v$(scripts/get_version.sh)" >> $GITHUB_ENV 49 | echo "RELEASE_NAME=$(scripts/get_version.sh)" >> $GITHUB_ENV 50 | echo "## Release notes" > NOTES.md 51 | sed -n '/^## /{n; :a; /^## /q; p; n; ba}' CHANGELOG.md >> NOTES.md 52 | # Create GitHub release 53 | - name: Create release 54 | id: create-release 55 | uses: softprops/action-gh-release@v1 56 | with: 57 | name: ${{ env.RELEASE_NAME }} 58 | tag_name: ${{ env.TAG_NAME }} 59 | append_body: true 60 | body_path: ./NOTES.md 61 | prerelease: false 62 | 63 | - name: Remove notes 64 | # Force to not error if it doesn't exist 65 | run: rm --force NOTES.md 66 | 67 | - name: Publish to crates.io 68 | run: cargo publish --token ${CARGO_REGISTRY_TOKEN} 69 | env: 70 | CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} 71 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | .profile/ 3 | .vscode/ 4 | 5 | target/ 6 | Cargo.lock 7 | **/*.rs.bk 8 | *.sw[po] 9 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## 0.26.2 - 2025-06-03 4 | 5 | ### Enhancements 6 | - Improved performance of live client by removing redundant state 7 | - Upgraded DBN version to 0.35.1 8 | 9 | ### Bug fixes 10 | - Fixed handling of `VersionUpgradePolicy` in `timeseries().get_range()` and 11 | `get_range_to_file()` 12 | - Bug fixes from DBN: 13 | - Fixed behavior where encoding metadata could lower the `version` 14 | - Changed `DbnFsm::data()` to exclude all processed data 15 | - Fixed `Metadata::upgrade()` behavior with `UpgradeToV2` 16 | 17 | ## 0.26.1 - 2025-05-30 18 | 19 | ### Bug fixes 20 | - Fixed handling of `VersionUpgradePolicy` in live client 21 | - Fixed default upgrade policies to `UpgradeToV3` to match announcement for 22 | version 0.26.0 23 | 24 | ## 0.26.0 - 2025-05-28 25 | 26 | This version marks the release of DBN version 3 (DBNv3), which is the new default. 27 | API methods and decoders support decoding all versions of DBN, but now default to 28 | upgrading data to version 3. 29 | 30 | ### Enhancements - Added `From` conversion for `DateTimeRange` 31 | - Added `is_last` field to live subscription requests which will be used to improve the 32 | handling of split subscription requests 33 | - Upgraded DBN version to 0.35.0: 34 | - Version 1 and 2 structs can be converted to version 3 structs with the `From` trait 35 | - Implemented conversion from `RecordRef` to `IoSlice` for use with 36 | `Write::write_vectored` 37 | 38 | ### Breaking changes 39 | - Breaking changes from DBN: 40 | - Definition schema: 41 | - Updated `InstrumentDefMsg` with new `leg_` fields to support multi-leg strategy 42 | definitions. 43 | - Expanded `asset` to 11 bytes and `ASSET_CSTR_LEN` to match 44 | - Expanded `raw_instrument_id` to 64 bits to support more venues. Like other 64-bit 45 | integer fields, its value will now be quoted in JSON 46 | - Removed `trading_reference_date`, `trading_reference_price`, and 47 | `settl_price_type` fields which will be normalized in the statistics schema 48 | - Removed `md_security_trading_status` better served by the status schema 49 | - Statistics schema: 50 | - Updated `StatMsg` has an expanded 64-bit `quantity` field. Like other 64-bit 51 | integer fields, its value will now be quoted in JSON 52 | - The previous `StatMsg` has been moved to `v2::StatMsg` or `StatMsgV2` 53 | - Changed the default `VersionUpgradePolicy` to `UpgradeToV3` 54 | - Updated the minimum supported `tokio` version to 1.38, which was released one year ago 55 | 56 | ## 0.25.0 - 2025-05-13 57 | 58 | ### Enhancements 59 | - Increased live subscription symbol chunking size 60 | - Upgraded DBN version to 0.34.0: 61 | - Added a `v3::StatMsg` record with an expanded 64-bit `quantity` field 62 | - Added `with_compression_level` methods to `DynWriter`, `AsyncDynWriter`, and 63 | `AsyncDynBufWriter` 64 | - Added `DBN_VERSION` constants to each version module: `v1`, `v2`, and `v3` 65 | - Added `UNDEF_STAT_QUANTITY` constants to each version module 66 | - Added statistics compatibility trait `StatRec` for generalizing across different 67 | versions of the statistics record 68 | - Added `AsRef<[u8]>` implementations for `RecordEnum` and `RecordRefEnum` 69 | - Added new off-market publishers for Eurex, and European Energy Exchange (EEX) 70 | 71 | ### Breaking changes 72 | - From DBN: 73 | - Made `Record` a subtrait of `AsRef<[u8]>` as all records should be convertible to 74 | bytes 75 | 76 | ## 0.24.0 - 2025-04-22 77 | 78 | ### Enhancements 79 | - Upgraded DBN version to 0.33.0: 80 | - Added `SystemCode` and `ErrorCode` enums to indicate types of system and error 81 | messages 82 | - Added `code()` methods to `SystemMsg` and `ErrorMsg` to retrieve the enum value if 83 | one exists and equivalent properties in Python 84 | - Converting a `v1::SystemMsg` to a `v2::SystemMsg` now sets to `code` to the 85 | heartbeat value 86 | - Added `ASSET_CSTR_LEN` constants for the size of `asset` field in `InstrumentDefMsg` 87 | in different DBN versions 88 | - Added `encode_record_with_sym()` method to `AsyncJsonEncoder` which encodes a 89 | record along with its text symbol to match the sync encoder 90 | 91 | ### Breaking changes 92 | - Breaking changes from DBN: 93 | - Added `code` parameter to `SystemCode::new()` and `ErrorMsg::new()` 94 | - Updated the `rtype_dispatch` and `schema_dispatch` macro invocations to look more 95 | like function invocation 96 | - Increased the size of `asset` field in `v3::InstrumentDefMsg` from 7 to 11. The 97 | `InstrumentDefMsgV3` message size remains 520 bytes. 98 | 99 | ## 0.23.0 - 2025-04-15 100 | 101 | ### Enhancements 102 | - Added `subscriptions` to `LiveClient` `Debug` implementation 103 | - Upgraded DBN version to 0.32.0: 104 | - Added `SystemCode` and `ErrorCode` enums to indicate types of system and error 105 | messages 106 | - Added `code()` methods to `SystemMsg` and `ErrorMsg` to retrieve the enum value if 107 | one exists and equivalent properties in Python 108 | - Converting a `v1::SystemMsg` to a `v2::SystemMsg` now sets to `code` to the heartbeat 109 | value 110 | - Added `Ord` and `PartialOrd` implementations for all enums and `FlagSet` to allow 111 | for use in ordered containers like `BTreeMap` 112 | - Added `decode_records()` method to `AsyncDbnDecoder` and `AsyncDbnRecordDecoder` 113 | which is similar to the sync decoder methods of the same name 114 | - Upgraded `pyo3` version to 0.24.1 115 | - Upgraded `time` version to 0.3.41 116 | 117 | ### Breaking changes 118 | - Added new `id` field to live `Subscription`, which will be used for improved error 119 | messages 120 | - Added new `id` parameter to `live::protocol::SubRequest::new()` method 121 | - Breaking changes from DBN: 122 | - Added `code` parameter to `SystemCode::new()` and `ErrorMsg::new()` 123 | - Updated the `rtype_dispatch` and `schema_dispatch` macro invocations to look more like 124 | function invocation 125 | - Removed deprecated `dataset` module. The top-level `Dataset` enum and its `const` `as_str()` 126 | method provide the same functionality for all datasets 127 | - Removed deprecated `SymbolIndex::get_for_rec_ref()` method 128 | 129 | ## 0.22.0 - 2025-04-01 130 | 131 | ### Enhancements 132 | - Added an implementation `From` for `DateRange` and `DateTimeRange` to make it 133 | simpler to request a single full day's worth of data 134 | - Added conversions between `DateRange` and `DateTimeRange` 135 | - Added conversions from `timeseries::GetRangeParams`, `timeseries::GetRangeToFileParams`, 136 | and `dbn::Metadata` to `symbology::ResolveParams` 137 | - Upgraded DBN version to 0.31.0: 138 | - Added support for mapping symbols from instrument definitions to `PitSymbolMap` 139 | with a new `on_instrument_def()` method 140 | - Added instrument definition compatibility trait `InstrumentDefRec` for generalizing 141 | across different versions of the instrument definition record 142 | - Added `Ord` and `PartialOrd` implementations for all enums and `FlagSet` to allow 143 | for use in ordered containers like `BTreeMap` 144 | - Added `decode_records()` method to `AsyncDbnDecoder` and `AsyncDbnRecordDecoder` 145 | which is similar to the sync decoder methods of the same name 146 | - Removed deprecated `dataset` module. The top-level `Dataset` enum and its `const` `as_str()` 147 | method provide the same functionality for all datasets 148 | - Removed deprecated `SymbolIndex::get_for_rec_ref()` method 149 | 150 | ## 0.21.0 - 2025-03-18 151 | 152 | ### Enhancements 153 | - Improved error when calling `LiveClient::next_record()` on an instance that hasn't 154 | been started 155 | - Improved error when calling `LiveClient::start()` on an instance that has already 156 | been started 157 | - Upgraded DBN version to 0.29.0: 158 | - Added new venues, datasets, and publishers for ICE Futures US, ICE Futures Europe 159 | (Financial products), Eurex, and European Energy Exchange (EEX) 160 | - Added new `SkipBytes` and `AsyncSkipBytes` traits which are a subset of the `Seek` 161 | and `AsyncSeek` traits respectively, only supporting seeking forward from the current 162 | position 163 | - Deprecated `AsyncRecordDecoder::get_mut()` and `AsyncDecoder::get_mut()` as modifying 164 | the inner reader after decoding any records could lead to a corrupted stream and 165 | decoding errors 166 | 167 | ## 0.20.0 - 2025-02-12 168 | 169 | ### Enhancements 170 | - Added `LiveClient::reconnect()` and `LiveClient::resubscribe()` methods to make it easier 171 | to resume a live session after losing the connection to the live gateway 172 | - Added `subscriptions()` and `subscriptions_mut()` getters to `LiveClient` for getting all 173 | active subscriptions 174 | - Added `shutdown()` method to `live::Protocol` to clean up the active session 175 | - Downgraded to tracing span level on `LiveClient::next_record()` to "debug" to reduce 176 | performance impact 177 | - Added `From<&[&str]>` and `From<[str; N]>` implementations for `Symbols` 178 | 179 | ### Breaking changes 180 | - Changed `LiveClient::close()` to take `&mut self` rather than an owned value to `self` now 181 | that clients can be reused through the `reconnect()` method 182 | - Changed `LiveClient::subscribe()` to take a `Subscription` parameter rather than a 183 | `&Subscription` because it will now store the `Subscription` struct internally 184 | - Upgraded DBN version to 0.28.0: 185 | - Added `CommoditySpot` `InstrumentClass` variant and made `InstrumentClass` 186 | non-exhaustive to allow future additions without breaking changes 187 | 188 | ## 0.19.0 - 2025-01-21 189 | 190 | ### Enhancements 191 | - Upgraded DBN version to 0.27.0: 192 | - Updated enumerations for unreleased US equities datasets and publishers 193 | - Added new venue `EQUS` for consolidated US equities 194 | - Added new dataset `EQUS.MINI` and new publishers `EQUS.MINI.EQUS` and 195 | `XNYS.TRADES.EQUS` 196 | 197 | ### Bug fixes 198 | - Changed historical metadata methods with `symbols` parameter to use a `POST` request 199 | to allow for requesting supported maximum of 2000 symbols 200 | 201 | ## 0.18.0 - 2025-01-08 202 | 203 | ### Enhancements 204 | - Upgraded DBN version to 0.26.0: 205 | - Added `v3` namespace in preparation for future DBN version 3 release. DBN version 2 206 | remains the current and default version 207 | - Added `v3::InstrumentDefMsg` record with new fields to support normalizing multi-leg 208 | strategy definitions 209 | - Removal of statistics-schema related fields `trading_reference_price`, 210 | `trading_reference_date`, and `settl_price_type` 211 | - Removal of the status-schema related field `md_security_trading_status` 212 | - Added initial support for merging DBN: 213 | - Decoding streams: `MergeDecoder` and `MergeRecordDecoder` structs 214 | - Metadata: `MergeDecoder` struct and `Metadata::merge()` method 215 | - In the CLI: specify more than one input file to initiate a merge 216 | - Eliminated `unsafe` in `From` implementations for record structs from different 217 | versions 218 | 219 | ## 0.17.0 - 2024-12-17 220 | 221 | ### Enhancements 222 | - Upgraded DBN version to 0.25.0: 223 | - Added `v1` and `v2` namespaces in DBN to allow unambiguously referring to the record 224 | types for a given DBN version regardless of whether the record type has changed 225 | - Changed `dataset()` method on `MetadataBuilder` to accept an `impl ToString` so now 226 | `Dataset` and `&str` can be passed directly 227 | - Changed async DBN decoding to return `Ok(None)` when an incomplete record remains in 228 | the stream. This matches the existing behavior of sync DBN decoding 229 | - Upgraded `thiserror` version to 2.0 230 | 231 | ### Breaking changes 232 | - Removed deprecated `Packaging` enum and `packaging` field that's no longer supported 233 | by the API 234 | - As part of the DBN version upgrade: 235 | - `VersionUpgradePolicy::Upgrade` was renamed to `UpgradeToV2` 236 | - Changed async DBN decoding to return `Ok(None)` when an incomplete record remains in 237 | the stream 238 | 239 | ## 0.16.0 - 2024-11-12 240 | 241 | #### Enhancements 242 | - Upgraded DBN version to 0.23.1: 243 | - Added floating-point getters for price fields 244 | - Added new IntelligentCross venues `ASPN`, `ASMT`, and `ASPI` 245 | - Upgraded `thiserror` version to 2.0 246 | 247 | #### Deprecations 248 | - Deprecated `Packaging` enum and `packaging` field on `SubmitJobParams` and `BatchJob`. 249 | These will be removed in a future version. All files from a batch job can be 250 | downloaded with the `batch().download()` method on the historical client 251 | 252 | ## 0.15.0 - 2024-10-22 253 | 254 | #### Enhancements 255 | - Upgraded DBN version to 0.23.0: 256 | - Added new `None` `Action` variant that will be gradually rolled out 257 | to historical and live `GLBX.MDP3` data 258 | - Added consistent escaping of non-printable and non-ASCII values when text encoding 259 | `c_char` fields 260 | - Implemented `Default` for `Action` and `Side` 261 | - Implemented missing `Serialize` for (with `serde` feature enabled) for `Venue`, 262 | `Dataset`, `Publisher`, `Compression`, `SType`, `Schema`, and `Encoding` 263 | 264 | ## 0.14.1 - 2024-10-08 265 | 266 | #### Enhancements 267 | - Upgraded DBN version to 0.22.1: 268 | - Fixed buffer overrun 269 | - Combined `_reserved3` and `reserved4` fields in `CbboMsg` 270 | 271 | ## 0.14.0 - 2024-10-01 272 | 273 | #### Enhancements 274 | - Made several previously internal functions public to allow advanced users more 275 | customization and piecemeal usage of the live API: 276 | - `ApiKey` 277 | - `Symbols::to_chunked_api_string()` 278 | - `live::protocol` module containing implementations of the raw API messages 279 | - Changed from `log` crate to `tracing` for better diagnostics 280 | 281 | ## 0.13.0 - 2024-09-25 282 | 283 | #### Enhancements 284 | - Upgraded DBN version to 0.21.0 for: 285 | - Changed the layout of `CbboMsg` to better match `BboMsg` 286 | - Renamed `Schema::Cbbo` to `Schema::Cmbp1` 287 | - Upgraded `typed-builder` version to 0.20 288 | 289 | #### Deprecations 290 | - Deprecated `Packaging::Tar`. Users should switch to `Packaging::Zip`. This variant 291 | will be removed in a future version when it is no longer supported by the API 292 | 293 | ## 0.12.1 - 2024-08-27 294 | 295 | #### Enhancements 296 | - Added `Intraday` variant to `DatasetCondition` in preparation for intraday data being 297 | available from the historical API 298 | - Upgraded DBN version to 0.20.1 for new publisher values for `XCIS.BBOTRADES` and 299 | `XNYS.BBOTRADES` 300 | 301 | ## 0.12.0 - 2024-07-30 302 | 303 | #### Breaking changes 304 | - Upgraded DBN version to 0.20.0: 305 | - Renamed `SType::Nasdaq` variant to `SType::NasdaqSymbol` 306 | - Renamed `SType::Cms` variant to `SType::CmsSymbol` 307 | 308 | ## 0.11.4 - 2024-07-16 309 | 310 | #### Enhancements 311 | - Upgraded DBN version to 0.19.1 with fixes for `BBOMsg` record struct 312 | 313 | ## 0.11.3 - 2024-07-09 314 | 315 | #### Enhancements 316 | - Upgraded DBN version to 0.19.0 with new `BBOMsg` record struct 317 | 318 | ## 0.11.2 - 2024-06-25 319 | 320 | #### Enhancements 321 | - Added `historical::timeseries::get_range_to_file` method to persist the data stream to 322 | a given path before returning an `AsyncDbnDecoder` 323 | - Upgraded DBN version to 0.18.2 324 | 325 | ## 0.11.1 - 2024-06-11 326 | 327 | #### Enhancements 328 | - Added getter for `heartbeat_interval` to `LiveClient` 329 | 330 | #### Bug fixes 331 | - Fixed potential incorrect DNS resolution when overriding the live gateway address 332 | with `live::Builder::addr` 333 | 334 | ## 0.11.0 - 2024-06-04 335 | 336 | #### Enhancements 337 | - Added configurable `heartbeat_interval` parameter for live client that determines the 338 | timeout before heartbeat `SystemMsg` records will be sent. It can be configured via 339 | the `heartbeat_interval` and `heartbeat_interval_s` methods of the 340 | `live::ClientBuilder` 341 | - Added `addr` function to `live::ClientBuilder` for configuring a custom gateway 342 | address without using `LiveClient::connect_with_addr` directly 343 | - Upgraded DBN version to 0.18.1 344 | 345 | #### Breaking changes 346 | - Added `heartbeat_interval` parameter to `LiveClient::connect` and 347 | `LiveClient::connect_with_addr` 348 | - Removed deprecated `start_date` and `end_date` fields from `DatasetRange` struct 349 | 350 | ## 0.10.0 - 2024-05-22 351 | 352 | #### Enhancements 353 | - Added `use_snapshot` attribute to `Subscription`, defaults to false 354 | - Upgraded reqwest version to 0.12 355 | 356 | #### Breaking changes 357 | - Upgraded DBN version to 0.18.0 358 | - Changed type of `flags` in `MboMsg`, `TradeMsg`, `Mbp1Msg`, `Mbp10Msg`, and `CbboMsg` 359 | from `u8` to a new `FlagSet` type with predicate methods for the various bit flags 360 | as well as setters. The `u8` value can still be obtained by calling the `raw()` method 361 | - Improved `Debug` formatting 362 | - Switched `DecodeStream` from `streaming_iterator` crate to `fallible_streaming_iterator` 363 | to allow better notification of errors 364 | - Changed default value for `stype_in` and `stype_out` in `SymbolMappingMsg` to 365 | `u8::MAX` to match C++ client and to reflect an unknown value. This also changes the 366 | value of these fields when upgrading a `SymbolMappingMsgV1` to DBNv2 367 | 368 | ## 0.9.1 - 2024-05-15 369 | 370 | #### Bug fixes 371 | - Fixed build when only `live` feature is enabled 372 | 373 | ## 0.9.0 - 2024-05-14 374 | 375 | #### Enhancements 376 | - Added `start` and `end` fields to the `DatasetRange` struct which provide time 377 | resolution and an exclusive end date 378 | - Upgraded DBN version to 0.17.1 379 | 380 | #### Deprecations 381 | - The `start_date` and `end_date` fields of the `DatasetRange` struct are deprecated and 382 | will be removed in a future release 383 | 384 | ## 0.8.0 - 2024-04-01 385 | 386 | #### Enhancements 387 | - Upgraded DBN version to 0.17.0 388 | - Added new record types and schema variants for consolidated BBO and subsampled BBO 389 | - Added `Volatility` and `Delta` `StatType` variants 390 | 391 | #### Breaking changes 392 | - Removed previously-deprecated `live::SymbolMap`. Please use 393 | `databento::dbn::PitSymbolMap` instead 394 | 395 | ## 0.7.1 - 2024-03-05 396 | 397 | #### Enhancements 398 | - Improve error handling when a historical HTTP error response is not in the 399 | expected JSON format 400 | 401 | ## 0.7.0 - 2024-03-01 402 | 403 | #### Enhancements 404 | - Document cancellation safety of `LiveClient` methods (credit: @yongqli) 405 | - Document `live::Subscription::start` is based on `ts_event` 406 | - Allow constructing a `DateRange` and `DateTimeRange` with an `end` based on a 407 | `time::Duration` 408 | - Implemented `Debug` for `LiveClient`, `live::ClientBuilder`, `HistoricalClient`, 409 | `historical::ClientBuilder`, `BatchClient`, `MetadataClient`, `SymbologyClient`, and 410 | `TimeseriesClient` 411 | - Derived `Clone` for `live::ClientBuilder` and `historical::ClientBuilder` 412 | - Added `ApiKey` type for safely deriving `Debug` for types containing an API key 413 | 414 | #### Breaking changes 415 | - Changed default `upgrade_policy` in `LiveBuilder` and `GetRangeParams` to `Upgrade` so 416 | by default the primary record types can always be used 417 | - Simplified `DateRange` and `DateTimeRange` by removing `FwdFill` variant that didn't 418 | work correctly 419 | - Upgraded DBN version to 0.16.0 420 | - Updated `StatusMsg` in preparation for status schema release 421 | - Fixed handling of `ts_out` when upgrading DBNv1 records to version 2 422 | - Fixed handling of `ErrorMsgV1` and `SystemMsgV1` in `rtype` dispatch macros 423 | 424 | ## 0.6.0 - 2024-01-16 425 | 426 | #### Enhancements 427 | - Relaxed version requirements for `tokio`, `tokio-util`, and `thiserror` 428 | 429 | #### Breaking changes 430 | - Upgraded DBN version to 0.15.0 431 | - Added support for larger `SystemMsg` and `ErrorMsg` records 432 | - Improved `Debug` implementations for records and `RecordRef` 433 | - Improved panic messages for `RecordRef::get` 434 | - Upgraded `typed-builder` to 0.18 435 | 436 | #### Bug fixes 437 | - Fixed documentation for `end` in `DateRange::Closed` and `DateTimeRange::Closed` 438 | 439 | ## 0.5.0 - 2023-11-23 440 | 441 | This release adds support for DBN v2. 442 | 443 | DBN v2 delivers improvements to the `Metadata` header symbology, new `stype_in` and `stype_out` 444 | fields for `SymbolMappingMsg`, and extends the symbol field length for `SymbolMappingMsg` and 445 | `InstrumentDefMsg`. The entire change notes are available [here](https://github.com/databento/dbn/releases/tag/v0.14.0). 446 | Users who wish to convert DBN v1 files to v2 can use the `dbn-cli` tool available in the [databento-dbn](https://github.com/databento/dbn/) crate. 447 | On a future date, the Databento live and historical APIs will stop serving DBN v1. 448 | 449 | This release is fully compatible with both DBN v1 and v2, and so should be seamless for most users. 450 | 451 | #### Enhancements 452 | - Made `LiveClient::next_record`, `dbn::decode::AsyncDbnDecoder::decode_record` and 453 | `decode_record_ref`, and `dbn::decode::AsyncRecordDecoder::decode` and `decode_ref` 454 | cancel safe. This makes them safe to use within a 455 | [`tokio::select!`](https://docs.rs/tokio/latest/tokio/macro.select.html) statement 456 | - Improved error reporting for `HistoricalClient` when receiving an error from 457 | Databento's API 458 | - Improved error messages around API keys 459 | - Improved performance of CSV and JSON encoding 460 | - Added support for emitting warnings from historical API response headers, such as for 461 | future deprecations 462 | - Added `symbol_map` method to the `Resolution` struct returned by `symbology::resolve` 463 | that returns a `TsSymbolMap` 464 | - Added `PartialEq` and `Eq` implementations for parameter builder classes 465 | - Added `upgrade_policy` setter to the `LiveClient` builder and a getter to the 466 | `LiveClient` 467 | - Added `upgrade_policy` optional setter to the `timeseries::GetRangeParams` builder 468 | 469 | #### Breaking changes 470 | - Upgraded `dbn` to 0.14.2. There are several breaking changes in this release as we 471 | begin migrating to DBN encoding version 2 (DBNv2) in order to support the ICE 472 | exchange: 473 | - Renamed `dbn::InstrumentDefMsg` to `dbn::compat::InstrumentDefMsgV1` and added a 474 | new `dbn::InstrumentDefMsg` with a longer `raw_symbol` field 475 | - Renamed `dbn::SymbolMappingMsg` to `dbn::compat::SymbolMappingMsgV1` and added a 476 | new `dbn::SymbolMappingMsg` with longer symbol fields and new `stype_in` and 477 | `stype_out` fields 478 | - Added `symbol_cstr_len` field to `dbn::Metadata` 479 | - Made `Error` non-exhaustive, meaning it no longer be exhaustively matched against, and 480 | new variants can be added in the future without a breaking change 481 | - Added an `upgrade_policy` parameter to `LiveClient::connect` and `connect_with_addr`. 482 | The builder provides a more stable API since new parameters are usually introduced as 483 | optional 484 | 485 | #### Deprecations 486 | - Deprecated `live::SymbolMap` in favor of `databento::dbn::PitSymbolMap` 487 | 488 | ## 0.4.2 - 2023-10-23 489 | 490 | #### Enhancemets 491 | - Upgraded `dbn` to 0.13.0 for improvements to symbology helpers 492 | - Upgraded `tokio` to 1.33 493 | - Upgraded `typed-builder` to 0.17 494 | 495 | #### Bug fixes 496 | - Fixed panic in `LiveClient` when gateway returned an auth response without the 497 | `success` key 498 | 499 | ## 0.4.1 - 2023-10-06 500 | 501 | #### Enhancements 502 | - Added support for changing datetime format used in batch job responses 503 | - Upgraded `dbn` to 0.11.1 504 | 505 | ## 0.4.0 - 2023-09-21 506 | 507 | #### Enhancements 508 | - Added `pretty_px` option for `batch::submit_job`, which formats prices to the correct 509 | scale using the fixed-precision scalar 1e-9 (available for CSV and JSON text 510 | encodings) 511 | - Added `pretty_ts` option for `batch::submit_job`, which formats timestamps as ISO 8601 512 | strings (available for CSV and JSON text encodings) 513 | - Added `map_symbols` option to `batch::submit_job`, which appends the raw symbol to 514 | every record (available for CSV and JSON text encodings) reducing the need to look at 515 | the `symbology.json` file 516 | - Added `split_symbols` option for `batch::submit_job`, which will split files by raw 517 | symbol 518 | - Added `encoding` option to `batch::submit_job` to allow requesting non-DBN encoded 519 | data through the client 520 | - Added `map_symbols`, `pretty_px`, and `pretty_ts` to `BatchJob` response 521 | - Added default `stype_in` of `RawSymbol` for live subscriptions to match behavior of 522 | the historical client and the Python client 523 | 524 | ## 0.3.0 - 2023-09-13 525 | 526 | #### Enhancements 527 | - Added `SymbolMap` type to help maintain up-to-date symbol mappings with live data 528 | - Added chunking to handle subscribing to many symbols for the Live client 529 | - Upgraded DBN version to 0.10.2 for easier historical symbology 530 | 531 | ## 0.2.1 - 2023-08-25 532 | 533 | #### Enhancements 534 | - Upgraded DBN version to 0.9.0 for publisher improvements to support OPRA 535 | 536 | ## 0.2.0 - 2023-08-10 537 | 538 | #### Breaking changes 539 | - Changed `metadata::list_publishers` to return a `Vec` 540 | - `metadata::list_fields`: 541 | - Changed return type to `Vec` 542 | - Made `encoding` and `schema` parameters required 543 | - Removed `dataset` parameter 544 | - `metadata::list_unit_prices`: 545 | - Changed return type to `Vec` 546 | - Made `dataset` parameter required 547 | - Removed `mode` and `schema` parameters 548 | 549 | ## 0.1.0 - 2023-08-02 550 | - Initial release with support for historical and live data 551 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | We as members, contributors, and leaders pledge to make participation in our 6 | community a harassment-free experience for everyone, regardless of age, body 7 | size, visible or invisible disability, ethnicity, sex characteristics, gender 8 | identity and expression, level of experience, education, socio-economic status, 9 | nationality, personal appearance, race, religion, or sexual identity 10 | and orientation. 11 | 12 | We pledge to act and interact in ways that contribute to an open, welcoming, 13 | diverse, inclusive, and healthy community. 14 | 15 | ## Our Standards 16 | 17 | Examples of behavior that contributes to a positive environment for our 18 | community include: 19 | 20 | * Demonstrating empathy and kindness toward other people 21 | * Being respectful of differing opinions, viewpoints, and experiences 22 | * Giving and gracefully accepting constructive feedback 23 | * Accepting responsibility and apologizing to those affected by our mistakes, 24 | and learning from the experience 25 | * Focusing on what is best not just for us as individuals, but for the 26 | overall community 27 | 28 | Examples of unacceptable behavior include: 29 | 30 | * The use of sexualized language or imagery, and sexual attention or 31 | advances of any kind 32 | * Trolling, insulting or derogatory comments, and personal or political attacks 33 | * Public or private harassment 34 | * Publishing others' private information, such as a physical or email 35 | address, without their explicit permission 36 | * Other conduct which could reasonably be considered inappropriate in a 37 | professional setting 38 | 39 | ## Enforcement Responsibilities 40 | 41 | Community leaders are responsible for clarifying and enforcing our standards of 42 | acceptable behavior and will take appropriate and fair corrective action in 43 | response to any behavior that they deem inappropriate, threatening, offensive, 44 | or harmful. 45 | 46 | Community leaders have the right and responsibility to remove, edit, or reject 47 | comments, commits, code, wiki edits, issues, and other contributions that are 48 | not aligned to this Code of Conduct, and will communicate reasons for moderation 49 | decisions when appropriate. 50 | 51 | ## Scope 52 | 53 | This Code of Conduct applies within all community spaces, and also applies when 54 | an individual is officially representing the community in public spaces. 55 | Examples of representing our community include using an official e-mail address, 56 | posting via an official social media account, or acting as an appointed 57 | representative at an online or offline event. 58 | 59 | ## Enforcement 60 | 61 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 62 | reported to the community leaders responsible for enforcement at 63 | info@nautechsystems.io. 64 | All complaints will be reviewed and investigated promptly and fairly. 65 | 66 | All community leaders are obligated to respect the privacy and security of the 67 | reporter of any incident. 68 | 69 | ## Enforcement Guidelines 70 | 71 | Community leaders will follow these Community Impact Guidelines in determining 72 | the consequences for any action they deem in violation of this Code of Conduct: 73 | 74 | ### 1. Correction 75 | 76 | **Community Impact**: Use of inappropriate language or other behavior deemed 77 | unprofessional or unwelcome in the community. 78 | 79 | **Consequence**: A private, written warning from community leaders, providing 80 | clarity around the nature of the violation and an explanation of why the 81 | behavior was inappropriate. A public apology may be requested. 82 | 83 | ### 2. Warning 84 | 85 | **Community Impact**: A violation through a single incident or series 86 | of actions. 87 | 88 | **Consequence**: A warning with consequences for continued behavior. No 89 | interaction with the people involved, including unsolicited interaction with 90 | those enforcing the Code of Conduct, for a specified period of time. This 91 | includes avoiding interactions in community spaces as well as external channels 92 | like social media. Violating these terms may lead to a temporary or 93 | permanent ban. 94 | 95 | ### 3. Temporary Ban 96 | 97 | **Community Impact**: A serious violation of community standards, including 98 | sustained inappropriate behavior. 99 | 100 | **Consequence**: A temporary ban from any sort of interaction or public 101 | communication with the community for a specified period of time. No public or 102 | private interaction with the people involved, including unsolicited interaction 103 | with those enforcing the Code of Conduct, is allowed during this period. 104 | Violating these terms may lead to a permanent ban. 105 | 106 | ### 4. Permanent Ban 107 | 108 | **Community Impact**: Demonstrating a pattern of violation of community 109 | standards, including sustained inappropriate behavior, harassment of an 110 | individual, or aggression toward or disparagement of classes of individuals. 111 | 112 | **Consequence**: A permanent ban from any sort of public interaction within 113 | the community. 114 | 115 | ## Attribution 116 | 117 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 118 | version 2.0, available at 119 | https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. 120 | 121 | Community Impact Guidelines were inspired by [Mozilla's code of conduct 122 | enforcement ladder](https://github.com/mozilla/diversity). 123 | 124 | [homepage]: https://www.contributor-covenant.org 125 | 126 | For answers to common questions about this code of conduct, see the FAQ at 127 | https://www.contributor-covenant.org/faq. Translations are available at 128 | https://www.contributor-covenant.org/translations. 129 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | Thank you for taking the time to contribute to our project. 2 | We welcome feedback through discussions and issues on GitHub, as well as our [community Slack](https://databento.com/support). 3 | While we don't merge pull requests directly due to the open-source repository being a downstream 4 | mirror of our internal codebase, we can commit the changes upstream with the original author. 5 | -------------------------------------------------------------------------------- /Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "databento" 3 | authors = ["Databento "] 4 | version = "0.26.2" 5 | edition = "2021" 6 | repository = "https://github.com/databento/databento-rs" 7 | description = "Official Databento client library" 8 | license = "Apache-2.0" 9 | # maximum of 5 10 | keywords = ["real-time", "historical", "market-data", "trading", "tick-data"] 11 | # see https://crates.io/category_slugs 12 | categories = ["api-bindings", "finance"] 13 | 14 | [package.metadata.docs.rs] 15 | # Document all features on docs.rs 16 | all-features = true 17 | # To build locally: `RUSTDOCFLAGS="--cfg docsrs" cargo +nightly doc --all-features --open` 18 | rustdoc-args = ["--cfg", "docsrs"] 19 | 20 | [features] 21 | default = ["historical", "live"] 22 | historical = [ 23 | "dep:async-compression", 24 | "dep:futures", 25 | "dep:reqwest", 26 | "dep:serde", 27 | "dep:tokio-util", 28 | "dep:serde_json", 29 | "tokio/fs" 30 | ] 31 | live = ["dep:hex", "dep:sha2", "tokio/net"] 32 | 33 | [dependencies] 34 | dbn = { version = "0.35.1", features = ["async", "serde"] } 35 | 36 | async-compression = { version = "0.4", features = ["tokio", "zstd"], optional = true } 37 | # Async stream trait 38 | futures = { version = "0.3", optional = true } 39 | # Used for Live authentication 40 | hex = { version = "0.4", optional = true } 41 | reqwest = { version = "0.12", optional = true, features = ["json", "stream"] } 42 | serde = { version = "1.0", optional = true, features = ["derive"] } 43 | serde_json = { version = "1.0", optional = true } 44 | # Used for Live authentication 45 | sha2 = { version = "0.10", optional = true } 46 | thiserror = "2.0" 47 | time = { version = ">=0.3.35", features = ["macros", "parsing", "serde"] } 48 | tokio = { version = ">=1.28", features = ["io-util", "macros"] } 49 | # Stream utils 50 | tokio-util = { version = "0.7", features = ["io"], optional = true } 51 | tracing = "0.1" 52 | typed-builder = "0.21" 53 | 54 | [dev-dependencies] 55 | anyhow = "1.0.98" 56 | async-compression = { version = "0.4", features = ["tokio", "zstd"] } 57 | clap = { version = "4.5.37", features = ["derive"] } 58 | rstest = "0.25.0" 59 | tempfile = "3.19.1" 60 | tokio = { version = "1.44", features = ["full"] } 61 | tracing-subscriber = "0.3.19" 62 | wiremock = "0.6" 63 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # databento-rs 2 | 3 | [![build](https://github.com/databento/databento-rs/actions/workflows/build.yaml/badge.svg)](https://github.com/databento/dbn/actions/workflows/build.yaml) 4 | [![Documentation](https://img.shields.io/docsrs/databento)](https://docs.rs/databento/latest/databento/) 5 | [![license](https://img.shields.io/github/license/databento/databento-rs?color=blue)](./LICENSE) 6 | [![Current Crates.io Version](https://img.shields.io/crates/v/databento.svg)](https://crates.io/crates/databento) 7 | [![Slack](https://img.shields.io/badge/join_Slack-community-darkblue.svg?logo=slack)](https://to.dbn.to/slack) 8 | 9 | The official Rust client library for [Databento](https://databento.com). 10 | The clients support fast and safe streaming of both real-time and historical market data 11 | through similar interfaces. 12 | 13 | ## Installation 14 | 15 | To add the crate to an existing project, run the following command: 16 | ```sh 17 | cargo add databento 18 | ``` 19 | 20 | ## Usage 21 | 22 | ### Live 23 | 24 | Real-time and intraday replay is provided through the Live clients. 25 | Here is a simple program that fetches the next ES mini futures trade: 26 | 27 | ```rust 28 | use std::error::Error; 29 | 30 | use databento::{ 31 | dbn::{Dataset, PitSymbolMap, SType, Schema, TradeMsg}, 32 | live::Subscription, 33 | LiveClient, 34 | }; 35 | 36 | #[tokio::main] 37 | async fn main() -> Result<(), Box> { 38 | let mut client = LiveClient::builder() 39 | .key_from_env()? 40 | .dataset(Dataset::GlbxMdp3) 41 | .build() 42 | .await?; 43 | client 44 | .subscribe( 45 | Subscription::builder() 46 | .symbols("ES.FUT") 47 | .schema(Schema::Trades) 48 | .stype_in(SType::Parent) 49 | .build(), 50 | ) 51 | .await 52 | .unwrap(); 53 | client.start().await?; 54 | 55 | let mut symbol_map = PitSymbolMap::new(); 56 | // Get the next trade 57 | while let Some(rec) = client.next_record().await? { 58 | if let Some(trade) = rec.get::() { 59 | let symbol = &symbol_map[trade]; 60 | println!("Received trade for {symbol}: {trade:?}"); 61 | break; 62 | } 63 | symbol_map.on_record(rec)?; 64 | } 65 | Ok(()) 66 | } 67 | ``` 68 | To run this program, set the `DATABENTO_API_KEY` environment variable with an API key and run `cargo run --example historical` 69 | 70 | ### Historical 71 | 72 | Here is a simple program that fetches 10 minutes worth of historical trades for the entire CME Globex market: 73 | ```rust 74 | use std::error::Error; 75 | 76 | use databento::{ 77 | dbn::{Schema, TradeMsg}, 78 | historical::timeseries::GetRangeParams, 79 | HistoricalClient, Symbols, 80 | }; 81 | use time::macros::{date, datetime}; 82 | 83 | #[tokio::main] 84 | async fn main() -> Result<(), Box> { 85 | let mut client = HistoricalClient::builder().key_from_env()?.build()?; 86 | let mut decoder = client 87 | .timeseries() 88 | .get_range( 89 | &GetRangeParams::builder() 90 | .dataset("GLBX.MDP3") 91 | .date_time_range(( 92 | datetime!(2022-06-10 14:30 UTC), 93 | datetime!(2022-06-10 14:40 UTC), 94 | )) 95 | .symbols(Symbols::All) 96 | .schema(Schema::Trades) 97 | .build(), 98 | ) 99 | .await?; 100 | let symbol_map = decoder 101 | .metadata() 102 | .symbol_map_for_date(date!(2022 - 06 - 10))?; 103 | while let Some(trade) = decoder.decode_record::().await? { 104 | let symbol = &symbol_map[trade]; 105 | println!("Received trade for {symbol}: {trade:?}"); 106 | } 107 | Ok(()) 108 | } 109 | ``` 110 | 111 | To run this program, set the `DATABENTO_API_KEY` environment variable with an API key and run `cargo bin --example live`. 112 | 113 | ## Documentation 114 | 115 | You can find more detailed examples and the full API documentation on the [Databento docs site](https://databento.com/docs/quickstart?historical=rust&live=rust). 116 | 117 | ## License 118 | 119 | Distributed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0.html). 120 | -------------------------------------------------------------------------------- /examples/historical.rs: -------------------------------------------------------------------------------- 1 | //! The example from README.md. Having it here ensures it compiles. 2 | use std::error::Error; 3 | 4 | use databento::{ 5 | dbn::{Schema, TradeMsg}, 6 | historical::timeseries::GetRangeParams, 7 | HistoricalClient, Symbols, 8 | }; 9 | use time::macros::{date, datetime}; 10 | 11 | #[tokio::main] 12 | async fn main() -> Result<(), Box> { 13 | let mut client = HistoricalClient::builder().key_from_env()?.build()?; 14 | let mut decoder = client 15 | .timeseries() 16 | .get_range( 17 | &GetRangeParams::builder() 18 | .dataset("GLBX.MDP3") 19 | .date_time_range(( 20 | datetime!(2022-06-10 14:30 UTC), 21 | datetime!(2022-06-10 14:40 UTC), 22 | )) 23 | .symbols(Symbols::All) 24 | .schema(Schema::Trades) 25 | .build(), 26 | ) 27 | .await?; 28 | let symbol_map = decoder 29 | .metadata() 30 | .symbol_map_for_date(date!(2022 - 06 - 10))?; 31 | while let Some(trade) = decoder.decode_record::().await? { 32 | let symbol = &symbol_map[trade]; 33 | println!("Received trade for {symbol}: {trade:?}"); 34 | } 35 | Ok(()) 36 | } 37 | -------------------------------------------------------------------------------- /examples/live.rs: -------------------------------------------------------------------------------- 1 | //! The example from README.md. Having it here ensures it compiles. 2 | use std::error::Error; 3 | 4 | use databento::{ 5 | dbn::{Dataset, PitSymbolMap, SType, Schema, TradeMsg}, 6 | live::Subscription, 7 | LiveClient, 8 | }; 9 | 10 | #[tokio::main] 11 | async fn main() -> Result<(), Box> { 12 | let mut client = LiveClient::builder() 13 | .key_from_env()? 14 | .dataset(Dataset::GlbxMdp3) 15 | .build() 16 | .await?; 17 | client 18 | .subscribe( 19 | Subscription::builder() 20 | .symbols("ES.FUT") 21 | .schema(Schema::Trades) 22 | .stype_in(SType::Parent) 23 | .build(), 24 | ) 25 | .await 26 | .unwrap(); 27 | client.start().await?; 28 | 29 | let mut symbol_map = PitSymbolMap::new(); 30 | // Get the next trade 31 | while let Some(rec) = client.next_record().await? { 32 | if let Some(trade) = rec.get::() { 33 | let symbol = &symbol_map[trade]; 34 | println!("Received trade for {symbol}: {trade:?}"); 35 | break; 36 | } 37 | symbol_map.on_record(rec)?; 38 | } 39 | Ok(()) 40 | } 41 | -------------------------------------------------------------------------------- /examples/live_smoke_test.rs: -------------------------------------------------------------------------------- 1 | use clap::Parser; 2 | use databento::{ 3 | dbn::{Dataset, ErrorMsg, MboMsg, RType, Record, SType, Schema}, 4 | live::Subscription, 5 | LiveClient, 6 | }; 7 | use time::{format_description::well_known::Rfc3339, OffsetDateTime}; 8 | 9 | #[derive(Debug, Parser)] 10 | #[clap(name = "Rust client", version, about)] 11 | pub struct Args { 12 | #[clap(help = "Gateway address", long)] 13 | pub gateway: String, 14 | 15 | #[clap(help = "Gateway port", long, default_value = "13000")] 16 | pub port: u16, 17 | 18 | #[clap(help = "API Key env var", long, default_value = "DATABENTO_API_KEY")] 19 | pub api_key_env_var: String, 20 | 21 | #[clap(help = "Dataset", long)] 22 | pub dataset: Dataset, 23 | 24 | #[clap(help = "Schema", long)] 25 | pub schema: Schema, 26 | 27 | #[clap(help = "SType", long)] 28 | pub stype: SType, 29 | 30 | #[clap(help = "Symbols", long)] 31 | pub symbols: String, 32 | 33 | #[clap(help = "Start time (rfc-3339)", long, default_value=None)] 34 | pub start: Option, 35 | 36 | #[clap(help = "Use snapshot", long, action)] 37 | pub use_snapshot: bool, 38 | } 39 | 40 | #[tokio::main] 41 | async fn main() -> anyhow::Result<()> { 42 | let args = Args::parse(); 43 | 44 | let client = LiveClient::builder() 45 | .addr((args.gateway.as_str(), args.port)) 46 | .await? 47 | .key(std::env::var(args.api_key_env_var.clone())?)? 48 | .dataset(args.dataset) 49 | .build() 50 | .await?; 51 | 52 | if args.use_snapshot { 53 | run_with_snapshot(args, client).await 54 | } else { 55 | run(args, client).await 56 | } 57 | } 58 | 59 | async fn run(args: Args, mut client: LiveClient) -> anyhow::Result<()> { 60 | let start = if let Some(start_time) = &args.start { 61 | if let Ok(ts) = start_time.parse::() { 62 | Some(OffsetDateTime::from_unix_timestamp_nanos(ts as i128)?) 63 | } else if let Ok(s) = OffsetDateTime::parse(start_time.as_str(), &Rfc3339) { 64 | Some(s) 65 | } else { 66 | return Err(anyhow::format_err!( 67 | "Timestamp {start_time} is neither nanoseconds from epoch nor in RFC 3339 format", 68 | )); 69 | } 70 | } else { 71 | None 72 | }; 73 | 74 | let builder = Subscription::builder() 75 | .schema(args.schema) 76 | .symbols(args.symbols.clone()) 77 | .stype_in(args.stype); 78 | 79 | let subscription = if let Some(s) = start { 80 | builder.start(s).build() 81 | } else { 82 | builder.build() 83 | }; 84 | 85 | client.subscribe(subscription).await?; 86 | 87 | // For start != 0 we stop at SymbolMappingMsg so that the tests can be run outside trading hours 88 | let expected_rtype: RType = if start 89 | .is_some_and(|s| s == OffsetDateTime::UNIX_EPOCH || args.stype == SType::InstrumentId) 90 | { 91 | args.schema.into() 92 | } else { 93 | RType::SymbolMapping 94 | }; 95 | 96 | client.start().await?; 97 | 98 | println!("Starting client...."); 99 | 100 | while let Some(record) = client.next_record().await? { 101 | if record.header().rtype == expected_rtype as u8 { 102 | println!("Received expected record {record:?}"); 103 | break; 104 | } else if let Some(msg) = record.get::() { 105 | // Unwrap because LSG should always return valid UTF-8 106 | panic!("Received error: {}", msg.err().unwrap()); 107 | } 108 | } 109 | 110 | client.close().await?; 111 | 112 | println!("Finished client"); 113 | 114 | Ok(()) 115 | } 116 | 117 | async fn run_with_snapshot(args: Args, mut client: LiveClient) -> anyhow::Result<()> { 118 | client 119 | .subscribe( 120 | Subscription::builder() 121 | .schema(args.schema) 122 | .symbols(args.symbols) 123 | .stype_in(args.stype) 124 | .use_snapshot() 125 | .build(), 126 | ) 127 | .await?; 128 | 129 | client.start().await?; 130 | 131 | println!("Starting client...."); 132 | 133 | let mut received_snapshot_record = false; 134 | 135 | while let Some(record) = client.next_record().await? { 136 | if let Some(msg) = record.get::() { 137 | if msg.flags.is_snapshot() { 138 | received_snapshot_record = true; 139 | } else { 140 | println!("Received expected record {record:?}"); 141 | break; 142 | } 143 | } else if let Some(msg) = record.get::() { 144 | // Unwrap because LSG should always return valid UTF-8 145 | panic!("Received error: {}", msg.err().unwrap()); 146 | } 147 | } 148 | 149 | client.close().await?; 150 | 151 | println!("Finished client"); 152 | 153 | assert!(received_snapshot_record, "Did not receive snapshot record"); 154 | 155 | Ok(()) 156 | } 157 | -------------------------------------------------------------------------------- /examples/split_symbols.rs: -------------------------------------------------------------------------------- 1 | //! An example program that splits a DBN file into several DBN files 2 | //! by parent symbol (from the `asset` field in the definitions schema). 3 | use std::collections::HashMap; 4 | 5 | use anyhow::Context; 6 | use async_compression::tokio::write::ZstdEncoder; 7 | use databento::{ 8 | dbn::{ 9 | decode::AsyncDbnDecoder, encode::AsyncDbnEncoder, InstrumentDefMsg, Metadata, Schema, 10 | SymbolIndex, 11 | }, 12 | historical::timeseries::GetRangeParams, 13 | HistoricalClient, 14 | }; 15 | use tokio::fs::File; 16 | 17 | #[tokio::main] 18 | async fn main() -> anyhow::Result<()> { 19 | if std::env::args().len() != 3 { 20 | anyhow::bail!( 21 | "Invalid number of arguments, expected: split_symbols FILE_PATH OUTPUT_PATTERN" 22 | ); 23 | } 24 | let file_path = std::env::args().nth(1).unwrap(); 25 | let output_pattern = std::env::args().nth(2).unwrap(); 26 | if !output_pattern.contains("{parent}") { 27 | anyhow::bail!("OUTPUT_PATTERN should contain {{parent}}"); 28 | } 29 | let mut decoder = AsyncDbnDecoder::from_zstd_file(file_path).await?; 30 | 31 | let metadata = decoder.metadata().clone(); 32 | let symbol_map = metadata.symbol_map()?; 33 | let symbols_to_parent = fetch_symbols_to_parent(&metadata).await?; 34 | let mut encoders = HashMap::>>::new(); 35 | while let Some(rec) = decoder.decode_record_ref().await? { 36 | let Some(symbol) = symbol_map.get_for_rec(&rec) else { 37 | eprintln!("Missing mapping for {rec:?}"); 38 | continue; 39 | }; 40 | let Some(parent) = symbols_to_parent.get(symbol) else { 41 | eprintln!("Couldn't find parent mapping for {symbol} with {rec:?}"); 42 | continue; 43 | }; 44 | if let Some(encoder) = encoders.get_mut(parent) { 45 | encoder.encode_record_ref(rec).await?; 46 | } else { 47 | let mut encoder = AsyncDbnEncoder::with_zstd( 48 | File::create_new(output_pattern.replace("{parent}", parent)) 49 | .await 50 | .with_context(|| format!("creating file for {parent}"))?, 51 | &metadata, 52 | ) 53 | .await?; 54 | encoder.encode_record_ref(rec).await?; 55 | encoders.insert(parent.clone(), encoder); 56 | }; 57 | } 58 | for (parent, encoder) in encoders { 59 | if let Err(e) = encoder.shutdown().await { 60 | eprintln!("Failed to shutdown encoder for {parent}: {e:?}"); 61 | } 62 | } 63 | 64 | Ok(()) 65 | } 66 | 67 | async fn fetch_symbols_to_parent(metadata: &Metadata) -> anyhow::Result> { 68 | let mut client = HistoricalClient::builder().key_from_env()?.build()?; 69 | let end = metadata.end().ok_or_else(|| { 70 | anyhow::format_err!("Missing end in metadata. This script is intended for historical data") 71 | })?; 72 | let mut res = HashMap::new(); 73 | // 2000 is the maximum number of symbols per request 74 | for chunk in metadata.symbols.chunks(2000) { 75 | let mut decoder = client 76 | .timeseries() 77 | .get_range( 78 | &GetRangeParams::builder() 79 | .dataset(metadata.dataset.clone()) 80 | .schema(Schema::Definition) 81 | .date_time_range((metadata.start(), end)) 82 | .symbols(Vec::from(chunk)) 83 | .build(), 84 | ) 85 | .await?; 86 | while let Some(def) = decoder.decode_record::().await? { 87 | res.insert(def.raw_symbol()?.to_owned(), def.asset()?.to_owned()); 88 | } 89 | } 90 | Ok(res) 91 | } 92 | -------------------------------------------------------------------------------- /scripts/build.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | set -e 3 | 4 | cargo --version 5 | echo build all 6 | cargo build --all-features 7 | echo build historical 8 | cargo build --no-default-features --features historical 9 | echo build live 10 | cargo build --no-default-features --features live 11 | echo build examples 12 | cargo build --examples 13 | -------------------------------------------------------------------------------- /scripts/format.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | cargo --version 3 | cargo fmt --check # fails if anything is misformatted 4 | -------------------------------------------------------------------------------- /scripts/get_version.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | SCRIPTS_DIR="$(cd "$(dirname "$0")" || exit; pwd -P)" 4 | PROJECT_ROOT_DIR="$(dirname "${SCRIPTS_DIR}")" 5 | grep -E '^version =' "${PROJECT_ROOT_DIR}/Cargo.toml" | cut -d'"' -f 2 6 | -------------------------------------------------------------------------------- /scripts/lint.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | set -e 3 | 4 | cargo --version 5 | cargo clippy --all-features -- --deny warnings 6 | # `cargo doc` does not have a `--deny warnings` flag like clippy, workaround from: 7 | # https://github.com/rust-lang/cargo/issues/8424#issuecomment-1070988443 8 | RUSTDOCFLAGS='--deny warnings' cargo doc --all-features 9 | -------------------------------------------------------------------------------- /scripts/test.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | set -e 3 | 4 | cargo --version 5 | cargo test --all-features 6 | -------------------------------------------------------------------------------- /src/error.rs: -------------------------------------------------------------------------------- 1 | //! Types for errors received from the API and occurring in the clients. 2 | use thiserror::Error; 3 | 4 | /// An error that can occur while working with Databento's API. 5 | #[derive(Debug, Error)] 6 | #[non_exhaustive] 7 | pub enum Error { 8 | /// An invalid argument was passed to a function. 9 | #[error("bad argument `{param_name}`: {desc}")] 10 | BadArgument { 11 | /// The name of the parameter to which the bad argument was passed. 12 | param_name: String, 13 | /// The description of why the argument was invalid. 14 | desc: String, 15 | }, 16 | /// An I/O error while reading or writing DBN or another encoding. 17 | #[error("I/O error: {0:?}")] 18 | Io(#[from] std::io::Error), 19 | /// An HTTP error. 20 | #[cfg(feature = "historical")] 21 | #[error("HTTP error: {0:?}")] 22 | Http(#[from] reqwest::Error), 23 | /// An error from the Databento API. 24 | #[cfg(feature = "historical")] 25 | #[error("API error: {0}")] 26 | Api(ApiError), 27 | /// An error internal to the client. 28 | #[error("internal error: {0}")] 29 | Internal(String), 30 | /// An error related to DBN encoding. 31 | #[error("DBN error: {0}")] 32 | Dbn(#[source] dbn::Error), 33 | /// An when authentication failed. 34 | #[error("authentication failed: {0}")] 35 | Auth(String), 36 | } 37 | /// An alias for a `Result` with [`databento::Error`](crate::Error) as the error type. 38 | pub type Result = std::result::Result; 39 | 40 | /// An error from the Databento API. 41 | #[cfg(feature = "historical")] 42 | #[derive(Debug)] 43 | pub struct ApiError { 44 | /// The request ID. 45 | pub request_id: Option, 46 | /// The HTTP status code of the response. 47 | pub status_code: reqwest::StatusCode, 48 | /// The message from the Databento API. 49 | pub message: String, 50 | /// The link to documentation related to the error. 51 | pub docs_url: Option, 52 | } 53 | 54 | impl Error { 55 | pub(crate) fn bad_arg(param_name: impl ToString, desc: impl ToString) -> Self { 56 | Self::BadArgument { 57 | param_name: param_name.to_string(), 58 | desc: desc.to_string(), 59 | } 60 | } 61 | 62 | pub(crate) fn internal(msg: impl ToString) -> Self { 63 | Self::Internal(msg.to_string()) 64 | } 65 | } 66 | 67 | impl From for Error { 68 | fn from(dbn_err: dbn::Error) -> Self { 69 | match dbn_err { 70 | // Convert to our own error type. 71 | dbn::Error::Io { source, .. } => Self::Io(source), 72 | dbn_err => Self::Dbn(dbn_err), 73 | } 74 | } 75 | } 76 | 77 | #[cfg(feature = "historical")] 78 | impl std::fmt::Display for ApiError { 79 | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { 80 | let doc = if let Some(ref docs_url) = self.docs_url { 81 | format!(" See {docs_url} for documentation.") 82 | } else { 83 | String::new() 84 | }; 85 | let status = self.status_code; 86 | let msg = &self.message; 87 | if let Some(ref request_id) = self.request_id { 88 | write!(f, "{request_id} failed with {status} {msg}{doc}") 89 | } else { 90 | write!(f, "{status} {msg}{doc}") 91 | } 92 | } 93 | } 94 | -------------------------------------------------------------------------------- /src/historical.rs: -------------------------------------------------------------------------------- 1 | //! Historical client and related API types. 2 | 3 | pub mod batch; 4 | mod client; 5 | mod deserialize; 6 | pub mod metadata; 7 | pub mod symbology; 8 | pub mod timeseries; 9 | 10 | pub use client::*; 11 | use time::{ 12 | format_description::BorrowedFormatItem, macros::format_description, Duration, Time, UtcOffset, 13 | }; 14 | 15 | use crate::{Error, Symbols}; 16 | 17 | /// The current Databento historical API version. 18 | pub const API_VERSION: u32 = 0; 19 | 20 | /// The Historical API gateway to use. 21 | #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] 22 | pub enum HistoricalGateway { 23 | /// The default gateway in Boston. 24 | #[default] 25 | Bo1, 26 | } 27 | 28 | /// A **half**-closed date interval with an inclusive start date and an exclusive end 29 | /// date. 30 | #[derive(Clone, Debug, PartialEq, Eq)] 31 | pub struct DateRange { 32 | /// The start date (inclusive). 33 | start: time::Date, 34 | /// The end date (exclusive). 35 | end: time::Date, 36 | } 37 | 38 | /// A **half**-closed datetime interval with an inclusive start time and an exclusive 39 | /// end time. 40 | #[derive(Clone, Debug, PartialEq, Eq)] 41 | pub struct DateTimeRange { 42 | /// The start date time (inclusive). 43 | start: time::OffsetDateTime, 44 | /// The end date time (exclusive). 45 | end: time::OffsetDateTime, 46 | } 47 | 48 | impl From<(time::Date, time::Date)> for DateRange { 49 | fn from(value: (time::Date, time::Date)) -> Self { 50 | Self { 51 | start: value.0, 52 | end: value.1, 53 | } 54 | } 55 | } 56 | 57 | impl From<(time::Date, time::Duration)> for DateRange { 58 | fn from(value: (time::Date, time::Duration)) -> Self { 59 | Self { 60 | start: value.0, 61 | end: value.0 + value.1, 62 | } 63 | } 64 | } 65 | 66 | impl From for DateRange { 67 | fn from(date: time::Date) -> Self { 68 | Self { 69 | start: date, 70 | end: date.next_day().unwrap(), 71 | } 72 | } 73 | } 74 | 75 | impl From for DateTimeRange { 76 | fn from(date: time::Date) -> Self { 77 | let start = date.with_time(Time::MIDNIGHT).assume_utc(); 78 | Self { 79 | start, 80 | end: start + Duration::DAY, 81 | } 82 | } 83 | } 84 | 85 | impl From for DateTimeRange { 86 | fn from(date_range: DateRange) -> Self { 87 | Self { 88 | start: date_range.start.with_time(Time::MIDNIGHT).assume_utc(), 89 | end: date_range.end.with_time(Time::MIDNIGHT).assume_utc(), 90 | } 91 | } 92 | } 93 | 94 | impl From for DateRange { 95 | fn from(dt_range: DateTimeRange) -> Self { 96 | let utc_end = dt_range.end.to_offset(UtcOffset::UTC); 97 | Self { 98 | start: dt_range.start.to_offset(UtcOffset::UTC).date(), 99 | // Round up end to nearest date 100 | end: if utc_end.time() == Time::MIDNIGHT { 101 | utc_end.date() 102 | } else { 103 | utc_end.date().next_day().unwrap() 104 | }, 105 | } 106 | } 107 | } 108 | 109 | pub(crate) const DATE_FORMAT: &[BorrowedFormatItem<'_>] = 110 | format_description!("[year]-[month]-[day]"); 111 | 112 | impl From<(time::OffsetDateTime, time::OffsetDateTime)> for DateTimeRange { 113 | fn from(value: (time::OffsetDateTime, time::OffsetDateTime)) -> Self { 114 | Self { 115 | start: value.0, 116 | end: value.1, 117 | } 118 | } 119 | } 120 | 121 | impl From<(time::OffsetDateTime, time::Duration)> for DateTimeRange { 122 | fn from(value: (time::OffsetDateTime, time::Duration)) -> Self { 123 | Self { 124 | start: value.0, 125 | end: value.0 + value.1, 126 | } 127 | } 128 | } 129 | 130 | impl TryFrom<(u64, u64)> for DateTimeRange { 131 | type Error = crate::Error; 132 | 133 | fn try_from(value: (u64, u64)) -> Result { 134 | let start = time::OffsetDateTime::from_unix_timestamp_nanos(value.0 as i128) 135 | .map_err(|e| Error::bad_arg("first UNIX nanos", format!("{e:?}")))?; 136 | let end = time::OffsetDateTime::from_unix_timestamp_nanos(value.1 as i128) 137 | .map_err(|e| Error::bad_arg("second UNIX nanos", format!("{e:?}")))?; 138 | Ok(Self { start, end }) 139 | } 140 | } 141 | 142 | trait AddToQuery { 143 | fn add_to_query(self, param: &T) -> Self; 144 | } 145 | 146 | impl AddToQuery for reqwest::RequestBuilder { 147 | fn add_to_query(self, param: &DateRange) -> Self { 148 | self.query(&[ 149 | ("start_date", param.start.format(DATE_FORMAT).unwrap()), 150 | ("end_date", param.end.format(DATE_FORMAT).unwrap()), 151 | ]) 152 | } 153 | } 154 | 155 | impl AddToQuery for reqwest::RequestBuilder { 156 | fn add_to_query(self, param: &DateTimeRange) -> Self { 157 | self.query(&[ 158 | ("start", param.start.unix_timestamp_nanos()), 159 | ("end", param.end.unix_timestamp_nanos()), 160 | ]) 161 | } 162 | } 163 | 164 | impl AddToQuery for reqwest::RequestBuilder { 165 | fn add_to_query(self, param: &Symbols) -> Self { 166 | self.query(&[("symbols", param.to_api_string())]) 167 | } 168 | } 169 | 170 | impl DateRange { 171 | pub(crate) fn add_to_form(&self, form: &mut Vec<(&'static str, String)>) { 172 | form.push(("start_date", self.start.format(DATE_FORMAT).unwrap())); 173 | form.push(("end_date", self.end.format(DATE_FORMAT).unwrap())); 174 | } 175 | } 176 | 177 | impl DateTimeRange { 178 | pub(crate) fn add_to_form(&self, form: &mut Vec<(&'static str, String)>) { 179 | form.push(("start", self.start.unix_timestamp_nanos().to_string())); 180 | form.push(("end", self.end.unix_timestamp_nanos().to_string())); 181 | } 182 | } 183 | 184 | #[cfg(test)] 185 | mod tests { 186 | use super::*; 187 | 188 | use time::macros::{date, datetime}; 189 | 190 | #[test] 191 | fn date_range_from_lt_day_duration() { 192 | let target = DateRange::from((date!(2024 - 02 - 16), time::Duration::SECOND)); 193 | assert_eq!( 194 | target, 195 | DateRange { 196 | start: date!(2024 - 02 - 16), 197 | end: date!(2024 - 02 - 16) 198 | } 199 | ) 200 | } 201 | 202 | #[test] 203 | fn single_date_conversion() { 204 | let date = date!(2025 - 03 - 27); 205 | assert_eq!( 206 | DateRange::from(date), 207 | DateRange::from((date!(2025 - 03 - 27), date!(2025 - 03 - 28))) 208 | ); 209 | assert_eq!( 210 | DateTimeRange::from(date), 211 | DateTimeRange::from(( 212 | datetime!(2025 - 03 - 27 00:00 UTC), 213 | datetime!(2025 - 03 - 28 00:00 UTC) 214 | )) 215 | ); 216 | } 217 | 218 | #[test] 219 | fn range_equivalency() { 220 | let date_range = DateRange::from((date!(2025 - 03 - 27), date!(2025 - 04 - 10))); 221 | assert_eq!( 222 | date_range, 223 | DateRange::from(DateTimeRange::from(date_range.clone())) 224 | ); 225 | } 226 | 227 | #[test] 228 | fn dt_offset_to_date_range() { 229 | assert_eq!( 230 | DateRange::from(DateTimeRange::from(( 231 | datetime!(2025-03-27 21:00 -4), 232 | datetime!(2025-03-28 20:00 -4) 233 | ))), 234 | DateRange::from((date!(2025 - 03 - 28), date!(2025 - 03 - 29))) 235 | ); 236 | assert_eq!( 237 | DateRange::from(DateTimeRange::from(( 238 | datetime!(2025-03-27 21:00 -4), 239 | datetime!(2025-03-28 20:30 -4) 240 | ))), 241 | DateRange::from((date!(2025 - 03 - 28), date!(2025 - 03 - 30))) 242 | ); 243 | } 244 | } 245 | -------------------------------------------------------------------------------- /src/historical/batch.rs: -------------------------------------------------------------------------------- 1 | //! The historical batch download API. 2 | 3 | use core::fmt; 4 | use std::{ 5 | collections::HashMap, 6 | fmt::Write, 7 | num::NonZeroU64, 8 | path::{Path, PathBuf}, 9 | str::FromStr, 10 | }; 11 | 12 | use dbn::{Compression, Encoding, SType, Schema}; 13 | use futures::StreamExt; 14 | use reqwest::RequestBuilder; 15 | use serde::{de, Deserialize, Deserializer}; 16 | use time::OffsetDateTime; 17 | use tokio::io::BufWriter; 18 | use tracing::info; 19 | use typed_builder::TypedBuilder; 20 | 21 | use crate::{historical::check_http_error, Error, Symbols}; 22 | 23 | use super::{ 24 | deserialize::{deserialize_date_time, deserialize_opt_date_time}, 25 | handle_response, DateTimeRange, 26 | }; 27 | 28 | /// A client for the batch group of Historical API endpoints. 29 | #[derive(Debug)] 30 | pub struct BatchClient<'a> { 31 | pub(crate) inner: &'a mut super::Client, 32 | } 33 | 34 | impl BatchClient<'_> { 35 | /// Submits a new batch job and returns a description and identifiers for the job. 36 | /// 37 | ///
38 | /// Calling this method will incur a cost. 39 | ///
40 | /// 41 | /// # Errors 42 | /// This function returns an error when it fails to communicate with the Databento API 43 | /// or the API indicates there's an issue with the request. 44 | pub async fn submit_job(&mut self, params: &SubmitJobParams) -> crate::Result { 45 | let mut form = vec![ 46 | ("dataset", params.dataset.to_string()), 47 | ("schema", params.schema.to_string()), 48 | ("encoding", params.encoding.to_string()), 49 | ("compression", params.compression.to_string()), 50 | ("pretty_px", params.pretty_px.to_string()), 51 | ("pretty_ts", params.pretty_ts.to_string()), 52 | ("map_symbols", params.map_symbols.to_string()), 53 | ("split_symbols", params.split_symbols.to_string()), 54 | ("split_duration", params.split_duration.to_string()), 55 | ("delivery", params.delivery.to_string()), 56 | ("stype_in", params.stype_in.to_string()), 57 | ("stype_out", params.stype_out.to_string()), 58 | ("symbols", params.symbols.to_api_string()), 59 | ]; 60 | params.date_time_range.add_to_form(&mut form); 61 | if let Some(split_size) = params.split_size { 62 | form.push(("split_size", split_size.to_string())); 63 | } 64 | if let Some(limit) = params.limit { 65 | form.push(("limit", limit.to_string())); 66 | } 67 | let builder = self.post("submit_job")?.form(&form); 68 | let resp = builder.send().await?; 69 | handle_response(resp).await 70 | } 71 | 72 | /// Lists previous batch jobs with filtering by `params`. 73 | /// 74 | /// # Errors 75 | /// This function returns an error when it fails to communicate with the Databento API 76 | /// or the API indicates there's an issue with the request. 77 | pub async fn list_jobs(&mut self, params: &ListJobsParams) -> crate::Result> { 78 | let mut builder = self.get("list_jobs")?; 79 | if let Some(ref states) = params.states { 80 | let states_str = states.iter().fold(String::new(), |mut acc, s| { 81 | if acc.is_empty() { 82 | s.as_str().to_owned() 83 | } else { 84 | write!(acc, ",{}", s.as_str()).unwrap(); 85 | acc 86 | } 87 | }); 88 | builder = builder.query(&[("states", states_str)]); 89 | } 90 | if let Some(ref since) = params.since { 91 | builder = builder.query(&[("since", &since.unix_timestamp_nanos().to_string())]); 92 | } 93 | let resp = builder.send().await?; 94 | handle_response(resp).await 95 | } 96 | 97 | /// Lists all files associated with the batch job with ID `job_id`. 98 | /// 99 | /// # Errors 100 | /// This function returns an error when it fails to communicate with the Databento API 101 | /// or the API indicates there's an issue with the request. 102 | pub async fn list_files(&mut self, job_id: &str) -> crate::Result> { 103 | let resp = self 104 | .get("list_files")? 105 | .query(&[("job_id", job_id)]) 106 | .send() 107 | .await?; 108 | handle_response(resp).await 109 | } 110 | 111 | /// Downloads the file specified in `params` or all files associated with the job ID. 112 | /// 113 | /// # Errors 114 | /// This function returns an error when it fails to communicate with the Databento API 115 | /// or the API indicates there's an issue with the request. It will also return an 116 | /// error if it encounters an issue downloading a file. 117 | pub async fn download(&mut self, params: &DownloadParams) -> crate::Result> { 118 | let job_dir = params.output_dir.join(¶ms.job_id); 119 | if job_dir.exists() { 120 | if !job_dir.is_dir() { 121 | return Err(Error::bad_arg( 122 | "output_dir", 123 | "exists but is not a directory", 124 | )); 125 | } 126 | } else { 127 | tokio::fs::create_dir_all(&job_dir).await?; 128 | } 129 | let job_files = self.list_files(¶ms.job_id).await?; 130 | if let Some(filename_to_download) = params.filename_to_download.as_ref() { 131 | let Some(file_desc) = job_files 132 | .iter() 133 | .find(|file| file.filename == *filename_to_download) 134 | else { 135 | return Err(Error::bad_arg( 136 | "filename_to_download", 137 | "not found for batch job", 138 | )); 139 | }; 140 | let output_path = job_dir.join(filename_to_download); 141 | let https_url = file_desc 142 | .urls 143 | .get("https") 144 | .ok_or_else(|| Error::internal("Missing https URL for batch file"))?; 145 | self.download_file(https_url, &output_path).await?; 146 | Ok(vec![output_path]) 147 | } else { 148 | let mut paths = Vec::new(); 149 | for file_desc in job_files.iter() { 150 | let output_path = params 151 | .output_dir 152 | .join(¶ms.job_id) 153 | .join(&file_desc.filename); 154 | let https_url = file_desc 155 | .urls 156 | .get("https") 157 | .ok_or_else(|| Error::internal("Missing https URL for batch file"))?; 158 | self.download_file(https_url, &output_path).await?; 159 | paths.push(output_path); 160 | } 161 | Ok(paths) 162 | } 163 | } 164 | 165 | async fn download_file(&mut self, url: &str, path: impl AsRef) -> crate::Result<()> { 166 | let url = reqwest::Url::parse(url) 167 | .map_err(|e| Error::internal(format!("Unable to parse URL: {e:?}")))?; 168 | let resp = self.inner.get_with_path(url.path())?.send().await?; 169 | let mut stream = check_http_error(resp).await?.bytes_stream(); 170 | info!(%url, path=%path.as_ref().display(), "Downloading file"); 171 | let mut output = BufWriter::new( 172 | tokio::fs::OpenOptions::new() 173 | .create(true) 174 | .truncate(true) 175 | .write(true) 176 | .open(path) 177 | .await?, 178 | ); 179 | while let Some(chunk) = stream.next().await { 180 | tokio::io::copy(&mut chunk?.as_ref(), &mut output).await?; 181 | } 182 | Ok(()) 183 | } 184 | 185 | const PATH_PREFIX: &'static str = "batch"; 186 | 187 | fn get(&mut self, slug: &str) -> crate::Result { 188 | self.inner.get(&format!("{}.{slug}", Self::PATH_PREFIX)) 189 | } 190 | 191 | fn post(&mut self, slug: &str) -> crate::Result { 192 | self.inner.post(&format!("{}.{slug}", Self::PATH_PREFIX)) 193 | } 194 | } 195 | 196 | /// The duration of time at which batch files will be split. 197 | #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] 198 | pub enum SplitDuration { 199 | /// One file per day. 200 | #[default] 201 | Day, 202 | /// One file per week. A week starts on Sunday UTC. 203 | Week, 204 | /// One file per month. 205 | Month, 206 | } 207 | 208 | /// How the batch job will be delivered. 209 | #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] 210 | pub enum Delivery { 211 | /// Via download from the Databento portal. 212 | #[default] 213 | Download, 214 | /// Via Amazon S3. 215 | S3, 216 | /// Via disk. 217 | Disk, 218 | } 219 | 220 | /// The state of a batch job. 221 | #[derive(Clone, Copy, Debug, PartialEq, Eq)] 222 | pub enum JobState { 223 | /// The job has been received (the initial state). 224 | Received, 225 | /// The job has been queued for processing. 226 | Queued, 227 | /// The job has begun processing. 228 | Processing, 229 | /// The job has finished processing and is ready for delivery. 230 | Done, 231 | /// The job is no longer available. 232 | Expired, 233 | } 234 | 235 | /// The parameters for [`BatchClient::submit_job()`]. Use [`SubmitJobParams::builder()`] to 236 | /// get a builder type with all the preset defaults. 237 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 238 | pub struct SubmitJobParams { 239 | /// The dataset code. 240 | #[builder(setter(transform = |dt: impl ToString| dt.to_string()))] 241 | pub dataset: String, 242 | /// The symbols to filter for. 243 | #[builder(setter(into))] 244 | pub symbols: Symbols, 245 | /// The data record schema. 246 | pub schema: Schema, 247 | /// The date time request range. 248 | /// Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. 249 | #[builder(setter(into))] 250 | pub date_time_range: DateTimeRange, 251 | /// The data encoding. Defaults to [`Dbn`](Encoding::Dbn). 252 | #[builder(default = Encoding::Dbn)] 253 | pub encoding: Encoding, 254 | /// The data compression mode. Defaults to [`ZStd`](Compression::ZStd). 255 | #[builder(default = Compression::ZStd)] 256 | pub compression: Compression, 257 | /// If `true`, prices will be formatted to the correct scale (using the fixed- 258 | /// precision scalar 1e-9). Only valid for [`Encoding::Csv`] and [`Encoding::Json`]. 259 | #[builder(default)] 260 | pub pretty_px: bool, 261 | /// If `true`, timestamps will be formatted as ISO 8601 strings. Only valid for 262 | /// [`Encoding::Csv`] and [`Encoding::Json`]. 263 | #[builder(default)] 264 | pub pretty_ts: bool, 265 | /// If `true`, a symbol field will be included with each text-encoded 266 | /// record, reducing the need to look at the `symbology.json`. Only valid for 267 | /// [`Encoding::Csv`] and [`Encoding::Json`]. 268 | #[builder(default)] 269 | pub map_symbols: bool, 270 | /// If `true`, files will be split by raw symbol. Cannot be requested with [`Symbols::All`]. 271 | #[builder(default)] 272 | pub split_symbols: bool, 273 | /// The maximum time duration before batched data is split into multiple files. 274 | /// Defaults to [`Day`](SplitDuration::Day). 275 | #[builder(default)] 276 | pub split_duration: SplitDuration, 277 | /// The optional maximum size (in bytes) of each batched data file before being split. 278 | /// Must be an integer between 1e9 and 10e9 inclusive (1GB - 10GB). Defaults to `None`. 279 | #[builder(default, setter(strip_option))] 280 | pub split_size: Option, 281 | /// The delivery mechanism for the batched data files once processed. Defaults to 282 | /// [`Download`](Delivery::Download). 283 | #[builder(default)] 284 | pub delivery: Delivery, 285 | /// The symbology type of the input `symbols`. Defaults to 286 | /// [`RawSymbol`](dbn::enums::SType::RawSymbol). 287 | #[builder(default = SType::RawSymbol)] 288 | pub stype_in: SType, 289 | /// The symbology type of the output `symbols`. Defaults to 290 | /// [`InstrumentId`](dbn::enums::SType::InstrumentId). 291 | #[builder(default = SType::InstrumentId)] 292 | pub stype_out: SType, 293 | /// The optional maximum number of records to return. Defaults to no limit. 294 | #[builder(default)] 295 | pub limit: Option, 296 | } 297 | 298 | /// The description of a submitted batch job. 299 | #[derive(Debug, Clone, Deserialize)] 300 | pub struct BatchJob { 301 | /// The unique job ID. 302 | pub id: String, 303 | /// The user ID of the user who submitted the job. 304 | pub user_id: Option, 305 | /// The bill ID (for internal use). 306 | pub bill_id: Option, 307 | /// The cost of the job in US dollars. Will be `None` until the job is processed. 308 | pub cost_usd: Option, 309 | /// The dataset code. 310 | pub dataset: String, 311 | /// The list of symbols specified in the request. 312 | pub symbols: Symbols, 313 | /// The symbology type of the input `symbols`. 314 | pub stype_in: SType, 315 | /// The symbology type of the output `symbols`. 316 | pub stype_out: SType, 317 | /// The data record schema. 318 | pub schema: Schema, 319 | /// The start of the request time range (inclusive). 320 | #[serde(deserialize_with = "deserialize_date_time")] 321 | pub start: OffsetDateTime, 322 | /// The end of the request time range (exclusive). 323 | #[serde(deserialize_with = "deserialize_date_time")] 324 | pub end: OffsetDateTime, 325 | /// The maximum number of records to return. 326 | pub limit: Option, 327 | /// The data encoding. 328 | pub encoding: Encoding, 329 | /// The data compression mode. 330 | #[serde(deserialize_with = "deserialize_compression")] 331 | pub compression: Compression, 332 | /// If prices are formatted to the correct scale (using the fixed-precision scalar 1e-9). 333 | pub pretty_px: bool, 334 | /// If timestamps are formatted as ISO 8601 strings. 335 | pub pretty_ts: bool, 336 | /// If a symbol field is included with each text-encoded record. 337 | pub map_symbols: bool, 338 | /// If files are split by raw symbol. 339 | pub split_symbols: bool, 340 | /// The maximum time interval for an individual file before splitting into multiple 341 | /// files. 342 | pub split_duration: SplitDuration, 343 | /// The maximum size for an individual file before splitting into multiple files. 344 | pub split_size: Option, 345 | /// The delivery mechanism of the batch data. 346 | pub delivery: Delivery, 347 | /// The number of data records (`None` until the job is processed). 348 | pub record_count: Option, 349 | /// The size of the raw binary data used to process the batch job (used for billing purposes). 350 | pub billed_size: Option, 351 | /// The total size of the result of the batch job after splitting and compression. 352 | pub actual_size: Option, 353 | /// The total size of the result of the batch job after any packaging (including metadata). 354 | pub package_size: Option, 355 | /// The current status of the batch job. 356 | pub state: JobState, 357 | /// The timestamp of when Databento received the batch job. 358 | #[serde(deserialize_with = "deserialize_date_time")] 359 | pub ts_received: OffsetDateTime, 360 | /// The timestamp of when the batch job was queued. 361 | #[serde(deserialize_with = "deserialize_opt_date_time")] 362 | pub ts_queued: Option, 363 | /// The timestamp of when the batch job began processing. 364 | #[serde(deserialize_with = "deserialize_opt_date_time")] 365 | pub ts_process_start: Option, 366 | /// The timestamp of when the batch job finished processing. 367 | #[serde(deserialize_with = "deserialize_opt_date_time")] 368 | pub ts_process_done: Option, 369 | /// The timestamp of when the batch job will expire from the Download center. 370 | #[serde(deserialize_with = "deserialize_opt_date_time")] 371 | pub ts_expiration: Option, 372 | } 373 | 374 | /// The parameters for [`BatchClient::list_jobs()`]. Use [`ListJobsParams::builder()`] to 375 | /// get a builder type with all the preset defaults. 376 | #[derive(Debug, Clone, Default, TypedBuilder, PartialEq, Eq)] 377 | pub struct ListJobsParams { 378 | /// The optional filter for job states. 379 | #[builder(default, setter(strip_option))] 380 | pub states: Option>, 381 | /// The optional filter for timestamp submitted (will not include jobs prior to 382 | /// this time). 383 | #[builder(default, setter(strip_option))] 384 | pub since: Option, 385 | } 386 | 387 | /// The file details for a batch job. 388 | #[derive(Debug, Clone, Deserialize)] 389 | pub struct BatchFileDesc { 390 | /// The file name. 391 | pub filename: String, 392 | /// The size of the file in bytes. 393 | pub size: u64, 394 | /// The SHA256 hash of the file. 395 | pub hash: String, 396 | /// A map of download protocol to URL. 397 | pub urls: HashMap, 398 | } 399 | 400 | /// The parameters for [`BatchClient::download()`]. Use [`DownloadParams::builder()`] to 401 | /// get a builder type with all the preset defaults. 402 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 403 | pub struct DownloadParams { 404 | /// The directory to download the file(s) to. 405 | #[builder(setter(transform = |dt: impl Into| dt.into()))] 406 | pub output_dir: PathBuf, 407 | /// The batch job identifier. 408 | #[builder(setter(transform = |dt: impl ToString| dt.to_string()))] 409 | pub job_id: String, 410 | /// `None` means all files associated with the job will be downloaded. 411 | #[builder(default, setter(strip_option))] 412 | pub filename_to_download: Option, 413 | } 414 | 415 | impl SplitDuration { 416 | /// Converts the enum to its `str` representation. 417 | pub const fn as_str(&self) -> &'static str { 418 | match self { 419 | SplitDuration::Day => "day", 420 | SplitDuration::Week => "week", 421 | SplitDuration::Month => "month", 422 | } 423 | } 424 | } 425 | 426 | impl fmt::Display for SplitDuration { 427 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { 428 | f.write_str(self.as_str()) 429 | } 430 | } 431 | 432 | impl FromStr for SplitDuration { 433 | type Err = crate::Error; 434 | 435 | fn from_str(s: &str) -> Result { 436 | match s { 437 | "day" => Ok(SplitDuration::Day), 438 | "week" => Ok(SplitDuration::Week), 439 | "month" => Ok(SplitDuration::Month), 440 | _ => Err(crate::Error::bad_arg( 441 | "s", 442 | format!( 443 | "{s} does not correspond with any {} variant", 444 | std::any::type_name::() 445 | ), 446 | )), 447 | } 448 | } 449 | } 450 | 451 | impl<'de> Deserialize<'de> for SplitDuration { 452 | fn deserialize>(deserializer: D) -> Result { 453 | let str = String::deserialize(deserializer)?; 454 | FromStr::from_str(&str).map_err(de::Error::custom) 455 | } 456 | } 457 | 458 | impl Delivery { 459 | /// Converts the enum to its `str` representation. 460 | pub const fn as_str(&self) -> &'static str { 461 | match self { 462 | Delivery::Download => "download", 463 | Delivery::S3 => "s3", 464 | Delivery::Disk => "disk", 465 | } 466 | } 467 | } 468 | 469 | impl fmt::Display for Delivery { 470 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { 471 | f.write_str(self.as_str()) 472 | } 473 | } 474 | 475 | impl FromStr for Delivery { 476 | type Err = crate::Error; 477 | 478 | fn from_str(s: &str) -> Result { 479 | match s { 480 | "download" => Ok(Delivery::Download), 481 | "s3" => Ok(Delivery::S3), 482 | "disk" => Ok(Delivery::Disk), 483 | _ => Err(crate::Error::bad_arg( 484 | "s", 485 | format!( 486 | "{s} does not correspond with any {} variant", 487 | std::any::type_name::() 488 | ), 489 | )), 490 | } 491 | } 492 | } 493 | 494 | impl<'de> Deserialize<'de> for Delivery { 495 | fn deserialize>(deserializer: D) -> Result { 496 | let str = String::deserialize(deserializer)?; 497 | FromStr::from_str(&str).map_err(de::Error::custom) 498 | } 499 | } 500 | 501 | impl JobState { 502 | /// Converts the enum to its `str` representation. 503 | pub const fn as_str(&self) -> &'static str { 504 | match self { 505 | JobState::Received => "received", 506 | JobState::Queued => "queued", 507 | JobState::Processing => "processing", 508 | JobState::Done => "done", 509 | JobState::Expired => "expired", 510 | } 511 | } 512 | } 513 | 514 | impl fmt::Display for JobState { 515 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { 516 | f.write_str(self.as_str()) 517 | } 518 | } 519 | 520 | impl FromStr for JobState { 521 | type Err = crate::Error; 522 | 523 | fn from_str(s: &str) -> Result { 524 | match s { 525 | "received" => Ok(JobState::Received), 526 | "queued" => Ok(JobState::Queued), 527 | "processing" => Ok(JobState::Processing), 528 | "done" => Ok(JobState::Done), 529 | "expired" => Ok(JobState::Expired), 530 | _ => Err(crate::Error::bad_arg( 531 | "s", 532 | format!( 533 | "{s} does not correspond with any {} variant", 534 | std::any::type_name::() 535 | ), 536 | )), 537 | } 538 | } 539 | } 540 | 541 | impl<'de> Deserialize<'de> for JobState { 542 | fn deserialize>(deserializer: D) -> Result { 543 | let str = String::deserialize(deserializer)?; 544 | FromStr::from_str(&str).map_err(de::Error::custom) 545 | } 546 | } 547 | 548 | // Handles Compression::None being serialized as null in JSON 549 | fn deserialize_compression<'de, D: serde::Deserializer<'de>>( 550 | deserializer: D, 551 | ) -> Result { 552 | let opt = Option::::deserialize(deserializer)?; 553 | Ok(opt.unwrap_or(Compression::None)) 554 | } 555 | 556 | #[cfg(test)] 557 | mod tests { 558 | use reqwest::StatusCode; 559 | use serde_json::json; 560 | use time::macros::datetime; 561 | use wiremock::{ 562 | matchers::{basic_auth, method, path, query_param_is_missing}, 563 | Mock, MockServer, ResponseTemplate, 564 | }; 565 | 566 | use super::*; 567 | use crate::{ 568 | body_contains, 569 | historical::{HistoricalGateway, API_VERSION}, 570 | HistoricalClient, 571 | }; 572 | 573 | const API_KEY: &str = "test-batch"; 574 | 575 | #[tokio::test] 576 | async fn test_submit_job() -> crate::Result<()> { 577 | const START: time::OffsetDateTime = datetime!(2023 - 06 - 14 00:00 UTC); 578 | const END: time::OffsetDateTime = datetime!(2023 - 06 - 17 00:00 UTC); 579 | const SCHEMA: Schema = Schema::Trades; 580 | 581 | let mock_server = MockServer::start().await; 582 | Mock::given(method("POST")) 583 | .and(basic_auth(API_KEY, "")) 584 | .and(path(format!("/v{API_VERSION}/batch.submit_job"))) 585 | .and(body_contains("dataset", "XNAS.ITCH")) 586 | .and(body_contains("schema", "trades")) 587 | .and(body_contains("symbols", "TSLA")) 588 | .and(body_contains( 589 | "start", 590 | START.unix_timestamp_nanos().to_string(), 591 | )) 592 | .and(body_contains("encoding", "dbn")) 593 | .and(body_contains("compression", "zstd")) 594 | .and(body_contains("map_symbols", "false")) 595 | .and(body_contains("end", END.unix_timestamp_nanos().to_string())) 596 | // // default 597 | .and(body_contains("stype_in", "raw_symbol")) 598 | .and(body_contains("stype_out", "instrument_id")) 599 | .respond_with( 600 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!({ 601 | "id": "123", 602 | "user_id": "test_user", 603 | "bill_id": "345", 604 | "cost_usd": 10.50, 605 | "dataset": "XNAS.ITCH", 606 | "symbols": ["TSLA"], 607 | "stype_in": "raw_symbol", 608 | "stype_out": "instrument_id", 609 | "schema": SCHEMA.as_str(), 610 | "start": "2023-06-14T00:00:00.000000000Z", 611 | "end": "2023-06-17 00:00:00.000000+00:00", 612 | "limit": null, 613 | "encoding": "dbn", 614 | "compression": "zstd", 615 | "pretty_px": false, 616 | "pretty_ts": false, 617 | "map_symbols": false, 618 | "split_symbols": false, 619 | "split_duration": "day", 620 | "split_size": null, 621 | "delivery": "download", 622 | "state": "queued", 623 | "ts_received": "2023-07-19T23:00:04.095538123Z", 624 | "ts_queued": null, 625 | "ts_process_start": null, 626 | "ts_process_done": null, 627 | "ts_expiration": null 628 | })), 629 | ) 630 | .mount(&mock_server) 631 | .await; 632 | let mut target = HistoricalClient::with_url( 633 | mock_server.uri(), 634 | API_KEY.to_owned(), 635 | HistoricalGateway::Bo1, 636 | )?; 637 | let job_desc = target 638 | .batch() 639 | .submit_job( 640 | &SubmitJobParams::builder() 641 | .dataset(dbn::Dataset::XnasItch) 642 | .schema(SCHEMA) 643 | .symbols("TSLA") 644 | .date_time_range((START, END)) 645 | .build(), 646 | ) 647 | .await?; 648 | assert_eq!(job_desc.dataset, dbn::Dataset::XnasItch.as_str()); 649 | Ok(()) 650 | } 651 | 652 | #[tokio::test] 653 | async fn test_list_jobs() -> crate::Result<()> { 654 | const SCHEMA: Schema = Schema::Trades; 655 | 656 | let mock_server = MockServer::start().await; 657 | Mock::given(method("GET")) 658 | .and(basic_auth(API_KEY, "")) 659 | .and(path(format!("/v{API_VERSION}/batch.list_jobs"))) 660 | .and(query_param_is_missing("states")) 661 | .and(query_param_is_missing("since")) 662 | .respond_with( 663 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!([{ 664 | "id": "123", 665 | "user_id": "test_user", 666 | "bill_id": "345", 667 | "cost_usd": 10.50, 668 | "dataset": "XNAS.ITCH", 669 | "symbols": "TSLA", 670 | "stype_in": "raw_symbol", 671 | "stype_out": "instrument_id", 672 | "schema": SCHEMA.as_str(), 673 | // test both time formats 674 | "start": "2023-06-14 00:00:00+00:00", 675 | "end": "2023-06-17T00:00:00.012345678Z", 676 | "limit": null, 677 | "encoding": "json", 678 | "compression": "zstd", 679 | "pretty_px": true, 680 | "pretty_ts": false, 681 | "map_symbols": true, 682 | "split_symbols": false, 683 | "split_duration": "day", 684 | "split_size": null, 685 | "delivery": "download", 686 | "state": "processing", 687 | "ts_received": "2023-07-19 23:00:04.095538+00:00", 688 | "ts_queued": "2023-07-19T23:00:08.095538123Z", 689 | "ts_process_start": "2023-07-19 23:01:04.000000+00:00", 690 | "ts_process_done": null, 691 | "ts_expiration": null 692 | }])), 693 | ) 694 | .mount(&mock_server) 695 | .await; 696 | let mut target = HistoricalClient::with_url( 697 | mock_server.uri(), 698 | API_KEY.to_owned(), 699 | HistoricalGateway::Bo1, 700 | )?; 701 | let job_descs = target.batch().list_jobs(&ListJobsParams::default()).await?; 702 | assert_eq!(job_descs.len(), 1); 703 | let job_desc = &job_descs[0]; 704 | assert_eq!( 705 | job_desc.ts_queued.unwrap(), 706 | datetime!(2023-07-19 23:00:08.095538123 UTC) 707 | ); 708 | assert_eq!( 709 | job_desc.ts_process_start.unwrap(), 710 | datetime!(2023-07-19 23:01:04 UTC) 711 | ); 712 | assert_eq!(job_desc.encoding, Encoding::Json); 713 | assert!(job_desc.pretty_px); 714 | assert!(!job_desc.pretty_ts); 715 | assert!(job_desc.map_symbols); 716 | Ok(()) 717 | } 718 | 719 | #[test] 720 | fn test_deserialize_compression() { 721 | #[derive(serde::Deserialize)] 722 | struct Test { 723 | #[serde(deserialize_with = "deserialize_compression")] 724 | compression: Compression, 725 | } 726 | 727 | const JSON: &str = 728 | r#"[{"compression":null}, {"compression":"none"}, {"compression":"zstd"}]"#; 729 | let res: Vec = serde_json::from_str(JSON).unwrap(); 730 | assert_eq!( 731 | res.into_iter().map(|t| t.compression).collect::>(), 732 | vec![Compression::None, Compression::None, Compression::ZStd] 733 | ); 734 | } 735 | } 736 | -------------------------------------------------------------------------------- /src/historical/client.rs: -------------------------------------------------------------------------------- 1 | use reqwest::{header::ACCEPT, IntoUrl, RequestBuilder, Url}; 2 | use serde::Deserialize; 3 | use tracing::warn; 4 | 5 | use crate::{error::ApiError, ApiKey, Error}; 6 | 7 | use super::{ 8 | batch::BatchClient, metadata::MetadataClient, symbology::SymbologyClient, 9 | timeseries::TimeseriesClient, HistoricalGateway, API_VERSION, 10 | }; 11 | 12 | /// The Historical client. Used for symbology resolutions, metadata requests, Historical 13 | /// data older than 24 hours, and submitting batch downloads. 14 | /// 15 | /// Use [`HistoricalClient::builder()`](Client::builder) to get a type-safe builder for 16 | /// initializing the required parameters for the client. 17 | /// 18 | /// individual API methods are accessed through its four subclients: 19 | /// - [`metadata()`](Self::metadata) 20 | /// - [`timeseries()`](Self::timeseries) 21 | /// - [`symbology()`](Self::symbology) 22 | /// - [`batch()`](Self::batch) 23 | #[derive(Debug, Clone)] 24 | pub struct Client { 25 | key: ApiKey, 26 | base_url: Url, 27 | gateway: HistoricalGateway, 28 | client: reqwest::Client, 29 | } 30 | 31 | #[derive(Debug, Deserialize)] 32 | #[serde(untagged)] 33 | pub(crate) enum ApiErrorResponse { 34 | Simple { detail: String }, 35 | Business { detail: BusinessErrorDetails }, 36 | } 37 | 38 | #[derive(Debug, Deserialize)] 39 | pub(crate) struct BusinessErrorDetails { 40 | message: String, 41 | docs: String, 42 | } 43 | 44 | const USER_AGENT: &str = concat!("Databento/", env!("CARGO_PKG_VERSION"), " Rust"); 45 | const WARNING_HEADER: &str = "X-Warning"; 46 | const REQUEST_ID_HEADER: &str = "request-id"; 47 | 48 | impl Client { 49 | /// Returns a type-safe builder for setting the required parameters 50 | /// for initializing a [`HistoricalClient`](Client). 51 | pub fn builder() -> ClientBuilder { 52 | ClientBuilder::default() 53 | } 54 | 55 | /// Creates a new client with the given API key. 56 | /// 57 | /// # Errors 58 | /// This function returns an error when it fails to build the HTTP client. 59 | pub fn new(key: String, gateway: HistoricalGateway) -> crate::Result { 60 | let url = match gateway { 61 | HistoricalGateway::Bo1 => "https://hist.databento.com", 62 | }; 63 | Self::with_url(url, key, gateway) 64 | } 65 | 66 | /// Creates a new client with a specific API URL. This is an advanced method and 67 | /// [`builder()`](Self::builder) or [`new()`](Self::new) should be used instead. 68 | /// 69 | /// # Errors 70 | /// This function returns an error when the `url` is invalid. 71 | pub fn with_url( 72 | url: impl IntoUrl, 73 | key: String, 74 | gateway: HistoricalGateway, 75 | ) -> crate::Result { 76 | let base_url = url 77 | .into_url() 78 | .map_err(|e| Error::bad_arg("url", format!("{e:?}")))?; 79 | let mut headers = reqwest::header::HeaderMap::new(); 80 | headers.insert(ACCEPT, "application/json".parse().unwrap()); 81 | Ok(Self { 82 | key: ApiKey(key), 83 | base_url, 84 | gateway, 85 | client: reqwest::ClientBuilder::new() 86 | .user_agent(USER_AGENT) 87 | .default_headers(headers) 88 | .build()?, 89 | }) 90 | } 91 | 92 | /// Returns the API key used by the instance of the client. 93 | pub fn key(&self) -> &str { 94 | &self.key.0 95 | } 96 | 97 | /// Returns the configured Historical gateway. 98 | pub fn gateway(&self) -> HistoricalGateway { 99 | self.gateway 100 | } 101 | 102 | /// Returns the batch subclient. 103 | pub fn batch(&mut self) -> BatchClient { 104 | BatchClient { inner: self } 105 | } 106 | 107 | /// Returns the metadata subclient. 108 | pub fn metadata(&mut self) -> MetadataClient { 109 | MetadataClient { inner: self } 110 | } 111 | 112 | /// Returns the symbology subclient. 113 | pub fn symbology(&mut self) -> SymbologyClient { 114 | SymbologyClient { inner: self } 115 | } 116 | 117 | /// Returns the timeseries subclient. 118 | pub fn timeseries(&mut self) -> TimeseriesClient { 119 | TimeseriesClient { inner: self } 120 | } 121 | 122 | pub(crate) fn get(&mut self, slug: &str) -> crate::Result { 123 | self.request(reqwest::Method::GET, slug) 124 | } 125 | 126 | pub(crate) fn get_with_path(&mut self, path: &str) -> crate::Result { 127 | Ok(self 128 | .client 129 | .get( 130 | self.base_url 131 | .join(path) 132 | .map_err(|e| Error::Internal(format!("created invalid URL: {e:?}")))?, 133 | ) 134 | .basic_auth(self.key(), Option::<&str>::None)) 135 | } 136 | 137 | pub(crate) fn post(&mut self, slug: &str) -> crate::Result { 138 | self.request(reqwest::Method::POST, slug) 139 | } 140 | 141 | fn request(&mut self, method: reqwest::Method, slug: &str) -> crate::Result { 142 | Ok(self 143 | .client 144 | .request( 145 | method, 146 | self.base_url 147 | .join(&format!("v{API_VERSION}/{slug}")) 148 | .map_err(|e| Error::Internal(format!("created invalid URL: {e:?}")))?, 149 | ) 150 | .basic_auth(self.key(), Option::<&str>::None)) 151 | } 152 | } 153 | 154 | pub(crate) async fn check_http_error( 155 | response: reqwest::Response, 156 | ) -> crate::Result { 157 | if response.status().is_success() { 158 | Ok(response) 159 | } else { 160 | let request_id = response 161 | .headers() 162 | .get(REQUEST_ID_HEADER) 163 | .and_then(|header| header.to_str().ok().map(ToOwned::to_owned)); 164 | let status_code = response.status(); 165 | let body = response.text().await.unwrap_or_default(); 166 | let err = match serde_json::from_str::(&body) { 167 | Ok(ApiErrorResponse::Simple { detail: message }) => ApiError { 168 | request_id, 169 | status_code, 170 | message, 171 | docs_url: None, 172 | }, 173 | Ok(ApiErrorResponse::Business { detail }) => ApiError { 174 | request_id, 175 | status_code, 176 | message: detail.message, 177 | docs_url: Some(detail.docs), 178 | }, 179 | Err(e) => { 180 | warn!("Failed to deserialize error response to expected JSON format: {e:?}"); 181 | ApiError { 182 | request_id, 183 | status_code, 184 | message: body, 185 | docs_url: None, 186 | } 187 | } 188 | }; 189 | Err(Error::Api(err)) 190 | } 191 | } 192 | 193 | pub(crate) async fn handle_response( 194 | response: reqwest::Response, 195 | ) -> crate::Result { 196 | check_warnings(&response); 197 | let response = check_http_error(response).await?; 198 | Ok(response.json::().await?) 199 | } 200 | 201 | fn check_warnings(response: &reqwest::Response) { 202 | if let Some(header) = response.headers().get(WARNING_HEADER) { 203 | match serde_json::from_slice::>(header.as_bytes()) { 204 | Ok(warnings) => { 205 | for warning in warnings { 206 | warn!("{warning}"); 207 | } 208 | } 209 | Err(err) => { 210 | warn!(?err, "Failed to parse server warnings from HTTP header"); 211 | } 212 | }; 213 | }; 214 | } 215 | 216 | #[doc(hidden)] 217 | #[derive(Debug, Copy, Clone)] 218 | pub struct Unset; 219 | 220 | /// A type-safe builder for the [`HistoricalClient`](Client). It will not allow you to 221 | /// call [`Self::build()`] before setting the required `key` field. 222 | #[derive(Clone)] 223 | pub struct ClientBuilder { 224 | key: AK, 225 | base_url: Option, 226 | gateway: HistoricalGateway, 227 | } 228 | 229 | impl Default for ClientBuilder { 230 | fn default() -> Self { 231 | Self { 232 | key: Unset, 233 | base_url: None, 234 | gateway: HistoricalGateway::default(), 235 | } 236 | } 237 | } 238 | 239 | impl ClientBuilder { 240 | /// Overrides the base URL to be used for the Historical API. Normally this is 241 | /// derived from the gateway. 242 | pub fn base_url(mut self, url: Url) -> Self { 243 | self.base_url = Some(url); 244 | self 245 | } 246 | 247 | /// Sets the historical gateway to use. 248 | pub fn gateway(mut self, gateway: HistoricalGateway) -> Self { 249 | self.gateway = gateway; 250 | self 251 | } 252 | } 253 | 254 | impl ClientBuilder { 255 | /// Creates a new [`ClientBuilder`]. 256 | pub fn new() -> Self { 257 | Self::default() 258 | } 259 | 260 | /// Sets the API key. 261 | /// 262 | /// # Errors 263 | /// This function returns an error when the API key is invalid. 264 | pub fn key(self, key: impl ToString) -> crate::Result> { 265 | Ok(ClientBuilder { 266 | key: ApiKey::new(key.to_string())?, 267 | base_url: self.base_url, 268 | gateway: self.gateway, 269 | }) 270 | } 271 | 272 | /// Sets the API key reading it from the `DATABENTO_API_KEY` environment 273 | /// variable. 274 | /// 275 | /// # Errors 276 | /// This function returns an error when the environment variable is not set or the 277 | /// API key is invalid. 278 | pub fn key_from_env(self) -> crate::Result> { 279 | let key = crate::key_from_env()?; 280 | self.key(key) 281 | } 282 | } 283 | 284 | impl ClientBuilder { 285 | /// Initializes the client. 286 | /// 287 | /// # Errors 288 | /// This function returns an error when it fails to build the HTTP client. 289 | pub fn build(self) -> crate::Result { 290 | if let Some(url) = self.base_url { 291 | Client::with_url(url, self.key.0, self.gateway) 292 | } else { 293 | Client::new(self.key.0, self.gateway) 294 | } 295 | } 296 | } 297 | 298 | #[cfg(test)] 299 | mod tests { 300 | use reqwest::StatusCode; 301 | use wiremock::{matchers::method, Mock, MockServer, ResponseTemplate}; 302 | 303 | use super::*; 304 | 305 | #[tokio::test] 306 | async fn check_http_error_non_json() { 307 | const BODY: &str = "

502 Bad Gateway

308 | The server returned an invalid or incomplete response. 309 | "; 310 | let mock_server = MockServer::start().await; 311 | 312 | Mock::given(method("GET")) 313 | .respond_with( 314 | ResponseTemplate::new(StatusCode::BAD_GATEWAY.as_u16()).set_body_string(BODY), 315 | ) 316 | .mount(&mock_server) 317 | .await; 318 | let resp = reqwest::get(mock_server.uri()).await.unwrap(); 319 | let err = check_http_error(resp).await.unwrap_err(); 320 | assert!( 321 | matches!(err, Error::Api(api_err) if api_err.status_code == StatusCode::BAD_GATEWAY && api_err.message == BODY && api_err.docs_url.is_none()) 322 | ); 323 | } 324 | } 325 | -------------------------------------------------------------------------------- /src/historical/deserialize.rs: -------------------------------------------------------------------------------- 1 | //! Custom deserializers 2 | use serde::Deserialize; 3 | use time::format_description::well_known::iso8601::Iso8601; 4 | 5 | const LEGACY_DATE_TIME_FORMAT: &[time::format_description::BorrowedFormatItem<'static>] = 6 | time::macros::format_description!("[year]-[month]-[day] [hour]:[minute]:[second][optional [.[subsecond digits:6]]][optional [+[offset_hour]:[offset_minute]]]"); 7 | 8 | pub(crate) fn deserialize_date_time<'de, D: serde::Deserializer<'de>>( 9 | deserializer: D, 10 | ) -> Result { 11 | let dt_str = String::deserialize(deserializer)?; 12 | time::PrimitiveDateTime::parse(&dt_str, &Iso8601::DEFAULT) 13 | .map(|dt| dt.assume_utc()) 14 | .or_else(|_| time::OffsetDateTime::parse(&dt_str, LEGACY_DATE_TIME_FORMAT)) 15 | .map_err(serde::de::Error::custom) 16 | } 17 | 18 | pub(crate) fn deserialize_opt_date_time<'de, D: serde::Deserializer<'de>>( 19 | deserializer: D, 20 | ) -> Result, D::Error> { 21 | if let Some(dt_str) = Option::::deserialize(deserializer)? { 22 | time::PrimitiveDateTime::parse(&dt_str, &Iso8601::DEFAULT) 23 | .map(|dt| dt.assume_utc()) 24 | .or_else(|_| time::OffsetDateTime::parse(&dt_str, LEGACY_DATE_TIME_FORMAT)) 25 | .map(Some) 26 | .map_err(serde::de::Error::custom) 27 | } else { 28 | Ok(None) 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /src/historical/metadata.rs: -------------------------------------------------------------------------------- 1 | //! The historical metadata download API. 2 | 3 | use std::{collections::HashMap, num::NonZeroU64, str::FromStr}; 4 | 5 | use dbn::{Encoding, SType, Schema}; 6 | use reqwest::RequestBuilder; 7 | use serde::{Deserialize, Deserializer}; 8 | use typed_builder::TypedBuilder; 9 | 10 | use crate::Symbols; 11 | 12 | use super::{ 13 | deserialize::deserialize_date_time, handle_response, AddToQuery, DateRange, DateTimeRange, 14 | }; 15 | 16 | /// A client for the metadata group of Historical API endpoints. 17 | #[derive(Debug)] 18 | pub struct MetadataClient<'a> { 19 | pub(crate) inner: &'a mut super::Client, 20 | } 21 | 22 | impl MetadataClient<'_> { 23 | /// Lists the details of all publishers. 24 | /// 25 | /// # Errors 26 | /// This function returns an error when it fails to communicate with the Databento API. 27 | pub async fn list_publishers(&mut self) -> crate::Result> { 28 | let resp = self.get("list_publishers")?.send().await?; 29 | handle_response(resp).await 30 | } 31 | 32 | /// Lists all available dataset codes on Databento. 33 | /// 34 | /// # Errors 35 | /// This function returns an error when it fails to communicate with the Databento API 36 | /// or the API indicates there's an issue with the request. 37 | pub async fn list_datasets( 38 | &mut self, 39 | date_range: Option, 40 | ) -> crate::Result> { 41 | let mut builder = self.get("list_datasets")?; 42 | if let Some(date_range) = date_range { 43 | builder = builder.add_to_query(&date_range); 44 | } 45 | let resp = builder.send().await?; 46 | handle_response(resp).await 47 | } 48 | 49 | /// Lists all available schemas for the given `dataset`. 50 | /// 51 | /// # Errors 52 | /// This function returns an error when it fails to communicate with the Databento API 53 | /// or the API indicates there's an issue with the request. 54 | pub async fn list_schemas(&mut self, dataset: &str) -> crate::Result> { 55 | let resp = self 56 | .get("list_schemas")? 57 | .query(&[("dataset", dataset)]) 58 | .send() 59 | .await?; 60 | handle_response(resp).await 61 | } 62 | 63 | /// Lists all fields for a schema and encoding. 64 | /// 65 | /// # Errors 66 | /// This function returns an error when it fails to communicate with the Databento API 67 | /// or the API indicates there's an issue with the request. 68 | pub async fn list_fields( 69 | &mut self, 70 | params: &ListFieldsParams, 71 | ) -> crate::Result> { 72 | let builder = self.get("list_fields")?.query(&[ 73 | ("encoding", params.encoding.as_str()), 74 | ("schema", params.schema.as_str()), 75 | ]); 76 | let resp = builder.send().await?; 77 | handle_response(resp).await 78 | } 79 | 80 | /// Lists unit prices for each data schema and feed mode in US dollars per gigabyte. 81 | /// 82 | /// # Errors 83 | /// This function returns an error when it fails to communicate with the Databento API 84 | /// or the API indicates there's an issue with the request. 85 | pub async fn list_unit_prices( 86 | &mut self, 87 | dataset: &str, 88 | ) -> crate::Result> { 89 | let builder = self 90 | .get("list_unit_prices")? 91 | .query(&[("dataset", &dataset)]); 92 | let resp = builder.send().await?; 93 | handle_response(resp).await 94 | } 95 | 96 | /// Gets the dataset condition from Databento. 97 | /// 98 | /// Use this method to discover data availability and quality. 99 | /// 100 | /// # Errors 101 | /// This function returns an error when it fails to communicate with the Databento API 102 | /// or the API indicates there's an issue with the request. 103 | pub async fn get_dataset_condition( 104 | &mut self, 105 | params: &GetDatasetConditionParams, 106 | ) -> crate::Result> { 107 | let mut builder = self 108 | .get("get_dataset_condition")? 109 | .query(&[("dataset", ¶ms.dataset)]); 110 | if let Some(ref date_range) = params.date_range { 111 | builder = builder.add_to_query(date_range); 112 | } 113 | let resp = builder.send().await?; 114 | handle_response(resp).await 115 | } 116 | 117 | /// Gets the available range for the dataset given the user's entitlements. 118 | /// 119 | /// Use this method to discover data availability. 120 | /// 121 | /// # Errors 122 | /// This function returns an error when it fails to communicate with the Databento API 123 | /// or the API indicates there's an issue with the request. 124 | pub async fn get_dataset_range(&mut self, dataset: &str) -> crate::Result { 125 | let resp = self 126 | .get("get_dataset_range")? 127 | .query(&[("dataset", dataset)]) 128 | .send() 129 | .await?; 130 | handle_response(resp).await 131 | } 132 | 133 | /// Gets the record count of the time series data query. 134 | /// 135 | /// # Errors 136 | /// This function returns an error when it fails to communicate with the Databento API 137 | /// or the API indicates there's an issue with the request. 138 | pub async fn get_record_count(&mut self, params: &GetRecordCountParams) -> crate::Result { 139 | let mut form = Vec::new(); 140 | params.add_to_form(&mut form); 141 | let resp = self.post("get_record_count")?.form(&form).send().await?; 142 | handle_response(resp).await 143 | } 144 | 145 | /// Gets the billable uncompressed raw binary size for historical streaming or 146 | /// batched files. 147 | /// 148 | /// # Errors 149 | /// This function returns an error when it fails to communicate with the Databento API 150 | /// or the API indicates there's an issue with the request. 151 | pub async fn get_billable_size( 152 | &mut self, 153 | params: &GetBillableSizeParams, 154 | ) -> crate::Result { 155 | let mut form = Vec::new(); 156 | params.add_to_form(&mut form); 157 | let resp = self.post("get_billable_size")?.form(&form).send().await?; 158 | handle_response(resp).await 159 | } 160 | 161 | /// Gets the cost in US dollars for a historical streaming or batch download 162 | /// request. This cost respects any discounts provided by flat rate plans. 163 | /// 164 | /// # Errors 165 | /// This function returns an error when it fails to communicate with the Databento API 166 | /// or the API indicates there's an issue with the request. 167 | pub async fn get_cost(&mut self, params: &GetCostParams) -> crate::Result { 168 | let mut form = Vec::new(); 169 | params.add_to_form(&mut form); 170 | let resp = self.post("get_cost")?.form(&form).send().await?; 171 | handle_response(resp).await 172 | } 173 | 174 | fn get(&mut self, slug: &str) -> crate::Result { 175 | self.inner.get(&format!("metadata.{slug}")) 176 | } 177 | 178 | fn post(&mut self, slug: &str) -> crate::Result { 179 | self.inner.post(&format!("metadata.{slug}")) 180 | } 181 | } 182 | 183 | /// A type of data feed. 184 | #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] 185 | pub enum FeedMode { 186 | /// The historical batch data feed. 187 | Historical, 188 | /// The historical streaming data feed. 189 | HistoricalStreaming, 190 | /// The Live data feed for real-time and intraday historical. 191 | Live, 192 | } 193 | 194 | /// The condition of a dataset on a day. 195 | #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] 196 | pub enum DatasetCondition { 197 | /// The data is available with no known issues. 198 | Available, 199 | /// The data is available, but there may be missing data or other correctness 200 | /// issues. 201 | Degraded, 202 | /// The data is not yet available, but may be available soon. 203 | Pending, 204 | /// The data is not available. 205 | Missing, 206 | /// The data is available intraday, which may have different licensing. 207 | Intraday, 208 | } 209 | 210 | /// The details about a publisher. 211 | #[derive(Debug, Clone, Deserialize, PartialEq, Eq)] 212 | pub struct PublisherDetail { 213 | /// The publisher ID assigned by Databento, which denotes the dataset and venue. 214 | pub publisher_id: u16, 215 | /// The dataset code for the publisher. 216 | pub dataset: String, 217 | /// The venue for the publisher. 218 | pub venue: String, 219 | /// The publisher description. 220 | pub description: String, 221 | } 222 | 223 | /// The parameters for [`MetadataClient::list_fields()`]. Use 224 | /// [`ListFieldsParams::builder()`] to get a builder type with all the preset defaults. 225 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 226 | pub struct ListFieldsParams { 227 | /// The encoding to request fields for. 228 | pub encoding: Encoding, 229 | /// The data record schema to request fields for. 230 | pub schema: Schema, 231 | } 232 | 233 | /// The details about a field in a schema. 234 | #[derive(Debug, Clone, Deserialize, PartialEq, Eq)] 235 | pub struct FieldDetail { 236 | /// The field name. 237 | pub name: String, 238 | /// The field type name. 239 | #[serde(rename = "type")] 240 | pub type_name: String, 241 | } 242 | 243 | /// The unit prices for a particular [`FeedMode`]. 244 | #[derive(Debug, Clone, Deserialize, PartialEq)] 245 | pub struct UnitPricesForMode { 246 | /// The data feed mode. 247 | pub mode: FeedMode, 248 | /// The unit prices in US dollars by data record schema. 249 | pub unit_prices: HashMap, 250 | } 251 | 252 | /// The parameters for [`MetadataClient::get_dataset_condition()`]. Use 253 | /// [`GetDatasetConditionParams::builder()`] to get a builder type with all the preset 254 | /// defaults. 255 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 256 | pub struct GetDatasetConditionParams { 257 | /// The dataset code. 258 | #[builder(setter(transform = |dataset: impl ToString| dataset.to_string()))] 259 | pub dataset: String, 260 | /// The optional filter by UTC date range. 261 | #[builder(default, setter(transform = |dr: impl Into| Some(dr.into())))] 262 | pub date_range: Option, 263 | } 264 | 265 | /// The condition of a dataset on a particular day. 266 | #[derive(Debug, Clone, Deserialize, PartialEq, Eq)] 267 | pub struct DatasetConditionDetail { 268 | /// The day of the described data. 269 | #[serde(deserialize_with = "deserialize_date")] 270 | pub date: time::Date, 271 | /// The condition code describing the quality and availability of the data on the 272 | /// given day. 273 | pub condition: DatasetCondition, 274 | /// The date when any schemna in the dataset on the given day was last generated or 275 | /// modified. 276 | #[serde(deserialize_with = "deserialize_date")] 277 | pub last_modified_date: time::Date, 278 | } 279 | 280 | /// The available range for a dataset. 281 | #[derive(Debug, Clone, PartialEq, Eq)] 282 | pub struct DatasetRange { 283 | /// The start of the available range. 284 | pub start: time::OffsetDateTime, 285 | /// The end of the available range (exclusive). 286 | pub end: time::OffsetDateTime, 287 | } 288 | 289 | impl From for DateTimeRange { 290 | fn from(DatasetRange { start, end }: DatasetRange) -> Self { 291 | Self { start, end } 292 | } 293 | } 294 | 295 | impl<'de> Deserialize<'de> for DatasetRange { 296 | fn deserialize(deserializer: D) -> std::result::Result 297 | where 298 | D: Deserializer<'de>, 299 | { 300 | #[derive(Deserialize)] 301 | struct Helper { 302 | #[serde(deserialize_with = "deserialize_date_time")] 303 | start: time::OffsetDateTime, 304 | #[serde(deserialize_with = "deserialize_date_time")] 305 | end: time::OffsetDateTime, 306 | } 307 | let partial = Helper::deserialize(deserializer)?; 308 | 309 | Ok(DatasetRange { 310 | start: partial.start, 311 | end: partial.end, 312 | }) 313 | } 314 | } 315 | 316 | /// The parameters for several metadata requests. 317 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 318 | pub struct GetQueryParams { 319 | /// The dataset code. 320 | #[builder(setter(transform = |dataset: impl ToString| dataset.to_string()))] 321 | pub dataset: String, 322 | /// The symbols to filter for. 323 | #[builder(setter(into))] 324 | pub symbols: Symbols, 325 | /// The data record schema. 326 | pub schema: Schema, 327 | /// The request time range. 328 | #[builder(setter(into))] 329 | pub date_time_range: DateTimeRange, 330 | /// The symbology type of the input `symbols`. Defaults to 331 | /// [`RawSymbol`](dbn::enums::SType::RawSymbol). 332 | #[builder(default = SType::RawSymbol)] 333 | pub stype_in: SType, 334 | /// The optional maximum number of records to return. Defaults to no limit. 335 | #[builder(default)] 336 | pub limit: Option, 337 | } 338 | 339 | /// The parameters for [`MetadataClient::get_record_count()`]. Use 340 | /// [`GetRecordCountParams::builder()`] to get a builder type with all the preset 341 | /// defaults. 342 | pub type GetRecordCountParams = GetQueryParams; 343 | /// The parameters for [`MetadataClient::get_billable_size()`]. Use 344 | /// [`GetBillableSizeParams::builder()`] to get a builder type with all the preset 345 | /// defaults. 346 | pub type GetBillableSizeParams = GetQueryParams; 347 | /// The parameters for [`MetadataClient::get_cost()`]. Use 348 | /// [`GetCostParams::builder()`] to get a builder type with all the preset 349 | /// defaults. 350 | pub type GetCostParams = GetQueryParams; 351 | 352 | impl AsRef for FeedMode { 353 | fn as_ref(&self) -> &str { 354 | self.as_str() 355 | } 356 | } 357 | 358 | impl FeedMode { 359 | /// Converts the enum to its `str` representation. 360 | pub const fn as_str(&self) -> &'static str { 361 | match self { 362 | FeedMode::Historical => "historical", 363 | FeedMode::HistoricalStreaming => "historical-streaming", 364 | FeedMode::Live => "live", 365 | } 366 | } 367 | } 368 | 369 | impl FromStr for FeedMode { 370 | type Err = crate::Error; 371 | 372 | fn from_str(s: &str) -> Result { 373 | match s { 374 | "historical" => Ok(Self::Historical), 375 | "historical-streaming" => Ok(Self::HistoricalStreaming), 376 | "live" => Ok(Self::Live), 377 | _ => Err(crate::Error::internal(format_args!( 378 | "Unabled to convert {s} to FeedMode" 379 | ))), 380 | } 381 | } 382 | } 383 | 384 | impl<'de> Deserialize<'de> for FeedMode { 385 | fn deserialize>(deserializer: D) -> Result { 386 | let str = String::deserialize(deserializer)?; 387 | FromStr::from_str(&str).map_err(serde::de::Error::custom) 388 | } 389 | } 390 | 391 | impl AsRef for DatasetCondition { 392 | fn as_ref(&self) -> &str { 393 | self.as_str() 394 | } 395 | } 396 | 397 | impl DatasetCondition { 398 | /// Converts the enum to its `str` representation. 399 | pub const fn as_str(&self) -> &'static str { 400 | match self { 401 | DatasetCondition::Available => "available", 402 | DatasetCondition::Degraded => "degraded", 403 | DatasetCondition::Pending => "pending", 404 | DatasetCondition::Missing => "missing", 405 | DatasetCondition::Intraday => "intraday", 406 | } 407 | } 408 | } 409 | 410 | impl FromStr for DatasetCondition { 411 | type Err = crate::Error; 412 | 413 | fn from_str(s: &str) -> Result { 414 | match s { 415 | "available" => Ok(DatasetCondition::Available), 416 | "degraded" => Ok(DatasetCondition::Degraded), 417 | "pending" => Ok(DatasetCondition::Pending), 418 | "missing" => Ok(DatasetCondition::Missing), 419 | "intraday" => Ok(DatasetCondition::Intraday), 420 | _ => Err(crate::Error::internal(format_args!( 421 | "Unabled to convert {s} to DatasetCondition" 422 | ))), 423 | } 424 | } 425 | } 426 | 427 | impl<'de> Deserialize<'de> for DatasetCondition { 428 | fn deserialize>(deserializer: D) -> Result { 429 | let str = String::deserialize(deserializer)?; 430 | FromStr::from_str(&str).map_err(serde::de::Error::custom) 431 | } 432 | } 433 | 434 | fn deserialize_date<'de, D: serde::Deserializer<'de>>( 435 | deserializer: D, 436 | ) -> Result { 437 | let dt_str = String::deserialize(deserializer)?; 438 | time::Date::parse(&dt_str, super::DATE_FORMAT).map_err(serde::de::Error::custom) 439 | } 440 | impl GetQueryParams { 441 | fn add_to_form(&self, form: &mut Vec<(&'static str, String)>) { 442 | form.push(("dataset", self.dataset.to_string())); 443 | form.push(("schema", self.schema.to_string())); 444 | form.push(("stype_in", self.stype_in.to_string())); 445 | form.push(("symbols", self.symbols.to_api_string())); 446 | self.date_time_range.add_to_form(form); 447 | if let Some(limit) = self.limit { 448 | form.push(("limit", limit.get().to_string())) 449 | } 450 | } 451 | } 452 | 453 | #[cfg(test)] 454 | mod tests { 455 | use reqwest::StatusCode; 456 | use serde_json::json; 457 | use time::macros::{date, datetime}; 458 | use wiremock::{ 459 | matchers::{basic_auth, method, path, query_param}, 460 | Mock, MockServer, ResponseTemplate, 461 | }; 462 | 463 | use super::*; 464 | use crate::{ 465 | historical::{HistoricalGateway, API_VERSION}, 466 | HistoricalClient, 467 | }; 468 | 469 | const API_KEY: &str = "test-metadata"; 470 | 471 | #[tokio::test] 472 | async fn test_list_fields() { 473 | const ENC: Encoding = Encoding::Csv; 474 | const SCHEMA: Schema = Schema::Ohlcv1S; 475 | let mock_server = MockServer::start().await; 476 | Mock::given(method("GET")) 477 | .and(basic_auth(API_KEY, "")) 478 | .and(path(format!("/v{API_VERSION}/metadata.list_fields"))) 479 | .and(query_param("encoding", ENC.as_str())) 480 | .and(query_param("schema", SCHEMA.as_str())) 481 | .respond_with( 482 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!([ 483 | {"name":"ts_event", "type": "uint64_t"}, 484 | {"name":"rtype", "type": "uint8_t"}, 485 | {"name":"open", "type": "int64_t"}, 486 | {"name":"high", "type": "int64_t"}, 487 | {"name":"low", "type": "int64_t"}, 488 | {"name":"close", "type": "int64_t"}, 489 | {"name":"volume", "type": "uint64_t"}, 490 | ])), 491 | ) 492 | .mount(&mock_server) 493 | .await; 494 | let mut target = HistoricalClient::with_url( 495 | mock_server.uri(), 496 | API_KEY.to_owned(), 497 | HistoricalGateway::Bo1, 498 | ) 499 | .unwrap(); 500 | let fields = target 501 | .metadata() 502 | .list_fields( 503 | &ListFieldsParams::builder() 504 | .encoding(ENC) 505 | .schema(SCHEMA) 506 | .build(), 507 | ) 508 | .await 509 | .unwrap(); 510 | let exp = vec![ 511 | FieldDetail { 512 | name: "ts_event".to_owned(), 513 | type_name: "uint64_t".to_owned(), 514 | }, 515 | FieldDetail { 516 | name: "rtype".to_owned(), 517 | type_name: "uint8_t".to_owned(), 518 | }, 519 | FieldDetail { 520 | name: "open".to_owned(), 521 | type_name: "int64_t".to_owned(), 522 | }, 523 | FieldDetail { 524 | name: "high".to_owned(), 525 | type_name: "int64_t".to_owned(), 526 | }, 527 | FieldDetail { 528 | name: "low".to_owned(), 529 | type_name: "int64_t".to_owned(), 530 | }, 531 | FieldDetail { 532 | name: "close".to_owned(), 533 | type_name: "int64_t".to_owned(), 534 | }, 535 | FieldDetail { 536 | name: "volume".to_owned(), 537 | type_name: "uint64_t".to_owned(), 538 | }, 539 | ]; 540 | assert_eq!(*fields, exp); 541 | } 542 | 543 | #[tokio::test] 544 | async fn test_list_unit_prices() { 545 | const SCHEMA: Schema = Schema::Tbbo; 546 | const DATASET: &str = "GLBX.MDP3"; 547 | let mock_server = MockServer::start().await; 548 | Mock::given(method("GET")) 549 | .and(basic_auth(API_KEY, "")) 550 | .and(path(format!("/v{API_VERSION}/metadata.list_unit_prices"))) 551 | .and(query_param("dataset", DATASET)) 552 | .respond_with( 553 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!([ 554 | { 555 | "mode": "historical", 556 | "unit_prices": { 557 | SCHEMA.as_str(): 17.89 558 | } 559 | }, 560 | { 561 | "mode": "live", 562 | "unit_prices": { 563 | SCHEMA.as_str(): 34.22 564 | } 565 | } 566 | ])), 567 | ) 568 | .mount(&mock_server) 569 | .await; 570 | let mut target = HistoricalClient::with_url( 571 | mock_server.uri(), 572 | API_KEY.to_owned(), 573 | HistoricalGateway::Bo1, 574 | ) 575 | .unwrap(); 576 | let prices = target.metadata().list_unit_prices(DATASET).await.unwrap(); 577 | assert_eq!( 578 | prices, 579 | vec![ 580 | UnitPricesForMode { 581 | mode: FeedMode::Historical, 582 | unit_prices: HashMap::from([(SCHEMA, 17.89)]) 583 | }, 584 | UnitPricesForMode { 585 | mode: FeedMode::Live, 586 | unit_prices: HashMap::from([(SCHEMA, 34.22)]) 587 | } 588 | ] 589 | ); 590 | } 591 | 592 | #[tokio::test] 593 | async fn test_get_dataset_condition() { 594 | const DATASET: &str = "GLBX.MDP3"; 595 | let mock_server = MockServer::start().await; 596 | Mock::given(method("GET")) 597 | .and(basic_auth(API_KEY, "")) 598 | .and(path(format!( 599 | "/v{API_VERSION}/metadata.get_dataset_condition" 600 | ))) 601 | .and(query_param("dataset", DATASET)) 602 | .and(query_param("start_date", "2022-05-17")) 603 | .and(query_param("end_date", "2022-05-18")) 604 | .respond_with( 605 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!([ 606 | { 607 | "date": "2022-05-17", 608 | "condition": "available", 609 | "last_modified_date": "2023-07-11", 610 | }, 611 | { 612 | "date": "2022-05-18", 613 | "condition": "degraded", 614 | "last_modified_date": "2022-05-19", 615 | } 616 | ])), 617 | ) 618 | .mount(&mock_server) 619 | .await; 620 | let mut target = HistoricalClient::with_url( 621 | mock_server.uri(), 622 | API_KEY.to_owned(), 623 | HistoricalGateway::Bo1, 624 | ) 625 | .unwrap(); 626 | let condition = target 627 | .metadata() 628 | .get_dataset_condition( 629 | &GetDatasetConditionParams::builder() 630 | .dataset(DATASET.to_owned()) 631 | .date_range((date!(2022 - 05 - 17), time::Duration::DAY)) 632 | .build(), 633 | ) 634 | .await 635 | .unwrap(); 636 | assert_eq!(condition.len(), 2); 637 | assert_eq!( 638 | condition[0], 639 | DatasetConditionDetail { 640 | date: date!(2022 - 05 - 17), 641 | condition: DatasetCondition::Available, 642 | last_modified_date: date!(2023 - 07 - 11) 643 | } 644 | ); 645 | assert_eq!( 646 | condition[1], 647 | DatasetConditionDetail { 648 | date: date!(2022 - 05 - 18), 649 | condition: DatasetCondition::Degraded, 650 | last_modified_date: date!(2022 - 05 - 19) 651 | } 652 | ); 653 | } 654 | 655 | #[tokio::test] 656 | async fn test_get_dataset_range() { 657 | const DATASET: &str = "XNAS.ITCH"; 658 | let mock_server = MockServer::start().await; 659 | Mock::given(method("GET")) 660 | .and(basic_auth(API_KEY, "")) 661 | .and(path(format!("/v{API_VERSION}/metadata.get_dataset_range"))) 662 | .and(query_param("dataset", DATASET)) 663 | .respond_with( 664 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!({ 665 | "start": "2019-07-07T00:00:00.000000000Z", 666 | // test both time formats 667 | "end": "2023-07-20T00:00:00.000000000Z", 668 | })), 669 | ) 670 | .mount(&mock_server) 671 | .await; 672 | let mut target = HistoricalClient::with_url( 673 | mock_server.uri(), 674 | API_KEY.to_owned(), 675 | HistoricalGateway::Bo1, 676 | ) 677 | .unwrap(); 678 | let range = target.metadata().get_dataset_range(DATASET).await.unwrap(); 679 | assert_eq!(range.start, datetime!(2019 - 07 - 07 00:00:00+00:00)); 680 | assert_eq!(range.end, datetime!(2023 - 07 - 20 00:00:00.000000+00:00)); 681 | } 682 | 683 | #[tokio::test] 684 | async fn test_get_dataset_range_no_dates() { 685 | const DATASET: &str = "XNAS.ITCH"; 686 | let mock_server = MockServer::start().await; 687 | Mock::given(method("GET")) 688 | .and(basic_auth(API_KEY, "")) 689 | .and(path(format!("/v{API_VERSION}/metadata.get_dataset_range"))) 690 | .and(query_param("dataset", DATASET)) 691 | .respond_with( 692 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!({ 693 | "start": "2019-07-07T00:00:00.000000000Z", 694 | "end": "2023-07-20T00:00:00.000000000Z", 695 | })), 696 | ) 697 | .mount(&mock_server) 698 | .await; 699 | let mut target = HistoricalClient::with_url( 700 | mock_server.uri(), 701 | API_KEY.to_owned(), 702 | HistoricalGateway::Bo1, 703 | ) 704 | .unwrap(); 705 | let range = target.metadata().get_dataset_range(DATASET).await.unwrap(); 706 | assert_eq!(range.start, datetime!(2019 - 07 - 07 00:00:00+00:00)); 707 | assert_eq!(range.end, datetime!(2023 - 07 - 20 00:00:00.000000+00:00)); 708 | } 709 | } 710 | -------------------------------------------------------------------------------- /src/historical/symbology.rs: -------------------------------------------------------------------------------- 1 | //! The historical symbology API. 2 | 3 | use std::{collections::HashMap, sync::Arc}; 4 | 5 | use dbn::{MappingInterval, Metadata, SType, TsSymbolMap}; 6 | use reqwest::RequestBuilder; 7 | use serde::Deserialize; 8 | use typed_builder::TypedBuilder; 9 | 10 | use crate::Symbols; 11 | 12 | use super::{handle_response, timeseries, DateRange, DateTimeRange}; 13 | 14 | /// A client for the symbology group of Historical API endpoints. 15 | #[derive(Debug)] 16 | pub struct SymbologyClient<'a> { 17 | pub(crate) inner: &'a mut super::Client, 18 | } 19 | 20 | impl SymbologyClient<'_> { 21 | /// Resolves a list of symbols from an input symbology type to an output one. 22 | /// 23 | /// For example, resolves a raw symbol to an instrument ID: `ESM2` → `3403`. 24 | /// 25 | /// # Errors 26 | /// This function returns an error when it fails to communicate with the Databento API 27 | /// or the API indicates there's an issue with the request. 28 | pub async fn resolve(&mut self, params: &ResolveParams) -> crate::Result { 29 | let mut form = vec![ 30 | ("dataset", params.dataset.to_string()), 31 | ("stype_in", params.stype_in.to_string()), 32 | ("stype_out", params.stype_out.to_string()), 33 | ("symbols", params.symbols.to_api_string()), 34 | ]; 35 | params.date_range.add_to_form(&mut form); 36 | let resp = self.post("resolve")?.form(&form).send().await?; 37 | let ResolutionResp { 38 | mappings, 39 | partial, 40 | not_found, 41 | } = handle_response(resp).await?; 42 | Ok(Resolution { 43 | mappings, 44 | partial, 45 | not_found, 46 | stype_in: params.stype_in, 47 | stype_out: params.stype_out, 48 | }) 49 | } 50 | 51 | fn post(&mut self, slug: &str) -> crate::Result { 52 | self.inner.post(&format!("symbology.{slug}")) 53 | } 54 | } 55 | 56 | /// The parameters for [`SymbologyClient::resolve()`]. Use [`ResolveParams::builder()`] 57 | /// to get a builder type with all the preset defaults. 58 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 59 | pub struct ResolveParams { 60 | /// The dataset code. 61 | #[builder(setter(transform = |dt: impl ToString| dt.to_string()))] 62 | pub dataset: String, 63 | /// The symbols to resolve. 64 | #[builder(setter(into))] 65 | pub symbols: Symbols, 66 | /// The symbology type of the input `symbols`. Defaults to 67 | /// [`RawSymbol`](dbn::enums::SType::RawSymbol). 68 | #[builder(default = SType::RawSymbol)] 69 | pub stype_in: SType, 70 | /// The symbology type of the output `symbols`. Defaults to 71 | /// [`InstrumentId`](dbn::enums::SType::InstrumentId). 72 | #[builder(default = SType::InstrumentId)] 73 | pub stype_out: SType, 74 | /// The date range of the resolution. 75 | #[builder(setter(into))] 76 | pub date_range: DateRange, 77 | } 78 | 79 | /// Primarily intended for requesting mappings for historical ALL_SYMBOLS requests, 80 | /// which currently don't return mappings on their own. 81 | impl TryFrom for ResolveParams { 82 | type Error = crate::Error; 83 | 84 | fn try_from(metadata: Metadata) -> Result { 85 | let stype_in = metadata 86 | .stype_in 87 | .ok_or_else(|| crate::Error::bad_arg("metadata", "stype_in must be Some value"))?; 88 | let end = metadata 89 | .end() 90 | .ok_or_else(|| crate::Error::bad_arg("metadata", "end must be Some value"))?; 91 | let dt_range = DateTimeRange::from((metadata.start(), end)); 92 | Ok(Self { 93 | dataset: metadata.dataset, 94 | symbols: Symbols::Symbols(metadata.symbols), 95 | stype_in, 96 | stype_out: metadata.stype_out, 97 | date_range: DateRange::from(dt_range), 98 | }) 99 | } 100 | } 101 | 102 | impl From for ResolveParams { 103 | fn from(get_range_params: timeseries::GetRangeParams) -> Self { 104 | Self { 105 | dataset: get_range_params.dataset, 106 | symbols: get_range_params.symbols, 107 | stype_in: get_range_params.stype_in, 108 | stype_out: get_range_params.stype_out, 109 | date_range: DateRange::from(get_range_params.date_time_range), 110 | } 111 | } 112 | } 113 | 114 | impl From for ResolveParams { 115 | fn from(get_range_to_file_params: timeseries::GetRangeToFileParams) -> Self { 116 | Self::from(timeseries::GetRangeParams::from(get_range_to_file_params)) 117 | } 118 | } 119 | 120 | /// A symbology resolution from one symbology type to another. 121 | #[derive(Debug, Clone)] 122 | pub struct Resolution { 123 | /// A mapping from input symbol to a list of resolved symbols in the output 124 | /// symbology. 125 | pub mappings: HashMap>, 126 | /// A list of symbols that were resolved for part, but not all of the date range 127 | /// from the request. 128 | pub partial: Vec, 129 | /// A list of symbols that were not resolved. 130 | pub not_found: Vec, 131 | /// The input symbology type. 132 | pub stype_in: SType, 133 | /// The output symbology type. 134 | pub stype_out: SType, 135 | } 136 | 137 | impl Resolution { 138 | /// Creates a symbology mapping from instrument ID and date to text symbol. 139 | /// 140 | /// # Errors 141 | /// This function returns an error if it's unable to parse a symbol into an 142 | /// instrument ID. 143 | pub fn symbol_map(&self) -> crate::Result { 144 | let mut map = TsSymbolMap::new(); 145 | if self.stype_in == SType::InstrumentId { 146 | for (iid, intervals) in self.mappings.iter() { 147 | let iid = iid.parse().map_err(|_| { 148 | crate::Error::internal(format!("Unable to parse '{iid}' to an instrument ID",)) 149 | })?; 150 | for interval in intervals { 151 | map.insert( 152 | iid, 153 | interval.start_date, 154 | interval.end_date, 155 | Arc::new(interval.symbol.clone()), 156 | )?; 157 | } 158 | } 159 | } else { 160 | for (raw_symbol, intervals) in self.mappings.iter() { 161 | let raw_symbol = Arc::new(raw_symbol.clone()); 162 | for interval in intervals { 163 | let iid = interval.symbol.parse().map_err(|_| { 164 | crate::Error::internal(format!( 165 | "Unable to parse '{}' to an instrument ID", 166 | interval.symbol 167 | )) 168 | })?; 169 | map.insert( 170 | iid, 171 | interval.start_date, 172 | interval.end_date, 173 | raw_symbol.clone(), 174 | )?; 175 | } 176 | } 177 | } 178 | Ok(map) 179 | } 180 | } 181 | 182 | #[derive(Debug, Clone, Deserialize)] 183 | struct ResolutionResp { 184 | #[serde(rename = "result")] 185 | pub mappings: HashMap>, 186 | pub partial: Vec, 187 | pub not_found: Vec, 188 | } 189 | 190 | #[cfg(test)] 191 | mod tests { 192 | use reqwest::StatusCode; 193 | use serde_json::json; 194 | use time::macros::date; 195 | use wiremock::{ 196 | matchers::{basic_auth, method, path}, 197 | Mock, MockServer, ResponseTemplate, 198 | }; 199 | 200 | use super::*; 201 | use crate::{ 202 | body_contains, 203 | historical::{HistoricalGateway, API_VERSION}, 204 | HistoricalClient, 205 | }; 206 | 207 | const API_KEY: &str = "test-API"; 208 | 209 | #[tokio::test] 210 | async fn test_resolve() { 211 | let mock_server = MockServer::start().await; 212 | Mock::given(method("POST")) 213 | .and(basic_auth(API_KEY, "")) 214 | .and(path(format!("/v{API_VERSION}/symbology.resolve"))) 215 | .and(body_contains("dataset", "GLBX.MDP3")) 216 | .and(body_contains("symbols", "ES.c.0%2CES.d.0")) 217 | .and(body_contains("stype_in", "continuous")) 218 | // default 219 | .and(body_contains("stype_out", "instrument_id")) 220 | .and(body_contains("start_date", "2023-06-14")) 221 | .and(body_contains("end_date", "2023-06-17")) 222 | .respond_with( 223 | ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_json(json!({ 224 | "result": { 225 | "ES.c.0": [ 226 | { 227 | "d0": "2023-06-14", 228 | "d1": "2023-06-15", 229 | "s": "10245" 230 | }, 231 | { 232 | "d0": "2023-06-15", 233 | "d1": "2023-06-16", 234 | "s": "10248" 235 | } 236 | ] 237 | }, 238 | "partial": [], 239 | "not_found": ["ES.d.0"] 240 | })), 241 | ) 242 | .mount(&mock_server) 243 | .await; 244 | let mut target = HistoricalClient::with_url( 245 | mock_server.uri(), 246 | API_KEY.to_owned(), 247 | HistoricalGateway::Bo1, 248 | ) 249 | .unwrap(); 250 | let res = target 251 | .symbology() 252 | .resolve( 253 | &ResolveParams::builder() 254 | .dataset(dbn::Dataset::GlbxMdp3) 255 | .symbols(vec!["ES.c.0", "ES.d.0"]) 256 | .stype_in(SType::Continuous) 257 | .date_range((date!(2023 - 06 - 14), date!(2023 - 06 - 17))) 258 | .build(), 259 | ) 260 | .await 261 | .unwrap(); 262 | assert_eq!( 263 | *res.mappings.get("ES.c.0").unwrap(), 264 | vec![ 265 | MappingInterval { 266 | start_date: time::macros::date!(2023 - 06 - 14), 267 | end_date: time::macros::date!(2023 - 06 - 15), 268 | symbol: "10245".to_owned() 269 | }, 270 | MappingInterval { 271 | start_date: time::macros::date!(2023 - 06 - 15), 272 | end_date: time::macros::date!(2023 - 06 - 16), 273 | symbol: "10248".to_owned() 274 | }, 275 | ] 276 | ); 277 | assert!(res.partial.is_empty()); 278 | assert_eq!(res.not_found, vec!["ES.d.0"]); 279 | } 280 | } 281 | -------------------------------------------------------------------------------- /src/historical/timeseries.rs: -------------------------------------------------------------------------------- 1 | //! The historical timeseries API. 2 | 3 | use std::{num::NonZeroU64, path::PathBuf}; 4 | 5 | use dbn::{encode::AsyncDbnEncoder, Compression, Encoding, SType, Schema, VersionUpgradePolicy}; 6 | use futures::{Stream, TryStreamExt}; 7 | use reqwest::{header::ACCEPT, RequestBuilder}; 8 | use tokio::{ 9 | fs::File, 10 | io::{AsyncReadExt, AsyncWriteExt, BufReader, BufWriter}, 11 | }; 12 | use tokio_util::{bytes::Bytes, io::StreamReader}; 13 | use typed_builder::TypedBuilder; 14 | 15 | use crate::Symbols; 16 | 17 | use super::{check_http_error, DateTimeRange}; 18 | 19 | // Re-export because it's returned. 20 | pub use dbn::decode::AsyncDbnDecoder; 21 | 22 | /// A client for the timeseries group of Historical API endpoints. 23 | #[derive(Debug)] 24 | pub struct TimeseriesClient<'a> { 25 | pub(crate) inner: &'a mut super::Client, 26 | } 27 | 28 | impl TimeseriesClient<'_> { 29 | /// Makes a streaming request for timeseries data from Databento. 30 | /// 31 | /// This method returns a stream decoder. For larger requests, consider using 32 | /// [`BatchClient::submit_job()`](super::batch::BatchClient::submit_job()). 33 | /// 34 | ///
35 | /// Calling this method will incur a cost. 36 | ///
37 | /// 38 | /// # Errors 39 | /// This function returns an error when it fails to communicate with the Databento API 40 | /// or the API indicates there's an issue with the request. 41 | pub async fn get_range( 42 | &mut self, 43 | params: &GetRangeParams, 44 | ) -> crate::Result> { 45 | let reader = self 46 | .get_range_impl( 47 | ¶ms.dataset, 48 | params.schema, 49 | params.stype_in, 50 | params.stype_out, 51 | ¶ms.symbols, 52 | ¶ms.date_time_range, 53 | params.limit, 54 | ) 55 | .await?; 56 | Ok( 57 | AsyncDbnDecoder::with_upgrade_policy(zstd_decoder(reader), params.upgrade_policy) 58 | .await?, 59 | ) 60 | } 61 | 62 | /// Makes a streaming request for timeseries data from Databento. 63 | /// 64 | /// This method returns a stream decoder. For larger requests, consider using 65 | /// [`BatchClient::submit_job()`](super::batch::BatchClient::submit_job()). 66 | /// 67 | ///
68 | /// Calling this method will incur a cost. 69 | ///
70 | /// 71 | /// # Errors 72 | /// This function returns an error when it fails to communicate with the Databento API 73 | /// or the API indicates there's an issue with the request. An error will also be returned 74 | /// if it fails to create a new file at `path`. 75 | pub async fn get_range_to_file( 76 | &mut self, 77 | params: &GetRangeToFileParams, 78 | ) -> crate::Result> { 79 | let reader = self 80 | .get_range_impl( 81 | ¶ms.dataset, 82 | params.schema, 83 | params.stype_in, 84 | params.stype_out, 85 | ¶ms.symbols, 86 | ¶ms.date_time_range, 87 | params.limit, 88 | ) 89 | .await?; 90 | let mut http_decoder = 91 | AsyncDbnDecoder::with_upgrade_policy(zstd_decoder(reader), params.upgrade_policy) 92 | .await?; 93 | let file = BufWriter::new(File::create(¶ms.path).await?); 94 | let mut encoder = AsyncDbnEncoder::with_zstd(file, http_decoder.metadata()).await?; 95 | while let Some(rec_ref) = http_decoder.decode_record_ref().await? { 96 | encoder.encode_record_ref(rec_ref).await?; 97 | } 98 | encoder.get_mut().shutdown().await?; 99 | Ok(AsyncDbnDecoder::with_upgrade_policy( 100 | zstd_decoder(BufReader::new(File::open(¶ms.path).await?)), 101 | // Applied upgrade policy during initial decoding 102 | VersionUpgradePolicy::AsIs, 103 | ) 104 | .await?) 105 | } 106 | 107 | #[allow(clippy::too_many_arguments)] // private method 108 | async fn get_range_impl( 109 | &mut self, 110 | dataset: &str, 111 | schema: Schema, 112 | stype_in: SType, 113 | stype_out: SType, 114 | symbols: &Symbols, 115 | date_time_range: &DateTimeRange, 116 | limit: Option, 117 | ) -> crate::Result>, Bytes>> { 118 | let mut form = vec![ 119 | ("dataset", dataset.to_owned()), 120 | ("schema", schema.to_string()), 121 | ("encoding", Encoding::Dbn.to_string()), 122 | ("compression", Compression::ZStd.to_string()), 123 | ("stype_in", stype_in.to_string()), 124 | ("stype_out", stype_out.to_string()), 125 | ("symbols", symbols.to_api_string()), 126 | ]; 127 | date_time_range.add_to_form(&mut form); 128 | if let Some(limit) = limit { 129 | form.push(("limit", limit.to_string())); 130 | } 131 | let resp = self 132 | .post("get_range")? 133 | // unlike almost every other request, it's not JSON 134 | .header(ACCEPT, "application/octet-stream") 135 | .form(&form) 136 | .send() 137 | .await?; 138 | let stream = check_http_error(resp) 139 | .await? 140 | .error_for_status()? 141 | .bytes_stream() 142 | .map_err(std::io::Error::other); 143 | Ok(tokio_util::io::StreamReader::new(stream)) 144 | } 145 | 146 | fn post(&mut self, slug: &str) -> crate::Result { 147 | self.inner.post(&format!("timeseries.{slug}")) 148 | } 149 | } 150 | 151 | /// The parameters for [`TimeseriesClient::get_range()`]. Use 152 | /// [`GetRangeParams::builder()`] to get a builder type with all the preset defaults. 153 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 154 | pub struct GetRangeParams { 155 | /// The dataset code. 156 | #[builder(setter(transform = |dt: impl ToString| dt.to_string()))] 157 | pub dataset: String, 158 | /// The symbols to filter for. 159 | #[builder(setter(into))] 160 | pub symbols: Symbols, 161 | /// The data record schema. 162 | pub schema: Schema, 163 | /// The date time request range. 164 | /// Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. 165 | #[builder(setter(into))] 166 | pub date_time_range: DateTimeRange, 167 | /// The symbology type of the input `symbols`. Defaults to 168 | /// [`RawSymbol`](dbn::enums::SType::RawSymbol). 169 | #[builder(default = SType::RawSymbol)] 170 | pub stype_in: SType, 171 | /// The symbology type of the output `symbols`. Defaults to 172 | /// [`InstrumentId`](dbn::enums::SType::InstrumentId). 173 | #[builder(default = SType::InstrumentId)] 174 | pub stype_out: SType, 175 | /// The optional maximum number of records to return. Defaults to no limit. 176 | #[builder(default)] 177 | pub limit: Option, 178 | /// How to decode DBN from prior versions. Defaults to upgrade. 179 | #[builder(default)] 180 | pub upgrade_policy: VersionUpgradePolicy, 181 | } 182 | 183 | /// The parameters for [`TimeseriesClient::get_range_to_file()`]. Use 184 | /// [`GetRangeToFileParams::builder()`] to get a builder type with all the preset defaults. 185 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 186 | pub struct GetRangeToFileParams { 187 | /// The dataset code. 188 | #[builder(setter(transform = |dt: impl ToString| dt.to_string()))] 189 | pub dataset: String, 190 | /// The symbols to filter for. 191 | #[builder(setter(into))] 192 | pub symbols: Symbols, 193 | /// The data record schema. 194 | pub schema: Schema, 195 | /// The date time request range. 196 | /// Filters on `ts_recv` if it exists in the schema, otherwise `ts_event`. 197 | #[builder(setter(into))] 198 | pub date_time_range: DateTimeRange, 199 | /// The symbology type of the input `symbols`. Defaults to 200 | /// [`RawSymbol`](dbn::enums::SType::RawSymbol). 201 | #[builder(default = SType::RawSymbol)] 202 | pub stype_in: SType, 203 | /// The symbology type of the output `symbols`. Defaults to 204 | /// [`InstrumentId`](dbn::enums::SType::InstrumentId). 205 | #[builder(default = SType::InstrumentId)] 206 | pub stype_out: SType, 207 | /// The optional maximum number of records to return. Defaults to no limit. 208 | #[builder(default)] 209 | pub limit: Option, 210 | /// How to decode DBN from prior versions. Defaults to upgrade. 211 | #[builder(default)] 212 | pub upgrade_policy: VersionUpgradePolicy, 213 | /// The file path to persist the stream data to. 214 | #[builder(default, setter(transform = |p: impl Into| p.into()))] 215 | pub path: PathBuf, 216 | } 217 | 218 | impl From for GetRangeParams { 219 | fn from(value: GetRangeToFileParams) -> Self { 220 | Self { 221 | dataset: value.dataset, 222 | symbols: value.symbols, 223 | schema: value.schema, 224 | date_time_range: value.date_time_range, 225 | stype_in: value.stype_in, 226 | stype_out: value.stype_out, 227 | limit: value.limit, 228 | upgrade_policy: value.upgrade_policy, 229 | } 230 | } 231 | } 232 | 233 | impl GetRangeParams { 234 | /// Converts these parameters into a request that will be persisted to a file 235 | /// at `path`. Used in conjunction with [`TimeseriesClient::get_range_to_file()``]. 236 | pub fn with_path(self, path: impl Into) -> GetRangeToFileParams { 237 | GetRangeToFileParams { 238 | dataset: self.dataset, 239 | symbols: self.symbols, 240 | schema: self.schema, 241 | date_time_range: self.date_time_range, 242 | stype_in: self.stype_in, 243 | stype_out: self.stype_out, 244 | limit: self.limit, 245 | upgrade_policy: self.upgrade_policy, 246 | path: path.into(), 247 | } 248 | } 249 | } 250 | 251 | fn zstd_decoder(reader: R) -> async_compression::tokio::bufread::ZstdDecoder 252 | where 253 | R: tokio::io::AsyncBufReadExt + Unpin, 254 | { 255 | let mut zstd_decoder = async_compression::tokio::bufread::ZstdDecoder::new(reader); 256 | // explicitly enable decoding multiple frames 257 | zstd_decoder.multiple_members(true); 258 | zstd_decoder 259 | } 260 | 261 | #[cfg(test)] 262 | mod tests { 263 | use dbn::{record::TradeMsg, Dataset}; 264 | use reqwest::StatusCode; 265 | use rstest::*; 266 | use time::macros::datetime; 267 | use wiremock::{ 268 | matchers::{basic_auth, method, path}, 269 | Mock, MockServer, ResponseTemplate, 270 | }; 271 | 272 | use super::*; 273 | use crate::{ 274 | body_contains, 275 | historical::{HistoricalGateway, API_VERSION}, 276 | zst_test_data_path, HistoricalClient, 277 | }; 278 | 279 | const API_KEY: &str = "test-API"; 280 | 281 | #[rstest] 282 | #[case(VersionUpgradePolicy::AsIs, 1)] 283 | #[case(VersionUpgradePolicy::UpgradeToV2, 2)] 284 | #[case(VersionUpgradePolicy::UpgradeToV3, 3)] 285 | #[tokio::test] 286 | async fn test_get_range(#[case] upgrade_policy: VersionUpgradePolicy, #[case] exp_version: u8) { 287 | const START: time::OffsetDateTime = datetime!(2023 - 06 - 14 00:00 UTC); 288 | const END: time::OffsetDateTime = datetime!(2023 - 06 - 17 00:00 UTC); 289 | const SCHEMA: Schema = Schema::Trades; 290 | 291 | let mock_server = MockServer::start().await; 292 | let bytes = tokio::fs::read(zst_test_data_path(SCHEMA)).await.unwrap(); 293 | Mock::given(method("POST")) 294 | .and(basic_auth(API_KEY, "")) 295 | .and(path(format!("/v{API_VERSION}/timeseries.get_range"))) 296 | .and(body_contains("dataset", "XNAS.ITCH")) 297 | .and(body_contains("schema", "trades")) 298 | .and(body_contains("symbols", "SPOT%2CAAPL")) 299 | .and(body_contains( 300 | "start", 301 | START.unix_timestamp_nanos().to_string(), 302 | )) 303 | .and(body_contains("end", END.unix_timestamp_nanos().to_string())) 304 | // // default 305 | .and(body_contains("stype_in", "raw_symbol")) 306 | .and(body_contains("stype_out", "instrument_id")) 307 | .respond_with(ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_bytes(bytes)) 308 | .mount(&mock_server) 309 | .await; 310 | let mut target = HistoricalClient::with_url( 311 | mock_server.uri(), 312 | API_KEY.to_owned(), 313 | HistoricalGateway::Bo1, 314 | ) 315 | .unwrap(); 316 | let mut decoder = target 317 | .timeseries() 318 | .get_range( 319 | &GetRangeParams::builder() 320 | .dataset(dbn::Dataset::XnasItch) 321 | .schema(SCHEMA) 322 | .symbols(vec!["SPOT", "AAPL"]) 323 | .date_time_range((START, END)) 324 | .upgrade_policy(upgrade_policy) 325 | .build(), 326 | ) 327 | .await 328 | .unwrap(); 329 | let metadata = decoder.metadata(); 330 | assert_eq!(metadata.schema.unwrap(), SCHEMA); 331 | assert_eq!(metadata.version, exp_version); 332 | // Two records 333 | decoder.decode_record::().await.unwrap().unwrap(); 334 | decoder.decode_record::().await.unwrap().unwrap(); 335 | assert!(decoder.decode_record::().await.unwrap().is_none()); 336 | } 337 | 338 | #[rstest] 339 | #[case(VersionUpgradePolicy::AsIs, 1)] 340 | #[case(VersionUpgradePolicy::UpgradeToV2, 2)] 341 | #[case(VersionUpgradePolicy::UpgradeToV3, 3)] 342 | #[tokio::test] 343 | async fn test_get_range_to_file( 344 | #[case] upgrade_policy: VersionUpgradePolicy, 345 | #[case] exp_version: u8, 346 | ) { 347 | const START: time::OffsetDateTime = datetime!(2024 - 05 - 17 00:00 UTC); 348 | const END: time::OffsetDateTime = datetime!(2024 - 05 - 18 00:00 UTC); 349 | const SCHEMA: Schema = Schema::Trades; 350 | const DATASET: &str = Dataset::IfeuImpact.as_str(); 351 | 352 | let mock_server = MockServer::start().await; 353 | let temp_dir = tempfile::TempDir::new().unwrap(); 354 | let bytes = tokio::fs::read(zst_test_data_path(SCHEMA)).await.unwrap(); 355 | Mock::given(method("POST")) 356 | .and(basic_auth(API_KEY, "")) 357 | .and(path(format!("/v{API_VERSION}/timeseries.get_range"))) 358 | .and(body_contains("dataset", DATASET)) 359 | .and(body_contains("schema", "trades")) 360 | .and(body_contains("symbols", "BRN.FUT")) 361 | .and(body_contains( 362 | "start", 363 | START.unix_timestamp_nanos().to_string(), 364 | )) 365 | .and(body_contains("end", END.unix_timestamp_nanos().to_string())) 366 | // // default 367 | .and(body_contains("stype_in", "parent")) 368 | .and(body_contains("stype_out", "instrument_id")) 369 | .respond_with(ResponseTemplate::new(StatusCode::OK.as_u16()).set_body_bytes(bytes)) 370 | .mount(&mock_server) 371 | .await; 372 | let mut target = HistoricalClient::with_url( 373 | mock_server.uri(), 374 | API_KEY.to_owned(), 375 | HistoricalGateway::Bo1, 376 | ) 377 | .unwrap(); 378 | let path = temp_dir.path().join("test.dbn.zst"); 379 | let mut decoder = target 380 | .timeseries() 381 | .get_range_to_file( 382 | &GetRangeToFileParams::builder() 383 | .dataset(DATASET) 384 | .schema(SCHEMA) 385 | .symbols(vec!["BRN.FUT"]) 386 | .stype_in(SType::Parent) 387 | .date_time_range((START, END)) 388 | .path(path.clone()) 389 | .upgrade_policy(upgrade_policy) 390 | .build(), 391 | ) 392 | .await 393 | .unwrap(); 394 | let metadata = decoder.metadata(); 395 | assert_eq!(metadata.schema.unwrap(), SCHEMA); 396 | assert_eq!(metadata.version, exp_version); 397 | // Two records 398 | decoder.decode_record::().await.unwrap().unwrap(); 399 | decoder.decode_record::().await.unwrap().unwrap(); 400 | assert!(decoder.decode_record::().await.unwrap().is_none()); 401 | } 402 | } 403 | -------------------------------------------------------------------------------- /src/lib.rs: -------------------------------------------------------------------------------- 1 | //! The official [Databento](https://databento.com) client library. 2 | //! It provides clients for fast, safe streaming of both real-time and historical market data through 3 | //! similar interfaces. 4 | //! The library is built on top of the tokio asynchronous runtime and 5 | //! [Databento's efficient binary encoding](https://databento.com/docs/standards-and-conventions/databento-binary-encoding). 6 | //! 7 | //! You can find getting started tutorials, full API method documentation, examples 8 | //! with output on the [Databento docs site](https://databento.com/docs/?historical=rust&live=rust). 9 | //! 10 | //! # Feature flags 11 | //! By default both features are enabled. 12 | //! - `historical`: enables the [historical client](HistoricalClient) for data older than 24 hours 13 | //! - `live`: enables the [live client](LiveClient) for real-time and intraday 14 | //! historical data 15 | 16 | // Experimental feature to allow docs.rs to display features 17 | #![cfg_attr(docsrs, feature(doc_auto_cfg))] 18 | #![deny(missing_docs)] 19 | #![deny(rustdoc::broken_intra_doc_links)] 20 | #![deny(clippy::missing_errors_doc)] 21 | 22 | pub mod error; 23 | #[cfg(feature = "historical")] 24 | pub mod historical; 25 | #[cfg(feature = "live")] 26 | pub mod live; 27 | 28 | pub use error::{Error, Result}; 29 | #[cfg(feature = "historical")] 30 | pub use historical::Client as HistoricalClient; 31 | #[cfg(feature = "live")] 32 | pub use live::Client as LiveClient; 33 | // Re-export to keep versions synchronized 34 | pub use dbn; 35 | 36 | use std::fmt::{self, Display, Write}; 37 | 38 | #[cfg(feature = "historical")] 39 | use serde::{Deserialize, Deserializer}; 40 | use tracing::error; 41 | 42 | /// A set of symbols for a particular [`SType`](dbn::enums::SType). 43 | #[derive(Debug, Clone, PartialEq, Eq)] 44 | pub enum Symbols { 45 | /// Sentinel value for all symbols in a dataset. 46 | All, 47 | /// A set of symbols identified by their instrument IDs. 48 | Ids(Vec), 49 | /// A set of symbols. 50 | Symbols(Vec), 51 | } 52 | 53 | const ALL_SYMBOLS: &str = "ALL_SYMBOLS"; 54 | const API_KEY_LENGTH: usize = 32; 55 | 56 | impl Symbols { 57 | /// Returns the string representation for sending to the API. 58 | pub fn to_api_string(&self) -> String { 59 | match self { 60 | Symbols::All => ALL_SYMBOLS.to_owned(), 61 | Symbols::Ids(ids) => ids.iter().fold(String::new(), |mut acc, s| { 62 | if acc.is_empty() { 63 | s.to_string() 64 | } else { 65 | write!(acc, ",{s}").unwrap(); 66 | acc 67 | } 68 | }), 69 | Symbols::Symbols(symbols) => symbols.join(","), 70 | } 71 | } 72 | 73 | #[cfg(feature = "live")] 74 | /// Splits the symbol into chunks to stay within the message length requirements of 75 | /// the live gateway. 76 | pub fn to_chunked_api_string(&self) -> Vec { 77 | const CHUNK_SIZE: usize = 500; 78 | match self { 79 | Symbols::All => vec![ALL_SYMBOLS.to_owned()], 80 | Symbols::Ids(ids) => ids 81 | .chunks(CHUNK_SIZE) 82 | .map(|chunk| { 83 | chunk.iter().fold(String::new(), |mut acc, s| { 84 | if acc.is_empty() { 85 | s.to_string() 86 | } else { 87 | write!(acc, ",{s}").unwrap(); 88 | acc 89 | } 90 | }) 91 | }) 92 | .collect(), 93 | Symbols::Symbols(symbols) => symbols 94 | .chunks(CHUNK_SIZE) 95 | .map(|chunk| chunk.join(",")) 96 | .collect(), 97 | } 98 | } 99 | } 100 | 101 | impl Display for Symbols { 102 | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { 103 | match self { 104 | Symbols::All => f.write_str(ALL_SYMBOLS), 105 | Symbols::Ids(ids) => { 106 | for (i, id) in ids.iter().enumerate() { 107 | if i == 0 { 108 | write!(f, "{id}")?; 109 | } else { 110 | write!(f, ", {id}")?; 111 | } 112 | } 113 | Ok(()) 114 | } 115 | Symbols::Symbols(symbols) => { 116 | for (i, sym) in symbols.iter().enumerate() { 117 | if i == 0 { 118 | write!(f, "{sym}")?; 119 | } else { 120 | write!(f, ", {sym}")?; 121 | } 122 | } 123 | Ok(()) 124 | } 125 | } 126 | } 127 | } 128 | 129 | impl From<&str> for Symbols { 130 | fn from(value: &str) -> Self { 131 | Symbols::Symbols(vec![value.to_owned()]) 132 | } 133 | } 134 | 135 | impl From for Symbols { 136 | fn from(value: u32) -> Self { 137 | Symbols::Ids(vec![value]) 138 | } 139 | } 140 | 141 | impl From> for Symbols { 142 | fn from(value: Vec) -> Self { 143 | Symbols::Ids(value) 144 | } 145 | } 146 | 147 | impl From for Symbols { 148 | fn from(value: String) -> Self { 149 | Symbols::Symbols(vec![value]) 150 | } 151 | } 152 | 153 | impl From> for Symbols { 154 | fn from(value: Vec) -> Self { 155 | Symbols::Symbols(value) 156 | } 157 | } 158 | 159 | impl From<[&str; N]> for Symbols { 160 | fn from(value: [&str; N]) -> Self { 161 | Symbols::Symbols(value.iter().map(ToString::to_string).collect()) 162 | } 163 | } 164 | 165 | impl From<&[&str]> for Symbols { 166 | fn from(value: &[&str]) -> Self { 167 | Symbols::Symbols(value.iter().map(ToString::to_string).collect()) 168 | } 169 | } 170 | 171 | impl From> for Symbols { 172 | fn from(value: Vec<&str>) -> Self { 173 | Symbols::Symbols(value.into_iter().map(ToOwned::to_owned).collect()) 174 | } 175 | } 176 | 177 | pub(crate) fn key_from_env() -> crate::Result { 178 | std::env::var("DATABENTO_API_KEY").map_err(|e| { 179 | Error::bad_arg( 180 | "key", 181 | match e { 182 | std::env::VarError::NotPresent => "tried to read API key from environment variable DATABENTO_API_KEY but it is not set", 183 | std::env::VarError::NotUnicode(_) => { 184 | "environment variable DATABENTO_API_KEY contains invalid unicode" 185 | } 186 | }, 187 | ) 188 | }) 189 | } 190 | 191 | #[cfg(feature = "historical")] 192 | impl<'de> Deserialize<'de> for Symbols { 193 | fn deserialize(deserializer: D) -> std::result::Result 194 | where 195 | D: Deserializer<'de>, 196 | { 197 | #[derive(Deserialize)] 198 | #[serde(untagged)] 199 | enum Helper { 200 | Id(u32), 201 | Ids(Vec), 202 | Symbol(String), 203 | Symbols(Vec), 204 | } 205 | let ir = Helper::deserialize(deserializer)?; 206 | Ok(match ir { 207 | Helper::Id(id) => Symbols::Ids(vec![id]), 208 | Helper::Ids(ids) => Symbols::Ids(ids), 209 | Helper::Symbol(symbol) if symbol == ALL_SYMBOLS => Symbols::All, 210 | Helper::Symbol(symbol) => Symbols::Symbols(vec![symbol]), 211 | Helper::Symbols(symbols) => Symbols::Symbols(symbols), 212 | }) 213 | } 214 | } 215 | 216 | /// A struct for holding an API key that implements Debug, but will only print the last 217 | /// five characters of the key. 218 | #[derive(Clone)] 219 | pub struct ApiKey(String); 220 | 221 | pub(crate) const BUCKET_ID_LENGTH: usize = 5; 222 | 223 | impl fmt::Debug for ApiKey { 224 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { 225 | write!( 226 | f, 227 | "\"…{}\"", 228 | &self.0[self.0.len().saturating_sub(BUCKET_ID_LENGTH)..] 229 | ) 230 | } 231 | } 232 | 233 | impl ApiKey { 234 | /// Validates `key` meets requirements of an API key. 235 | /// 236 | /// # Errors 237 | /// This function returns an error if the key is invalid. 238 | pub fn new(key: String) -> crate::Result { 239 | if key == "$YOUR_API_KEY" { 240 | Err(Error::bad_arg( 241 | "key", 242 | "got placeholder API key '$YOUR_API_KEY'. Please pass a real API key", 243 | )) 244 | } else if key.len() != API_KEY_LENGTH { 245 | Err(Error::bad_arg( 246 | "key", 247 | format!( 248 | "expected to be 32-characters long, got {} characters", 249 | key.len() 250 | ), 251 | )) 252 | } else if !key.is_ascii() { 253 | error!("API key '{key}' contains non-ASCII characters"); 254 | Err(Error::bad_arg( 255 | "key", 256 | "expected to be composed of only ASCII characters", 257 | )) 258 | } else { 259 | Ok(ApiKey(key)) 260 | } 261 | } 262 | 263 | /// Returns a slice of the last 5 characters of the key. 264 | #[cfg(feature = "live")] 265 | pub fn bucket_id(&self) -> &str { 266 | // Safe to splice because validated as only containing ASCII characters in [`Self::new()`] 267 | &self.0[API_KEY_LENGTH - BUCKET_ID_LENGTH..] 268 | } 269 | 270 | /// Returns the entire key as a slice. 271 | pub fn as_str(&self) -> &str { 272 | self.0.as_str() 273 | } 274 | } 275 | 276 | #[cfg(test)] 277 | const TEST_DATA_PATH: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/tests/data"); 278 | #[cfg(test)] 279 | pub(crate) fn zst_test_data_path(schema: dbn::enums::Schema) -> String { 280 | format!("{TEST_DATA_PATH}/test_data.{}.dbn.zst", schema.as_str()) 281 | } 282 | #[cfg(test)] 283 | pub(crate) fn body_contains( 284 | key: impl Display, 285 | val: impl Display, 286 | ) -> wiremock::matchers::BodyContainsMatcher { 287 | wiremock::matchers::body_string_contains(format!("{key}={val}")) 288 | } 289 | 290 | #[cfg(test)] 291 | mod tests { 292 | use super::*; 293 | 294 | #[test] 295 | fn test_deserialize_symbols() { 296 | const JSON: &str = r#"["ALL_SYMBOLS", [1, 2, 3], ["ESZ3", "CLZ3"], "TSLA", 1001]"#; 297 | let symbol_res: Vec = serde_json::from_str(JSON).unwrap(); 298 | assert_eq!(symbol_res.len(), 5); 299 | assert_eq!(symbol_res[0], Symbols::All); 300 | assert_eq!(symbol_res[1], Symbols::Ids(vec![1, 2, 3])); 301 | assert_eq!( 302 | symbol_res[2], 303 | Symbols::Symbols(vec!["ESZ3".to_owned(), "CLZ3".to_owned()]) 304 | ); 305 | assert_eq!(symbol_res[3], Symbols::Symbols(vec!["TSLA".to_owned()])); 306 | assert_eq!(symbol_res[4], Symbols::Ids(vec![1001])); 307 | } 308 | 309 | #[test] 310 | fn test_key_debug_truncates() { 311 | assert_eq!( 312 | format!("{:?}", ApiKey("abcdefghijklmnopqrstuvwxyz".to_owned())), 313 | "\"…vwxyz\"" 314 | ); 315 | } 316 | 317 | #[test] 318 | fn test_key_debug_doesnt_underflow() { 319 | assert_eq!(format!("{:?}", ApiKey("test".to_owned())), "\"…test\""); 320 | } 321 | } 322 | -------------------------------------------------------------------------------- /src/live.rs: -------------------------------------------------------------------------------- 1 | //! The Live client and related API types. Used for both real-time data and intraday historical. 2 | 3 | mod client; 4 | pub mod protocol; 5 | 6 | use std::{net::SocketAddr, sync::Arc}; 7 | 8 | use dbn::{SType, Schema, VersionUpgradePolicy}; 9 | use time::{Duration, OffsetDateTime}; 10 | use tokio::net::{lookup_host, ToSocketAddrs}; 11 | use tracing::warn; 12 | use typed_builder::TypedBuilder; 13 | 14 | use crate::{ApiKey, Symbols}; 15 | 16 | pub use client::Client; 17 | 18 | /// A subscription for real-time or intraday historical data. 19 | #[derive(Debug, Clone, TypedBuilder, PartialEq, Eq)] 20 | pub struct Subscription { 21 | /// The symbols of the instruments to subscribe to. 22 | #[builder(setter(into))] 23 | pub symbols: Symbols, 24 | /// The data record schema of data to subscribe to. 25 | pub schema: Schema, 26 | /// The symbology type of the symbols in [`symbols`](Self::symbols). 27 | #[builder(default = SType::RawSymbol)] 28 | pub stype_in: SType, 29 | /// If specified, requests available data since that time (inclusive), based on 30 | /// [`ts_event`](dbn::RecordHeader::ts_event). When `None`, only real-time data is sent. 31 | /// 32 | /// Setting this field is not supported once the session has been started with 33 | /// [`LiveClient::start`](crate::LiveClient::start). 34 | #[builder(default, setter(strip_option))] 35 | pub start: Option, 36 | #[doc(hidden)] 37 | /// Request subscription with snapshot. Defaults to `false`. Conflicts with the `start` parameter. 38 | #[builder(setter(strip_bool))] 39 | pub use_snapshot: bool, 40 | /// The optional numerical identifier associated with this subscription. 41 | #[builder(default, setter(strip_option))] 42 | pub id: Option, 43 | } 44 | 45 | #[doc(hidden)] 46 | #[derive(Debug, Copy, Clone)] 47 | pub struct Unset; 48 | 49 | /// A type-safe builder for the [`LiveClient`](Client). It will not allow you to call 50 | /// [`Self::build()`] before setting the required fields: 51 | /// - `key` 52 | /// - `dataset` 53 | #[derive(Debug, Clone)] 54 | pub struct ClientBuilder { 55 | addr: Option>>, 56 | key: AK, 57 | dataset: D, 58 | send_ts_out: bool, 59 | upgrade_policy: VersionUpgradePolicy, 60 | heartbeat_interval: Option, 61 | } 62 | 63 | impl Default for ClientBuilder { 64 | fn default() -> Self { 65 | Self { 66 | addr: None, 67 | key: Unset, 68 | dataset: Unset, 69 | send_ts_out: false, 70 | upgrade_policy: VersionUpgradePolicy::default(), 71 | heartbeat_interval: None, 72 | } 73 | } 74 | } 75 | 76 | impl ClientBuilder { 77 | /// Sets `ts_out`, which when enabled instructs the gateway to send a send timestamp 78 | /// after every record. These can be decoded with the special [`WithTsOut`](dbn::record::WithTsOut) type. 79 | pub fn send_ts_out(mut self, send_ts_out: bool) -> Self { 80 | self.send_ts_out = send_ts_out; 81 | self 82 | } 83 | 84 | /// Sets `upgrade_policy`, which controls how to decode data from prior DBN 85 | /// versions. The current default is to upgrade them to the latest version while 86 | /// decoding. 87 | pub fn upgrade_policy(mut self, upgrade_policy: VersionUpgradePolicy) -> Self { 88 | self.upgrade_policy = upgrade_policy; 89 | self 90 | } 91 | 92 | /// Sets `heartbeat_interval`, which controls the interval at which the gateway 93 | /// will send heartbeat records if no other data records are sent. If no heartbeat 94 | /// interval is configured, the gateway default will be used. 95 | /// 96 | /// Note that granularity of less than a second is not supported and will be 97 | /// ignored. 98 | pub fn heartbeat_interval(mut self, heartbeat_interval: Duration) -> Self { 99 | if heartbeat_interval.subsec_nanoseconds() > 0 { 100 | warn!( 101 | "heartbeat_interval subsecond precision ignored: {}ns", 102 | heartbeat_interval.subsec_nanoseconds() 103 | ) 104 | } 105 | self.heartbeat_interval = Some(heartbeat_interval); 106 | self 107 | } 108 | 109 | /// Overrides the address of the gateway the client will connect to. This is an 110 | /// advanced method. 111 | /// 112 | /// # Errors 113 | /// This function returns an error when `addr` fails to resolve. 114 | pub async fn addr(mut self, addr: impl ToSocketAddrs) -> crate::Result { 115 | const PARAM_NAME: &str = "addr"; 116 | let addrs: Vec<_> = lookup_host(addr) 117 | .await 118 | .map_err(|e| crate::Error::bad_arg(PARAM_NAME, format!("{e}")))? 119 | .collect(); 120 | self.addr = Some(Arc::new(addrs)); 121 | Ok(self) 122 | } 123 | } 124 | 125 | impl ClientBuilder { 126 | /// Creates a new [`ClientBuilder`]. 127 | pub fn new() -> Self { 128 | Self::default() 129 | } 130 | } 131 | 132 | impl ClientBuilder { 133 | /// Sets the API key. 134 | /// 135 | /// # Errors 136 | /// This function returns an error when the API key is invalid. 137 | pub fn key(self, key: impl ToString) -> crate::Result> { 138 | Ok(ClientBuilder { 139 | addr: self.addr, 140 | key: ApiKey::new(key.to_string())?, 141 | dataset: self.dataset, 142 | send_ts_out: self.send_ts_out, 143 | upgrade_policy: self.upgrade_policy, 144 | heartbeat_interval: self.heartbeat_interval, 145 | }) 146 | } 147 | 148 | /// Sets the API key reading it from the `DATABENTO_API_KEY` environment 149 | /// variable. 150 | /// 151 | /// # Errors 152 | /// This function returns an error when the environment variable is not set or the 153 | /// API key is invalid. 154 | pub fn key_from_env(self) -> crate::Result> { 155 | let key = crate::key_from_env()?; 156 | self.key(key) 157 | } 158 | } 159 | 160 | impl ClientBuilder { 161 | /// Sets the dataset. 162 | pub fn dataset(self, dataset: impl ToString) -> ClientBuilder { 163 | ClientBuilder { 164 | addr: self.addr, 165 | key: self.key, 166 | dataset: dataset.to_string(), 167 | send_ts_out: self.send_ts_out, 168 | upgrade_policy: self.upgrade_policy, 169 | heartbeat_interval: self.heartbeat_interval, 170 | } 171 | } 172 | } 173 | 174 | impl ClientBuilder { 175 | /// Initializes the client and attempts to connect to the gateway. 176 | /// 177 | /// # Errors 178 | /// This function returns an error when its unable 179 | /// to connect and authenticate with the Live gateway. 180 | pub async fn build(self) -> crate::Result { 181 | if let Some(addr) = self.addr { 182 | Client::connect_with_addr( 183 | addr.as_slice(), 184 | self.key.0, 185 | self.dataset, 186 | self.send_ts_out, 187 | self.upgrade_policy, 188 | self.heartbeat_interval, 189 | ) 190 | .await 191 | } else { 192 | Client::connect( 193 | self.key.0, 194 | self.dataset, 195 | self.send_ts_out, 196 | self.upgrade_policy, 197 | self.heartbeat_interval, 198 | ) 199 | .await 200 | } 201 | } 202 | } 203 | -------------------------------------------------------------------------------- /src/live/protocol.rs: -------------------------------------------------------------------------------- 1 | //! Lower-level live interfaces exposed for those who want more customization or 2 | //! control. 3 | //! 4 | //! As these are not part of the primary live API, they are less documented and 5 | //! subject to change without warning. 6 | 7 | use std::{ 8 | collections::HashMap, 9 | fmt::{Debug, Display}, 10 | }; 11 | 12 | use dbn::{SType, Schema}; 13 | use hex::ToHex; 14 | use sha2::{Digest, Sha256}; 15 | use tokio::io::{AsyncBufReadExt, AsyncWriteExt}; 16 | use tracing::{debug, error, instrument}; 17 | 18 | use crate::{ApiKey, Error}; 19 | 20 | use super::Subscription; 21 | 22 | /// Returns the host and port for the live gateway for the given dataset. 23 | /// 24 | /// Performs no validation on `dataset`. 25 | pub fn determine_gateway(dataset: &str) -> String { 26 | const DEFAULT_PORT: u16 = 13_000; 27 | 28 | let dataset_subdomain: String = dataset.replace('.', "-").to_ascii_lowercase(); 29 | format!("{dataset_subdomain}.lsg.databento.com:{DEFAULT_PORT}") 30 | } 31 | 32 | /// The core live API protocol. 33 | pub struct Protocol { 34 | sender: W, 35 | } 36 | 37 | impl Protocol 38 | where 39 | W: AsyncWriteExt + Unpin, 40 | { 41 | /// Creates a new instance of the live API protocol that will send raw API messages 42 | /// to `sender`. 43 | pub fn new(sender: W) -> Self { 44 | Self { sender } 45 | } 46 | 47 | /// Conducts CRAM authentication with the live gateway. Returns the session ID. 48 | /// 49 | /// # Errors 50 | /// This function returns an error if the gateway fails to respond or the authentication 51 | /// request is rejected. 52 | /// 53 | /// # Cancel safety 54 | /// This method is not cancellation safe. If this method is used in a 55 | /// [`tokio::select!`] statement and another branch completes first, the 56 | /// authentication may have been only partially sent, resulting in the gateway 57 | /// rejecting the authentication and closing the connection. 58 | #[instrument(skip(self, recver, key))] 59 | pub async fn authenticate( 60 | &mut self, 61 | recver: &mut R, 62 | key: &ApiKey, 63 | dataset: &str, 64 | send_ts_out: bool, 65 | heartbeat_interval_s: Option, 66 | ) -> crate::Result 67 | where 68 | R: AsyncBufReadExt + Unpin, 69 | { 70 | let mut greeting = String::new(); 71 | // Greeting 72 | recver.read_line(&mut greeting).await?; 73 | greeting.pop(); // remove newline 74 | 75 | debug!(greeting); 76 | let mut response = String::new(); 77 | // Challenge 78 | recver.read_line(&mut response).await?; 79 | response.pop(); // remove newline 80 | 81 | // Parse challenge 82 | let challenge = Challenge::parse(&response).inspect_err(|_| { 83 | error!(?response, "No CRAM challenge in response from gateway"); 84 | })?; 85 | debug!(%challenge, "Received CRAM challenge"); 86 | 87 | // Send CRAM reply/auth request 88 | let auth_req = 89 | AuthRequest::new(key, dataset, send_ts_out, heartbeat_interval_s, &challenge); 90 | debug!(?auth_req, "Sending CRAM reply"); 91 | self.sender.write_all(auth_req.as_bytes()).await.unwrap(); 92 | 93 | response.clear(); 94 | recver.read_line(&mut response).await?; 95 | debug!( 96 | auth_resp = &response[..response.len() - 1], 97 | "Received auth response" 98 | ); 99 | response.pop(); // remove newline 100 | 101 | let auth_resp = AuthResponse::parse(&response)?; 102 | Ok(auth_resp 103 | .0 104 | .get("session_id") 105 | .map(|sid| (*sid).to_owned()) 106 | .unwrap_or_default()) 107 | } 108 | 109 | /// Sends one or more subscription messages for `sub` depending on the number of symbols. 110 | /// 111 | /// # Errors 112 | /// This function returns an error if it's unable to communicate with the gateway. 113 | /// 114 | /// # Cancel safety 115 | /// This method is not cancellation safe. If this method is used in a 116 | /// [`tokio::select!`] statement and another branch completes first, the subscription 117 | /// may have been partially sent, resulting in the gateway rejecting the 118 | /// subscription, sending an error, and closing the connection. 119 | pub async fn subscribe(&mut self, sub: &Subscription) -> crate::Result<()> { 120 | let Subscription { 121 | schema, 122 | stype_in, 123 | start, 124 | use_snapshot, 125 | .. 126 | } = ⊂ 127 | 128 | if *use_snapshot && start.is_some() { 129 | return Err(Error::BadArgument { 130 | param_name: "use_snapshot".to_string(), 131 | desc: "cannot request snapshot with start time".to_string(), 132 | }); 133 | } 134 | let start_nanos = sub.start.as_ref().map(|start| start.unix_timestamp_nanos()); 135 | 136 | let symbol_chunks = sub.symbols.to_chunked_api_string(); 137 | let last_chunk_idx = symbol_chunks.len() - 1; 138 | for (i, sym_str) in symbol_chunks.into_iter().enumerate() { 139 | let sub_req = SubRequest::new( 140 | *schema, 141 | *stype_in, 142 | start_nanos, 143 | *use_snapshot, 144 | sub.id, 145 | &sym_str, 146 | i == last_chunk_idx, 147 | ); 148 | debug!(?sub_req, "Sending subscription request"); 149 | self.sender.write_all(sub_req.as_bytes()).await?; 150 | } 151 | Ok(()) 152 | } 153 | 154 | /// Sends a start session message to the live gateway. 155 | /// 156 | /// # Errors 157 | /// This function returns an error if it's unable to communicate with 158 | /// the gateway. 159 | /// 160 | /// # Cancel safety 161 | /// This method is not cancellation safe. If this method is used in a 162 | /// [`tokio::select!`] statement and another branch completes first, the live 163 | /// gateway may only receive a partial message, resulting in it sending an error and 164 | /// closing the connection. 165 | pub async fn start_session(&mut self) -> crate::Result<()> { 166 | Ok(self.sender.write_all(StartRequest.as_bytes()).await?) 167 | } 168 | 169 | /// Shuts down the inner writer. 170 | /// 171 | /// # Errors 172 | /// This function returns an error if the shut down did not complete successfully. 173 | pub async fn shutdown(&mut self) -> crate::Result<()> { 174 | Ok(self.sender.shutdown().await?) 175 | } 176 | 177 | /// Consumes the protocol instance and returns the inner sender. 178 | pub fn into_inner(self) -> W { 179 | self.sender 180 | } 181 | } 182 | 183 | /// A challenge request from the live gateway. 184 | /// 185 | /// See the [raw API documentation](https://databento.com/docs/api-reference-live/gateway-control-messages/challenge-request?live=raw) 186 | /// for more information. 187 | #[derive(Debug)] 188 | pub struct Challenge<'a>(&'a str); 189 | 190 | impl<'a> Challenge<'a> { 191 | /// Parses a challenge request from the given raw response. 192 | /// 193 | /// # Errors 194 | /// Returns an error if the response does not begin with "cram=". 195 | // Can't use `FromStr` with lifetime 196 | pub fn parse(response: &'a str) -> crate::Result { 197 | if response.starts_with("cram=") { 198 | Ok(Self(response.split_once('=').unwrap().1)) 199 | } else { 200 | Err(Error::internal( 201 | "no CRAM challenge in response from gateway", 202 | )) 203 | } 204 | } 205 | } 206 | 207 | impl Display for Challenge<'_> { 208 | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { 209 | write!(f, "{}", self.0) 210 | } 211 | } 212 | 213 | /// An authentication request to be sent to the live gateway. 214 | /// 215 | /// See the [raw API documentation](https://databento.com/docs/api-reference-live/client-control-messages/authentication-request?live=raw) 216 | /// for more information. 217 | pub struct AuthRequest(String); 218 | 219 | impl AuthRequest { 220 | /// Creates the raw API authentication request message from the given parameters. 221 | pub fn new( 222 | key: &ApiKey, 223 | dataset: &str, 224 | send_ts_out: bool, 225 | heartbeat_interval_s: Option, 226 | challenge: &Challenge, 227 | ) -> Self { 228 | let challenge_key = format!("{challenge}|{}", key.0); 229 | let mut hasher = Sha256::new(); 230 | hasher.update(challenge_key.as_bytes()); 231 | let hashed = hasher.finalize(); 232 | let bucket_id = key.bucket_id(); 233 | let encoded_response = hashed.encode_hex::(); 234 | let send_ts_out = send_ts_out as u8; 235 | let mut req = 236 | format!("auth={encoded_response}-{bucket_id}|dataset={dataset}|encoding=dbn|ts_out={send_ts_out}|client=Rust {}", env!("CARGO_PKG_VERSION")); 237 | if let Some(heartbeat_interval_s) = heartbeat_interval_s { 238 | req = format!("{req}|heartbeat_interval_s={heartbeat_interval_s}"); 239 | } 240 | req.push('\n'); 241 | Self(req) 242 | } 243 | 244 | /// Returns the string slice of the request. 245 | pub fn as_str(&self) -> &str { 246 | self.0.as_str() 247 | } 248 | 249 | /// Returns the request as a byte slice. 250 | pub fn as_bytes(&self) -> &[u8] { 251 | self.0.as_bytes() 252 | } 253 | } 254 | 255 | impl Debug for AuthRequest { 256 | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { 257 | // Should never be empty 258 | write!(f, "{}", &self.0[..self.0.len() - 1]) 259 | } 260 | } 261 | 262 | /// An authentication response from the live gateway. 263 | /// 264 | /// See the [raw API documentation](https://databento.com/docs/api-reference-live/gateway-control-messages/authentication-response?live=raw) 265 | /// for more information. 266 | pub struct AuthResponse<'a>(HashMap<&'a str, &'a str>); 267 | 268 | impl<'a> AuthResponse<'a> { 269 | /// Parses a challenge request from the given raw response. 270 | /// 271 | /// # Errors 272 | /// Returns an error if the response does not begin with "cram=". 273 | // Can't use `FromStr` with lifetime 274 | pub fn parse(response: &'a str) -> crate::Result { 275 | let auth_keys: HashMap<&'a str, &'a str> = response 276 | .split('|') 277 | .filter_map(|kvp| kvp.split_once('=')) 278 | .collect(); 279 | // Lack of success key also indicates something went wrong 280 | if auth_keys.get("success").map(|v| *v != "1").unwrap_or(true) { 281 | return Err(Error::Auth( 282 | auth_keys 283 | .get("error") 284 | .map(|msg| (*msg).to_owned()) 285 | .unwrap_or_else(|| response.to_owned()), 286 | )); 287 | } 288 | Ok(Self(auth_keys)) 289 | } 290 | 291 | /// Returns a reference to the key-value pairs. 292 | pub fn get_ref(&self) -> &HashMap<&'a str, &'a str> { 293 | &self.0 294 | } 295 | } 296 | 297 | /// A subscription request to be sent to the live gateway. 298 | /// 299 | /// See the [raw API documentation](https://databento.com/docs/api-reference-live/client-control-messages/subscription-request?live=raw) 300 | /// for more information. 301 | pub struct SubRequest(String); 302 | 303 | impl SubRequest { 304 | /// Creates the raw API authentication request message from the given parameters. 305 | /// `symbols` is expected to already be a valid length, such as from 306 | /// [`Symbols::to_chunked_api_string()`](crate::Symbols::to_chunked_api_string). 307 | pub fn new( 308 | schema: Schema, 309 | stype_in: SType, 310 | start_nanos: Option, 311 | use_snapshot: bool, 312 | id: Option, 313 | symbols: &str, 314 | is_last: bool, 315 | ) -> Self { 316 | let use_snapshot = use_snapshot as u8; 317 | let is_last = is_last as u8; 318 | let mut args = format!( 319 | "schema={schema}|stype_in={stype_in}|symbols={symbols}|snapshot={use_snapshot}|is_last={is_last}" 320 | ); 321 | 322 | if let Some(start) = start_nanos { 323 | args = format!("{args}|start={start}"); 324 | } 325 | if let Some(id) = id { 326 | args = format!("{args}|id={id}"); 327 | } 328 | args.push('\n'); 329 | Self(args) 330 | } 331 | 332 | /// Returns the string slice of the request. 333 | pub fn as_str(&self) -> &str { 334 | self.0.as_str() 335 | } 336 | 337 | /// Returns the request as a byte slice. 338 | pub fn as_bytes(&self) -> &[u8] { 339 | self.0.as_bytes() 340 | } 341 | } 342 | 343 | impl Debug for SubRequest { 344 | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { 345 | // Should never be empty 346 | write!(f, "{}", &self.0[..self.0.len() - 1]) 347 | } 348 | } 349 | 350 | /// A request to begin the session to be sent to the live gateway. 351 | /// 352 | /// See the [raw API documentation](https://databento.com/docs/api-reference-live/client-control-messages/session-start?live=raw) 353 | /// for more information. 354 | pub struct StartRequest; 355 | 356 | impl StartRequest { 357 | /// Returns the string slice of the request. 358 | pub fn as_str(&self) -> &str { 359 | "start_session\n" 360 | } 361 | 362 | /// Returns the request as a byte slice. 363 | pub fn as_bytes(&self) -> &[u8] { 364 | self.as_str().as_bytes() 365 | } 366 | } 367 | -------------------------------------------------------------------------------- /tests/data/test_data.definition.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.definition.dbn -------------------------------------------------------------------------------- /tests/data/test_data.definition.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.definition.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.imbalance.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.imbalance.dbn -------------------------------------------------------------------------------- /tests/data/test_data.imbalance.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.imbalance.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.mbo.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.mbo.dbn -------------------------------------------------------------------------------- /tests/data/test_data.mbo.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.mbo.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.mbp-1.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.mbp-1.dbn -------------------------------------------------------------------------------- /tests/data/test_data.mbp-1.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.mbp-1.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.mbp-10.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.mbp-10.dbn -------------------------------------------------------------------------------- /tests/data/test_data.mbp-10.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.mbp-10.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1d.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1d.dbn -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1d.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1d.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1h.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1h.dbn -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1h.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1h.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1m.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1m.dbn -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1m.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1m.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1s.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1s.dbn -------------------------------------------------------------------------------- /tests/data/test_data.ohlcv-1s.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.ohlcv-1s.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.statistics.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.statistics.dbn -------------------------------------------------------------------------------- /tests/data/test_data.statistics.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.statistics.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.tbbo.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.tbbo.dbn -------------------------------------------------------------------------------- /tests/data/test_data.tbbo.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.tbbo.dbn.zst -------------------------------------------------------------------------------- /tests/data/test_data.trades.dbn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.trades.dbn -------------------------------------------------------------------------------- /tests/data/test_data.trades.dbn.zst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/databento/databento-rs/c778b930b218620899f8248aef8e6747d2aa4d1e/tests/data/test_data.trades.dbn.zst --------------------------------------------------------------------------------