├── .gitignore ├── LICENSE ├── README.md ├── bgpkit ├── __init__.py ├── bgpkit_broker.py ├── bgpkit_parser.py ├── bgpkit_roas.py └── test_integration.py ├── pyproject.toml ├── setup.py └── test.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Generated by Cargo 2 | # will have compiled files and executables 3 | /target/ 4 | 5 | # Remove Cargo.lock from gitignore if creating an executable, leave it for libraries 6 | # More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html 7 | Cargo.lock 8 | 9 | # These are backup files generated by rustfmt 10 | **/*.rs.bk 11 | 12 | build 13 | dist 14 | *.egg-info 15 | .idea 16 | 17 | *.gz 18 | venv 19 | __pycache__ 20 | 21 | .venv 22 | test*.py 23 | cache 24 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 BGPKIT 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PyBGPKIT 2 | 3 | Python bindings for BGPKIT software. For all software offerings, please check out our GitHub 4 | repository at . 5 | 6 | ## SDKs 7 | 8 | ### BGPKIT Parser 9 | 10 | Original Rust BGPKIT Parser code available at: 11 | 12 | Example: 13 | ```python 14 | import bgpkit 15 | parser = bgpkit.Parser(url="https://spaces.bgpkit.org/parser/update-example", 16 | filters={"peer_ips": "185.1.8.65, 2001:7f8:73:0:3:fa4:0:1"}) 17 | count = 0 18 | for elem in parser: 19 | count += 1 20 | print(elem) 21 | assert count == 4227 22 | ``` 23 | 24 | The `Parser` constructor takes the following parameters: 25 | - `url`: the URL or local file path toward an MRT file 26 | - `filters`: optional a dictionary of filters, available filters are: 27 | - `origin_asn`: origin AS number 28 | - `prefix`: exact match prefix 29 | - `prefix_super`: exact prefix and its super prefixes 30 | - `prefix_sub`: exact prefix and its sub prefixes 31 | - `prefix_super_sub`: exact prefix and its super and sub prefixes 32 | - `peer_ip`: peer's IP address 33 | - `peer_ips`: peers' IP addresses 34 | - `peer_asn`: peer's ASN 35 | - `type`: message type (`withdraw` or `announce`) 36 | - `ts_start`: start unix timestamp 37 | - `ts_end`: end unix timestamp 38 | - `as_path`: regular expression for AS path string 39 | - `cache_dir`: optional string for specifying a download cache directory 40 | 41 | 42 | Each returning item has the following field: 43 | - `timestamp`: float, unix timestamp 44 | - `elem_type`: str, `A`, announcement; `W`, withdrawn 45 | - `peer_ip`: str, peer IP address 46 | - `peer_asn`: int, peer ASN 47 | - `prefix`: str, the announced/withdrawn IP prefix 48 | - `next_hop`: str or None, next hop IP address 49 | - `as_path`: str or None, AS path str, e.g. `60924 6939 58715 63969 135490` 50 | - `origin_asns`: [int] or None, array of originating ASNs of the prefix 51 | - `origin`: str or None, `IGP`, `EGP`, or `INCOMPLETE` 52 | - `local_pref`: int or None, local preference 53 | - `med`: int or None, multi-exitmultiple exit discriminator 54 | - `communities`: [str] or None, community values, e.g. `['60924:6', '60924:150', '60924:502', '60924:2002', 'ecop:67:0:000000000000']` 55 | - `atomic`: str, `AG` for atomic aggregate, and `NAG` for non-atomic aggregate 56 | - `aggr_ip`: str or None, aggregator IP address 57 | - `aggr_asn`: int or None, aggregator ASN 58 | 59 | 60 | 61 | ### BGPKIT Broker 62 | 63 | Original Rust version BGPKIT Broker code available at: 64 | 65 | Example: 66 | ```python 67 | import bgpkit 68 | broker = bgpkit.Broker() 69 | items = broker.query(ts_start="1634693400", ts_end="2021-10-20T01:30:00") 70 | for item in items: 71 | print(item) 72 | print(len(items)) 73 | assert len(items) == 58 74 | ``` 75 | 76 | Available fields: 77 | 78 | - `Broker()` 79 | - `api_url`: the base URL for the BGPKIT Broker instance. Default: `https://api.broker.bgpkit.com/v2` 80 | - `page_size`: the number of items per API call (no need to change it). Default: 100. 81 | - `query()` 82 | - `ts_start`: start timestamp for MRT file, UNIX timestamp or string format 83 | - `ts_end`: end timestamp for MRT file, UNIX timestamp or string format 84 | - `collector_id`: collector name, e.g. `rrc00` or `route-views2` 85 | - `data_type`: `rib` or `update` 86 | 87 | ### BGPKIT ROAS Lookup 88 | 89 | BGPKIT ROAS lookup API provides lookup for historical RPKI ROAS data lookup. The following example shows a query that 90 | asks for all the validated ROA payload for RIPE NCC on the date of `2018-01-01`. 91 | 92 | ```python 93 | import bgpkit 94 | roas = bgpkit.Roas() 95 | data = roas.query(debug=True, asn=3333, date="2018-01-01") 96 | for entry in data: 97 | print(entry) 98 | assert len(data) == 10 99 | ``` 100 | 101 | ``` 102 | {'tal': 'ripencc', 'prefix': '193.0.0.0/21', 'asn': 3333, 'max_len': 21, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 103 | {'tal': 'ripencc', 'prefix': '193.0.10.0/23', 'asn': 3333, 'max_len': 23, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 104 | {'tal': 'ripencc', 'prefix': '193.0.12.0/23', 'asn': 3333, 'max_len': 23, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 105 | {'tal': 'ripencc', 'prefix': '193.0.18.0/23', 'asn': 3333, 'max_len': 23, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 106 | {'tal': 'ripencc', 'prefix': '193.0.20.0/23', 'asn': 3333, 'max_len': 23, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 107 | {'tal': 'ripencc', 'prefix': '193.0.22.0/23', 'asn': 3333, 'max_len': 23, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 108 | {'tal': 'ripencc', 'prefix': '193.0.24.0/22', 'asn': 3333, 'max_len': 26, 'date_ranges': [['2017-01-14', '2018-12-27'], ['2019-01-03', '2019-06-24']]} 109 | {'tal': 'ripencc', 'prefix': '193.0.24.0/24', 'asn': 3333, 'max_len': 24, 'date_ranges': [['2017-02-25', '2018-12-27'], ['2019-01-03', '2019-06-24']]} 110 | {'tal': 'ripencc', 'prefix': '2001:610:240::/42', 'asn': 3333, 'max_len': 42, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 111 | {'tal': 'ripencc', 'prefix': '2001:67c:2e8::/48', 'asn': 3333, 'max_len': 48, 'date_ranges': [['2015-03-10', '2016-01-26'], ['2016-01-30', '2018-12-27'], ['2019-01-03', '2019-10-21'], ['2019-10-23', '2020-02-23'], ['2020-02-25', '2020-04-05'], ['2020-04-07', '2020-08-02'], ['2020-08-04', '2021-04-21'], ['2021-04-23', '2021-04-24'], ['2021-04-28', '2022-02-26']]} 112 | ``` 113 | 114 | Available query fields: 115 | 116 | - `Roas()` 117 | - `api_url`: the base URL for the BGPKIT ROAS instance. Default: `https://api.roas.bgpkit.com` 118 | - `query()` 119 | - `prefix`: prefix to query in `str` 120 | - `asn`: AS number to query in `int` 121 | - `tal`: trust anchor to query in `str`, available values: `ripencc`, `arin`, `apnic`, `afrinic`, `lacnic` 122 | - `date`: date to query, format: `YYYY-MM-DD`, e.g. `2022-01-01` 123 | - `max_len`: filter results to only VRP's with specific max length 124 | - `debug`: boolean toggle to display debug information, default `False` 125 | 126 | ## Build and Upload 127 | 128 | Install `python-build` module: 129 | ``` bash 130 | python3 -m pip install build twine 131 | ``` 132 | 133 | Build current package: 134 | ``` bash 135 | python3 -m build 136 | ``` 137 | 138 | Upload to PyPi (needs credentials) 139 | ``` bash 140 | python3 -m twine upload dist/* 141 | ``` 142 | 143 | -------------------------------------------------------------------------------- /bgpkit/__init__.py: -------------------------------------------------------------------------------- 1 | from .bgpkit_parser import Parser 2 | from .bgpkit_broker import Broker 3 | from .bgpkit_roas import Roas 4 | -------------------------------------------------------------------------------- /bgpkit/bgpkit_broker.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | 3 | import requests as requests 4 | import urllib3 5 | 6 | 7 | def check_type(value: any, ty: type) -> bool: 8 | try: 9 | ty(value) 10 | return True 11 | except ValueError: 12 | raise ValueError("invalid option input") 13 | 14 | 15 | @dataclass 16 | class BrokerItem: 17 | ts_start: str 18 | ts_end: str 19 | collector_id: str 20 | data_type: str 21 | url: str 22 | rough_size: int 23 | exact_size: int 24 | 25 | 26 | class Broker: 27 | 28 | def __init__(self, api_url: str = "https://api.bgpkit.com/broker", page_size: int = 100, verify=True): 29 | self.base_url = api_url.strip() 30 | self.page_size = int(page_size) 31 | self.verify = verify 32 | if not verify: 33 | # if a user disable SSL verification on-purpose, do not warn the user 34 | urllib3.disable_warnings() 35 | 36 | def query(self, 37 | ts_start: str = None, 38 | ts_end: str = None, 39 | collector_id: str = None, 40 | project: str = None, 41 | data_type: str = None, 42 | print_url: bool = False, 43 | ) -> [BrokerItem]: 44 | params = [] 45 | if ts_start: 46 | params.append(f"ts_start={ts_start}") 47 | if ts_end: 48 | params.append(f"ts_end={ts_end}") 49 | if collector_id: 50 | check_type(collector_id, str) 51 | params.append(f"collector_id={collector_id}") 52 | if project: 53 | check_type(project, str) 54 | params.append(f"project={project}") 55 | if data_type: 56 | check_type(data_type, str) 57 | params.append(f"data_type={data_type}") 58 | params.append(f"page_size={self.page_size}") 59 | 60 | api_url = f"{self.base_url}/search?" + "&".join(params) 61 | page = 1 62 | 63 | data_items = [] 64 | if print_url: 65 | print(api_url) 66 | res = requests.get(api_url, verify=self.verify).json() 67 | while res: 68 | if res["count"] > 0: 69 | data_items.extend([BrokerItem(**i) for i in res["data"]]) 70 | 71 | if res["count"] < res["page_size"]: 72 | break 73 | 74 | page += 1 75 | query_url = f"{api_url}&page={page}" 76 | if print_url: 77 | print(query_url) 78 | res = requests.get(query_url, verify=self.verify).json() 79 | else: 80 | break 81 | 82 | return data_items 83 | -------------------------------------------------------------------------------- /bgpkit/bgpkit_parser.py: -------------------------------------------------------------------------------- 1 | from pybgpkit_parser import Parser -------------------------------------------------------------------------------- /bgpkit/bgpkit_roas.py: -------------------------------------------------------------------------------- 1 | from dataclasses import dataclass 2 | from typing import Optional, List 3 | 4 | import requests as requests 5 | 6 | 7 | def check_type(value: any, ty: type) -> bool: 8 | try: 9 | ty(value) 10 | return True 11 | except ValueError: 12 | raise ValueError("invalid option input") 13 | 14 | 15 | @dataclass 16 | class RoasItem: 17 | prefix: str 18 | asn: int 19 | tal: str 20 | date_ranges: List[List[str]] 21 | 22 | 23 | @dataclass 24 | class RoasRes: 25 | limit: int 26 | count: int 27 | data: List[RoasItem] 28 | next_page_num: Optional[int] 29 | next_page: Optional[str] 30 | error: Optional[str] 31 | 32 | 33 | class Roas: 34 | 35 | def __init__(self, api_url: str = "https://api.roas.bgpkit.com"): 36 | self.base_url = api_url.strip() 37 | 38 | def query(self, 39 | prefix: str = None, 40 | asn: int = None, 41 | tal: str = None, 42 | date: str = None, 43 | max_len: int = None, 44 | debug: bool = False, 45 | ) -> [RoasItem]: 46 | 47 | if not (prefix or asn or tal or date or max_len): 48 | print("ERROR: must specify at least one query parameter: prefix, asn, tal, date, max_len") 49 | return [] 50 | 51 | params = [] 52 | if prefix: 53 | check_type(prefix, str) 54 | params.append(f"prefix={prefix}") 55 | if asn: 56 | check_type(asn, int) 57 | params.append(f"asn={asn}") 58 | if tal: 59 | check_type(tal, str) 60 | params.append(f"tal={tal}") 61 | if date: 62 | check_type(date, str) 63 | params.append(f"date={date}") 64 | if max_len: 65 | check_type(max_len, int) 66 | params.append(f"max_len={max_len}") 67 | 68 | api_url = f"{self.base_url}/lookup?" + "&".join(params) 69 | data_items = [] 70 | if debug: 71 | print(api_url) 72 | res = RoasRes(**requests.get(api_url).json()) 73 | while res.data: 74 | data_items.extend(res.data) 75 | if res.next_page: 76 | res = RoasRes(**requests.get(res.next_page).json()) 77 | else: 78 | break 79 | 80 | return data_items 81 | -------------------------------------------------------------------------------- /bgpkit/test_integration.py: -------------------------------------------------------------------------------- 1 | import json 2 | import unittest 3 | 4 | import bgpkit 5 | 6 | 7 | class TestIntegration(unittest.TestCase): 8 | 9 | def test_parser(self): 10 | parser = bgpkit.Parser(url="https://spaces.bgpkit.org/parser/update-example", 11 | filters={"peer_ips": "185.1.8.65, 2001:7f8:73:0:3:fa4:0:1"}) 12 | elems = parser.parse_all() 13 | assert len(elems) == 4227 14 | 15 | def test_broker(self): 16 | # filter by only time 17 | broker = bgpkit.Broker() 18 | items = broker.query(ts_start="1643760000", ts_end="2022-02-02T00:20:00") 19 | assert len(items) == 290 20 | 21 | # filter by both time and collector 22 | broker = bgpkit.Broker() 23 | items = broker.query(ts_start="1643760000", ts_end="2022-02-02T00:20:00", collector_id="rrc00") 24 | assert len(items) == 7 25 | 26 | # specify API endpoint and filter by time string with timezones (+ and - zones) 27 | broker = bgpkit.Broker("https://api.bgpkit.com/broker") 28 | items = broker.query(ts_start="2022-02-02T00:00:00-00:00", ts_end="2022-02-02T00:20:00.123000+00:00", 29 | collector_id="rrc00") 30 | assert len(items) == 7 31 | 32 | def test_broker_no_verify(self): 33 | broker = bgpkit.Broker(verify=False) 34 | items = broker.query(ts_start="1643760000", ts_end="2022-02-02T00:20:00", collector_id="rrc00") 35 | assert len(items) == 7 36 | 37 | def test_roas(self): 38 | roas = bgpkit.Roas() 39 | data = roas.query(debug=True, asn=3333, date="2018-01-01") 40 | for entry in data: 41 | print(entry) 42 | assert len(data) == 10 43 | 44 | assert len(roas.query()) == 0 45 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "setuptools>=42", 4 | "wheel" 5 | ] 6 | build-backend = "setuptools.build_meta" -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | # read the contents of your README file 4 | from pathlib import Path 5 | this_directory = Path(__file__).parent 6 | long_description = (this_directory / "README.md").read_text() 7 | 8 | setuptools.setup( 9 | name='pybgpkit', 10 | version='0.5.2', 11 | description='BGPKIT tools Python bindings', 12 | url='https://github.com/bgpkit/pybgpkit', 13 | author='Mingwei Zhang', 14 | author_email='mingwei@bgpkit.com', 15 | packages=setuptools.find_packages(), 16 | include_package_data=True, 17 | long_description=long_description, 18 | long_description_content_type='text/markdown', 19 | install_requires=[ 20 | # available on pip 21 | 'dataclasses_json', 22 | 'pybgpkit-parser==0.5.2', 23 | 'requests', 24 | ] 25 | ) 26 | -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | import bgpkit 2 | parser = bgpkit.Parser(url="https://spaces.bgpkit.org/parser/update-example", 3 | filters={"peer_ips": "185.1.8.65, 2001:7f8:73:0:3:fa4:0:1"}, 4 | cache_dir="cache" 5 | ) 6 | elems = parser.parse_all() 7 | assert len(elems) == 4227 8 | import json 9 | print(json.dumps(elems[0], indent=4)) 10 | --------------------------------------------------------------------------------