├── debian ├── compat ├── dirs ├── docs ├── source │ └── format ├── rules ├── manpages ├── control ├── copyright └── changelog ├── tests ├── 20190719 │ ├── idn_vs_tld.dat │ ├── ipv6_rsn_abusers_accum.dat │ ├── opcode.dat │ ├── chaos_types_and_names.dat │ ├── client_subnet_count.dat │ ├── do_bit.dat │ ├── idn_qname.dat │ ├── rcode.dat │ ├── rd_bit.dat │ ├── edns_version.dat │ ├── ipv6_rsn_abusers_count.dat │ ├── dnssec_qtype.dat │ ├── client_subnet_accum.dat │ ├── qtype.dat │ ├── client_subnet2_count.dat │ ├── client_subnet2_trace.dat │ ├── direction_vs_ipproto.dat │ ├── transport_vs_qtype.dat │ ├── certain_qnames_vs_qtype.dat │ ├── pcap_stats.dat │ ├── edns_bufsiz.dat │ ├── client_subnet2_accum.dat │ ├── qtype_vs_tld.dat │ ├── client_addr_vs_rcode_accum.dat │ ├── qtype_vs_qnamelen.dat │ ├── client_port_range.dat │ └── rcode_vs_replylen.dat ├── dsc-datatool.py ├── test_main.py ├── broken.xml ├── dsc-datatool ├── test.gold4 ├── utf8.xml ├── test_objects.py ├── coverage.sh ├── test.sh ├── test.gold3 ├── ipv6-unicast-address-assignments.csv ├── test.gold2 ├── 1458044657.xml ├── test.gold ├── labler.yaml └── 1563520620.dscdata.xml ├── sonar-project.properties.local ├── MANIFEST.in ├── setup.cfg ├── .gitignore ├── man ├── man5 │ └── dsc-datatool.conf.5 ├── man7 │ ├── dsc-datatool-output-influxdb.7 │ ├── dsc-datatool-transformer-labler.7 │ ├── dsc-datatool-transformer-netremap.7 │ ├── dsc-datatool-generator-client_subnet_country.7 │ ├── dsc-datatool-transformer-reranger.7 │ ├── dsc-datatool-generator-client_subnet_authority.7 │ └── dsc-datatool-output-prometheus.7 └── man1 │ └── dsc-datatool.1 ├── setup.py ├── .forgejo └── workflows │ └── test.yml ├── .copr └── Makefile ├── README.md ├── contrib └── iana-dns-params-toyaml.py ├── LICENSE ├── dsc_datatool ├── transformer │ ├── labler.py │ ├── net_remap.py │ └── re_ranger.py ├── input │ ├── xml.py │ └── dat.py ├── output │ ├── influxdb.py │ └── prometheus.py ├── generator │ ├── client_subnet_country.py │ └── client_subnet_authority.py └── __init__.py ├── CHANGES └── rpm └── dsc-datatool.spec /debian/compat: -------------------------------------------------------------------------------- 1 | 10 2 | -------------------------------------------------------------------------------- /debian/dirs: -------------------------------------------------------------------------------- 1 | usr/bin 2 | -------------------------------------------------------------------------------- /tests/20190719/idn_vs_tld.dat: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /debian/docs: -------------------------------------------------------------------------------- 1 | README.md 2 | CHANGES 3 | -------------------------------------------------------------------------------- /debian/source/format: -------------------------------------------------------------------------------- 1 | 3.0 (quilt) 2 | -------------------------------------------------------------------------------- /tests/20190719/ipv6_rsn_abusers_accum.dat: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /sonar-project.properties.local: -------------------------------------------------------------------------------- 1 | sonar.sources=dsc_datatool 2 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft dsc_datatool 2 | global-exclude *.pyc 3 | include CHANGES 4 | -------------------------------------------------------------------------------- /tests/20190719/opcode.dat: -------------------------------------------------------------------------------- 1 | 1563520560 0 345 2 | #MD5 3a960c4f0c7ba6d9062f78477f11f1e8 3 | -------------------------------------------------------------------------------- /tests/20190719/chaos_types_and_names.dat: -------------------------------------------------------------------------------- 1 | 1563520560 2 | #MD5 d0a25752e8594d8c013f127947cbc5f9 3 | -------------------------------------------------------------------------------- /tests/20190719/client_subnet_count.dat: -------------------------------------------------------------------------------- 1 | 1563520560 2 2 | #MD5 387ea7cf05e936539c3ec40a63d29dbf 3 | -------------------------------------------------------------------------------- /tests/20190719/do_bit.dat: -------------------------------------------------------------------------------- 1 | 1563520560 set 134 clr 211 2 | #MD5 7bb564b9b0ebd370f10339a349e82e42 3 | -------------------------------------------------------------------------------- /tests/20190719/idn_qname.dat: -------------------------------------------------------------------------------- 1 | 1563520560 normal 345 2 | #MD5 df6cf377b07e083c57953eda1e9100ae 3 | -------------------------------------------------------------------------------- /tests/20190719/rcode.dat: -------------------------------------------------------------------------------- 1 | 1563520560 0 416 3 8 2 6 2 | #MD5 7ae0768474205df1d88a4fae8d03b56d 3 | -------------------------------------------------------------------------------- /tests/20190719/rd_bit.dat: -------------------------------------------------------------------------------- 1 | 1563520560 clr 159 set 186 2 | #MD5 2dd58158fe3ebb4176faabd9199d06fe 3 | -------------------------------------------------------------------------------- /tests/20190719/edns_version.dat: -------------------------------------------------------------------------------- 1 | 1563520560 0 141 none 204 2 | #MD5 4c2f5b7124dea738c503bb5dccb232e2 3 | -------------------------------------------------------------------------------- /tests/20190719/ipv6_rsn_abusers_count.dat: -------------------------------------------------------------------------------- 1 | 1563520560 0 2 | #MD5 9b8bcef0308351ed7d24cbed894c2217 3 | -------------------------------------------------------------------------------- /tests/20190719/dnssec_qtype.dat: -------------------------------------------------------------------------------- 1 | 1563520560 else 315 48 25 43 5 2 | #MD5 2c7018cb7254ee62b7097fcb6239a57d 3 | -------------------------------------------------------------------------------- /tests/20190719/client_subnet_accum.dat: -------------------------------------------------------------------------------- 1 | 2620:ff:c000:: 5 2 | 64.191.0.0 198 3 | #MD5 7b9f4f6c1fad290e8d9f0f6ef2905b0f 4 | -------------------------------------------------------------------------------- /tests/20190719/qtype.dat: -------------------------------------------------------------------------------- 1 | 1563520560 12 78 2 7 6 4 15 9 else 48 1 128 28 71 2 | #MD5 afc958bfa6de0678f87f62fe9c9c134f 3 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [tool:pytest] 2 | testpaths = tests 3 | 4 | [coverage:run] 5 | branch = True 6 | source = 7 | dsc_datatool 8 | -------------------------------------------------------------------------------- /tests/20190719/client_subnet2_count.dat: -------------------------------------------------------------------------------- 1 | 1563520560 rfc1918-ptr 1 ok 2 non-auth-tld 1 2 | #MD5 e883a01b1b022817133db1995aa2c10b 3 | -------------------------------------------------------------------------------- /tests/20190719/client_subnet2_trace.dat: -------------------------------------------------------------------------------- 1 | 1563520560 non-auth-tld 18 rfc1918-ptr 1 ok 185 2 | #MD5 b4a4ddc379bfe93124277fc6e2c330cb 3 | -------------------------------------------------------------------------------- /tests/20190719/direction_vs_ipproto.dat: -------------------------------------------------------------------------------- 1 | 1563520560 recv udp:219 sent udp:289 else udp:267 2 | #MD5 425e11a1b3d2f17c5e87e54ff5ecbeb3 3 | -------------------------------------------------------------------------------- /tests/20190719/transport_vs_qtype.dat: -------------------------------------------------------------------------------- 1 | 1563520560 udp else:48:28:71:1:128:15:9:12:78:2:7:6:4 2 | #MD5 9712453c813073917ee807bfb91dc07a 3 | -------------------------------------------------------------------------------- /tests/20190719/certain_qnames_vs_qtype.dat: -------------------------------------------------------------------------------- 1 | 1563520560 else else:48:1:128:28:71:15:9:12:78:2:7:6:4 2 | #MD5 4bc33435cfa929c19592fa5bbfd4bb69 3 | -------------------------------------------------------------------------------- /tests/dsc-datatool.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | import sys 3 | from dsc_datatool import main 4 | 5 | if __name__ == "__main__": 6 | sys.exit(main()) 7 | -------------------------------------------------------------------------------- /tests/20190719/pcap_stats.dat: -------------------------------------------------------------------------------- 1 | 1563520560 filter_received eth0:5625 kernel_dropped eth0:731 pkts_captured eth0:4894 2 | #MD5 6dbbefe12290b8e7f589f86ec9c30556 3 | -------------------------------------------------------------------------------- /tests/test_main.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | import dsc_datatool as app 3 | 4 | 5 | def test_main(): 6 | with pytest.raises(Exception): 7 | app.main() 8 | -------------------------------------------------------------------------------- /tests/broken.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /tests/20190719/qtype_vs_qnamelen.dat: -------------------------------------------------------------------------------- 1 | 2 23 1 2 | 2 9 1 3 | 2 21 2 4 | 2 12 3 5 | 43 26 2 6 | 43 17 1 7 | 43 15 2 8 | 12 28 1 9 | 12 72 2 10 | 12 23 35 11 | 12 26 16 12 | 12 25 21 13 | 12 27 1 14 | 12 24 2 15 | 6 9 1 16 | 6 21 2 17 | 6 17 1 18 | 1 18 1 19 | 1 46 1 20 | 1 12 1 21 | 1 20 84 22 | 1 23 1 23 | 1 16 8 24 | 1 24 2 25 | 1 15 28 26 | 1 22 1 27 | 1 28 1 28 | 28 16 7 29 | 28 24 2 30 | 28 15 12 31 | 28 18 1 32 | 28 35 1 33 | 28 20 45 34 | 28 26 1 35 | 28 23 1 36 | 28 59 1 37 | 48 12 2 38 | 48 13 10 39 | 48 23 11 40 | 48 17 2 41 | 16 74 9 42 | 16 15 1 43 | 16 21 8 44 | 15 26 1 45 | 15 13 8 46 | #MD5 1cf280838630abd4d8cc5fe322bbd8ff 47 | -------------------------------------------------------------------------------- /man/man5/dsc-datatool.conf.5: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool.conf" "5" 2 | .SH NAME 3 | dsc-datatool.conf \- Configuration file for dsc-datatool. 4 | .SH DESCRIPTION 5 | Not implemented. 6 | .SH "SEE ALSO" 7 | .BR dsc-datatool (1) 8 | .SH AUTHORS 9 | Jerry Lundström, DNS-OARC 10 | .LP 11 | Maintained by DNS-OARC 12 | .LP 13 | .RS 14 | .I https://www.dns-oarc.net/tools/dsc 15 | .RE 16 | .LP 17 | .SH BUGS 18 | For issues and feature requests please use: 19 | .LP 20 | .RS 21 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 22 | .RE 23 | .LP 24 | For question and help please use: 25 | .LP 26 | .RS 27 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 28 | .RE 29 | .LP 30 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import find_packages, setup 2 | 3 | setup( 4 | name='dsc_datatool', 5 | version='1.4.2', 6 | packages=find_packages(), 7 | include_package_data=True, 8 | zip_safe=False, 9 | python_requires='>=3.5.1', 10 | install_requires=[ 11 | 'maxminddb>=1.2.0', 12 | 'PyYAML>=3.11', 13 | ], 14 | extras_require={ 15 | 'dev': [ 16 | 'pytest>=4', 17 | 'coverage', 18 | 'watchdog', 19 | ], 20 | }, 21 | entry_points={ 22 | 'console_scripts': [ 23 | 'dsc-datatool = dsc_datatool:main', 24 | ], 25 | }, 26 | scripts=[ 27 | ], 28 | ) 29 | -------------------------------------------------------------------------------- /debian/control: -------------------------------------------------------------------------------- 1 | Source: dsc-datatool 2 | Section: python 3 | Priority: optional 4 | Maintainer: Jerry Lundström 5 | Build-Depends: debhelper (>= 10), python3 (>= 3.5), python3-maxminddb, 6 | python3-yaml, python3-setuptools, dh-python 7 | Standards-Version: 3.9.4 8 | Homepage: https://www.dns-oarc.net/oarc/data/dsc 9 | Vcs-Git: https://codeberg.org/DNS-OARC/dsc-datatool.git 10 | Vcs-Browser: https://codeberg.org/DNS-OARC/dsc-datatool 11 | X-Python3-Version: >= 3.5 12 | 13 | Package: dsc-datatool 14 | Architecture: all 15 | Depends: ${misc:Depends}, ${python3:Depends} 16 | Suggests: dsc 17 | Description: Export DSC data to other formats and/or databases 18 | Tool for converting, exporting, merging and transforming DSC data. 19 | -------------------------------------------------------------------------------- /.forgejo/workflows/test.yml: -------------------------------------------------------------------------------- 1 | name: test 2 | on: pull_request 3 | jobs: 4 | test: 5 | runs-on: docker 6 | steps: 7 | - uses: actions/checkout@v4 8 | - name: Set up Python 3.13 9 | uses: actions/setup-python@v5 10 | - name: Install dependencies 11 | run: | 12 | apt-get update 13 | apt-get install -y python3-maxminddb python3-yaml wget python3-pip 14 | python3 -m pip install --upgrade pip 15 | pip install . 16 | - name: Test 17 | run: | 18 | cd tests 19 | rm ipv4-address-space.csv ipv6-unicast-address-assignments.csv 20 | wget https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv 21 | wget https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv 22 | ./test.sh 23 | -------------------------------------------------------------------------------- /tests/20190719/client_port_range.dat: -------------------------------------------------------------------------------- 1 | 1563520560 58368-59391 8 1024-2047 3 45056-46079 7 31744-32767 5 15360-16383 7 30720-31743 4 9216-10239 3 5120-6143 4 25600-26623 4 41984-43007 9 43008-44031 6 62464-63487 4 22528-23551 4 10240-11263 4 39936-40959 5 34816-35839 7 16384-17407 5 32768-33791 6 37888-38911 7 53248-54271 3 3072-4095 7 47104-48127 5 21504-22527 2 49152-50175 6 6144-7167 2 33792-34815 12 61440-62463 8 51200-52223 2 27648-28671 3 4096-5119 3 46080-47103 7 56320-57343 11 63488-64511 8 50176-51199 4 2048-3071 5 8192-9215 3 57344-58367 3 14336-15359 6 52224-53247 6 55296-56319 10 12288-13311 5 24576-25599 11 60416-61439 1 26624-27647 4 17408-18431 4 7168-8191 4 48128-49151 4 38912-39935 4 23552-24575 4 28672-29695 5 20480-21503 3 40960-41983 7 54272-55295 5 13312-14335 6 11264-12287 4 19456-20479 8 36864-37887 6 29696-30719 4 35840-36863 12 18432-19455 4 44032-45055 3 59392-60415 7 2 | #MD5 21f1a5b2da887ab9b0c608f34992d7de 3 | -------------------------------------------------------------------------------- /.copr/Makefile: -------------------------------------------------------------------------------- 1 | top=.. 2 | 3 | all: srpm 4 | 5 | prereq: $(top)/rpmbuild 6 | rpm -q git rpm-build >/dev/null || dnf -y install git rpm-build 7 | 8 | update-dist-tools: $(top)/dist-tools 9 | ( cd "$(top)/dist-tools" && git pull ) 10 | 11 | $(top)/dist-tools: 12 | git clone https://github.com/jelu/dist-tools.git "$(top)/dist-tools" 13 | 14 | $(top)/rpmbuild: 15 | mkdir -p "$(top)"/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} 16 | 17 | srpm: prereq update-dist-tools 18 | test -f .gitmodules && git submodule update --init || true 19 | echo "$(spec)" | grep -q "develop.spec" && auto_build_number=`date --utc +%s` message="Auto build `date --utc --iso-8601=seconds`" "$(top)/dist-tools/spec-new-changelog-entry" || true 20 | overwrite=yes nosign=yes "$(top)/dist-tools/create-source-packages" rpm 21 | cp ../*.orig.tar.gz "$(top)/rpmbuild/SOURCES/" 22 | echo "$(spec)" | grep -q "develop.spec" && rpmbuild -bs --define "%_topdir $(top)/rpmbuild" --undefine=dist rpm/*.spec || rpmbuild -bs --define "%_topdir $(top)/rpmbuild" --undefine=dist "$(spec)" 23 | cp "$(top)"/rpmbuild/SRPMS/*.src.rpm "$(outdir)" 24 | -------------------------------------------------------------------------------- /tests/20190719/rcode_vs_replylen.dat: -------------------------------------------------------------------------------- 1 | 2 31 6 2 | 0 293 3 3 | 0 112 1 4 | 0 232 31 5 | 0 396 2 6 | 0 89 1 7 | 0 302 1 8 | 0 94 1 9 | 0 298 3 10 | 0 1118 10 11 | 0 85 1 12 | 0 563 6 13 | 0 132 2 14 | 0 245 3 15 | 0 233 7 16 | 0 389 1 17 | 0 174 1 18 | 0 88 4 19 | 0 241 4 20 | 0 253 2 21 | 0 412 1 22 | 0 42 4 23 | 0 104 4 24 | 0 75 40 25 | 0 66 45 26 | 0 54 84 27 | 0 155 1 28 | 0 131 2 29 | 0 151 1 30 | 0 229 1 31 | 0 328 1 32 | 0 362 1 33 | 0 1100 2 34 | 0 98 1 35 | 0 722 4 36 | 0 368 7 37 | 0 111 1 38 | 0 237 1 39 | 0 235 1 40 | 0 1141 1 41 | 0 62 3 42 | 0 99 2 43 | 0 84 2 44 | 0 102 2 45 | 0 345 1 46 | 0 61 2 47 | 0 239 2 48 | 0 261 2 49 | 0 259 1 50 | 0 309 5 51 | 0 244 17 52 | 0 80 1 53 | 0 69 1 54 | 0 1472 6 55 | 0 587 3 56 | 0 77 1 57 | 0 125 1 58 | 0 127 1 59 | 0 110 2 60 | 0 324 1 61 | 0 591 1 62 | 0 96 2 63 | 0 271 9 64 | 0 50 2 65 | 0 296 8 66 | 0 202 1 67 | 0 299 1 68 | 0 268 6 69 | 0 1146 1 70 | 0 272 11 71 | 0 41 2 72 | 0 57 1 73 | 0 123 2 74 | 0 49 7 75 | 0 294 1 76 | 0 478 3 77 | 0 326 1 78 | 0 82 5 79 | 0 379 4 80 | 0 64 1 81 | 0 87 2 82 | 0 574 2 83 | 3 130 1 84 | 3 101 2 85 | 3 92 2 86 | 3 795 1 87 | 3 694 2 88 | #MD5 f7092a6510cfbece8835bc3b41f8e20d 89 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DSC DataTool 2 | 3 | [![Bugs](https://sonarcloud.io/api/project_badges/measure?project=dns-oarc%3Adsc-datatool&metric=bugs)](https://sonarcloud.io/summary/new_code?id=dns-oarc%3Adsc-datatool) [![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=dns-oarc%3Adsc-datatool&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=dns-oarc%3Adsc-datatool) 4 | 5 | Tool for converting, exporting, merging and transforming DSC data. 6 | 7 | Please have a look at the man-page(s) `dsc-datatool` (1) on how to use or 8 | [the wiki article](https://codeberg.org/DNS-OARC/dsc-datatool/wiki/Setting-up-a-test-Grafana) 9 | on how to set this up using Influx DB and Grafana. 10 | 11 | More information about DSC may be found here: 12 | - https://www.dns-oarc.net/tools/dsc 13 | - https://www.dns-oarc.net/oarc/data/dsc 14 | 15 | Issues should be reported here: 16 | - https://codeberg.org/DNS-OARC/dsc-datatool/issues 17 | 18 | General support and discussion: 19 | - Mattermost: https://chat.dns-oarc.net/community/channels/oarc-software 20 | 21 | ## Dependencies 22 | 23 | `dsc-datatool` requires the following Python libraries: 24 | - PyYAML 25 | - maxminddb 26 | 27 | ## Python Development Environment 28 | 29 | Using Ubuntu/Debian: 30 | 31 | ``` 32 | sudo apt-get install python3-maxminddb python3-yaml python3-venv 33 | python3 -m venv venv --system-site-packages 34 | . venv/bin/activate 35 | pip install -e . --no-deps 36 | ``` 37 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-output-influxdb.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-output influxdb" "7" 2 | .SH NAME 3 | InfluxDB \- InfluxDB output. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-output 7 | .I """;InfluxDB;""" 8 | .YS 9 | .SH DESCRIPTION 10 | This output generates InfluxDB importable output to stdout or to a specified 11 | file. 12 | .SH OPTIONS 13 | .TP 14 | .BR timestamp =[start|stop] 15 | Choose which timestamp from the dataset to use in the InfluxDB output to 16 | specify when the metrics took place. 17 | 18 | Default to 19 | .I start 20 | timestamp. 21 | .TP 22 | .BR file = 23 | Specify a file to output to instead of stdout. 24 | .TP 25 | .BR append 26 | If given, the output will be appended to the file specified rather then 27 | overwritten. 28 | .TP 29 | .BR dml 30 | Add a 31 | .I "# DML" 32 | header to the output. 33 | .TP 34 | .BR database =name 35 | If 36 | .I dml 37 | is used, this can be used to specify which database the output should be 38 | imported into. 39 | .LP 40 | .SH "SEE ALSO" 41 | .BR dsc-datatool (1), 42 | .BR influx (1) 43 | .SH AUTHORS 44 | Jerry Lundström, DNS-OARC 45 | .LP 46 | Maintained by DNS-OARC 47 | .LP 48 | .RS 49 | .I https://www.dns-oarc.net/tools/dsc 50 | .RE 51 | .LP 52 | .SH BUGS 53 | For issues and feature requests please use: 54 | .LP 55 | .RS 56 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 57 | .RE 58 | .LP 59 | For question and help please use: 60 | .LP 61 | .RS 62 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 63 | .RE 64 | .LP 65 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-transformer-labler.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-transformer labler" "7" 2 | .SH NAME 3 | Labler \- Rewrite numeric labels to textual labels using the provided YAML data as lookup tables. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-transform 7 | .I """;Labler;;yaml=""" 8 | .YS 9 | .SH DESCRIPTION 10 | This transformer rewrites labels in datasets based on information provided 11 | in a YAML file. 12 | 13 | The YAML structure is as follows: 14 | .EX 15 | --- 16 | dataset_name: 17 | DimentionName: 18 | DimentionValue: RewriteTo 19 | .EE 20 | 21 | The 22 | .I dataset_name 23 | is the name of the dataset, 24 | .I DimentionName 25 | is the name of the dimension to rewrite and 26 | .I DimensionValue 27 | is the value to change to 28 | .IR RewriteTo . 29 | .SH OPTIONS 30 | .TP 31 | .B dataset 32 | See 33 | .IR dsc-datatool (1) 34 | on how to specify which dataset(s) to run the transformer on. 35 | .TP 36 | .BR yaml = 37 | The YAML file to load rewrite data from (required). 38 | .LP 39 | .SH "SEE ALSO" 40 | .BR dsc-datatool (1) 41 | .SH AUTHORS 42 | Jerry Lundström, DNS-OARC 43 | .LP 44 | Maintained by DNS-OARC 45 | .LP 46 | .RS 47 | .I https://www.dns-oarc.net/tools/dsc 48 | .RE 49 | .LP 50 | .SH BUGS 51 | For issues and feature requests please use: 52 | .LP 53 | .RS 54 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 55 | .RE 56 | .LP 57 | For question and help please use: 58 | .LP 59 | .RS 60 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 61 | .RE 62 | .LP 63 | -------------------------------------------------------------------------------- /contrib/iana-dns-params-toyaml.py: -------------------------------------------------------------------------------- 1 | import yaml 2 | import csv 3 | from urllib.request import Request, urlopen 4 | from io import StringIO 5 | 6 | rcode = {} 7 | qtype = {} 8 | opcode = {} 9 | 10 | for row in csv.reader(StringIO(urlopen(Request('http://www.iana.org/assignments/dns-parameters/dns-parameters-6.csv')).read().decode('utf-8'))): 11 | if row[0] == 'RCODE': 12 | continue 13 | rcode[row[0]] = row[1] 14 | 15 | for row in csv.reader(StringIO(urlopen(Request('http://www.iana.org/assignments/dns-parameters/dns-parameters-4.csv')).read().decode('utf-8'))): 16 | if row[0] == 'TYPE': 17 | continue 18 | qtype[row[1]] = row[0] 19 | 20 | for row in csv.reader(StringIO(urlopen(Request('http://www.iana.org/assignments/dns-parameters/dns-parameters-5.csv')).read().decode('utf-8'))): 21 | if row[0] == 'OpCode': 22 | continue 23 | opcode[row[0]] = row[1] 24 | 25 | y = {} 26 | 27 | for n in ['rcode', 'client_addr_vs_rcode', 'rcode_vs_replylen']: 28 | y[n] = { 'Rcode': {} } 29 | for k, v in rcode.items(): 30 | y[n]['Rcode'][k] = v 31 | 32 | for n in ['qtype', 'transport_vs_qtype', 'certain_qnames_vs_qtype', 'qtype_vs_tld', 'qtype_vs_qnamelen', 'chaos_types_and_names', 'dns_ip_version_vs_qtype']: 33 | y[n] = { 'Qtype': {} } 34 | for k, v in qtype.items(): 35 | if v == '*': 36 | v = 'wildcard' 37 | y[n]['Qtype'][k] = v 38 | 39 | for n in ['opcode']: 40 | y[n] = { 'Opcode': {} } 41 | for k, v in rcode.items(): 42 | y[n]['Opcode'][k] = v 43 | 44 | print(yaml.dump(y, explicit_start=True, default_flow_style=False)) 45 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | DSC DataTool 2 | 3 | Copyright (c) 2016-2024 OARC, Inc. 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions 8 | are met: 9 | 10 | 1. Redistributions of source code must retain the above copyright 11 | notice, this list of conditions and the following disclaimer. 12 | 13 | 2. Redistributions in binary form must reproduce the above copyright 14 | notice, this list of conditions and the following disclaimer in 15 | the documentation and/or other materials provided with the 16 | distribution. 17 | 18 | 3. Neither the name of the copyright holder nor the names of its 19 | contributors may be used to endorse or promote products derived 20 | from this software without specific prior written permission. 21 | 22 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25 | FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26 | COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27 | INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 28 | BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 30 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 31 | LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 32 | ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 33 | POSSIBILITY OF SUCH DAMAGE. 34 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-transformer-netremap.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-transformer netremap" "7" 2 | .SH NAME 3 | NetRemap \- Remap network addresses to other ranges/subnets. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-transform 7 | .I """;NetRemap;;""" 8 | .YS 9 | .SH DESCRIPTION 10 | This transformer can remap network ranges on IP addresses and subnets. 11 | .SH OPTIONS 12 | .TP 13 | .B dataset 14 | See 15 | .IR dsc-datatool (1) 16 | on how to specify which dataset(s) to run the transformer on. 17 | .TP 18 | .BR net = 19 | The network to remap both IPv4 and IPv6 addresses and networks, specify 20 | only the number in a short-form network (e.g. "/"). 21 | .TP 22 | .BR v4net = 23 | The IPv4 network to use for IPv4 addresses and networks, default to 24 | .I net 25 | if that is specified. 26 | .TP 27 | .BR v6net = 28 | The IPv6 network to use for IPv6 addresses and networks, default to 29 | .I net 30 | if that is specified. 31 | .TP 32 | .BR nonstrict = 33 | If set (to any value) then a non-strict mode is enabled which will disregard 34 | any bad data in the dataset, skipping it completely. 35 | Default is strict mode. 36 | .LP 37 | .SH "SEE ALSO" 38 | .BR dsc-datatool (1) 39 | .SH AUTHORS 40 | Jerry Lundström, DNS-OARC 41 | .LP 42 | Maintained by DNS-OARC 43 | .LP 44 | .RS 45 | .I https://www.dns-oarc.net/tools/dsc 46 | .RE 47 | .LP 48 | .SH BUGS 49 | For issues and feature requests please use: 50 | .LP 51 | .RS 52 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 53 | .RE 54 | .LP 55 | For question and help please use: 56 | .LP 57 | .RS 58 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 59 | .RE 60 | .LP 61 | -------------------------------------------------------------------------------- /dsc_datatool/transformer/labler.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.transformer.labler 2 | 3 | See `man dsc-datatool-transformer labler`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import yaml 11 | 12 | from dsc_datatool import Transformer, encoding 13 | 14 | 15 | def _process(label, d): 16 | l = label.get(d.name, None) 17 | if d.values: 18 | if l is None: 19 | return 20 | 21 | values = d.values 22 | d.values = {} 23 | 24 | for k, v in values.items(): 25 | nk = l.get(k, None) 26 | d.values[nk or k] = v 27 | 28 | return 29 | 30 | if l: 31 | v = l.get(d.value, None) 32 | if v: 33 | d.value = v 34 | for d2 in d.dimensions: 35 | _process(label, d2) 36 | 37 | 38 | class Labler(Transformer): 39 | label = None 40 | 41 | 42 | def __init__(self, opts): 43 | Transformer.__init__(self, opts) 44 | if not 'yaml' in opts: 45 | raise Exception('yaml=file option required') 46 | f = open(opts.get('yaml'), 'r', encoding=encoding) 47 | self.label = yaml.safe_load(f) 48 | f.close() 49 | 50 | 51 | def process(self, datasets): 52 | if self.label is None: 53 | return 54 | 55 | for dataset in datasets: 56 | label = self.label.get(dataset.name, None) 57 | if label is None: 58 | continue 59 | 60 | for d in dataset.dimensions: 61 | _process(label, d) 62 | 63 | 64 | import sys 65 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 66 | Transformer.__init_subclass__(Labler) 67 | -------------------------------------------------------------------------------- /tests/test_objects.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | from dsc_datatool import Dataset, Dimension, Input, Output, Generator, Transformer 3 | 4 | 5 | def test_dataset(): 6 | o = Dataset() 7 | assert '%r' % o == '' 8 | 9 | 10 | def test_dimension(): 11 | o = Dimension('test') 12 | assert '%r' % o == '' 13 | 14 | 15 | def test_input(): 16 | o = Input() 17 | with pytest.raises(Exception): 18 | o.process("test") 19 | 20 | class Input1(Input): 21 | def process(self, file): 22 | pass 23 | with pytest.raises(Exception): 24 | class Input1(Input): 25 | def process(self, file): 26 | pass 27 | 28 | 29 | def test_output(): 30 | o = Output({}) 31 | with pytest.raises(Exception): 32 | o.process([]) 33 | 34 | class Output1(Output): 35 | def process(self, file): 36 | pass 37 | with pytest.raises(Exception): 38 | class Output1(Output): 39 | def process(self, file): 40 | pass 41 | 42 | 43 | def test_generator(): 44 | o = Generator({}) 45 | with pytest.raises(Exception): 46 | o.process([]) 47 | 48 | class Generator1(Generator): 49 | def process(self, file): 50 | pass 51 | with pytest.raises(Exception): 52 | class Generator1(Generator): 53 | def process(self, file): 54 | pass 55 | 56 | 57 | def test_transformer(): 58 | o = Transformer({}) 59 | with pytest.raises(Exception): 60 | o.process([]) 61 | 62 | class Transformer1(Transformer): 63 | def process(self, file): 64 | pass 65 | with pytest.raises(Exception): 66 | class Transformer1(Transformer): 67 | def process(self, file): 68 | pass 69 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-generator-client_subnet_country.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-generator client_subnet_country" "7" 2 | .SH NAME 3 | client_subnet_country \- Generates country information based on client subnet using Maxmind database. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-generator 7 | .I """;client_subnet_country;""" 8 | .YS 9 | .SH DESCRIPTION 10 | This generator looks up country code in a Maxmind database for the subnets 11 | in the 12 | .I client_subnet 13 | dataset and the dataset 14 | .I client_subnet_country 15 | from that. 16 | .SH OPTIONS 17 | .TP 18 | .BR path = 19 | Search for the Maxmind database in the specified 20 | .IR directory . 21 | This option can be given multiple times to search in multiple directories. 22 | 23 | Default search paths are: 24 | .IR /var/lib/GeoIP , 25 | .IR /usr/share/GeoIP , 26 | .IR /usr/local/share/GeoIP . 27 | .TP 28 | .BR filename =filename 29 | The database filename to search for, default to 30 | .IR GeoLite2-Country.mmdb . 31 | .TP 32 | .BR db =/path/to/database.mmdb 33 | Full path to Maxmind database, this option overrides 34 | .I path 35 | and 36 | .IR filename . 37 | .TP 38 | .BR nonstrict = 39 | If set (to any value) then a non-strict mode is enabled which will disregard 40 | any bad data in the dataset, skipping it completely. 41 | Default is strict mode. 42 | .LP 43 | .SH "SEE ALSO" 44 | .BR dsc-datatool (1), 45 | .BR geoipupdate (1) 46 | .SH AUTHORS 47 | Jerry Lundström, DNS-OARC 48 | .LP 49 | Maintained by DNS-OARC 50 | .LP 51 | .RS 52 | .I https://www.dns-oarc.net/tools/dsc 53 | .RE 54 | .LP 55 | .SH BUGS 56 | For issues and feature requests please use: 57 | .LP 58 | .RS 59 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 60 | .RE 61 | .LP 62 | For question and help please use: 63 | .LP 64 | .RS 65 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 66 | .RE 67 | .LP 68 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-transformer-reranger.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-transformer reranger" "7" 2 | .SH NAME 3 | ReRanger \- Rewrite ranged or numerical statistics into other ranges. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-transform 7 | .I """;ReRanger;;""" 8 | .YS 9 | .SH DESCRIPTION 10 | This transformer can re-range values in datasets to new rangers. 11 | 12 | It supports both values that are a single numerical value or one that is 13 | already a range itself 14 | .RI ( num\-num ). 15 | .SH OPTIONS 16 | .TP 17 | .B dataset 18 | See 19 | .IR dsc-datatool (1) 20 | on how to specify which dataset(s) to run the transformer on. 21 | .TP 22 | .BR key =[low|mid|high] 23 | Specify what value to use when re-ranging an already ranged value 24 | .RI (low\-high). 25 | 26 | Default to 27 | .IR mid , 28 | which is high minus low then divided by 2. 29 | .TP 30 | .BR func = 31 | The function to use when aggregating multiple values into a new ranger, 32 | default to 33 | .IR sum . 34 | 35 | Currently only one function exists, 36 | .IR sum , 37 | which adds the sum of all values into the new range. 38 | .TP 39 | .B allow_invalid_keys 40 | If given the "invalid" keys/values that is not numerical or a range will be 41 | passed through unmodified. 42 | .TP 43 | .BR range = 44 | The new range given as 45 | .I /number 46 | (required). 47 | 48 | For example 49 | .I range=/64 50 | will re-range all values into buckets of 64, 0-63, 64-127 and so on. 51 | .LP 52 | .SH "SEE ALSO" 53 | .BR dsc-datatool (1) 54 | .SH AUTHORS 55 | Jerry Lundström, DNS-OARC 56 | .LP 57 | Maintained by DNS-OARC 58 | .LP 59 | .RS 60 | .I https://www.dns-oarc.net/tools/dsc 61 | .RE 62 | .LP 63 | .SH BUGS 64 | For issues and feature requests please use: 65 | .LP 66 | .RS 67 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 68 | .RE 69 | .LP 70 | For question and help please use: 71 | .LP 72 | .RS 73 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 74 | .RE 75 | .LP 76 | -------------------------------------------------------------------------------- /debian/copyright: -------------------------------------------------------------------------------- 1 | Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ 2 | Upstream-Name: dsc-datatool 3 | Source: https://codeberg.org/DNS-OARC/dsc-datatool 4 | 5 | Files: * 6 | Copyright: 2016-2024 OARC, Inc. 7 | License: BSD-3-Clause 8 | 9 | Files: debian/* 10 | Copyright: 2024 Jerry Lundström 11 | License: BSD-3-Clause 12 | 13 | License: BSD-3-Clause 14 | Redistribution and use in source and binary forms, with or without 15 | modification, are permitted provided that the following conditions 16 | are met: 17 | . 18 | 1. Redistributions of source code must retain the above copyright 19 | notice, this list of conditions and the following disclaimer. 20 | . 21 | 2. Redistributions in binary form must reproduce the above copyright 22 | notice, this list of conditions and the following disclaimer in 23 | the documentation and/or other materials provided with the 24 | distribution. 25 | . 26 | 3. Neither the name of the copyright holder nor the names of its 27 | contributors may be used to endorse or promote products derived 28 | from this software without specific prior written permission. 29 | . 30 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 31 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 32 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 33 | FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 34 | COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 35 | INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 36 | BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 37 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 38 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 39 | LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 40 | ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 41 | POSSIBILITY OF SUCH DAMAGE. 42 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-generator-client_subnet_authority.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-generator client_subnet_authority" "7" 2 | .SH NAME 3 | client_subnet_authority \- Generates network authority information (RIR) based on client subnet. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-generator 7 | .I """;client_subnet_authority;""" 8 | .YS 9 | .SH DESCRIPTION 10 | This generator will lookup the network authority (RIR) based network 11 | information in the 12 | .I client_subnet 13 | dataset and generate a new dataset 14 | .IR client_subnet_authority . 15 | 16 | The network authority lookup will be done from imports of CSV files or by 17 | fetching the required data from IANA. 18 | .SH OPTIONS 19 | .TP 20 | .BR csv = 21 | Read the network authority information from the given 22 | .IR file(s) . 23 | This option can be a single file, a list of comma separated files or given 24 | multiple times. 25 | .TP 26 | .BR fetch =yes 27 | Fetch the network authority information from IANA IP address space registry 28 | for IPv4 and IPv6. 29 | .TP 30 | .BR urlv4 =url 31 | URL for IANA IPv4 address space registry, default to 32 | .IR https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv . 33 | .TP 34 | .BR urlv6 =url 35 | URL for IANA IPv6 address space registry, default to 36 | .IR https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv . 37 | .TP 38 | .BR nonstrict = 39 | If set (to any value) then a non-strict mode is enabled which will disregard 40 | any bad data in the dataset, skipping it completely. 41 | Default is strict mode. 42 | .LP 43 | .SH "SEE ALSO" 44 | .BR dsc-datatool (1) 45 | .SH AUTHORS 46 | Jerry Lundström, DNS-OARC 47 | .LP 48 | Maintained by DNS-OARC 49 | .LP 50 | .RS 51 | .I https://www.dns-oarc.net/tools/dsc 52 | .RE 53 | .LP 54 | .SH BUGS 55 | For issues and feature requests please use: 56 | .LP 57 | .RS 58 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 59 | .RE 60 | .LP 61 | For question and help please use: 62 | .LP 63 | .RS 64 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 65 | .RE 66 | .LP 67 | -------------------------------------------------------------------------------- /dsc_datatool/transformer/net_remap.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.transformer.net_remap 2 | 3 | See `man dsc-datatool-transformer netremap`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import ipaddress 11 | 12 | from dsc_datatool import Transformer, args 13 | 14 | 15 | class NetRemap(Transformer): 16 | v4net = None 17 | v6net = None 18 | nonstrict = False 19 | 20 | 21 | def __init__(self, opts): 22 | Transformer.__init__(self, opts) 23 | net = opts.get('net', None) 24 | self.v4net = opts.get('v4net', net) 25 | self.v6net = opts.get('v6net', net) 26 | 27 | if not self.v4net: 28 | raise Exception('v4net (or net) must be given') 29 | if not self.v6net: 30 | raise Exception('v6net (or net) must be given') 31 | 32 | if opts.get('nonstrict', False): 33 | self.nonstrict = True 34 | 35 | 36 | def _process(self, dimension): 37 | if not dimension.values: 38 | for d2 in dimension.dimensions: 39 | self._process(d2) 40 | return 41 | 42 | values = dimension.values 43 | dimension.values = {} 44 | 45 | for k, v in values.items(): 46 | if k == args.skipped_key: 47 | continue 48 | elif k == args.skipped_sum_key: 49 | dimension.values['0'] = v 50 | continue 51 | 52 | try: 53 | ip = ipaddress.ip_address(k) 54 | except Exception as e: 55 | if not self.nonstrict: 56 | raise e 57 | continue 58 | if ip.version == 4: 59 | nkey = str(ipaddress.IPv4Network('%s/%s' % (ip, self.v4net), strict=False).network_address) 60 | else: 61 | nkey = str(ipaddress.IPv6Network('%s/%s' % (ip, self.v6net), strict=False).network_address) 62 | 63 | if not nkey in dimension.values: 64 | dimension.values[nkey] = v 65 | else: 66 | dimension.values[nkey] += v 67 | 68 | 69 | def process(self, datasets): 70 | for dataset in datasets: 71 | for dimension in dataset.dimensions: 72 | self._process(dimension) 73 | 74 | 75 | import sys 76 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 77 | Transformer.__init_subclass__(NetRemap) 78 | -------------------------------------------------------------------------------- /man/man7/dsc-datatool-output-prometheus.7: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool-output prometheus" "7" 2 | .SH NAME 3 | Prometheus \- Prometheus output. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .B \-\-output 7 | .I """;Prometheus;""" 8 | .YS 9 | .SH DESCRIPTION 10 | This output generates Prometheus importable output to stdout or to a specified 11 | file. 12 | .SS Prometheus' node_exporter 13 | This output can be used together with Prometheus' 14 | .IR node_exporter 's 15 | Textfile Collector to automate statistics gathering but some specific 16 | setup and requirements must be meet. 17 | 18 | You must hide the timestamp with option 19 | .B timestamp=hide 20 | because timestamps are not supported by the Textfile Collector. 21 | 22 | You must make sure only one XML file from a server+node combination is 23 | processed at a time. 24 | Because otherwise you will get multiple data point for the same metric 25 | and this will generate errors from the Textfile Collector, since it does 26 | not support timestamps and cannot separate the measurements. 27 | 28 | You must make sure that only one file (per server+node combo) is generated 29 | for the Textfile Collector to read, and it should be the same between runs. 30 | See Textfile Collectors documentation how to setup that atomically. 31 | .SH OPTIONS 32 | .TP 33 | .BR timestamp =[hide|start|stop] 34 | Choose which timestamp from the dataset to use in the Prometheus output to 35 | specify when the metrics took place. 36 | 37 | Default to 38 | .I start 39 | timestamp. 40 | .TP 41 | .BR file = 42 | Specify a file to output to instead of stdout. 43 | .TP 44 | .BR append 45 | If given, the output will be appended to the file specified rather then 46 | overwritten. 47 | .TP 48 | .BR prefix = 49 | Use the given string as prefix on all metric names. 50 | .LP 51 | .SH "SEE ALSO" 52 | .BR dsc-datatool (1) 53 | 54 | .I https://prometheus.io/docs/guides/node-exporter/ 55 | 56 | .I https://github.com/prometheus/node_exporter#textfile-collector 57 | .SH AUTHORS 58 | Jerry Lundström, DNS-OARC 59 | .LP 60 | Maintained by DNS-OARC 61 | .LP 62 | .RS 63 | .I https://www.dns-oarc.net/tools/dsc 64 | .RE 65 | .LP 66 | .SH BUGS 67 | For issues and feature requests please use: 68 | .LP 69 | .RS 70 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 71 | .RE 72 | .LP 73 | For question and help please use: 74 | .LP 75 | .RS 76 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 77 | .RE 78 | .LP 79 | -------------------------------------------------------------------------------- /dsc_datatool/input/xml.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.input.xml 2 | 3 | Input plugin to generate `Dataset`'s from DSC XML files. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import logging 11 | from xml.dom import minidom 12 | import base64 13 | 14 | from dsc_datatool import Input, Dataset, Dimension, process_dataset 15 | 16 | 17 | class XML(Input): 18 | def process(self, file): 19 | dom = minidom.parse(file) 20 | datasets = [] 21 | for array in dom.getElementsByTagName('array'): 22 | if process_dataset and not array.getAttribute('name') in process_dataset: 23 | continue 24 | 25 | dataset = Dataset() 26 | dataset.name = array.getAttribute('name') 27 | dataset.start_time = int(array.getAttribute('start_time')) 28 | dataset.stop_time = int(array.getAttribute('stop_time')) 29 | 30 | dimensions = [None, None] 31 | for dimension in array.getElementsByTagName('dimension'): 32 | if dimension.getAttribute('number') == '1': 33 | if dimensions[0]: 34 | logging.warning('Overwriting dimension 1 for %s' % dataset.name) 35 | dimensions[0] = dimension.getAttribute('type') 36 | elif dimension.getAttribute('number') == '2': 37 | if dimensions[1]: 38 | logging.warning('Overwriting dimension 2 for %s' % dataset.name) 39 | dimensions[1] = dimension.getAttribute('type') 40 | else: 41 | logging.warning('Invalid dimension number %r for %s' % (dimension.getAttribute('number'), dataset.name)) 42 | 43 | for node1 in array.getElementsByTagName(dimensions[0]): 44 | d1 = Dimension(dimensions[0]) 45 | d1.value = node1.getAttribute('val') 46 | try: 47 | if node1.getAttribute('base64'): 48 | d1.value = base64.b64decode(d1.value).decode('utf-8') 49 | except Exception as e: 50 | pass 51 | dataset.dimensions.append(d1) 52 | 53 | d2 = Dimension(dimensions[1]) 54 | d1.dimensions.append(d2) 55 | for node2 in node1.getElementsByTagName(dimensions[1]): 56 | val = node2.getAttribute('val') 57 | try: 58 | if node2.getAttribute('base64'): 59 | val = base64.b64decode(val).decode('utf-8') 60 | except Exception as e: 61 | pass 62 | d2.values[val] = int(node2.getAttribute('count')) 63 | 64 | datasets.append(dataset) 65 | 66 | return datasets 67 | 68 | 69 | import sys 70 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 71 | Input.__init_subclass__(XML) 72 | -------------------------------------------------------------------------------- /dsc_datatool/output/influxdb.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.output.influxdb 2 | 3 | See `man dsc-datatool-output influxdb`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import re 11 | import sys 12 | import atexit 13 | 14 | from dsc_datatool import Output, args, encoding 15 | 16 | 17 | _re = re.compile(r'([,=\\\s])') 18 | 19 | 20 | def _key(key): 21 | return re.sub(_re, r'\\\1', key) 22 | 23 | 24 | def _val(val): 25 | ret = re.sub(_re, r'\\\1', val) 26 | if ret == '': 27 | return '""' 28 | return ret 29 | 30 | 31 | def _process(tags, timestamp, dimension, fh): 32 | if dimension.dimensions is None: 33 | return 34 | 35 | if len(dimension.dimensions) > 0: 36 | if not (dimension.name == 'All' and dimension.value == 'ALL'): 37 | tags += ',%s=%s' % (_key(dimension.name.lower()), _val(dimension.value)) 38 | for d2 in dimension.dimensions: 39 | _process(tags, timestamp, d2, fh) 40 | return 41 | 42 | if dimension.values is None: 43 | return 44 | 45 | if len(dimension.values) > 0: 46 | tags += ',%s=' % _key(dimension.name.lower()) 47 | 48 | for k, v in dimension.values.items(): 49 | print('%s%s value=%s %s' % (tags, _val(k), v, timestamp), file=fh) 50 | 51 | 52 | class InfluxDB(Output): 53 | start_timestamp = True 54 | fh = None 55 | 56 | 57 | def __init__(self, opts): 58 | Output.__init__(self, opts) 59 | timestamp = opts.get('timestamp', 'start') 60 | if timestamp == 'start': 61 | pass 62 | elif timestamp == 'stop': 63 | self.start_timestamp = False 64 | else: 65 | raise Exception('timestamp option invalid') 66 | file = opts.get('file', None) 67 | append = opts.get('append', False) 68 | if file: 69 | if append: 70 | self.fh = open(file, 'a', encoding=encoding) 71 | else: 72 | self.fh = open(file, 'w', encoding=encoding) 73 | atexit.register(self.close) 74 | else: 75 | self.fh = sys.stdout 76 | 77 | if opts.get('dml', False): 78 | print('# DML', file=self.fh) 79 | database = opts.get('database', None) 80 | if database: 81 | print('# CONTEXT-DATABASE: %s' % database, file=self.fh) 82 | 83 | 84 | def close(self): 85 | if self.fh: 86 | self.fh.close() 87 | self.fh = None 88 | 89 | 90 | def process(self, datasets): 91 | for dataset in datasets: 92 | tags = '%s,server=%s,node=%s' % (_key(dataset.name.lower()), args.server, args.node) 93 | if self.start_timestamp: 94 | timestamp = dataset.start_time * 1000000000 95 | else: 96 | timestamp = dataset.end_time * 1000000000 97 | 98 | for d in dataset.dimensions: 99 | _process(tags, timestamp, d, self.fh) 100 | 101 | 102 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 103 | Output.__init_subclass__(InfluxDB) 104 | -------------------------------------------------------------------------------- /dsc_datatool/generator/client_subnet_country.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.generator.client_subnet_country 2 | 3 | See `man dsc-datatool-generator client_subnet_country`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import maxminddb 11 | import os 12 | import logging 13 | 14 | from dsc_datatool import Generator, Dataset, Dimension, args 15 | 16 | 17 | class client_subnet_country(Generator): 18 | reader = None 19 | nonstrict = False 20 | 21 | 22 | def __init__(self, opts): 23 | Generator.__init__(self, opts) 24 | paths = opts.get('path', ['/var/lib/GeoIP', '/usr/share/GeoIP', '/usr/local/share/GeoIP']) 25 | if not isinstance(paths, list): 26 | paths = [ paths ] 27 | filename = opts.get('filename', 'GeoLite2-Country.mmdb') 28 | db = opts.get('db', None) 29 | 30 | if db is None: 31 | for path in paths: 32 | db = '%s/%s' % (path, filename) 33 | if os.path.isfile(db) and os.access(db, os.R_OK): 34 | break 35 | db = None 36 | if db is None: 37 | raise Exception('Please specify valid Maxmind database with path=,filename= or db=') 38 | 39 | logging.info('Using %s' % db) 40 | self.reader = maxminddb.open_database(db) 41 | 42 | if opts.get('nonstrict', False): 43 | self.nonstrict = True 44 | 45 | 46 | def process(self, datasets): 47 | gen_datasets = [] 48 | 49 | for dataset in datasets: 50 | if dataset.name != 'client_subnet': 51 | continue 52 | 53 | subnets = {} 54 | for d1 in dataset.dimensions: 55 | for d2 in d1.dimensions: 56 | for k, v in d2.values.items(): 57 | if k == args.skipped_key: 58 | continue 59 | elif k == args.skipped_sum_key: 60 | continue 61 | 62 | if k in subnets: 63 | subnets[k] += v 64 | else: 65 | subnets[k] = v 66 | 67 | cc = {} 68 | for subnet in subnets: 69 | try: 70 | c = self.reader.get(subnet) 71 | except Exception as e: 72 | if not self.nonstrict: 73 | raise e 74 | continue 75 | if c: 76 | iso_code = c.get('country', {}).get('iso_code', '??') 77 | if iso_code in cc: 78 | cc[iso_code] += subnets[subnet] 79 | else: 80 | cc[iso_code] = subnets[subnet] 81 | 82 | if cc: 83 | ccd = Dataset() 84 | ccd.name = 'client_subnet_country' 85 | ccd.start_time = dataset.start_time 86 | ccd.stop_time = dataset.stop_time 87 | gen_datasets.append(ccd) 88 | 89 | ccd1 = Dimension('ClientCountry') 90 | ccd1.values = cc 91 | ccd.dimensions.append(ccd1) 92 | 93 | return gen_datasets 94 | 95 | 96 | import sys 97 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 98 | Generator.__init_subclass__(client_subnet_country) 99 | -------------------------------------------------------------------------------- /tests/coverage.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -xe 4 | 5 | base=`dirname $0` 6 | 7 | export PATH="$base:$PATH" 8 | 9 | "$base/test.sh" 10 | 11 | dsc-datatool \ 12 | -vvv \ 13 | -s test-server \ 14 | -n test-node \ 15 | --output ";InfluxDB;dml=1;database=dsc" \ 16 | --transform ";Labler;*;yaml=$base/labler.yaml" \ 17 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 18 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 19 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 20 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 21 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 22 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 23 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 24 | --generator ";client_subnet_country;path=$HOME/GeoIP" \ 25 | --xml "$base/1563520620.dscdata.xml" >/dev/null 26 | 27 | ! dsc-datatool \ 28 | -vvv \ 29 | -s test-server \ 30 | -n test-node \ 31 | --output ";InfluxDB;dml=1;database=dsc" \ 32 | --transform ";Labler;*;yaml=$base/labler.yaml" \ 33 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 34 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 35 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 36 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 37 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 38 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 39 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 40 | --generator ";client_subnet_country;path=$HOME/GeoIP" \ 41 | --xml "$base" >/dev/null 42 | 43 | dsc-datatool \ 44 | -vvv \ 45 | -s test-server \ 46 | -n test-node \ 47 | --output ";InfluxDB;dml=1;database=dsc" \ 48 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 49 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 50 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 51 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 52 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 53 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 54 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 55 | --generator ";client_subnet_country;path=$HOME/GeoIP" \ 56 | --dat "$base/20190719" >/dev/null 57 | 58 | dsc-datatool -vvvvvvv --list >/dev/null 59 | ! dsc-datatool -s test -n test --output ";InfluxDB;test=a;test=b;test=c" >/dev/null 60 | ! dsc-datatool -s test -n test --generator does_not_exist >/dev/null 61 | ! dsc-datatool -s test -n test --generator does_not_exist,really_does_not_exist >/dev/null 62 | ! dsc-datatool -s test -n test --transform ";does_not_exist;*" >/dev/null 63 | ! dsc-datatool -s test -n test --transform ";ReRanger;a,a,a;range=/8" >/dev/null 64 | ! dsc-datatool -s test -n test --output does_not_exists >/dev/null 65 | ! dsc-datatool -s test -n test --dataset a --dataset b --dataset c,d,e >/dev/null 66 | ! dsc-datatool -s test -n test --dat "$base/coverage.sh" >/dev/null 67 | -------------------------------------------------------------------------------- /dsc_datatool/output/prometheus.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.output.prometheus 2 | 3 | See `man dsc-datatool-output prometheus`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import re 11 | import sys 12 | import atexit 13 | 14 | from dsc_datatool import Output, args, encoding 15 | 16 | 17 | _re = re.compile(r'([\\\n"])') 18 | 19 | 20 | def _key(key): 21 | return re.sub(_re, r'\\\1', key) 22 | 23 | 24 | def _val(val): 25 | ret = re.sub(_re, r'\\\1', val) 26 | if ret == '': 27 | return '""' 28 | return '"%s"' % ret 29 | 30 | 31 | class Prometheus(Output): 32 | show_timestamp = True 33 | start_timestamp = True 34 | fh = None 35 | type_def = '' 36 | type_printed = False 37 | prefix = '' 38 | 39 | 40 | def __init__(self, opts): 41 | Output.__init__(self, opts) 42 | timestamp = opts.get('timestamp', 'start') 43 | if timestamp == 'hide': 44 | self.show_timestamp = False 45 | elif timestamp == 'start': 46 | pass 47 | elif timestamp == 'stop': 48 | self.start_timestamp = False 49 | else: 50 | raise Exception('timestamp option invalid') 51 | file = opts.get('file', None) 52 | append = opts.get('append', False) 53 | if file: 54 | if append: 55 | self.fh = open(file, 'a', encoding=encoding) 56 | else: 57 | self.fh = open(file, 'w', encoding=encoding) 58 | atexit.register(self.close) 59 | else: 60 | self.fh = sys.stdout 61 | self.prefix = opts.get('prefix', '') 62 | 63 | 64 | def close(self): 65 | if self.fh: 66 | self.fh.close() 67 | self.fh = None 68 | 69 | 70 | def _process(self, tags, timestamp, dimension, fh): 71 | if dimension.dimensions is None: 72 | return 73 | 74 | if len(dimension.dimensions) > 0: 75 | if not (dimension.name == 'All' and dimension.value == 'ALL'): 76 | tags += ',%s=%s' % (_key(dimension.name.lower()), _val(dimension.value)) 77 | for d2 in dimension.dimensions: 78 | self._process(tags, timestamp, d2, fh) 79 | return 80 | 81 | if dimension.values is None: 82 | return 83 | 84 | if len(dimension.values) > 0: 85 | tags += ',%s=' % _key(dimension.name.lower()) 86 | 87 | for k, v in dimension.values.items(): 88 | if not self.type_printed: 89 | print(self.type_def, file=fh) 90 | self.type_printed = True 91 | if self.show_timestamp: 92 | print('%s%s} %s %s' % (tags, _val(k), v, timestamp), file=fh) 93 | else: 94 | print('%s%s} %s' % (tags, _val(k), v), file=fh) 95 | 96 | 97 | def process(self, datasets): 98 | for dataset in datasets: 99 | self.type_def = '# TYPE %s gauge' % _key(dataset.name.lower()) 100 | self.type_printed = False 101 | tags = '%s%s{server=%s,node=%s' % (self.prefix, _key(dataset.name.lower()), _val(args.server), _val(args.node)) 102 | if self.start_timestamp: 103 | timestamp = dataset.start_time * 1000 104 | else: 105 | timestamp = dataset.end_time * 1000 106 | 107 | for d in dataset.dimensions: 108 | self._process(tags, timestamp, d, self.fh) 109 | 110 | 111 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 112 | Output.__init_subclass__(Prometheus) 113 | -------------------------------------------------------------------------------- /dsc_datatool/transformer/re_ranger.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.transformer.re_ranger 2 | 3 | See `man dsc-datatool-transformer reranger`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import re 11 | 12 | from dsc_datatool import Transformer, args 13 | 14 | 15 | _key_re = re.compile(r'^(?:(\d+)|(\d+)-(\d+))$') 16 | 17 | 18 | class ReRanger(Transformer): 19 | key = None 20 | func = None 21 | allow_invalid_keys = None 22 | range = None 23 | split_by = None 24 | 25 | 26 | def __init__(self, opts): 27 | Transformer.__init__(self, opts) 28 | self.key = opts.get('key', 'mid') 29 | self.func = opts.get('func', 'sum') 30 | self.allow_invalid_keys = opts.get('allow_invalid_keys', False) 31 | self.range = opts.get('range', None) 32 | 33 | if self.allow_invalid_keys != False: 34 | self.allow_invalid_keys = True 35 | 36 | if self.range is None: 37 | raise Exception('range must be given') 38 | m = re.match(r'^/(\d+)$', self.range) 39 | if m is None: 40 | raise Exception('invalid range') 41 | self.split_by = int(m.group(1)) 42 | 43 | if self.key != 'low' and self.key != 'mid' and self.key != 'high': 44 | raise Exception('invalid key %r' % self.key) 45 | 46 | if self.func != 'sum': 47 | raise Exception('invalid func %r' % self.func) 48 | 49 | 50 | def _process(self, dimension): 51 | global _key_re 52 | 53 | if not dimension.values: 54 | for d2 in dimension.dimensions: 55 | self._process(d2) 56 | return 57 | 58 | values = dimension.values 59 | dimension.values = {} 60 | skipped = None 61 | 62 | for k, v in values.items(): 63 | low = None 64 | high = None 65 | 66 | m = _key_re.match(k) 67 | if m: 68 | low, low2, high = m.group(1, 2, 3) 69 | if high is None: 70 | low = int(low) 71 | high = low 72 | else: 73 | low = int(low2) 74 | high = int(high) 75 | elif k == args.skipped_key: 76 | continue 77 | elif k == args.skipped_sum_key: 78 | if skipped is None: 79 | skipped = v 80 | else: 81 | skipped += v 82 | continue 83 | elif self.allow_invalid_keys: 84 | dimension.values[k] = v 85 | continue 86 | else: 87 | raise Exception('invalid key %r' % k) 88 | 89 | if self.key == 'low': 90 | nkey = low 91 | elif self.key == 'mid': 92 | nkey = int(low + ( (high - low) / 2 )) 93 | else: 94 | nkey = high 95 | 96 | nkey = int(nkey / self.split_by) * self.split_by 97 | low = nkey 98 | high = nkey + self.split_by - 1 99 | 100 | if self.func == 'sum': 101 | if low != high: 102 | nkey = '%d-%d' % (low, high) 103 | else: 104 | nkey = str(nkey) 105 | 106 | if nkey in dimension.values: 107 | dimension.values[nkey] += v 108 | else: 109 | dimension.values[nkey] = v 110 | 111 | if skipped: 112 | dimension.values['skipped'] = skipped 113 | 114 | 115 | def process(self, datasets): 116 | for dataset in datasets: 117 | for dimension in dataset.dimensions: 118 | self._process(dimension) 119 | 120 | 121 | import sys 122 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 123 | Transformer.__init_subclass__(ReRanger) 124 | -------------------------------------------------------------------------------- /tests/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh -xe 2 | 3 | base=`dirname $0` 4 | 5 | dsc-datatool \ 6 | -vvv \ 7 | -s test-server \ 8 | -n test-node \ 9 | --output ";InfluxDB;dml=1;database=dsc" \ 10 | --transform ";Labler;*;yaml=$base/labler.yaml" \ 11 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 12 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 13 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 14 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 15 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 16 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 17 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 18 | --xml "$base/1563520620.dscdata.xml" | sort -s > "$base/test.out" 19 | 20 | sort -s "$base/test.gold" > "$base/test.gold.tmp" 21 | diff -u "$base/test.gold.tmp" "$base/test.out" 22 | 23 | dsc-datatool \ 24 | -vvv \ 25 | -s test-server \ 26 | -n test-node \ 27 | --output ";InfluxDB;dml=1;database=dsc" \ 28 | --transform ";Labler;*;yaml=$base/labler.yaml" \ 29 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 30 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 31 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 32 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 33 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 34 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 35 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 36 | --dat "$base/20190719" | sort -s > "$base/test.out" 37 | 38 | sort -s "$base/test.gold2" > "$base/test.gold2.tmp" 39 | diff -u "$base/test.gold2.tmp" "$base/test.out" 40 | 41 | dsc-datatool \ 42 | -vvv \ 43 | -s test-server \ 44 | -n test-node \ 45 | --output ";InfluxDB;dml=1;database=dsc" \ 46 | --transform ";Labler;*;yaml=$base/labler.yaml" \ 47 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 48 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 49 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 50 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 51 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 52 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 53 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 54 | --xml "$base/1458044657.xml" | sort -s > "$base/test.out" 55 | 56 | sort -s "$base/test.gold3" > "$base/test.gold3.tmp" 57 | diff -u "$base/test.gold3.tmp" "$base/test.out" 58 | 59 | dsc-datatool \ 60 | -vvv \ 61 | -s test-server-åäö \ 62 | -n test-node \ 63 | --output ";InfluxDB;dml=1;database=dsc" \ 64 | --transform ";Labler;*;yaml=$base/labler.yaml" \ 65 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \ 66 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \ 67 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \ 68 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \ 69 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \ 70 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=16" \ 71 | --generator ";client_subnet_authority;csv=$base/ipv4-address-space.csv;csv=$base/ipv6-unicast-address-assignments.csv" \ 72 | --xml "$base/utf8.xml" | sort -s > "$base/test.out" 73 | 74 | sort -s "$base/test.gold4" > "$base/test.gold4.tmp" 75 | diff -u "$base/test.gold4.tmp" "$base/test.out" 76 | -------------------------------------------------------------------------------- /tests/test.gold3: -------------------------------------------------------------------------------- 1 | # DML 2 | # CONTEXT-DATABASE: dsc 3 | pcap_stats,server=test-server,node=test-node,ifname=./1458044657.pcap.dist,pcap_stat=pkts_captured value=8 1458044655000000000 4 | label_count,server=test-server,node=test-node,labelcount=3 value=4 1458044655000000000 5 | label_count,server=test-server,node=test-node,labelcount=6 value=4 1458044655000000000 6 | third_ld_vs_rcode,server=test-server,node=test-node,rcode=0,thirdld=216.in-addr.arpa value=2 1458044655000000000 7 | third_ld_vs_rcode,server=test-server,node=test-node,rcode=0,thirdld=www.google.se value=1 1458044655000000000 8 | third_ld_vs_rcode,server=test-server,node=test-node,rcode=0,thirdld=www.google.com value=1 1458044655000000000 9 | second_ld_vs_rcode,server=test-server,node=test-node,rcode=0,secondld=in-addr.arpa value=2 1458044655000000000 10 | second_ld_vs_rcode,server=test-server,node=test-node,rcode=0,secondld=google.com value=1 1458044655000000000 11 | second_ld_vs_rcode,server=test-server,node=test-node,rcode=0,secondld=www.google.se value=1 1458044655000000000 12 | server,server=test-server,node=test-node,ip=8.8.8.8 value=8 1458044655000000000 13 | qr_aa_bits,server=test-server,node=test-node,direction=else,qraabits=qr\=0\,aa\=0 value=4 1458044655000000000 14 | qr_aa_bits,server=test-server,node=test-node,direction=else,qraabits=qr\=1\,aa\=0 value=4 1458044655000000000 15 | qname,server=test-server,node=test-node,name=100.209.58.216.in-addr.arpa value=2 1458044655000000000 16 | qname,server=test-server,node=test-node,name=www.google.se value=2 1458044655000000000 17 | qname,server=test-server,node=test-node,name=131.209.58.216.in-addr.arpa value=2 1458044655000000000 18 | qname,server=test-server,node=test-node,name=www.google.com value=2 1458044655000000000 19 | qclass,server=test-server,node=test-node,class=1 value=8 1458044655000000000 20 | dns_ip_version,server=test-server,node=test-node,version=IPv4 value=8 1458044655000000000 21 | ip_version,server=test-server,node=test-node,version=IPv4 value=8 1458044655000000000 22 | direction_vs_ipproto,server=test-server,node=test-node,direction=else,ipproto=udp value=8 1458044655000000000 23 | client_port,server=test-server,node=test-node,port=59978 value=4 1458044655000000000 24 | client_port,server=test-server,node=test-node,port=0 value=1 1458044655000000000 25 | client_port,server=test-server,node=test-node,port=53 value=1 1458044655000000000 26 | client_port,server=test-server,node=test-node,port=44275 value=1 1458044655000000000 27 | client_port,server=test-server,node=test-node,port=57483 value=1 1458044655000000000 28 | client_port_range,server=test-server,node=test-node,portrange=43008-45055 value=1 1458044655000000000 29 | client_port_range,server=test-server,node=test-node,portrange=55296-57343 value=1 1458044655000000000 30 | client_port_range,server=test-server,node=test-node,portrange=57344-59391 value=2 1458044655000000000 31 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=A value=2 1458044655000000000 32 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=PTR value=2 1458044655000000000 33 | tc_bit,server=test-server,node=test-node,tc=clr value=8 1458044655000000000 34 | rd_bit,server=test-server,node=test-node,rd=set value=4 1458044655000000000 35 | do_bit,server=test-server,node=test-node,d0=clr value=4 1458044655000000000 36 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=None value=4 1458044655000000000 37 | edns_version,server=test-server,node=test-node,ednsversion=none value=4 1458044655000000000 38 | idn_qname,server=test-server,node=test-node,idnqname=normal value=4 1458044655000000000 39 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=NoError,clientaddr=172.17.0.0 value=4 1458044655000000000 40 | client_subnet2,server=test-server,node=test-node,class=ok,clientsubnet=172.17.0.0 value=3 1458044655000000000 41 | client_subnet2,server=test-server,node=test-node,class=non-auth-tld,clientsubnet=172.17.0.0 value=1 1458044655000000000 42 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=A value=2 1458044655000000000 43 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=PTR value=2 1458044655000000000 44 | qtype_vs_tld,server=test-server,node=test-node,qtype=A,tld=com value=1 1458044655000000000 45 | qtype_vs_tld,server=test-server,node=test-node,qtype=A,tld=google.se value=1 1458044655000000000 46 | qtype_vs_tld,server=test-server,node=test-node,qtype=PTR,tld=arpa value=2 1458044655000000000 47 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=A,qnamelen=0-15 value=2 1458044655000000000 48 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=PTR,qnamelen=16-31 value=2 1458044655000000000 49 | client_subnet,server=test-server,node=test-node,clientsubnet=172.17.0.0 value=4 1458044655000000000 50 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=128-191 value=2 1458044655000000000 51 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=0-63 value=2 1458044655000000000 52 | opcode,server=test-server,node=test-node,opcode=NoError value=4 1458044655000000000 53 | rcode,server=test-server,node=test-node,rcode=NoError value=4 1458044655000000000 54 | qtype,server=test-server,node=test-node,qtype=A value=2 1458044655000000000 55 | qtype,server=test-server,node=test-node,qtype=PTR value=2 1458044655000000000 56 | client_subnet_authority,server=test-server,node=test-node,clientauthority=ARIN value=4 1458044655000000000 57 | -------------------------------------------------------------------------------- /man/man1/dsc-datatool.1: -------------------------------------------------------------------------------- 1 | .TH "dsc-datatool" "1" 2 | .SH NAME 3 | dsc-datatool \- Tool for converting, exporting, merging and transforming DSC data. 4 | .SH SYNOPSIS 5 | .SY dsc-datatool 6 | .OP \-h 7 | .OP \-c CONF 8 | .OP \-s SERVER 9 | .OP \-n NODE 10 | .OP \-x XML 11 | .OP \-d DAT 12 | .OP \-\-dataset DATASET 13 | .OP \-o OUTPUT 14 | .OP \-t TRANSFORM] 15 | .OP \-g GENERATOR 16 | .OP \-\-list 17 | .OP \-\-skipped\-key SKIPPED_KEY 18 | .OP \-\-skipped\-sum\-key SKIPPED_SUM_KEY 19 | .OP \-v 20 | .OP \-V 21 | .YS 22 | .SH DESCRIPTION 23 | Tool for converting, exporting, merging and transforming DSC data. 24 | 25 | Please have a look at the wiki article on how to set this up using 26 | Influx DB and Grafana. 27 | 28 | https://codeberg.org/DNS-OARC/dsc-datatool/wiki/Setting-up-a-test-Grafana 29 | .SH OPTIONS 30 | .TP 31 | .B -h, --help 32 | show this help message and exit 33 | .TP 34 | .BI "-c " CONF ", --conf " CONF 35 | Not implemented 36 | .TP 37 | .BI "-s " SERVER ", --server " SERVER 38 | Specify the server for where the data comes from. (required) 39 | .TP 40 | .BI "-n " NODE ", --node " NODE 41 | Specify the node for where the data comes from. (required) 42 | .TP 43 | .BI "-x " XML ", --xml " XML 44 | Read DSC data from the given file or directory, can be specified multiple 45 | times. 46 | If a directory is given then all files ending with .xml will be read. 47 | .TP 48 | .BI "-d " DAT ", --dat " DAT 49 | Read DSC data from the given directory, can be specified multiple times. 50 | Note that the DAT format is depended on the filename to know what type of 51 | data it is. 52 | .TP 53 | .BI "--dataset " DATASET 54 | Specify that only the list of datasets will be processed, the list is 55 | comma separated and the option can be given multiple times. 56 | .TP 57 | .BI "-o " OUTPUT ", --output " OUTPUT 58 | .I OUTPUT 59 | has the following format that uses 60 | .I output 61 | to specify the output module and 62 | .I sep 63 | as an options separator. 64 | 65 | .EX 66 | [option=value...]> 67 | .EE 68 | 69 | Can be specified multiple times to output to more then one. 70 | 71 | Use 72 | .B dsc-datatool --list 73 | to see a list of modules and the man-page of each output for information 74 | about options. 75 | .TP 76 | .BI "-t " TRANSFORM ", --transform " TRANSFORM 77 | .I TRANSFORM 78 | has the following format that uses 79 | .I name 80 | to specify the transformer module and 81 | .I sep 82 | as an options separator. 83 | The 84 | .I datasets 85 | field can specify which dataset to run the transformer on, or use 86 | .I * 87 | to specify all datasets. 88 | 89 | .EX 90 | [option=value...]> 91 | .EE 92 | 93 | Can be specific multiple times to chain transformation, the chain will be 94 | executed in the order on command line with one exception. 95 | All transformations specified for dataset 96 | .I * 97 | will be executed before named dataset transformations. 98 | 99 | Use 100 | .B dsc-datatool --list 101 | to see a list of modules and the man-page of each transformer for 102 | information about options. 103 | For a list of datasets see the DSC configuration that creates the XML files 104 | or the documentation for the Presenter that creates the DAT files. 105 | .TP 106 | .BI "-g " GENERATOR ", --generator " GENERATOR 107 | .I GENERATOR 108 | has two formats, one to specify a comma separated list of generators 109 | and one that uses 110 | .I name 111 | to specify the generator module and 112 | .I sep 113 | as an options separator. 114 | 115 | .EX 116 | [,,...] 117 | 118 | [option=value...]> 119 | .EE 120 | 121 | This option can be given multiple times. 122 | 123 | Use 124 | .B dsc-datatool --list 125 | to see a list of modules and the man-page of each generator for 126 | information about options. 127 | .TP 128 | .B --list 129 | List the available generators, transformers and outputs then exit. 130 | .TP 131 | .BI "--skipped-key " SKIPPED_KEY 132 | Set the special DSC skipped key. (default to "-:SKIPPED:-") 133 | .TP 134 | .BI "--skipped-sum-key " SKIPPED_SUM_KEY 135 | Set the special DSC skipped sum key. (default to "-:SKIPPED_SUM:-") 136 | .TP 137 | .BI "--encoding " ENCODING 138 | Set the encoding to use when reading and writing files, default to utf-8. 139 | .TP 140 | .B -v, --verbose 141 | Increase the verbose level, can be given multiple times. 142 | .TP 143 | .B -V, --version 144 | Display version and exit. 145 | .LP 146 | .SH EXAMPLE 147 | .EX 148 | dsc-datatool \\ 149 | --server "$SERVER" \\ 150 | --node "$NODE" \\ 151 | --output ";InfluxDB;file=influx.txt;dml=1;database=dsc" \\ 152 | --transform ";Labler;*;yaml=$HOME/labler.yaml" \\ 153 | --transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \\ 154 | --transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \\ 155 | --transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \\ 156 | --transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \\ 157 | --transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \\ 158 | --transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=8" \\ 159 | --generator client_subnet_country \\ 160 | --generator ";client_subnet_authority;fetch=yes" \\ 161 | --xml "$XML" 162 | .EE 163 | .SH "SEE ALSO" 164 | .BR dsc-datatool.conf (5), 165 | .BI dsc-datatool- [ generator | transformer | output ] 166 | .BR (7) 167 | .SH AUTHORS 168 | Jerry Lundström, DNS-OARC 169 | .LP 170 | Maintained by DNS-OARC 171 | .LP 172 | .RS 173 | .I https://www.dns-oarc.net/tools/dsc 174 | .RE 175 | .LP 176 | .SH BUGS 177 | For issues and feature requests please use: 178 | .LP 179 | .RS 180 | \fIhttps://codeberg.org/DNS-OARC/dsc-datatool/issues\fP 181 | .RE 182 | .LP 183 | For question and help please use: 184 | .LP 185 | .RS 186 | \fIhttps://chat.dns-oarc.net/community/channels/oarc-software\fP 187 | .RE 188 | .LP 189 | -------------------------------------------------------------------------------- /dsc_datatool/input/dat.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.input.dat 2 | 3 | Input plugin to generate `Dataset`'s from DSC DAT files. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import re 11 | 12 | from dsc_datatool import Input, Dataset, Dimension, process_dataset, encoding 13 | 14 | 15 | _dataset1d = [ 16 | 'client_subnet_count', 17 | 'ipv6_rsn_abusers_count', 18 | ] 19 | 20 | _dataset2d = { 21 | 'qtype': 'Qtype', 22 | 'rcode': 'Rcode', 23 | 'do_bit': 'D0', 24 | 'rd_bit': 'RD', 25 | 'opcode': 'Opcode', 26 | 'dnssec_qtype': 'Qtype', 27 | 'edns_version': 'EDNSVersion', 28 | 'client_subnet2_count': 'Class', 29 | 'client_subnet2_trace': 'Class', 30 | 'edns_bufsiz': 'EDNSBufSiz', 31 | 'idn_qname': 'IDNQname', 32 | 'client_port_range': 'PortRange', 33 | 'priming_responses': 'ReplyLen', 34 | } 35 | 36 | _dataset3d = { 37 | 'chaos_types_and_names': [ 'Qtype', 'Qname' ], 38 | 'certain_qnames_vs_qtype': [ 'CertainQnames', 'Qtype' ], 39 | 'direction_vs_ipproto': [ 'Direction', 'IPProto' ], 40 | 'pcap_stats': [ 'pcap_stat', 'ifname' ], 41 | 'transport_vs_qtype': [ 'Transport', 'Qtype' ], 42 | 'dns_ip_version': [ 'IPVersion', 'Qtype' ], 43 | 'priming_queries': [ 'Transport', 'EDNSBufSiz' ], 44 | 'qr_aa_bits': [ 'Direction', 'QRAABits' ], 45 | } 46 | 47 | 48 | class DAT(Input): 49 | def process(self, dir): 50 | global _dataset1d, _dataset2d, _dataset3d 51 | 52 | datasets = [] 53 | 54 | for d in _dataset1d: 55 | if process_dataset and not d in process_dataset: 56 | continue 57 | try: 58 | datasets += self.process1d('%s/%s.dat' % (dir, d), d) 59 | except FileNotFoundError: 60 | pass 61 | for k, v in _dataset2d.items(): 62 | if process_dataset and not k in process_dataset: 63 | continue 64 | try: 65 | datasets += self.process2d('%s/%s.dat' % (dir, k), k, v) 66 | except FileNotFoundError: 67 | pass 68 | for k, v in _dataset3d.items(): 69 | if process_dataset and not k in process_dataset: 70 | continue 71 | try: 72 | datasets += self.process3d('%s/%s.dat' % (dir, k), k, v[0], v[1]) 73 | except FileNotFoundError: 74 | pass 75 | 76 | return datasets 77 | 78 | 79 | def process1d(self, file, name): 80 | datasets = [] 81 | with open(file, 'r', encoding=encoding) as f: 82 | for l in f.readlines(): 83 | if re.match(r'^#', l): 84 | continue 85 | l = re.sub(r'[\r\n]+$', '', l) 86 | dat = re.split(r'\s+', l) 87 | if len(dat) != 2: 88 | raise Exception('DAT %r dataset %r: invalid number of elements for a 1d dataset' % (file, name)) 89 | 90 | dataset = Dataset() 91 | dataset.name = name 92 | dataset.start_time = int(dat.pop(0)) 93 | dataset.stop_time = dataset.start_time + 60 94 | 95 | d1 = Dimension('All') 96 | d1.values = { 'ALL': int(dat[0]) } 97 | dataset.dimensions.append(d1) 98 | 99 | datasets.append(dataset) 100 | 101 | return datasets 102 | 103 | 104 | def process2d(self, file, name, field): 105 | datasets = [] 106 | with open(file, 'r', encoding=encoding) as f: 107 | for l in f.readlines(): 108 | if re.match(r'^#', l): 109 | continue 110 | l = re.sub(r'[\r\n]+$', '', l) 111 | dat = re.split(r'\s+', l) 112 | 113 | dataset = Dataset() 114 | dataset.name = name 115 | dataset.start_time = int(dat.pop(0)) 116 | dataset.stop_time = dataset.start_time + 60 117 | 118 | d1 = Dimension('All') 119 | d1.value = 'ALL' 120 | dataset.dimensions.append(d1) 121 | 122 | d2 = Dimension(field) 123 | while dat: 124 | if len(dat) < 2: 125 | raise Exception('DAT %r dataset %r: invalid number of elements for a 2d dataset' % (file, name)) 126 | k = dat.pop(0) 127 | v = dat.pop(0) 128 | d2.values[k] = int(v) 129 | d1.dimensions.append(d2) 130 | 131 | datasets.append(dataset) 132 | 133 | return datasets 134 | 135 | 136 | def process3d(self, file, name, first, second): 137 | datasets = [] 138 | with open(file, 'r', encoding=encoding) as f: 139 | for l in f.readlines(): 140 | if re.match(r'^#', l): 141 | continue 142 | l = re.sub(r'[\r\n]+$', '', l) 143 | dat = re.split(r'\s+', l) 144 | 145 | dataset = Dataset() 146 | dataset.name = name 147 | dataset.start_time = int(dat.pop(0)) 148 | dataset.stop_time = dataset.start_time + 60 149 | 150 | while dat: 151 | if len(dat) < 2: 152 | raise Exception('DAT %r dataset %r: invalid number of elements for a 2d dataset' % (file, name)) 153 | k = dat.pop(0) 154 | v = dat.pop(0) 155 | 156 | d1 = Dimension(first) 157 | d1.value = k 158 | dataset.dimensions.append(d1) 159 | 160 | d2 = Dimension(second) 161 | dat2 = v.split(':') 162 | while dat2: 163 | if len(dat2) < 2: 164 | raise Exception('DAT %r dataset %r: invalid number of elements for a 2d dataset' % (file, name)) 165 | k2 = dat2.pop(0) 166 | v2 = dat2.pop(0) 167 | d2.values[k2] = int(v2) 168 | d1.dimensions.append(d2) 169 | 170 | datasets.append(dataset) 171 | 172 | return datasets 173 | 174 | 175 | import sys 176 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 177 | Input.__init_subclass__(DAT) 178 | -------------------------------------------------------------------------------- /dsc_datatool/generator/client_subnet_authority.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool.generator.client_subnet_authority 2 | 3 | See `man dsc-datatool-generator client_subnet_authority`. 4 | 5 | Part of dsc_datatool. 6 | 7 | :copyright: 2024 OARC, Inc. 8 | """ 9 | 10 | import csv 11 | import ipaddress 12 | import logging 13 | from urllib.request import Request, urlopen 14 | from io import StringIO 15 | 16 | from dsc_datatool import Generator, Dataset, Dimension, args, encoding 17 | 18 | 19 | _whois2rir = { 20 | 'whois.apnic.net': 'APNIC', 21 | 'whois.arin.net': 'ARIN', 22 | 'whois.ripe.net': 'RIPE NCC', 23 | 'whois.lacnic.net': 'LACNIC', 24 | 'whois.afrinic.net': 'AFRINIC', 25 | } 26 | 27 | _desig2rir = { 28 | 'apnic': 'APNIC', 29 | 'arin': 'ARIN', 30 | 'ripe ncc': 'RIPE NCC', 31 | 'lacnic': 'LACNIC', 32 | 'afrinic': 'AFRINIC', 33 | 'iana': 'IANA', 34 | '6to4': 'IANA', 35 | } 36 | 37 | class client_subnet_authority(Generator): 38 | auth = None 39 | nonstrict = False 40 | 41 | 42 | def _read(self, input): 43 | global _whois2rir, _desig2rir 44 | for row in csv.reader(input): 45 | prefix, designation, date, whois, rdap, status, note = row 46 | if prefix == 'Prefix': 47 | continue 48 | rir = designation.replace('Administered by ', '').lower() 49 | 50 | whois = whois.lower() 51 | if whois in _whois2rir: 52 | rir = _whois2rir[whois] 53 | else: 54 | if rir in _desig2rir: 55 | rir = _desig2rir[rir] 56 | else: 57 | found = None 58 | for k, v in _desig2rir.items(): 59 | if k in rir: 60 | found = v 61 | break 62 | if found: 63 | rir = found 64 | else: 65 | if status == 'RESERVED': 66 | rir = 'IANA' 67 | elif designation == 'Segment Routing (SRv6) SIDs': 68 | # TODO: How to better handle this weird allocation? 69 | rir = 'IANA' 70 | else: 71 | raise Exception('Unknown whois/designation: %r/%r' % (whois, designation)) 72 | 73 | try: 74 | net = ipaddress.ip_network(prefix) 75 | except Exception: 76 | ip, net = prefix.split('/') 77 | net = ipaddress.ip_network('%s.0.0.0/%s' % (int(ip), net)) 78 | 79 | if net.version == 4: 80 | idx = ipaddress.ip_network('%s/8' % net.network_address, strict=False) 81 | else: 82 | idx = ipaddress.ip_network('%s/24' % net.network_address, strict=False) 83 | 84 | if idx.network_address in self.auth: 85 | self.auth[idx.network_address].append({'net': net, 'auth': rir}) 86 | else: 87 | self.auth[idx.network_address] = [{'net': net, 'auth': rir}] 88 | 89 | 90 | def __init__(self, opts): 91 | Generator.__init__(self, opts) 92 | self.auth = {} 93 | csvs = opts.get('csv', None) 94 | urlv4 = opts.get('urlv4', 'https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv') 95 | urlv6 = opts.get('urlv6', 'https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv') 96 | if opts.get('nonstrict', False): 97 | self.nonstrict = True 98 | 99 | if csvs: 100 | if not isinstance(csvs, list): 101 | csvs = [ csvs ] 102 | for file in csvs: 103 | with open(file, newline='', encoding=encoding) as csvfile: 104 | self._read(csvfile) 105 | elif opts.get('fetch', 'no').lower() == 'yes': 106 | urls = opts.get('url', [ urlv4, urlv6 ]) 107 | if urls and not isinstance(urls, list): 108 | urls = [ urls ] 109 | logging.info('bootstrapping client subnet authority using URLs') 110 | for url in urls: 111 | logging.info('fetching %s' % url) 112 | self._read(StringIO(urlopen(Request(url)).read().decode('utf-8'))) 113 | else: 114 | raise Exception('No authorities bootstrapped, please specify csv= or fetch=yes') 115 | 116 | 117 | def process(self, datasets): 118 | gen_datasets = [] 119 | 120 | for dataset in datasets: 121 | if dataset.name != 'client_subnet': 122 | continue 123 | 124 | subnets = {} 125 | for d1 in dataset.dimensions: 126 | for d2 in d1.dimensions: 127 | for k, v in d2.values.items(): 128 | if k == args.skipped_key: 129 | continue 130 | elif k == args.skipped_sum_key: 131 | continue 132 | 133 | if k in subnets: 134 | subnets[k] += v 135 | else: 136 | subnets[k] = v 137 | 138 | auth = {} 139 | for subnet in subnets: 140 | try: 141 | ip = ipaddress.ip_address(subnet) 142 | except Exception as e: 143 | if not self.nonstrict: 144 | raise e 145 | continue 146 | if ip.version == 4: 147 | idx = ipaddress.ip_network('%s/8' % ip, strict=False) 148 | ip = ipaddress.ip_network('%s/32' % ip) 149 | else: 150 | idx = ipaddress.ip_network('%s/24' % ip, strict=False) 151 | ip = ipaddress.ip_network('%s/128' % ip) 152 | if not idx.network_address in self.auth: 153 | idx = '??' 154 | else: 155 | for entry in self.auth[idx.network_address]: 156 | if entry['net'].overlaps(ip): 157 | idx = entry['auth'] 158 | break 159 | 160 | if idx in auth: 161 | auth[idx] += subnets[subnet] 162 | else: 163 | auth[idx] = subnets[subnet] 164 | 165 | if auth: 166 | authd = Dataset() 167 | authd.name = 'client_subnet_authority' 168 | authd.start_time = dataset.start_time 169 | authd.stop_time = dataset.stop_time 170 | gen_datasets.append(authd) 171 | 172 | authd1 = Dimension('ClientAuthority') 173 | authd1.values = auth 174 | authd.dimensions.append(authd1) 175 | 176 | return gen_datasets 177 | 178 | 179 | import sys 180 | if sys.version_info[0] == 3 and sys.version_info[1] == 5: # pragma: no cover 181 | Generator.__init_subclass__(client_subnet_authority) 182 | -------------------------------------------------------------------------------- /tests/ipv6-unicast-address-assignments.csv: -------------------------------------------------------------------------------- 1 | Prefix,Designation,Date,WHOIS,RDAP,Status,Note 2 | 2001:0000::/23,IANA,1999-07-01,whois.iana.org,,ALLOCATED,"2001:0000::/23 is reserved for IETF Protocol Assignments [RFC2928]. 3 | 2001:0000::/32 is reserved for TEREDO [RFC4380]. 4 | 2001:1::1/128 is reserved for Port Control Protocol Anycast [RFC7723]. 5 | 2001:1::2/128 is reserved for Traversal Using Relays around NAT Anycast [RFC8155]. 6 | 2001:1::3/128 is reserved for DNS-SD Service Registration Protocol Anycast [RFC-ietf-dnssd-srp-25]. 7 | 2001:2::/48 is reserved for Benchmarking [RFC5180][RFC Errata 1752]. 8 | 2001:3::/32 is reserved for AMT [RFC7450]. 9 | 2001:4:112::/48 is reserved for AS112-v6 [RFC7535]. 10 | 2001:10::/28 is deprecated (previously ORCHID) [RFC4843]. 11 | 2001:20::/28 is reserved for ORCHIDv2 [RFC7343]. 12 | 2001:30::/28 is reserved for Drone Remote ID Protocol Entity Tags (DETs) [RFC9374]. 13 | For complete registration details, see [IANA registry iana-ipv6-special-registry]." 14 | 2001:0200::/23,APNIC,1999-07-01,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED, 15 | 2001:0400::/23,ARIN,1999-07-01,whois.arin.net,"https://rdap.arin.net/registry 16 | http://rdap.arin.net/registry",ALLOCATED, 17 | 2001:0600::/23,RIPE NCC,1999-07-01,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 18 | 2001:0800::/22,RIPE NCC,2002-11-02,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED,2001:0800::/23 was allocated on 2002-05-02. The more recent allocation (2002-11-02) incorporates the previous allocation. 19 | 2001:0c00::/23,APNIC,2002-05-02,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED,"2001:db8::/32 reserved for Documentation [RFC3849]. 20 | For complete registration details, see [IANA registry iana-ipv6-special-registry]." 21 | 2001:0e00::/23,APNIC,2003-01-01,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED, 22 | 2001:1200::/23,LACNIC,2002-11-01,whois.lacnic.net,https://rdap.lacnic.net/rdap/,ALLOCATED, 23 | 2001:1400::/22,RIPE NCC,2003-07-01,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED,2001:1400::/23 was allocated on 2003-02-01. The more recent allocation (2003-07-01) incorporates the previous allocation. 24 | 2001:1800::/23,ARIN,2003-04-01,whois.arin.net,"https://rdap.arin.net/registry 25 | http://rdap.arin.net/registry",ALLOCATED, 26 | 2001:1a00::/23,RIPE NCC,2004-01-01,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 27 | 2001:1c00::/22,RIPE NCC,2004-05-04,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 28 | 2001:2000::/19,RIPE NCC,2019-03-12,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED,"2001:2000::/20, 2001:3000::/21, and 2001:3800::/22 were allocated on 2004-05-04. The more recent allocation (2019-03-12) incorporates all these previous allocations." 29 | 2001:4000::/23,RIPE NCC,2004-06-11,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 30 | 2001:4200::/23,AFRINIC,2004-06-01,whois.afrinic.net,"https://rdap.afrinic.net/rdap/ 31 | http://rdap.afrinic.net/rdap/",ALLOCATED, 32 | 2001:4400::/23,APNIC,2004-06-11,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED, 33 | 2001:4600::/23,RIPE NCC,2004-08-17,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 34 | 2001:4800::/23,ARIN,2004-08-24,whois.arin.net,"https://rdap.arin.net/registry 35 | http://rdap.arin.net/registry",ALLOCATED, 36 | 2001:4a00::/23,RIPE NCC,2004-10-15,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 37 | 2001:4c00::/23,RIPE NCC,2004-12-17,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 38 | 2001:5000::/20,RIPE NCC,2004-09-10,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 39 | 2001:8000::/19,APNIC,2004-11-30,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED, 40 | 2001:a000::/20,APNIC,2004-11-30,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED, 41 | 2001:b000::/20,APNIC,2006-03-08,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED, 42 | 2002:0000::/16,6to4,2001-02-01,,,ALLOCATED,"2002::/16 is reserved for 6to4 [RFC3056]. 43 | For complete registration details, see [IANA registry iana-ipv6-special-registry]." 44 | 2003:0000::/18,RIPE NCC,2005-01-12,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 45 | 2400:0000::/12,APNIC,2006-10-03,whois.apnic.net,https://rdap.apnic.net/,ALLOCATED,"2400:0000::/19 was allocated on 2005-05-20. 2400:2000::/19 was allocated on 2005-07-08. 2400:4000::/21 was 46 | allocated on 2005-08-08. 2404:0000::/23 was allocated on 2006-01-19. The more recent allocation (2006-10-03) 47 | incorporates all these previous allocations." 48 | 2600:0000::/12,ARIN,2006-10-03,whois.arin.net,"https://rdap.arin.net/registry 49 | http://rdap.arin.net/registry",ALLOCATED,"2600:0000::/22, 2604:0000::/22, 2608:0000::/22 and 260c:0000::/22 were allocated on 2005-04-19. The more 50 | recent allocation (2006-10-03) incorporates all these previous allocations." 51 | 2610:0000::/23,ARIN,2005-11-17,whois.arin.net,"https://rdap.arin.net/registry 52 | http://rdap.arin.net/registry",ALLOCATED, 53 | 2620:0000::/23,ARIN,2006-09-12,whois.arin.net,"https://rdap.arin.net/registry 54 | http://rdap.arin.net/registry",ALLOCATED, 55 | 2630:0000::/12,ARIN,2019-11-06,whois.arin.net,"https://rdap.arin.net/registry 56 | http://rdap.arin.net/registry",ALLOCATED, 57 | 2800:0000::/12,LACNIC,2006-10-03,whois.lacnic.net,https://rdap.lacnic.net/rdap/,ALLOCATED,"2800:0000::/23 was allocated on 2005-11-17. The more recent allocation (2006-10-03) incorporates the 58 | previous allocation." 59 | 2a00:0000::/12,RIPE NCC,2006-10-03,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED,"2a00:0000::/21 was originally allocated on 2005-04-19. 2a01:0000::/23 was allocated on 2005-07-14. 60 | 2a01:0000::/16 (incorporating the 2a01:0000::/23) was allocated on 2005-12-15. The more recent allocation 61 | (2006-10-03) incorporates these previous allocations." 62 | 2a10:0000::/12,RIPE NCC,2019-06-05,whois.ripe.net,https://rdap.db.ripe.net/,ALLOCATED, 63 | 2c00:0000::/12,AFRINIC,2006-10-03,whois.afrinic.net,"https://rdap.afrinic.net/rdap/ 64 | http://rdap.afrinic.net/rdap/",ALLOCATED, 65 | 2d00:0000::/8,IANA,1999-07-01,,,RESERVED, 66 | 2e00:0000::/7,IANA,1999-07-01,,,RESERVED, 67 | 3000:0000::/4,IANA,1999-07-01,,,RESERVED, 68 | 3ffe::/16,IANA,2008-04,,,RESERVED,"3ffe:831f::/32 was used for Teredo in some old but widely distributed networking stacks. This usage is 69 | deprecated in favor of 2001::/32, which was allocated for the purpose in [RFC4380]. 70 | 3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]" 71 | 5f00::/16,Segment Routing (SRv6) SIDs,2024-04-23,,,ALLOCATED,"5f00::/16 is reserved for Segment Routing (SRv6) SIDs [RFC-ietf-6man-sids-06]. 72 | For complete registration details, see [IANA registry iana-ipv6-special-registry]." 73 | 5f01::/16,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 74 | 5f02::/15,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 75 | 5f04::/14,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 76 | 5f08::/13,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 77 | 5f10::/12,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 78 | 5f20::/11,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 79 | 5f40::/10,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 80 | 5f80::/9,IANA,2008-04,,,RESERVED,3ffe::/16 and 5f00::/8 were used for the 6bone but were returned. [RFC5156]. 81 | -------------------------------------------------------------------------------- /CHANGES: -------------------------------------------------------------------------------- 1 | 2024-06-18 Jerry Lundström 2 | 3 | Release 1.4.2 4 | 5 | This release fixes issues with IANA's IPv6 parameters file, 6 | dsc-datatool expected a RIR in the Designation field but IANA recently 7 | added a title for SRv6 reservation which caused an exception. 8 | 9 | Other updates are related to packages and GitHub workflows. 10 | 11 | 7560d82 Tests 12 | 8568c84 Fix client subnet authority 13 | a8c58a9 Workflow 14 | fd8915c RPM SUSE 15 | 16 | 2023-12-06 Jerry Lundström 17 | 18 | Release 1.4.1 19 | 20 | This release fixes issue with InfluxDB quoting, was missing to quote 21 | the quote character. 22 | 23 | Other changes: 24 | - Dependency correction for SLE 15.5 25 | - Tweaks to test layouts 26 | 27 | b44b874 Tests 28 | eef3ae0 SLE 15.5 29 | 75c7fc1 Influx quoting 30 | 31 | 2023-06-15 Jerry Lundström 32 | 33 | Release 1.4.0 34 | 35 | This release adds the option `--encoding` to set an encoding to use 36 | for reading and writing files. 37 | 38 | f64c8b6 encoding man-page 39 | 09c0ce9 Encoding 40 | 41 | 2022-11-10 Jerry Lundström 42 | 43 | Release 1.3.0 44 | 45 | This release adds option `nonstrict` to `client_subnet_authority` 46 | generator for skipping bad data in datasets. 47 | 48 | The contrib DSC+Grafana test site dashboards has been moved to its 49 | own repository, feel free to contribute your own creations to it: 50 | https://github.com/DNS-OARC/dsc-datatool-grafana 51 | 52 | 90b232d Add CodeQL workflow for GitHub code scanning 53 | e4fa3b0 Test site 54 | 474f97d client_subnet_authority non-strict mode 55 | 56 | 2022-06-13 Jerry Lundström 57 | 58 | Release 1.2.0 59 | 60 | This release fixes handling of base64'ed strings in DSC XML and will 61 | now decode them back into text when reading, the selected output will 62 | then handling any quoting or escaping needed. 63 | Added a new option for Prometheus output to set a prefix for metrics so 64 | that they can be easily separated from other metrics if needed, see 65 | `man dsc-datatool-output prometheus`. 66 | 67 | 5f9f972 Fix COPR 68 | 3d72019 Prometheus metric prefix 69 | bdc992e base64 labels 70 | 71 | 2022-04-05 Jerry Lundström 72 | 73 | Release 1.1.0 74 | 75 | This release adds support for Prometheus' node_exporter using it's 76 | Textfile Collector (see `man dsc-datatool-output prometheus`) and 77 | fixes a bug in InfluxDB output when selecting what timestamp to use. 78 | Also updates packages and Grafana test site dashboards. 79 | 80 | 4381541 RPM 81 | 19bc153 Typo/clarification 82 | 2a32dd8 Prometheus, InfluxDB, Copyright 83 | dd5323e debhelper 84 | 7352c1e Bye Travis 85 | 32b3bbe Grafana dashboards 86 | 304ab76 Info 87 | 88 | 2020-10-21 Jerry Lundström 89 | 90 | Release 1.0.2 91 | 92 | This release fixed a bug in DAT file parsing that was discovered when 93 | adding coverage tests. 94 | 95 | 45b1aa3 Coverage 96 | 7aedc1a Coverage 97 | 64957b9 DAT, Coverage 98 | 370fb86 Coverage 99 | 891cb7c Coverage 100 | 9374faa Coverage 101 | 102 | 2020-08-07 Jerry Lundström 103 | 104 | Release 1.0.1 105 | 106 | This release adds compatibility with Python v3.5 which allows 107 | packages to be built for Ubuntu Xenial. 108 | 109 | bc0be5b python 3.5 110 | 111 | 2020-08-03 Jerry Lundström 112 | 113 | Release 1.0.0 114 | 115 | This release brings a complete rewrite of the tool, from Perl to 116 | Python. This rewrite was made possible thanks to funding from EURid, 117 | and will help with maintainability and packaging. 118 | 119 | Core design and command line syntax is kept the same but as the 120 | libraries the generators use have been changed additional command line 121 | options must be used. 122 | 123 | - client_subnet_authority (generator) 124 | 125 | This generator now uses IANA's IP address space registry CSVs to 126 | look up the network authority, therefor it needs either to fetch 127 | the CSV files or be given them on command line. 128 | 129 | See `man dsc-datatool-generator client_subnet_authority` for more 130 | information. 131 | 132 | - client_subnet_country (generator) 133 | 134 | This generator now uses MaxMind databases to look up country based 135 | on subnet. 136 | 137 | See `man dsc-datatool generator client_subnet_country` for more 138 | information and setup guide of the MaxMind databases. 139 | 140 | 589ea8b Badges 141 | c32038b nonstrict 142 | 0ea3e32 LGTM 143 | cff2e1c COPR 144 | 02c31b0 COPR 145 | e8332fd COPR 146 | 6d9f71c Input, YAML 147 | 93ba755 EPEL 8 packages 148 | 3e2df6f Authority 149 | f5d023f Debian packaging 150 | 1a59f09 Documentation 151 | 85cb1e1 restructure 152 | decd3f6 man-pages, URLs 153 | f264854 man-pages 154 | d73c319 man-pages 155 | f5ca007 man-pages 156 | 7bfaf53 Fedora dependencies 157 | 3452b48 RPM dependencies 158 | 7a4edbc Test 159 | ed43406 client_subnet_authority 160 | 62c7d9d Server, node 161 | e0c6419 RPM package 162 | 938f154 Rewrite 163 | 5400464 README 164 | 968ccb1 COPR, spec 165 | 14d987f RPM requires 166 | ee10efb Package 167 | a25870f Funding 168 | 169 | Revision history for App::DSC::DataTool 170 | 171 | 0.05 2019-05-31 172 | Release 0.05 173 | 174 | Fixed issue with empty values in InfluxDB output, they are now 175 | quoted as an empty string. 176 | 177 | 9917c4e InfluxDB quote keys/values 178 | 179 | 0.04 2019-01-21 180 | Release 0.04 181 | 182 | Package dependency fix and update of example Grafana dashboards. 183 | 184 | d3babc9 Copyright years 185 | 9955c88 Travis Perl versions 186 | 134a8b3 Debian dependency 187 | 2d2114d Fix #23: Rework Grafana dashboards to hopefully show more 188 | correct numbers and also split them up. 189 | 9bca0d3 Prepare SPEC for OSB/COPR 190 | 191 | 0.03 2016-12-16 192 | Release 0.03 193 | 194 | Support processing of 25 of the 37 DAT files that the Extractor 195 | can produce, the others can not be converted into time series data 196 | since they lack timestamps. Processing of XML is the recommended 197 | approach to secure all information. 198 | 199 | 72e829c Implement processing of DAT directories 200 | 45294d0 RPM spec 201 | 4e8ff69 Fix 5.24 forbidden keys usage 202 | 7589ad2 Use perl 5.24 also 203 | cfac110 Fix #16: Handle directories in --xml and warn that --dat is 204 | not implemented yet 205 | 206 | 0.02 2016-11-11 207 | Release 0.02 208 | 209 | First release of `dsc-datatool` with support for: 210 | - Reading DSC XML files 211 | - Transformer: 212 | - Labler: convert indexes/keys to textual names such as QTYPE 213 | - ReRanger: (re)compile lists/ranges/buckets into new buckets 214 | - NetRemap: (re)compile IP addresses and subets into new subnets 215 | - Generator: 216 | - client_subnet_authority: Create a dataset with IP Authority for subnets 217 | - client_subnet_country: Create a dataset with Countries for subnets 218 | - Output: 219 | - Graphite 220 | - InfluxDB 221 | 222 | See `dsc-datatool -h` for options and wiki article: 223 | https://github.com/DNS-OARC/dsc-datatool/wiki/Setting-up-a-test-Grafana 224 | -------------------------------------------------------------------------------- /tests/test.gold2: -------------------------------------------------------------------------------- 1 | # DML 2 | # CONTEXT-DATABASE: dsc 3 | client_subnet_count,server=test-server,node=test-node,all=ALL value=2 1563520560000000000 4 | ipv6_rsn_abusers_count,server=test-server,node=test-node,all=ALL value=0 1563520560000000000 5 | qtype,server=test-server,node=test-node,qtype=PTR value=78 1563520560000000000 6 | qtype,server=test-server,node=test-node,qtype=NS value=7 1563520560000000000 7 | qtype,server=test-server,node=test-node,qtype=SOA value=4 1563520560000000000 8 | qtype,server=test-server,node=test-node,qtype=MX value=9 1563520560000000000 9 | qtype,server=test-server,node=test-node,qtype=else value=48 1563520560000000000 10 | qtype,server=test-server,node=test-node,qtype=A value=128 1563520560000000000 11 | qtype,server=test-server,node=test-node,qtype=AAAA value=71 1563520560000000000 12 | rcode,server=test-server,node=test-node,rcode=NoError value=416 1563520560000000000 13 | rcode,server=test-server,node=test-node,rcode=NXDomain value=8 1563520560000000000 14 | rcode,server=test-server,node=test-node,rcode=ServFail value=6 1563520560000000000 15 | do_bit,server=test-server,node=test-node,d0=set value=134 1563520560000000000 16 | do_bit,server=test-server,node=test-node,d0=clr value=211 1563520560000000000 17 | rd_bit,server=test-server,node=test-node,rd=clr value=159 1563520560000000000 18 | rd_bit,server=test-server,node=test-node,rd=set value=186 1563520560000000000 19 | opcode,server=test-server,node=test-node,opcode=NoError value=345 1563520560000000000 20 | dnssec_qtype,server=test-server,node=test-node,qtype=else value=315 1563520560000000000 21 | dnssec_qtype,server=test-server,node=test-node,qtype=48 value=25 1563520560000000000 22 | dnssec_qtype,server=test-server,node=test-node,qtype=43 value=5 1563520560000000000 23 | edns_version,server=test-server,node=test-node,ednsversion=0 value=141 1563520560000000000 24 | edns_version,server=test-server,node=test-node,ednsversion=none value=204 1563520560000000000 25 | client_subnet2_count,server=test-server,node=test-node,class=rfc1918-ptr value=1 1563520560000000000 26 | client_subnet2_count,server=test-server,node=test-node,class=ok value=2 1563520560000000000 27 | client_subnet2_count,server=test-server,node=test-node,class=non-auth-tld value=1 1563520560000000000 28 | client_subnet2_trace,server=test-server,node=test-node,class=non-auth-tld value=18 1563520560000000000 29 | client_subnet2_trace,server=test-server,node=test-node,class=rfc1918-ptr value=1 1563520560000000000 30 | client_subnet2_trace,server=test-server,node=test-node,class=ok value=185 1563520560000000000 31 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=3584-4095 value=12 1563520560000000000 32 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=4096-4607 value=73 1563520560000000000 33 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=2048-2559 value=1 1563520560000000000 34 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=512-1023 value=26 1563520560000000000 35 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=None value=204 1563520560000000000 36 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=8192-8703 value=1 1563520560000000000 37 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=1536-2047 value=6 1563520560000000000 38 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=1024-1535 value=22 1563520560000000000 39 | idn_qname,server=test-server,node=test-node,idnqname=normal value=345 1563520560000000000 40 | client_port_range,server=test-server,node=test-node,portrange=57344-59391 value=11 1563520560000000000 41 | client_port_range,server=test-server,node=test-node,portrange=0-2047 value=3 1563520560000000000 42 | client_port_range,server=test-server,node=test-node,portrange=45056-47103 value=14 1563520560000000000 43 | client_port_range,server=test-server,node=test-node,portrange=30720-32767 value=9 1563520560000000000 44 | client_port_range,server=test-server,node=test-node,portrange=14336-16383 value=13 1563520560000000000 45 | client_port_range,server=test-server,node=test-node,portrange=8192-10239 value=6 1563520560000000000 46 | client_port_range,server=test-server,node=test-node,portrange=4096-6143 value=7 1563520560000000000 47 | client_port_range,server=test-server,node=test-node,portrange=24576-26623 value=15 1563520560000000000 48 | client_port_range,server=test-server,node=test-node,portrange=40960-43007 value=16 1563520560000000000 49 | client_port_range,server=test-server,node=test-node,portrange=43008-45055 value=9 1563520560000000000 50 | client_port_range,server=test-server,node=test-node,portrange=61440-63487 value=12 1563520560000000000 51 | client_port_range,server=test-server,node=test-node,portrange=22528-24575 value=8 1563520560000000000 52 | client_port_range,server=test-server,node=test-node,portrange=10240-12287 value=8 1563520560000000000 53 | client_port_range,server=test-server,node=test-node,portrange=38912-40959 value=9 1563520560000000000 54 | client_port_range,server=test-server,node=test-node,portrange=34816-36863 value=19 1563520560000000000 55 | client_port_range,server=test-server,node=test-node,portrange=16384-18431 value=9 1563520560000000000 56 | client_port_range,server=test-server,node=test-node,portrange=32768-34815 value=18 1563520560000000000 57 | client_port_range,server=test-server,node=test-node,portrange=36864-38911 value=13 1563520560000000000 58 | client_port_range,server=test-server,node=test-node,portrange=53248-55295 value=8 1563520560000000000 59 | client_port_range,server=test-server,node=test-node,portrange=2048-4095 value=12 1563520560000000000 60 | client_port_range,server=test-server,node=test-node,portrange=47104-49151 value=9 1563520560000000000 61 | client_port_range,server=test-server,node=test-node,portrange=20480-22527 value=5 1563520560000000000 62 | client_port_range,server=test-server,node=test-node,portrange=49152-51199 value=10 1563520560000000000 63 | client_port_range,server=test-server,node=test-node,portrange=6144-8191 value=6 1563520560000000000 64 | client_port_range,server=test-server,node=test-node,portrange=51200-53247 value=8 1563520560000000000 65 | client_port_range,server=test-server,node=test-node,portrange=26624-28671 value=7 1563520560000000000 66 | client_port_range,server=test-server,node=test-node,portrange=55296-57343 value=21 1563520560000000000 67 | client_port_range,server=test-server,node=test-node,portrange=63488-65535 value=8 1563520560000000000 68 | client_port_range,server=test-server,node=test-node,portrange=12288-14335 value=11 1563520560000000000 69 | client_port_range,server=test-server,node=test-node,portrange=59392-61439 value=8 1563520560000000000 70 | client_port_range,server=test-server,node=test-node,portrange=28672-30719 value=9 1563520560000000000 71 | client_port_range,server=test-server,node=test-node,portrange=18432-20479 value=12 1563520560000000000 72 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=else value=48 1563520560000000000 73 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=A value=128 1563520560000000000 74 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=AAAA value=71 1563520560000000000 75 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=MX value=9 1563520560000000000 76 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=PTR value=78 1563520560000000000 77 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=NS value=7 1563520560000000000 78 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=SOA value=4 1563520560000000000 79 | direction_vs_ipproto,server=test-server,node=test-node,direction=recv,ipproto=udp value=219 1563520560000000000 80 | direction_vs_ipproto,server=test-server,node=test-node,direction=sent,ipproto=udp value=289 1563520560000000000 81 | direction_vs_ipproto,server=test-server,node=test-node,direction=else,ipproto=udp value=267 1563520560000000000 82 | pcap_stats,server=test-server,node=test-node,pcap_stat=filter_received,ifname=eth0 value=5625 1563520560000000000 83 | pcap_stats,server=test-server,node=test-node,pcap_stat=kernel_dropped,ifname=eth0 value=731 1563520560000000000 84 | pcap_stats,server=test-server,node=test-node,pcap_stat=pkts_captured,ifname=eth0 value=4894 1563520560000000000 85 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=else value=48 1563520560000000000 86 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=AAAA value=71 1563520560000000000 87 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=A value=128 1563520560000000000 88 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=MX value=9 1563520560000000000 89 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=PTR value=78 1563520560000000000 90 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=NS value=7 1563520560000000000 91 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=SOA value=4 1563520560000000000 92 | -------------------------------------------------------------------------------- /debian/changelog: -------------------------------------------------------------------------------- 1 | dsc-datatool (1.4.2-1~unstable+1) unstable; urgency=low 2 | 3 | * Release 1.4.2 4 | 5 | This release fixes issues with IANA's IPv6 parameters file, 6 | dsc-datatool expected a RIR in the Designation field but IANA recently 7 | added a title for SRv6 reservation which caused an exception. 8 | 9 | Other updates are related to packages and GitHub workflows. 10 | 11 | 7560d82 Tests 12 | 8568c84 Fix client subnet authority 13 | a8c58a9 Workflow 14 | fd8915c RPM SUSE 15 | 16 | -- Jerry Lundström Tue, 18 Jun 2024 12:32:53 +0200 17 | 18 | dsc-datatool (1.4.1-1~unstable+1) unstable; urgency=low 19 | 20 | * Release 1.4.1 21 | 22 | This release fixes issue with InfluxDB quoting, was missing to quote 23 | the quote character. 24 | 25 | Other changes: 26 | - Dependency correction for SLE 15.5 27 | - Tweaks to test layouts 28 | 29 | b44b874 Tests 30 | eef3ae0 SLE 15.5 31 | 75c7fc1 Influx quoting 32 | 33 | -- Jerry Lundström Wed, 06 Dec 2023 09:02:27 +0100 34 | 35 | dsc-datatool (1.4.0-1~unstable+1) unstable; urgency=low 36 | 37 | * Release 1.4.0 38 | 39 | This release adds the option `--encoding` to set an encoding to use 40 | for reading and writing files. 41 | 42 | f64c8b6 encoding man-page 43 | 09c0ce9 Encoding 44 | 45 | -- Jerry Lundström Thu, 15 Jun 2023 10:36:37 +0200 46 | 47 | dsc-datatool (1.3.0-1~unstable+1) unstable; urgency=low 48 | 49 | * Release 1.3.0 50 | 51 | This release adds option `nonstrict` to `client_subnet_authority` 52 | generator for skipping bad data in datasets. 53 | 54 | The contrib DSC+Grafana test site dashboards has been moved to its 55 | own repository, feel free to contribute your own creations to it: 56 | https://github.com/DNS-OARC/dsc-datatool-grafana 57 | 58 | 90b232d Add CodeQL workflow for GitHub code scanning 59 | e4fa3b0 Test site 60 | 474f97d client_subnet_authority non-strict mode 61 | 62 | -- Jerry Lundström Thu, 10 Nov 2022 13:58:05 +0100 63 | 64 | dsc-datatool (1.2.0-1~unstable+1) unstable; urgency=low 65 | 66 | * Release 1.2.0 67 | 68 | This release fixes handling of base64'ed strings in DSC XML and will 69 | now decode them back into text when reading, the selected output will 70 | then handling any quoting or escaping needed. 71 | Added a new option for Prometheus output to set a prefix for metrics so 72 | that they can be easily separated from other metrics if needed, see 73 | `man dsc-datatool-output prometheus`. 74 | 75 | 5f9f972 Fix COPR 76 | 3d72019 Prometheus metric prefix 77 | bdc992e base64 labels 78 | 79 | -- Jerry Lundström Mon, 13 Jun 2022 15:25:09 +0200 80 | 81 | dsc-datatool (1.1.0-1~unstable+1) unstable; urgency=low 82 | 83 | * Release 1.1.0 84 | 85 | This releases adds support for Prometheus' node_exporter using it's 86 | Textfile Collector (see `man dsc-datatool-output prometheus`) and 87 | fixes a bug in InfluxDB output when selecting what timestamp to use. 88 | Also updates packages and Grafana test site dashboards. 89 | 90 | 4381541 RPM 91 | 19bc153 Typo/clarification 92 | 2a32dd8 Prometheus, InfluxDB, Copyright 93 | dd5323e debhelper 94 | 7352c1e Bye Travis 95 | 32b3bbe Grafana dashboards 96 | 304ab76 Info 97 | 98 | -- Jerry Lundström Tue, 05 Apr 2022 10:45:59 +0200 99 | 100 | dsc-datatool (1.0.2-1~unstable+1) unstable; urgency=low 101 | 102 | * Release 1.0.2 103 | 104 | This release fixed a bug in DAT file parsing that was discovered when 105 | adding coverage tests. 106 | 107 | 45b1aa3 Coverage 108 | 7aedc1a Coverage 109 | 64957b9 DAT, Coverage 110 | 370fb86 Coverage 111 | 891cb7c Coverage 112 | 9374faa Coverage 113 | 114 | -- Jerry Lundström Wed, 21 Oct 2020 14:05:50 +0200 115 | 116 | dsc-datatool (1.0.1-1~unstable+1) unstable; urgency=low 117 | 118 | * Release 1.0.1 119 | 120 | This release adds compatibility with Python v3.5 which allows 121 | packages to be built for Ubuntu Xenial. 122 | 123 | bc0be5b python 3.5 124 | 125 | -- Jerry Lundström Fri, 07 Aug 2020 10:51:33 +0200 126 | 127 | dsc-datatool (1.0.0-2~unstable+1) unstable; urgency=low 128 | 129 | * Release 1.0.0 130 | 131 | This release brings a complete rewrite of the tool, from Perl to 132 | Python. This rewrite was made possible thanks to funding from EURid, 133 | and will help with maintainability and packaging. 134 | 135 | Core design and command line syntax is kept the same but as the 136 | libraries the generators use have been changed additional command line 137 | options must be used. 138 | 139 | - client_subnet_authority (generator) 140 | 141 | This generator now uses IANA's IP address space registry CSVs to 142 | look up the network authority, therefor it needs either to fetch 143 | the CSV files or be given them on command line. 144 | 145 | See `man dsc-datatool-generator client_subnet_authority` for more 146 | information. 147 | 148 | - client_subnet_country (generator) 149 | 150 | This generator now uses MaxMind databases to look up country based 151 | on subnet. 152 | 153 | See `man dsc-datatool generator client_subnet_country` for more 154 | information and setup guide of the MaxMind databases. 155 | 156 | 589ea8b Badges 157 | c32038b nonstrict 158 | 0ea3e32 LGTM 159 | cff2e1c COPR 160 | 02c31b0 COPR 161 | e8332fd COPR 162 | 6d9f71c Input, YAML 163 | 93ba755 EPEL 8 packages 164 | 3e2df6f Authority 165 | f5d023f Debian packaging 166 | 1a59f09 Documentation 167 | 85cb1e1 restructure 168 | decd3f6 man-pages, URLs 169 | f264854 man-pages 170 | d73c319 man-pages 171 | f5ca007 man-pages 172 | 7bfaf53 Fedora dependencies 173 | 3452b48 RPM dependencies 174 | 7a4edbc Test 175 | ed43406 client_subnet_authority 176 | 62c7d9d Server, node 177 | e0c6419 RPM package 178 | 938f154 Rewrite 179 | 5400464 README 180 | 968ccb1 COPR, spec 181 | 14d987f RPM requires 182 | ee10efb Package 183 | a25870f Funding 184 | 185 | -- Jerry Lundström Mon, 03 Aug 2020 14:17:13 +0200 186 | 187 | dsc-datatool (0.05-1~unstable+1) unstable; urgency=low 188 | 189 | * Release 0.05 190 | 191 | Fixed issue with empty values in InfluxDB output, they are now 192 | quoted as an empty string. 193 | 194 | 9917c4e InfluxDB quote keys/values 195 | 196 | -- Jerry Lundström Fri, 31 May 2019 08:44:19 +0200 197 | 198 | dsc-datatool (0.04-1~unstable+1) unstable; urgency=low 199 | 200 | * Release 0.04 201 | 202 | Package dependency fix and update of example Grafana dashboards. 203 | 204 | d3babc9 Copyright years 205 | 9955c88 Travis Perl versions 206 | 134a8b3 Debian dependency 207 | 2d2114d Fix #23: Rework Grafana dashboards to hopefully show more 208 | correct numbers and also split them up. 209 | 9bca0d3 Prepare SPEC for OSB/COPR 210 | 211 | -- Jerry Lundström Mon, 21 Jan 2019 11:27:43 +0100 212 | 213 | dsc-datatool (0.03-1~unstable+1) unstable; urgency=low 214 | 215 | * Release 0.03 216 | 217 | Support processing of 25 of the 37 DAT files that the Extractor 218 | can produce, the others can not be converted into time series data 219 | since they lack timestamps. Processing of XML is the recommended 220 | approach to secure all information. 221 | 222 | 72e829c Implement processing of DAT directories 223 | 45294d0 RPM spec 224 | 4e8ff69 Fix 5.24 forbidden keys usage 225 | 7589ad2 Use perl 5.24 also 226 | cfac110 Fix #16: Handle directories in --xml and warn that --dat is 227 | not implemented yet 228 | 229 | -- Jerry Lundström Fri, 16 Dec 2016 08:28:56 +0100 230 | 231 | dsc-datatool (0.02-1~unstable+1) unstable; urgency=low 232 | 233 | * Release 0.02 234 | 235 | First release of `dsc-datatool` with support for: 236 | - Reading DSC XML files 237 | - Transformer: 238 | - Labler: convert indexes/keys to textual names such as QTYPE 239 | - ReRanger: (re)compile lists/ranges/buckets into new buckets 240 | - NetRemap: (re)compile IP addresses and subets into new subnets 241 | - Generator: 242 | - client_subnet_authority: Create a dataset with IP Authority for subnets 243 | - client_subnet_country: Create a dataset with Countries for subnets 244 | - Output: 245 | - Graphite 246 | - InfluxDB 247 | 248 | See `dsc-datatool -h` for options and wiki article: 249 | https://github.com/DNS-OARC/dsc-datatool/wiki/Setting-up-a-test-Grafana 250 | 251 | -- Jerry Lundström Fri, 11 Nov 2016 16:26:35 +0100 252 | -------------------------------------------------------------------------------- /rpm/dsc-datatool.spec: -------------------------------------------------------------------------------- 1 | Name: dsc-datatool 2 | Version: 1.4.2 3 | Release: 1%{?dist} 4 | Summary: Export DSC data to other formats and/or databases 5 | Group: Productivity/Networking/DNS/Utilities 6 | 7 | License: BSD-3-Clause 8 | URL: https://www.dns-oarc.net/oarc/data/dsc 9 | # Source needs to be generated by dist-tools/create-source-packages, see 10 | # https://github.com/jelu/dist-tools 11 | Source0: %{name}_%{version}.orig.tar.gz 12 | 13 | BuildArch: noarch 14 | 15 | BuildRequires: python3-devel 16 | BuildRequires: python3-setuptools 17 | BuildRequires: python-rpm-macros 18 | %if 0%{?el7} 19 | BuildRequires: python36-maxminddb 20 | BuildRequires: python36-PyYAML 21 | %else 22 | %if (0%{?sle_version} == 150500 && !0%{?is_opensuse}) || 0%{?sle_version} >= 150600 23 | BuildRequires: python311-maxminddb 24 | %else 25 | BuildRequires: python3-maxminddb 26 | %endif 27 | BuildRequires: python3-PyYAML 28 | %endif 29 | 30 | %if 0%{?el7} 31 | Requires: python36-maxminddb 32 | Requires: python36-PyYAML 33 | %else 34 | %if (0%{?sle_version} == 150500 && !0%{?is_opensuse}) || 0%{?sle_version} >= 150600 35 | Requires: python311-maxminddb 36 | %else 37 | Requires: python3-maxminddb 38 | %endif 39 | Requires: python3-PyYAML 40 | %endif 41 | 42 | %package doc 43 | Summary: Documentation files for %{name} 44 | Group: Documentation 45 | 46 | 47 | %description 48 | Tool for converting, exporting, merging and transforming DSC data. 49 | 50 | 51 | %description doc 52 | Tool for converting, exporting, merging and transforming DSC data. 53 | 54 | This package contains the documentation for dsc-datatool. 55 | 56 | 57 | %prep 58 | %setup -q -n %{name}_%{version} 59 | 60 | 61 | %build 62 | python3 setup.py build 63 | 64 | 65 | %install 66 | python3 setup.py install --prefix=%{_prefix} --root=%{buildroot} 67 | mkdir -p %{buildroot}%{_mandir}/man1/ 68 | install -m644 man/man1/dsc-datatool.1 %{buildroot}%{_mandir}/man1/ 69 | mkdir -p %{buildroot}%{_mandir}/man5/ 70 | install -m644 man/man5/dsc-datatool.conf.5 %{buildroot}%{_mandir}/man5/ 71 | mkdir -p %{buildroot}%{_mandir}/man7/ 72 | install -m644 man/man7/dsc-datatool-transformer-reranger.7 %{buildroot}%{_mandir}/man7/ 73 | install -m644 man/man7/dsc-datatool-generator-client_subnet_country.7 %{buildroot}%{_mandir}/man7/ 74 | install -m644 man/man7/dsc-datatool-generator-client_subnet_authority.7 %{buildroot}%{_mandir}/man7/ 75 | install -m644 man/man7/dsc-datatool-output-influxdb.7 %{buildroot}%{_mandir}/man7/ 76 | install -m644 man/man7/dsc-datatool-output-prometheus.7 %{buildroot}%{_mandir}/man7/ 77 | install -m644 man/man7/dsc-datatool-transformer-labler.7 %{buildroot}%{_mandir}/man7/ 78 | install -m644 man/man7/dsc-datatool-transformer-netremap.7 %{buildroot}%{_mandir}/man7/ 79 | 80 | 81 | %check 82 | true 83 | 84 | 85 | %files 86 | %license LICENSE 87 | %{_bindir}/dsc-datatool 88 | %{_mandir}/man1/dsc-datatool.1* 89 | %{_mandir}/man5/dsc-datatool.conf.5* 90 | %{_mandir}/man7/dsc-datatool*.7* 91 | %{python3_sitelib}/dsc_datatool* 92 | 93 | 94 | %files doc 95 | %doc CHANGES README.md 96 | %license LICENSE 97 | 98 | 99 | %changelog 100 | * Tue Jun 18 2024 Jerry Lundström 1.4.2-1 101 | - Release 1.4.2 102 | * This release fixes issues with IANA's IPv6 parameters file, 103 | dsc-datatool expected a RIR in the Designation field but IANA recently 104 | added a title for SRv6 reservation which caused an exception. 105 | * Other updates are related to packages and GitHub workflows. 106 | * Commits: 107 | 7560d82 Tests 108 | 8568c84 Fix client subnet authority 109 | a8c58a9 Workflow 110 | fd8915c RPM SUSE 111 | * Wed Dec 06 2023 Jerry Lundström 1.4.1-1 112 | - Release 1.4.1 113 | * This release fixes issue with InfluxDB quoting, was missing to quote 114 | the quote character. 115 | * Other changes: 116 | - Dependency correction for SLE 15.5 117 | - Tweaks to test layouts 118 | * Commits: 119 | b44b874 Tests 120 | eef3ae0 SLE 15.5 121 | 75c7fc1 Influx quoting 122 | * Thu Jun 15 2023 Jerry Lundström 1.4.0-1 123 | - Release 1.4.0 124 | * This release adds the option `--encoding` to set an encoding to use 125 | for reading and writing files. 126 | * Commits: 127 | f64c8b6 encoding man-page 128 | 09c0ce9 Encoding 129 | * Thu Nov 10 2022 Jerry Lundström 1.3.0-1 130 | - Release 1.3.0 131 | * This release adds option `nonstrict` to `client_subnet_authority` 132 | generator for skipping bad data in datasets. 133 | * The contrib DSC+Grafana test site dashboards has been moved to its 134 | own repository, feel free to contribute your own creations to it: 135 | https://github.com/DNS-OARC/dsc-datatool-grafana 136 | * Commits: 137 | 90b232d Add CodeQL workflow for GitHub code scanning 138 | e4fa3b0 Test site 139 | 474f97d client_subnet_authority non-strict mode 140 | * Mon Jun 13 2022 Jerry Lundström 1.2.0-1 141 | - Release 1.2.0 142 | * This release fixes handling of base64'ed strings in DSC XML and will 143 | now decode them back into text when reading, the selected output will 144 | then handling any quoting or escaping needed. 145 | * Added a new option for Prometheus output to set a prefix for metrics so 146 | that they can be easily separated from other metrics if needed, see 147 | `man dsc-datatool-output prometheus`. 148 | * Commits: 149 | 5f9f972 Fix COPR 150 | 3d72019 Prometheus metric prefix 151 | bdc992e base64 labels 152 | * Tue Apr 05 2022 Jerry Lundström 1.1.0-1 153 | - Release 1.1.0 154 | * This releases adds support for Prometheus' node_exporter using it's 155 | Textfile Collector (see `man dsc-datatool-output prometheus`) and 156 | fixes a bug in InfluxDB output when selecting what timestamp to use. 157 | Also updates packages and Grafana test site dashboards. 158 | * Commits: 159 | 4381541 RPM 160 | 19bc153 Typo/clarification 161 | 2a32dd8 Prometheus, InfluxDB, Copyright 162 | dd5323e debhelper 163 | 7352c1e Bye Travis 164 | 32b3bbe Grafana dashboards 165 | 304ab76 Info 166 | * Wed Oct 21 2020 Jerry Lundström 1.0.2-1 167 | - Release 1.0.2 168 | * This release fixed a bug in DAT file parsing that was discovered when 169 | adding coverage tests. 170 | * Commits: 171 | 45b1aa3 Coverage 172 | 7aedc1a Coverage 173 | 64957b9 DAT, Coverage 174 | 370fb86 Coverage 175 | 891cb7c Coverage 176 | 9374faa Coverage 177 | * Fri Aug 07 2020 Jerry Lundström 1.0.1-1 178 | - Release 1.0.1 179 | * This release adds compatibility with Python v3.5 which allows 180 | packages to be built for Ubuntu Xenial. 181 | * Commits: 182 | bc0be5b python 3.5 183 | * Mon Aug 03 2020 Jerry Lundström 1.0.0-2 184 | - Release 1.0.0 185 | * This release brings a complete rewrite of the tool, from Perl to 186 | Python. This rewrite was made possible thanks to funding from EURid, 187 | and will help with maintainability and packaging. 188 | * Core design and command line syntax is kept the same but as the 189 | libraries the generators use have been changed additional command line 190 | options must be used. 191 | - client_subnet_authority (generator) 192 | This generator now uses IANA's IP address space registry CSVs to 193 | look up the network authority, therefor it needs either to fetch 194 | the CSV files or be given them on command line. 195 | See `man dsc-datatool-generator client_subnet_authority` for more 196 | information. 197 | - client_subnet_country (generator) 198 | This generator now uses MaxMind databases to look up country based 199 | on subnet. 200 | See `man dsc-datatool generator client_subnet_country` for more 201 | information and setup guide of the MaxMind databases. 202 | * Commits: 203 | 589ea8b Badges 204 | c32038b nonstrict 205 | 0ea3e32 LGTM 206 | cff2e1c COPR 207 | 02c31b0 COPR 208 | e8332fd COPR 209 | 6d9f71c Input, YAML 210 | 93ba755 EPEL 8 packages 211 | 3e2df6f Authority 212 | f5d023f Debian packaging 213 | 1a59f09 Documentation 214 | 85cb1e1 restructure 215 | decd3f6 man-pages, URLs 216 | f264854 man-pages 217 | d73c319 man-pages 218 | f5ca007 man-pages 219 | 7bfaf53 Fedora dependencies 220 | 3452b48 RPM dependencies 221 | 7a4edbc Test 222 | ed43406 client_subnet_authority 223 | 62c7d9d Server, node 224 | e0c6419 RPM package 225 | 938f154 Rewrite 226 | 5400464 README 227 | 968ccb1 COPR, spec 228 | 14d987f RPM requires 229 | ee10efb Package 230 | a25870f Funding 231 | * Wed Apr 15 2020 Jerry Lundström 1.0.0-1 232 | - Prepare for v1.0.0 233 | * Fri May 31 2019 Jerry Lundström 0.05-1 234 | - Release 0.05 235 | * Fixed issue with empty values in InfluxDB output, they are now 236 | quoted as an empty string. 237 | * Commits: 238 | 9917c4e InfluxDB quote keys/values 239 | * Mon Jan 21 2019 Jerry Lundström 0.04-1 240 | - Release 0.04 241 | * Package dependency fix and update of example Grafana dashboards. 242 | * Commits: 243 | d3babc9 Copyright years 244 | 9955c88 Travis Perl versions 245 | 134a8b3 Debian dependency 246 | 2d2114d Fix #23: Rework Grafana dashboards to hopefully show more 247 | correct numbers and also split them up. 248 | 9bca0d3 Prepare SPEC for OSB/COPR 249 | * Fri Dec 16 2016 Jerry Lundström 0.03-1 250 | - Release 0.03 251 | * Support processing of 25 of the 37 DAT files that the Extractor 252 | can produce, the others can not be converted into time series data 253 | since they lack timestamps. Processing of XML is the recommended 254 | approach to secure all information. 255 | * Commits: 256 | 72e829c Implement processing of DAT directories 257 | 45294d0 RPM spec 258 | 4e8ff69 Fix 5.24 forbidden keys usage 259 | 7589ad2 Use perl 5.24 also 260 | cfac110 Fix #16: Handle directories in --xml and warn that --dat is 261 | not implemented yet 262 | * Thu Dec 15 2016 Jerry Lundström 0.02-1 263 | - Initial package 264 | -------------------------------------------------------------------------------- /tests/1458044657.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | -------------------------------------------------------------------------------- /tests/test.gold: -------------------------------------------------------------------------------- 1 | # DML 2 | # CONTEXT-DATABASE: dsc 3 | pcap_stats,server=test-server,node=test-node,ifname=eth0,pcap_stat=filter_received value=5625 1563520560000000000 4 | pcap_stats,server=test-server,node=test-node,ifname=eth0,pcap_stat=pkts_captured value=4894 1563520560000000000 5 | pcap_stats,server=test-server,node=test-node,ifname=eth0,pcap_stat=kernel_dropped value=731 1563520560000000000 6 | direction_vs_ipproto,server=test-server,node=test-node,direction=sent,ipproto=udp value=289 1563520560000000000 7 | direction_vs_ipproto,server=test-server,node=test-node,direction=recv,ipproto=udp value=219 1563520560000000000 8 | direction_vs_ipproto,server=test-server,node=test-node,direction=else,ipproto=udp value=267 1563520560000000000 9 | client_port_range,server=test-server,node=test-node,portrange=32768-34815 value=18 1563520560000000000 10 | client_port_range,server=test-server,node=test-node,portrange=34816-36863 value=19 1563520560000000000 11 | client_port_range,server=test-server,node=test-node,portrange=24576-26623 value=15 1563520560000000000 12 | client_port_range,server=test-server,node=test-node,portrange=55296-57343 value=21 1563520560000000000 13 | client_port_range,server=test-server,node=test-node,portrange=40960-43007 value=16 1563520560000000000 14 | client_port_range,server=test-server,node=test-node,portrange=18432-20479 value=12 1563520560000000000 15 | client_port_range,server=test-server,node=test-node,portrange=57344-59391 value=11 1563520560000000000 16 | client_port_range,server=test-server,node=test-node,portrange=61440-63487 value=12 1563520560000000000 17 | client_port_range,server=test-server,node=test-node,portrange=63488-65535 value=8 1563520560000000000 18 | client_port_range,server=test-server,node=test-node,portrange=2048-4095 value=12 1563520560000000000 19 | client_port_range,server=test-server,node=test-node,portrange=14336-16383 value=13 1563520560000000000 20 | client_port_range,server=test-server,node=test-node,portrange=36864-38911 value=13 1563520560000000000 21 | client_port_range,server=test-server,node=test-node,portrange=45056-47103 value=14 1563520560000000000 22 | client_port_range,server=test-server,node=test-node,portrange=59392-61439 value=8 1563520560000000000 23 | client_port_range,server=test-server,node=test-node,portrange=12288-14335 value=11 1563520560000000000 24 | client_port_range,server=test-server,node=test-node,portrange=43008-45055 value=9 1563520560000000000 25 | client_port_range,server=test-server,node=test-node,portrange=49152-51199 value=10 1563520560000000000 26 | client_port_range,server=test-server,node=test-node,portrange=51200-53247 value=8 1563520560000000000 27 | client_port_range,server=test-server,node=test-node,portrange=16384-18431 value=9 1563520560000000000 28 | client_port_range,server=test-server,node=test-node,portrange=28672-30719 value=9 1563520560000000000 29 | client_port_range,server=test-server,node=test-node,portrange=30720-32767 value=9 1563520560000000000 30 | client_port_range,server=test-server,node=test-node,portrange=38912-40959 value=9 1563520560000000000 31 | client_port_range,server=test-server,node=test-node,portrange=47104-49151 value=9 1563520560000000000 32 | client_port_range,server=test-server,node=test-node,portrange=53248-55295 value=8 1563520560000000000 33 | client_port_range,server=test-server,node=test-node,portrange=4096-6143 value=7 1563520560000000000 34 | client_port_range,server=test-server,node=test-node,portrange=6144-8191 value=6 1563520560000000000 35 | client_port_range,server=test-server,node=test-node,portrange=10240-12287 value=8 1563520560000000000 36 | client_port_range,server=test-server,node=test-node,portrange=22528-24575 value=8 1563520560000000000 37 | client_port_range,server=test-server,node=test-node,portrange=26624-28671 value=7 1563520560000000000 38 | client_port_range,server=test-server,node=test-node,portrange=0-2047 value=3 1563520560000000000 39 | client_port_range,server=test-server,node=test-node,portrange=8192-10239 value=6 1563520560000000000 40 | client_port_range,server=test-server,node=test-node,portrange=20480-22527 value=5 1563520560000000000 41 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=A value=128 1563520560000000000 42 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=PTR value=78 1563520560000000000 43 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=AAAA value=71 1563520560000000000 44 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=DNSKEY value=25 1563520560000000000 45 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=TXT value=18 1563520560000000000 46 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=MX value=9 1563520560000000000 47 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=NS value=7 1563520560000000000 48 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=DS value=5 1563520560000000000 49 | transport_vs_qtype,server=test-server,node=test-node,transport=udp,qtype=SOA value=4 1563520560000000000 50 | rd_bit,server=test-server,node=test-node,rd=set value=186 1563520560000000000 51 | rd_bit,server=test-server,node=test-node,rd=clr value=159 1563520560000000000 52 | do_bit,server=test-server,node=test-node,d0=clr value=211 1563520560000000000 53 | do_bit,server=test-server,node=test-node,d0=set value=134 1563520560000000000 54 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=None value=204 1563520560000000000 55 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=4096-4607 value=73 1563520560000000000 56 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=512-1023 value=26 1563520560000000000 57 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=1024-1535 value=22 1563520560000000000 58 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=3584-4095 value=12 1563520560000000000 59 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=1536-2047 value=6 1563520560000000000 60 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=2048-2559 value=1 1563520560000000000 61 | edns_bufsiz,server=test-server,node=test-node,ednsbufsiz=8192-8703 value=1 1563520560000000000 62 | edns_version,server=test-server,node=test-node,ednsversion=none value=204 1563520560000000000 63 | edns_version,server=test-server,node=test-node,ednsversion=0 value=141 1563520560000000000 64 | idn_qname,server=test-server,node=test-node,idnqname=normal value=345 1563520560000000000 65 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=NoError,clientaddr=64.191.0.0 value=78 1563520560000000000 66 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=NoError,clientaddr=2620:: value=50 1563520560000000000 67 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=NoError,clientaddr=0 value=131 1563520560000000000 68 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=NXDomain,clientaddr=64.191.0.0 value=3 1563520560000000000 69 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=NXDomain,clientaddr=2620:: value=2 1563520560000000000 70 | client_addr_vs_rcode,server=test-server,node=test-node,rcode=ServFail,clientaddr=64.191.0.0 value=6 1563520560000000000 71 | client_subnet2,server=test-server,node=test-node,class=ok,clientsubnet=64.191.0.0 value=180 1563520560000000000 72 | client_subnet2,server=test-server,node=test-node,class=ok,clientsubnet=2620:: value=5 1563520560000000000 73 | client_subnet2,server=test-server,node=test-node,class=non-auth-tld,clientsubnet=64.191.0.0 value=18 1563520560000000000 74 | client_subnet2,server=test-server,node=test-node,class=rfc1918-ptr,clientsubnet=2620:: value=1 1563520560000000000 75 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=A value=128 1563520560000000000 76 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=PTR value=78 1563520560000000000 77 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=AAAA value=71 1563520560000000000 78 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=DNSKEY value=25 1563520560000000000 79 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=TXT value=18 1563520560000000000 80 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=MX value=9 1563520560000000000 81 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=NS value=7 1563520560000000000 82 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=DS value=5 1563520560000000000 83 | certain_qnames_vs_qtype,server=test-server,node=test-node,certainqnames=else,qtype=SOA value=4 1563520560000000000 84 | qtype_vs_tld,server=test-server,node=test-node,qtype=A,tld=net value=128 1563520560000000000 85 | qtype_vs_tld,server=test-server,node=test-node,qtype=A,tld=... value=128 1563520560000000000 86 | qtype_vs_tld,server=test-server,node=test-node,qtype=PTR,tld=arpa value=78 1563520560000000000 87 | qtype_vs_tld,server=test-server,node=test-node,qtype=AAAA,tld=net value=71 1563520560000000000 88 | qtype_vs_tld,server=test-server,node=test-node,qtype=MX,tld=asia value=8 1563520560000000000 89 | qtype_vs_tld,server=test-server,node=test-node,qtype=MX,tld=net value=1 1563520560000000000 90 | qtype_vs_tld,server=test-server,node=test-node,qtype=NS,tld=arpa value=3 1563520560000000000 91 | qtype_vs_tld,server=test-server,node=test-node,qtype=NS,tld=net value=2 1563520560000000000 92 | qtype_vs_tld,server=test-server,node=test-node,qtype=NS,tld=com value=1 1563520560000000000 93 | qtype_vs_tld,server=test-server,node=test-node,qtype=NS,tld=org value=1 1563520560000000000 94 | qtype_vs_tld,server=test-server,node=test-node,qtype=SOA,tld=net value=2 1563520560000000000 95 | qtype_vs_tld,server=test-server,node=test-node,qtype=SOA,tld=arpa value=2 1563520560000000000 96 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=A,qnamelen=16-31 value=98 1563520560000000000 97 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=A,qnamelen=0-15 value=29 1563520560000000000 98 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=A,qnamelen=32-47 value=1 1563520560000000000 99 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=DS,qnamelen=0-15 value=2 1563520560000000000 100 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=DS,qnamelen=16-31 value=3 1563520560000000000 101 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=PTR,qnamelen=16-31 value=76 1563520560000000000 102 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=PTR,qnamelen=64-79 value=2 1563520560000000000 103 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=TXT,qnamelen=64-79 value=9 1563520560000000000 104 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=TXT,qnamelen=16-31 value=8 1563520560000000000 105 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=TXT,qnamelen=0-15 value=1 1563520560000000000 106 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=DNSKEY,qnamelen=16-31 value=13 1563520560000000000 107 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=DNSKEY,qnamelen=0-15 value=12 1563520560000000000 108 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=AAAA,qnamelen=16-31 value=57 1563520560000000000 109 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=AAAA,qnamelen=0-15 value=12 1563520560000000000 110 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=AAAA,qnamelen=32-47 value=1 1563520560000000000 111 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=AAAA,qnamelen=48-63 value=1 1563520560000000000 112 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=MX,qnamelen=0-15 value=8 1563520560000000000 113 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=MX,qnamelen=16-31 value=1 1563520560000000000 114 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=NS,qnamelen=0-15 value=4 1563520560000000000 115 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=NS,qnamelen=16-31 value=3 1563520560000000000 116 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=SOA,qnamelen=16-31 value=3 1563520560000000000 117 | qtype_vs_qnamelen,server=test-server,node=test-node,qtype=SOA,qnamelen=0-15 value=1 1563520560000000000 118 | client_subnet,server=test-server,node=test-node,clientsubnet=64.191.0.0 value=198 1563520560000000000 119 | client_subnet,server=test-server,node=test-node,clientsubnet=2620:: value=5 1563520560000000000 120 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=0-63 value=105 1563520560000000000 121 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=64-127 value=124 1563520560000000000 122 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=192-255 value=70 1563520560000000000 123 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=256-319 value=51 1563520560000000000 124 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=1088-1151 value=14 1563520560000000000 125 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=320-383 value=16 1563520560000000000 126 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=512-575 value=8 1563520560000000000 127 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=1472-1535 value=6 1563520560000000000 128 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=704-767 value=4 1563520560000000000 129 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=448-511 value=3 1563520560000000000 130 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=576-639 value=4 1563520560000000000 131 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=128-191 value=7 1563520560000000000 132 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NoError,replylen=384-447 value=4 1563520560000000000 133 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NXDomain,replylen=64-127 value=4 1563520560000000000 134 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NXDomain,replylen=640-703 value=2 1563520560000000000 135 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NXDomain,replylen=128-191 value=1 1563520560000000000 136 | rcode_vs_replylen,server=test-server,node=test-node,rcode=NXDomain,replylen=768-831 value=1 1563520560000000000 137 | rcode_vs_replylen,server=test-server,node=test-node,rcode=ServFail,replylen=0-63 value=6 1563520560000000000 138 | opcode,server=test-server,node=test-node,opcode=NoError value=345 1563520560000000000 139 | rcode,server=test-server,node=test-node,rcode=NoError value=416 1563520560000000000 140 | rcode,server=test-server,node=test-node,rcode=NXDomain value=8 1563520560000000000 141 | rcode,server=test-server,node=test-node,rcode=ServFail value=6 1563520560000000000 142 | qtype,server=test-server,node=test-node,qtype=A value=128 1563520560000000000 143 | qtype,server=test-server,node=test-node,qtype=PTR value=78 1563520560000000000 144 | qtype,server=test-server,node=test-node,qtype=AAAA value=71 1563520560000000000 145 | qtype,server=test-server,node=test-node,qtype=DNSKEY value=25 1563520560000000000 146 | qtype,server=test-server,node=test-node,qtype=TXT value=18 1563520560000000000 147 | qtype,server=test-server,node=test-node,qtype=MX value=9 1563520560000000000 148 | qtype,server=test-server,node=test-node,qtype=NS value=7 1563520560000000000 149 | qtype,server=test-server,node=test-node,qtype=DS value=5 1563520560000000000 150 | qtype,server=test-server,node=test-node,qtype=SOA value=4 1563520560000000000 151 | client_subnet_authority,server=test-server,node=test-node,clientauthority=ARIN value=203 1563520560000000000 152 | -------------------------------------------------------------------------------- /tests/labler.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | certain_qnames_vs_qtype: 3 | Qtype: 4 | '1': A 5 | '10': 'NULL' 6 | '100': UINFO 7 | '101': UID 8 | '102': GID 9 | '103': UNSPEC 10 | '104': NID 11 | '105': L32 12 | '106': L64 13 | '107': LP 14 | '108': EUI48 15 | '109': EUI64 16 | '11': WKS 17 | 110-248: Unassigned 18 | '12': PTR 19 | '13': HINFO 20 | '14': MINFO 21 | '15': MX 22 | '16': TXT 23 | '17': RP 24 | '18': AFSDB 25 | '19': X25 26 | '2': NS 27 | '20': ISDN 28 | '21': RT 29 | '22': NSAP 30 | '23': NSAP-PTR 31 | '24': SIG 32 | '249': TKEY 33 | '25': KEY 34 | '250': TSIG 35 | '251': IXFR 36 | '252': AXFR 37 | '253': MAILB 38 | '254': MAILA 39 | '255': wildcard 40 | '256': URI 41 | '257': CAA 42 | '258': AVC 43 | '259': DOA 44 | '26': PX 45 | '260': AMTRELAY 46 | 261-32767: Unassigned 47 | '27': GPOS 48 | '28': AAAA 49 | '29': LOC 50 | '3': MD 51 | '30': NXT 52 | '31': EID 53 | '32': NIMLOC 54 | '32768': TA 55 | '32769': DLV 56 | 32770-65279: Unassigned 57 | '33': SRV 58 | '34': ATMA 59 | '35': NAPTR 60 | '36': KX 61 | '37': CERT 62 | '38': A6 63 | '39': DNAME 64 | '4': MF 65 | '40': SINK 66 | '41': OPT 67 | '42': APL 68 | '43': DS 69 | '44': SSHFP 70 | '45': IPSECKEY 71 | '46': RRSIG 72 | '47': NSEC 73 | '48': DNSKEY 74 | '49': DHCID 75 | '5': CNAME 76 | '50': NSEC3 77 | '51': NSEC3PARAM 78 | '52': TLSA 79 | '53': SMIMEA 80 | '54': Unassigned 81 | '55': HIP 82 | '56': NINFO 83 | '57': RKEY 84 | '58': TALINK 85 | '59': CDS 86 | '6': SOA 87 | '60': CDNSKEY 88 | '61': OPENPGPKEY 89 | '62': CSYNC 90 | '63': ZONEMD 91 | 64-98: Unassigned 92 | 65280-65534: Private use 93 | '65535': Reserved 94 | '7': MB 95 | '8': MG 96 | '9': MR 97 | '99': SPF 98 | chaos_types_and_names: 99 | Qtype: 100 | '1': A 101 | '10': 'NULL' 102 | '100': UINFO 103 | '101': UID 104 | '102': GID 105 | '103': UNSPEC 106 | '104': NID 107 | '105': L32 108 | '106': L64 109 | '107': LP 110 | '108': EUI48 111 | '109': EUI64 112 | '11': WKS 113 | 110-248: Unassigned 114 | '12': PTR 115 | '13': HINFO 116 | '14': MINFO 117 | '15': MX 118 | '16': TXT 119 | '17': RP 120 | '18': AFSDB 121 | '19': X25 122 | '2': NS 123 | '20': ISDN 124 | '21': RT 125 | '22': NSAP 126 | '23': NSAP-PTR 127 | '24': SIG 128 | '249': TKEY 129 | '25': KEY 130 | '250': TSIG 131 | '251': IXFR 132 | '252': AXFR 133 | '253': MAILB 134 | '254': MAILA 135 | '255': wildcard 136 | '256': URI 137 | '257': CAA 138 | '258': AVC 139 | '259': DOA 140 | '26': PX 141 | '260': AMTRELAY 142 | 261-32767: Unassigned 143 | '27': GPOS 144 | '28': AAAA 145 | '29': LOC 146 | '3': MD 147 | '30': NXT 148 | '31': EID 149 | '32': NIMLOC 150 | '32768': TA 151 | '32769': DLV 152 | 32770-65279: Unassigned 153 | '33': SRV 154 | '34': ATMA 155 | '35': NAPTR 156 | '36': KX 157 | '37': CERT 158 | '38': A6 159 | '39': DNAME 160 | '4': MF 161 | '40': SINK 162 | '41': OPT 163 | '42': APL 164 | '43': DS 165 | '44': SSHFP 166 | '45': IPSECKEY 167 | '46': RRSIG 168 | '47': NSEC 169 | '48': DNSKEY 170 | '49': DHCID 171 | '5': CNAME 172 | '50': NSEC3 173 | '51': NSEC3PARAM 174 | '52': TLSA 175 | '53': SMIMEA 176 | '54': Unassigned 177 | '55': HIP 178 | '56': NINFO 179 | '57': RKEY 180 | '58': TALINK 181 | '59': CDS 182 | '6': SOA 183 | '60': CDNSKEY 184 | '61': OPENPGPKEY 185 | '62': CSYNC 186 | '63': ZONEMD 187 | 64-98: Unassigned 188 | 65280-65534: Private use 189 | '65535': Reserved 190 | '7': MB 191 | '8': MG 192 | '9': MR 193 | '99': SPF 194 | client_addr_vs_rcode: 195 | Rcode: 196 | '0': NoError 197 | '1': FormErr 198 | '10': NotZone 199 | '11': DSOTYPENI 200 | 12-15: Unassigned 201 | '16': BADSIG 202 | '17': BADKEY 203 | '18': BADTIME 204 | '19': BADMODE 205 | '2': ServFail 206 | '20': BADNAME 207 | '21': BADALG 208 | '22': BADTRUNC 209 | '23': BADCOOKIE 210 | 24-3840: Unassigned 211 | '3': NXDomain 212 | 3841-4095: Reserved for Private Use 213 | '4': NotImp 214 | 4096-65534: Unassigned 215 | '5': Refused 216 | '6': YXDomain 217 | '65535': Reserved, can be allocated by Standards Action 218 | '7': YXRRSet 219 | '8': NXRRSet 220 | '9': NotAuth 221 | dns_ip_version_vs_qtype: 222 | Qtype: 223 | '1': A 224 | '10': 'NULL' 225 | '100': UINFO 226 | '101': UID 227 | '102': GID 228 | '103': UNSPEC 229 | '104': NID 230 | '105': L32 231 | '106': L64 232 | '107': LP 233 | '108': EUI48 234 | '109': EUI64 235 | '11': WKS 236 | 110-248: Unassigned 237 | '12': PTR 238 | '13': HINFO 239 | '14': MINFO 240 | '15': MX 241 | '16': TXT 242 | '17': RP 243 | '18': AFSDB 244 | '19': X25 245 | '2': NS 246 | '20': ISDN 247 | '21': RT 248 | '22': NSAP 249 | '23': NSAP-PTR 250 | '24': SIG 251 | '249': TKEY 252 | '25': KEY 253 | '250': TSIG 254 | '251': IXFR 255 | '252': AXFR 256 | '253': MAILB 257 | '254': MAILA 258 | '255': wildcard 259 | '256': URI 260 | '257': CAA 261 | '258': AVC 262 | '259': DOA 263 | '26': PX 264 | '260': AMTRELAY 265 | 261-32767: Unassigned 266 | '27': GPOS 267 | '28': AAAA 268 | '29': LOC 269 | '3': MD 270 | '30': NXT 271 | '31': EID 272 | '32': NIMLOC 273 | '32768': TA 274 | '32769': DLV 275 | 32770-65279: Unassigned 276 | '33': SRV 277 | '34': ATMA 278 | '35': NAPTR 279 | '36': KX 280 | '37': CERT 281 | '38': A6 282 | '39': DNAME 283 | '4': MF 284 | '40': SINK 285 | '41': OPT 286 | '42': APL 287 | '43': DS 288 | '44': SSHFP 289 | '45': IPSECKEY 290 | '46': RRSIG 291 | '47': NSEC 292 | '48': DNSKEY 293 | '49': DHCID 294 | '5': CNAME 295 | '50': NSEC3 296 | '51': NSEC3PARAM 297 | '52': TLSA 298 | '53': SMIMEA 299 | '54': Unassigned 300 | '55': HIP 301 | '56': NINFO 302 | '57': RKEY 303 | '58': TALINK 304 | '59': CDS 305 | '6': SOA 306 | '60': CDNSKEY 307 | '61': OPENPGPKEY 308 | '62': CSYNC 309 | '63': ZONEMD 310 | 64-98: Unassigned 311 | 65280-65534: Private use 312 | '65535': Reserved 313 | '7': MB 314 | '8': MG 315 | '9': MR 316 | '99': SPF 317 | opcode: 318 | Opcode: 319 | '0': NoError 320 | '1': FormErr 321 | '10': NotZone 322 | '11': DSOTYPENI 323 | 12-15: Unassigned 324 | '16': BADSIG 325 | '17': BADKEY 326 | '18': BADTIME 327 | '19': BADMODE 328 | '2': ServFail 329 | '20': BADNAME 330 | '21': BADALG 331 | '22': BADTRUNC 332 | '23': BADCOOKIE 333 | 24-3840: Unassigned 334 | '3': NXDomain 335 | 3841-4095: Reserved for Private Use 336 | '4': NotImp 337 | 4096-65534: Unassigned 338 | '5': Refused 339 | '6': YXDomain 340 | '65535': Reserved, can be allocated by Standards Action 341 | '7': YXRRSet 342 | '8': NXRRSet 343 | '9': NotAuth 344 | qtype: 345 | Qtype: 346 | '1': A 347 | '10': 'NULL' 348 | '100': UINFO 349 | '101': UID 350 | '102': GID 351 | '103': UNSPEC 352 | '104': NID 353 | '105': L32 354 | '106': L64 355 | '107': LP 356 | '108': EUI48 357 | '109': EUI64 358 | '11': WKS 359 | 110-248: Unassigned 360 | '12': PTR 361 | '13': HINFO 362 | '14': MINFO 363 | '15': MX 364 | '16': TXT 365 | '17': RP 366 | '18': AFSDB 367 | '19': X25 368 | '2': NS 369 | '20': ISDN 370 | '21': RT 371 | '22': NSAP 372 | '23': NSAP-PTR 373 | '24': SIG 374 | '249': TKEY 375 | '25': KEY 376 | '250': TSIG 377 | '251': IXFR 378 | '252': AXFR 379 | '253': MAILB 380 | '254': MAILA 381 | '255': wildcard 382 | '256': URI 383 | '257': CAA 384 | '258': AVC 385 | '259': DOA 386 | '26': PX 387 | '260': AMTRELAY 388 | 261-32767: Unassigned 389 | '27': GPOS 390 | '28': AAAA 391 | '29': LOC 392 | '3': MD 393 | '30': NXT 394 | '31': EID 395 | '32': NIMLOC 396 | '32768': TA 397 | '32769': DLV 398 | 32770-65279: Unassigned 399 | '33': SRV 400 | '34': ATMA 401 | '35': NAPTR 402 | '36': KX 403 | '37': CERT 404 | '38': A6 405 | '39': DNAME 406 | '4': MF 407 | '40': SINK 408 | '41': OPT 409 | '42': APL 410 | '43': DS 411 | '44': SSHFP 412 | '45': IPSECKEY 413 | '46': RRSIG 414 | '47': NSEC 415 | '48': DNSKEY 416 | '49': DHCID 417 | '5': CNAME 418 | '50': NSEC3 419 | '51': NSEC3PARAM 420 | '52': TLSA 421 | '53': SMIMEA 422 | '54': Unassigned 423 | '55': HIP 424 | '56': NINFO 425 | '57': RKEY 426 | '58': TALINK 427 | '59': CDS 428 | '6': SOA 429 | '60': CDNSKEY 430 | '61': OPENPGPKEY 431 | '62': CSYNC 432 | '63': ZONEMD 433 | 64-98: Unassigned 434 | 65280-65534: Private use 435 | '65535': Reserved 436 | '7': MB 437 | '8': MG 438 | '9': MR 439 | '99': SPF 440 | qtype_vs_qnamelen: 441 | Qtype: 442 | '1': A 443 | '10': 'NULL' 444 | '100': UINFO 445 | '101': UID 446 | '102': GID 447 | '103': UNSPEC 448 | '104': NID 449 | '105': L32 450 | '106': L64 451 | '107': LP 452 | '108': EUI48 453 | '109': EUI64 454 | '11': WKS 455 | 110-248: Unassigned 456 | '12': PTR 457 | '13': HINFO 458 | '14': MINFO 459 | '15': MX 460 | '16': TXT 461 | '17': RP 462 | '18': AFSDB 463 | '19': X25 464 | '2': NS 465 | '20': ISDN 466 | '21': RT 467 | '22': NSAP 468 | '23': NSAP-PTR 469 | '24': SIG 470 | '249': TKEY 471 | '25': KEY 472 | '250': TSIG 473 | '251': IXFR 474 | '252': AXFR 475 | '253': MAILB 476 | '254': MAILA 477 | '255': wildcard 478 | '256': URI 479 | '257': CAA 480 | '258': AVC 481 | '259': DOA 482 | '26': PX 483 | '260': AMTRELAY 484 | 261-32767: Unassigned 485 | '27': GPOS 486 | '28': AAAA 487 | '29': LOC 488 | '3': MD 489 | '30': NXT 490 | '31': EID 491 | '32': NIMLOC 492 | '32768': TA 493 | '32769': DLV 494 | 32770-65279: Unassigned 495 | '33': SRV 496 | '34': ATMA 497 | '35': NAPTR 498 | '36': KX 499 | '37': CERT 500 | '38': A6 501 | '39': DNAME 502 | '4': MF 503 | '40': SINK 504 | '41': OPT 505 | '42': APL 506 | '43': DS 507 | '44': SSHFP 508 | '45': IPSECKEY 509 | '46': RRSIG 510 | '47': NSEC 511 | '48': DNSKEY 512 | '49': DHCID 513 | '5': CNAME 514 | '50': NSEC3 515 | '51': NSEC3PARAM 516 | '52': TLSA 517 | '53': SMIMEA 518 | '54': Unassigned 519 | '55': HIP 520 | '56': NINFO 521 | '57': RKEY 522 | '58': TALINK 523 | '59': CDS 524 | '6': SOA 525 | '60': CDNSKEY 526 | '61': OPENPGPKEY 527 | '62': CSYNC 528 | '63': ZONEMD 529 | 64-98: Unassigned 530 | 65280-65534: Private use 531 | '65535': Reserved 532 | '7': MB 533 | '8': MG 534 | '9': MR 535 | '99': SPF 536 | qtype_vs_tld: 537 | Qtype: 538 | '1': A 539 | '10': 'NULL' 540 | '100': UINFO 541 | '101': UID 542 | '102': GID 543 | '103': UNSPEC 544 | '104': NID 545 | '105': L32 546 | '106': L64 547 | '107': LP 548 | '108': EUI48 549 | '109': EUI64 550 | '11': WKS 551 | 110-248: Unassigned 552 | '12': PTR 553 | '13': HINFO 554 | '14': MINFO 555 | '15': MX 556 | '16': TXT 557 | '17': RP 558 | '18': AFSDB 559 | '19': X25 560 | '2': NS 561 | '20': ISDN 562 | '21': RT 563 | '22': NSAP 564 | '23': NSAP-PTR 565 | '24': SIG 566 | '249': TKEY 567 | '25': KEY 568 | '250': TSIG 569 | '251': IXFR 570 | '252': AXFR 571 | '253': MAILB 572 | '254': MAILA 573 | '255': wildcard 574 | '256': URI 575 | '257': CAA 576 | '258': AVC 577 | '259': DOA 578 | '26': PX 579 | '260': AMTRELAY 580 | 261-32767: Unassigned 581 | '27': GPOS 582 | '28': AAAA 583 | '29': LOC 584 | '3': MD 585 | '30': NXT 586 | '31': EID 587 | '32': NIMLOC 588 | '32768': TA 589 | '32769': DLV 590 | 32770-65279: Unassigned 591 | '33': SRV 592 | '34': ATMA 593 | '35': NAPTR 594 | '36': KX 595 | '37': CERT 596 | '38': A6 597 | '39': DNAME 598 | '4': MF 599 | '40': SINK 600 | '41': OPT 601 | '42': APL 602 | '43': DS 603 | '44': SSHFP 604 | '45': IPSECKEY 605 | '46': RRSIG 606 | '47': NSEC 607 | '48': DNSKEY 608 | '49': DHCID 609 | '5': CNAME 610 | '50': NSEC3 611 | '51': NSEC3PARAM 612 | '52': TLSA 613 | '53': SMIMEA 614 | '54': Unassigned 615 | '55': HIP 616 | '56': NINFO 617 | '57': RKEY 618 | '58': TALINK 619 | '59': CDS 620 | '6': SOA 621 | '60': CDNSKEY 622 | '61': OPENPGPKEY 623 | '62': CSYNC 624 | '63': ZONEMD 625 | 64-98: Unassigned 626 | 65280-65534: Private use 627 | '65535': Reserved 628 | '7': MB 629 | '8': MG 630 | '9': MR 631 | '99': SPF 632 | rcode: 633 | Rcode: 634 | '0': NoError 635 | '1': FormErr 636 | '10': NotZone 637 | '11': DSOTYPENI 638 | 12-15: Unassigned 639 | '16': BADSIG 640 | '17': BADKEY 641 | '18': BADTIME 642 | '19': BADMODE 643 | '2': ServFail 644 | '20': BADNAME 645 | '21': BADALG 646 | '22': BADTRUNC 647 | '23': BADCOOKIE 648 | 24-3840: Unassigned 649 | '3': NXDomain 650 | 3841-4095: Reserved for Private Use 651 | '4': NotImp 652 | 4096-65534: Unassigned 653 | '5': Refused 654 | '6': YXDomain 655 | '65535': Reserved, can be allocated by Standards Action 656 | '7': YXRRSet 657 | '8': NXRRSet 658 | '9': NotAuth 659 | rcode_vs_replylen: 660 | Rcode: 661 | '0': NoError 662 | '1': FormErr 663 | '10': NotZone 664 | '11': DSOTYPENI 665 | 12-15: Unassigned 666 | '16': BADSIG 667 | '17': BADKEY 668 | '18': BADTIME 669 | '19': BADMODE 670 | '2': ServFail 671 | '20': BADNAME 672 | '21': BADALG 673 | '22': BADTRUNC 674 | '23': BADCOOKIE 675 | 24-3840: Unassigned 676 | '3': NXDomain 677 | 3841-4095: Reserved for Private Use 678 | '4': NotImp 679 | 4096-65534: Unassigned 680 | '5': Refused 681 | '6': YXDomain 682 | '65535': Reserved, can be allocated by Standards Action 683 | '7': YXRRSet 684 | '8': NXRRSet 685 | '9': NotAuth 686 | transport_vs_qtype: 687 | Qtype: 688 | '1': A 689 | '10': 'NULL' 690 | '100': UINFO 691 | '101': UID 692 | '102': GID 693 | '103': UNSPEC 694 | '104': NID 695 | '105': L32 696 | '106': L64 697 | '107': LP 698 | '108': EUI48 699 | '109': EUI64 700 | '11': WKS 701 | 110-248: Unassigned 702 | '12': PTR 703 | '13': HINFO 704 | '14': MINFO 705 | '15': MX 706 | '16': TXT 707 | '17': RP 708 | '18': AFSDB 709 | '19': X25 710 | '2': NS 711 | '20': ISDN 712 | '21': RT 713 | '22': NSAP 714 | '23': NSAP-PTR 715 | '24': SIG 716 | '249': TKEY 717 | '25': KEY 718 | '250': TSIG 719 | '251': IXFR 720 | '252': AXFR 721 | '253': MAILB 722 | '254': MAILA 723 | '255': wildcard 724 | '256': URI 725 | '257': CAA 726 | '258': AVC 727 | '259': DOA 728 | '26': PX 729 | '260': AMTRELAY 730 | 261-32767: Unassigned 731 | '27': GPOS 732 | '28': AAAA 733 | '29': LOC 734 | '3': MD 735 | '30': NXT 736 | '31': EID 737 | '32': NIMLOC 738 | '32768': TA 739 | '32769': DLV 740 | 32770-65279: Unassigned 741 | '33': SRV 742 | '34': ATMA 743 | '35': NAPTR 744 | '36': KX 745 | '37': CERT 746 | '38': A6 747 | '39': DNAME 748 | '4': MF 749 | '40': SINK 750 | '41': OPT 751 | '42': APL 752 | '43': DS 753 | '44': SSHFP 754 | '45': IPSECKEY 755 | '46': RRSIG 756 | '47': NSEC 757 | '48': DNSKEY 758 | '49': DHCID 759 | '5': CNAME 760 | '50': NSEC3 761 | '51': NSEC3PARAM 762 | '52': TLSA 763 | '53': SMIMEA 764 | '54': Unassigned 765 | '55': HIP 766 | '56': NINFO 767 | '57': RKEY 768 | '58': TALINK 769 | '59': CDS 770 | '6': SOA 771 | '60': CDNSKEY 772 | '61': OPENPGPKEY 773 | '62': CSYNC 774 | '63': ZONEMD 775 | 64-98: Unassigned 776 | 65280-65534: Private use 777 | '65535': Reserved 778 | '7': MB 779 | '8': MG 780 | '9': MR 781 | '99': SPF 782 | 783 | -------------------------------------------------------------------------------- /dsc_datatool/__init__.py: -------------------------------------------------------------------------------- 1 | """dsc_datatool 2 | 3 | The main Python module for the command line tool `dsc-datatool`, see 4 | `man dsc-datatool` on how to run it. 5 | 6 | On runtime it will load all plugins under the following module path: 7 | - dsc_datatool.input 8 | - dsc_datatool.output 9 | - dsc_datatool.generator 10 | - dsc_datatool.transformer 11 | 12 | Each plugin category should base it class on one of the follow superclasses: 13 | - dsc_datatool.Input 14 | - dsc_datatool.Output 15 | - dsc_datatool.Generator 16 | - dsc_datatool.Transformer 17 | 18 | Doing so it will be automatically registered as available and indexed in 19 | the following public dicts using the class name: 20 | - inputs 21 | - outputs 22 | - generators 23 | - transformers 24 | 25 | Example of an output: 26 | 27 | from dsc_datatool import Output 28 | class ExampleOutput(Output): 29 | def process(self, datasets) 30 | ... 31 | 32 | :copyright: 2024 OARC, Inc. 33 | """ 34 | 35 | __version__ = '1.4.2' 36 | 37 | import argparse 38 | import logging 39 | import os 40 | import importlib 41 | import pkgutil 42 | import sys 43 | import traceback 44 | import re 45 | 46 | args = argparse.Namespace() 47 | inputs = {} 48 | outputs = {} 49 | generators = {} 50 | transformers = {} 51 | process_dataset = {} 52 | encoding = 'utf-8' 53 | 54 | 55 | class Dataset(object): 56 | """A representation of a DSC dataset 57 | 58 | A DSC dataset is one to two dimensional structure where the last 59 | dimension holds an array of values and counters. 60 | 61 | It is based on the XML structure of DSC: 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | Attributes: 76 | - name: The name of the dataset 77 | - start_time: The start time of the dataset in seconds 78 | - stop_time: The stop time of the dataset in seconds 79 | - dimensions: An array with `Dimension`, the first dimension 80 | """ 81 | name = None 82 | start_time = None 83 | stop_time = None 84 | dimensions = None 85 | 86 | 87 | def __init__(self): 88 | self.dimensions = [] 89 | 90 | 91 | def __repr__(self): 92 | return '' % (self.name, self.dimensions) 93 | 94 | 95 | class Dimension(object): 96 | """A representation of a DSC dimension 97 | 98 | A DSC dataset dimension which can be the first or second dimension, 99 | see `Dataset` for more information. 100 | 101 | Attributes: 102 | - name: The name of the dimension 103 | - value: Is set to the value of the dimension if it's the first dimension 104 | - values: A dict of values with corresponding counters if it's the second dimension 105 | """ 106 | name = None 107 | value = None 108 | values = None 109 | dimensions = None 110 | 111 | 112 | def __init__(self, name): 113 | self.name = name 114 | self.values = {} 115 | self.dimensions = [] 116 | 117 | 118 | def __repr__(self): 119 | return '' % (self.name, self.values or self.value, self.dimensions) 120 | 121 | 122 | class Input(object): 123 | """Base class of an input plugin""" 124 | 125 | 126 | def process(self, file): 127 | """Input.process(...) -> [ Dataset, ... ] 128 | 129 | Called to process a file and return an array of `Dataset`'s found in it. 130 | """ 131 | raise Exception('process() not overloaded') 132 | 133 | 134 | def __init_subclass__(cls): 135 | """This method is called when a class is subclassed and it will 136 | register the input plugin in `inputs`.""" 137 | global inputs 138 | if cls.__name__ in inputs: 139 | raise Exception('Duplicate input module: %s already exists' % cls.__name__) 140 | inputs[cls.__name__] = cls 141 | 142 | 143 | class Output(object): 144 | """Base class of an output plugin""" 145 | 146 | 147 | def process(self, datasets): 148 | """Output.process([ Dataset, ... ]) 149 | 150 | Called to output the `Dataset`'s in the given array.""" 151 | raise Exception('process() not overloaded') 152 | 153 | 154 | def __init__(self, opts): 155 | """instance = Output({ 'opt': value, ... }) 156 | 157 | Called to create an instance of the output plugin, will get a dict 158 | with options provided on command line.""" 159 | pass 160 | 161 | 162 | def __init_subclass__(cls): 163 | """This method is called when a class is subclassed and it will 164 | register the output plugin in `outputs`.""" 165 | global outputs 166 | if cls.__name__ in outputs: 167 | raise Exception('Duplicate output module: %s already exists' % cls.__name__) 168 | outputs[cls.__name__] = cls 169 | 170 | 171 | class Generator(object): 172 | """Base class of a generator plugin""" 173 | 174 | 175 | def process(self, datasets): 176 | """Generator.process([ Dataset, ... ]) -> [ Dataset, ... ] 177 | 178 | Called to generate additional `Dataset`'s based on the given array 179 | of `Dataset`'s.""" 180 | raise Exception('process() not overloaded') 181 | 182 | 183 | def __init__(self, opts): 184 | """instance = Generator({ 'opt': value, ... }) 185 | 186 | Called to create an instance of the generator plugin, will get a dict 187 | with options provided on command line.""" 188 | pass 189 | 190 | 191 | def __init_subclass__(cls): 192 | """This method is called when a class is subclassed and it will 193 | register the generator plugin in `generators`.""" 194 | global generators 195 | if cls.__name__ in generators: 196 | raise Exception('Duplicate generator module: %s already exists' % cls.__name__) 197 | generators[cls.__name__] = cls 198 | 199 | 200 | class Transformer(object): 201 | """Base class of a transformer plugin""" 202 | 203 | 204 | def process(self, datasets): 205 | """Transformer.process([ Dataset, ... ]) 206 | 207 | Called to do transformation of the given `Dataset`'s, as in modifying 208 | them directly.""" 209 | raise Exception('process() not overloaded') 210 | 211 | 212 | def __init__(self, opts): 213 | """instance = Transformer({ 'opt': value, ... }) 214 | 215 | Called to create an instance of the transformer plugin, will get a dict 216 | with options provided on command line.""" 217 | pass 218 | 219 | 220 | def __init_subclass__(cls): 221 | """This method is called when a class is subclassed and it will 222 | register the transformer plugin in `transformers`.""" 223 | global transformers 224 | if cls.__name__ in transformers: 225 | raise Exception('Duplicate transformer module: %s already exists' % cls.__name__) 226 | transformers[cls.__name__] = cls 227 | 228 | 229 | def main(): 230 | """Called when running `dsc-datatool`.""" 231 | def iter_namespace(ns_pkg): 232 | return pkgutil.iter_modules(ns_pkg.__path__, ns_pkg.__name__ + ".") 233 | 234 | 235 | def split_arg(arg, num=1): 236 | sep = arg[0] 237 | p = arg.split(sep) 238 | p.pop(0) 239 | ret = () 240 | while num > 0: 241 | ret += (p.pop(0),) 242 | num -= 1 243 | ret += (p,) 244 | return ret 245 | 246 | 247 | def parse_opts(opts): 248 | ret = {} 249 | for opt in opts: 250 | p = opt.split('=', maxsplit=1) 251 | if len(p) > 1: 252 | if p[0] in ret: 253 | if isinstance(ret[p[0]], list): 254 | ret[p[0]].append(p[1]) 255 | else: 256 | ret[p[0]] = [ ret[p[0]], p[1] ] 257 | else: 258 | ret[p[0]] = p[1] 259 | elif len(p) > 0: 260 | ret[p[0]] = True 261 | return ret 262 | 263 | 264 | def _process(datasets, generators, transformers, outputs): 265 | gen_datasets = [] 266 | for generator in generators: 267 | try: 268 | gen_datasets += generator.process(datasets) 269 | except Exception as e: 270 | logging.warning('Generator %s failed: %s' % (generator, e)) 271 | exc_type, exc_value, exc_traceback = sys.exc_info() 272 | for tb in traceback.format_tb(exc_traceback): 273 | logging.warning(str(tb)) 274 | return 2 275 | 276 | datasets += gen_datasets 277 | 278 | if '*' in transformers: 279 | for transformer in transformers['*']: 280 | try: 281 | transformer.process(datasets) 282 | except Exception as e: 283 | logging.warning('Transformer %s failed: %s' % (transformer, e)) 284 | exc_type, exc_value, exc_traceback = sys.exc_info() 285 | for tb in traceback.format_tb(exc_traceback): 286 | logging.warning(str(tb)) 287 | return 2 288 | for dataset in datasets: 289 | if dataset.name in transformers: 290 | for transformer in transformers[dataset.name]: 291 | try: 292 | transformer.process([dataset]) 293 | except Exception as e: 294 | logging.warning('Transformer %s failed: %s' % (transformer, e)) 295 | exc_type, exc_value, exc_traceback = sys.exc_info() 296 | for tb in traceback.format_tb(exc_traceback): 297 | logging.warning(str(tb)) 298 | return 2 299 | 300 | for output in outputs: 301 | try: 302 | output.process(datasets) 303 | except Exception as e: 304 | logging.warning('Output %s failed: %s' % (output, e)) 305 | exc_type, exc_value, exc_traceback = sys.exc_info() 306 | for tb in traceback.format_tb(exc_traceback): 307 | logging.warning(str(tb)) 308 | return 2 309 | 310 | return 0 311 | 312 | 313 | global args, inputs, outputs, generators, transformers, process_dataset 314 | 315 | parser = argparse.ArgumentParser(prog='dsc-datatool', 316 | description='Export DSC data into various formats and databases.', 317 | epilog='See man-page dsc-datatool(1) and dsc-datatool-[generator|transformer|output] (5) for more information') 318 | parser.add_argument('-c', '--conf', nargs=1, 319 | help='Not implemented') 320 | # help='Specify the YAML configuration file to use (default to ~/.dsc-datatool.conf), any command line option will override the options in the configuration file. See dsc-datatool.conf(5)for more information.') 321 | parser.add_argument('-s', '--server', nargs=1, 322 | help='Specify the server for where the data comes from. (required)') 323 | parser.add_argument('-n', '--node', nargs=1, 324 | help='Specify the node for where the data comes from. (required)') 325 | parser.add_argument('-x', '--xml', action='append', 326 | help='Read DSC data from the given file or directory, can be specified multiple times. If a directory is given then all files ending with .xml will be read.') 327 | parser.add_argument('-d', '--dat', action='append', 328 | help='Read DSC data from the given directory, can be specified multiple times. Note that the DAT format is depended on the filename to know what type of data it is.') 329 | parser.add_argument('--dataset', action='append', 330 | help='Specify that only the list of datasets will be processed, the list is comma separated and the option can be given multiple times.') 331 | parser.add_argument('-o', '--output', action='append', 332 | help='"[option=value...]>" Output data to and use as an options separator.') 333 | parser.add_argument('-t', '--transform', action='append', 334 | help='"[option=value...]>" Use the transformer to change the list of datasets in .') 335 | parser.add_argument('-g', '--generator', action='append', 336 | help='"[,,...]" or "[option=value...]>" Use the specified generators to generate additional datasets.') 337 | parser.add_argument('--list', action='store_true', 338 | help='List the available generators, transformers and outputs then exit.') 339 | parser.add_argument('--skipped-key', nargs=1, default='-:SKIPPED:-', 340 | help='Set the special DSC skipped key. (default to "-:SKIPPED:-")') 341 | parser.add_argument('--skipped-sum-key', nargs=1, default='-:SKIPPED_SUM:-', 342 | help='Set the special DSC skipped sum key. (default to "-:SKIPPED_SUM:-")') 343 | parser.add_argument('--encoding', nargs=1, default='utf-8', 344 | help='Encoding to use for all files, default utf-8.') 345 | parser.add_argument('-v', '--verbose', action='count', default=0, 346 | help='Increase the verbose level, can be given multiple times.') 347 | parser.add_argument('-V', '--version', action='version', version='%(prog)s v'+__version__, 348 | help='Display version and exit.') 349 | 350 | args = parser.parse_args() 351 | 352 | log_level = 30 - (args.verbose * 10) 353 | if log_level < 0: 354 | log_level = 0 355 | logging.basicConfig(format='%(asctime)s %(levelname)s %(module)s: %(message)s', level=log_level, stream=sys.stderr) 356 | 357 | import dsc_datatool.input 358 | import dsc_datatool.output 359 | import dsc_datatool.generator 360 | import dsc_datatool.transformer 361 | 362 | for finder, name, ispkg in iter_namespace(dsc_datatool.input): 363 | importlib.import_module(name) 364 | for finder, name, ispkg in iter_namespace(dsc_datatool.output): 365 | importlib.import_module(name) 366 | for finder, name, ispkg in iter_namespace(dsc_datatool.generator): 367 | importlib.import_module(name) 368 | for finder, name, ispkg in iter_namespace(dsc_datatool.transformer): 369 | importlib.import_module(name) 370 | 371 | if args.list: 372 | print('Generators:') 373 | for name in generators: 374 | print('',name) 375 | print('Transformers:') 376 | for name in transformers: 377 | print('',name) 378 | print('Outputs:') 379 | for name in outputs: 380 | print('',name) 381 | return 0 382 | 383 | if not args.server or not args.node: 384 | raise Exception('--server and --node must be given') 385 | 386 | if isinstance(args.server, list): 387 | args.server = ' '.join(args.server) 388 | elif not isinstance(args.server, str): 389 | raise Exception('Invalid argument for --server: %r' % args.server) 390 | if isinstance(args.node, list): 391 | args.node = ' '.join(args.node) 392 | elif not isinstance(args.node, str): 393 | raise Exception('Invalid argument for --node: %r' % args.node) 394 | 395 | gens = [] 396 | if args.generator: 397 | for arg in args.generator: 398 | if not re.match(r'^\w', arg): 399 | name, opts = split_arg(arg) 400 | if not name in generators: 401 | logging.critical('Generator %s does not exist' % name) 402 | return 1 403 | gens.append(generators[name](parse_opts(opts))) 404 | continue 405 | for name in arg.split(','): 406 | if not name in generators: 407 | logging.critical('Generator %s does not exist' % name) 408 | return 1 409 | gens.append(generators[name]({})) 410 | 411 | trans = {} 412 | if args.transform: 413 | for arg in args.transform: 414 | name, datasets, opts = split_arg(arg, num=2) 415 | if not name in transformers: 416 | logging.critical('Transformer %s does not exist' % name) 417 | return 1 418 | for dataset in datasets.split(','): 419 | if not dataset in trans: 420 | trans[dataset] = [] 421 | trans[dataset].append(transformers[name](parse_opts(opts))) 422 | 423 | out = [] 424 | if args.output: 425 | for arg in args.output: 426 | name, opts = split_arg(arg) 427 | if not name in outputs: 428 | logging.critical('Output %s does not exist' % name) 429 | return 1 430 | out.append(outputs[name](parse_opts(opts))) 431 | 432 | if args.dataset: 433 | for dataset in args.dataset: 434 | for p in dataset.split(','): 435 | process_dataset[p] = True 436 | 437 | xml = [] 438 | if args.xml: 439 | for entry in args.xml: 440 | if os.path.isfile(entry): 441 | xml.append(entry) 442 | elif os.path.isdir(entry): 443 | with os.scandir(entry) as dir: 444 | for file in dir: 445 | if not file.name.startswith('.') and file.is_file() and file.name.lower().endswith('.xml'): 446 | xml.append(file.path) 447 | else: 448 | logging.error('--xml %r is not a file or directory' % entry) 449 | 450 | dat = [] 451 | if args.dat: 452 | for entry in args.dat: 453 | if os.path.isdir(entry): 454 | dat.append(entry) 455 | else: 456 | logging.error('--dat %r is not a directory' % entry) 457 | 458 | if not xml and not dat: 459 | logging.error('No valid --xml or --dat given') 460 | return 1 461 | 462 | xml_input = inputs['XML']() 463 | for file in xml: 464 | try: 465 | datasets = xml_input.process(file) 466 | except Exception as e: 467 | logging.critical('Unable to process XML file %s: %s' % (file, e)) 468 | return 1 469 | 470 | ret = _process(datasets, gens, trans, out) 471 | if ret > 0: 472 | return ret 473 | 474 | dat_input = inputs['DAT']() 475 | for dir in dat: 476 | try: 477 | datasets = dat_input.process(dir) 478 | except Exception as e: 479 | logging.critical('Unable to process DAT files in %s: %s' % (dir, e)) 480 | return 1 481 | 482 | ret = _process(datasets, gens, trans, out) 483 | if ret > 0: 484 | return ret 485 | -------------------------------------------------------------------------------- /tests/1563520620.dscdata.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | 348 | 349 | 350 | 351 | 352 | 353 | 354 | 355 | 356 | 357 | 358 | 359 | 360 | 361 | 362 | 363 | 364 | 365 | 366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 376 | 377 | 378 | 379 | 380 | 381 | 382 | 383 | 384 | 385 | 386 | 387 | 388 | 389 | 390 | 391 | 392 | 393 | 394 | 395 | 396 | 397 | 398 | 399 | 400 | 401 | 402 | 403 | 404 | 405 | 406 | 407 | 408 | 409 | 410 | 411 | 412 | 413 | 414 | 415 | 416 | 417 | 418 | 419 | 420 | 421 | 422 | 423 | 424 | 425 | 426 | 427 | 428 | 429 | 430 | 431 | 432 | 433 | 434 | 435 | 436 | 437 | 438 | 439 | 440 | 441 | 442 | 443 | 444 | 445 | 446 | 447 | 448 | 449 | 450 | 451 | 452 | 453 | 454 | 455 | 456 | 457 | 458 | 459 | 460 | 461 | 462 | 463 | 464 | 465 | 466 | 467 | 468 | 469 | 470 | 471 | 472 | 473 | 474 | 475 | 476 | 477 | 478 | 479 | 480 | 481 | 482 | 483 | 484 | 485 | 486 | 487 | 488 | 489 | 490 | 491 | 492 | 493 | 494 | 495 | 496 | 497 | 498 | 499 | 500 | 501 | 502 | 503 | 504 | 505 | 506 | 507 | 508 | --------------------------------------------------------------------------------