├── tests ├── __init__.py ├── README.rst ├── test_ccadb_date.py └── test_ccadb_misc.py ├── .gitignore ├── pyproject.toml ├── doc ├── Makefile ├── links.md └── admin-guide.rst ├── TODO.rst ├── bin ├── cert-fingerprints.sh ├── fling.py ├── chainring.py ├── link.py ├── sprocket.py ├── chain.py └── guard.py ├── pan_chainguard ├── __init__.py ├── crtsh.py ├── mozilla.py ├── util.py └── ccadb.py ├── setup.cfg ├── LICENSE.txt ├── README.rst ├── HISTORY.rst └── etc └── derailer.py /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .panrc 2 | *~ 3 | *.py[co] 4 | __pycache__ 5 | *.json 6 | *.xml 7 | *.html 8 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools", "wheel"] 3 | build-backend = "setuptools.build_meta" 4 | -------------------------------------------------------------------------------- /doc/Makefile: -------------------------------------------------------------------------------- 1 | RST2HTML = rst2html 2 | OPTIONS = 3 | SOURCE = admin-guide.html 4 | 5 | .SUFFIXES: .rst .html 6 | .rst.html: 7 | $(RST2HTML) $(OPTIONS) $< $@ 8 | 9 | all: $(SOURCE) 10 | 11 | clean: 12 | rm -f $(SOURCE) 13 | -------------------------------------------------------------------------------- /TODO.rst: -------------------------------------------------------------------------------- 1 | pan-chainguard To Do List 2 | ========================= 3 | 4 | - Verify pan-chainguard derived root store with each vendor list. 5 | For Mozilla also verify pan-chainguard intermediates with 6 | preloaded list. 7 | 8 | - Integrate with SCM API (e.g., SCM capable guard.py). 9 | 10 | - Update admin guide with use cases (e.g., update root store 11 | only). 12 | 13 | - Optimise XML API usage for increased performance; consider use 14 | of multi-config. 15 | 16 | - Retry transient XML API errors when possible. 17 | 18 | - Use Python logging module. 19 | -------------------------------------------------------------------------------- /bin/cert-fingerprints.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | usage() { 4 | echo "usage: $(basename $0) cert-directory" 5 | exit 1 6 | } 7 | 8 | # Usage: fingerprints directory 9 | fingerprints() { 10 | dir=$1 11 | 12 | echo '"type","sha256"' 13 | for file in $(ls $dir/*.cer); do 14 | fp=$(openssl x509 -noout -fingerprint -sha256 -in $file) 15 | fp=$(echo $fp | sed -e 's/.*=//') 16 | fp=$(echo $fp | sed -e 's/://g') 17 | echo \"root\",\"$fp\" 18 | done 19 | } 20 | 21 | if [ $# != 1 ] || [ $1 = '--help' ]; then 22 | usage 23 | fi 24 | if [ ! -d $1 ]; then 25 | echo "$1: not directory" 26 | exit 1 27 | fi 28 | 29 | fingerprints $1 30 | -------------------------------------------------------------------------------- /tests/README.rst: -------------------------------------------------------------------------------- 1 | pan-chainguard Tests 2 | ==================== 3 | 4 | ``pan-chainguard`` tests use the Python 5 | `unit testing framework 6 | `_. 7 | 8 | Run Tests 9 | --------- 10 | 11 | To run all tests from the top-level directory: 12 | :: 13 | 14 | $ python3 -m unittest discover -v -s tests 15 | 16 | To run a specific test from the top-level directory: 17 | :: 18 | 19 | $ python3 -m unittest discover -v -s tests -p test_ccadb_date.py 20 | 21 | To run all tests from the ``tests/`` directory: 22 | :: 23 | 24 | $ python3 -m unittest discover -v -t .. 25 | 26 | To run a specific test from the ``tests/`` directory: 27 | :: 28 | 29 | $ python3 -m unittest discover -v -t .. -p test_ccadb_date.py 30 | -------------------------------------------------------------------------------- /pan_chainguard/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) 2024 Palo Alto Networks, Inc. 3 | # 4 | # Permission to use, copy, modify, and distribute this software for any 5 | # purpose with or without fee is hereby granted, provided that the above 6 | # copyright notice and this permission notice appear in all copies. 7 | # 8 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 9 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 10 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 11 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 12 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 13 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 14 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 15 | # 16 | 17 | __version__ = '0.12.0' 18 | title = 'pan-chainguard' 19 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = pan-chainguard 3 | version = attr: pan_chainguard.__version__ 4 | author = Palo Alto Networks, Inc. 5 | author_email = devrel@paloaltonetworks.com 6 | description = Manage Root Store and Intermediate Certificate Chains on PAN-OS 7 | long_description = file: README.rst 8 | long_description_content_type = text/x-rst 9 | url = https://github.com/PaloAltoNetworks/pan-chainguard 10 | license = ISC 11 | classifiers = 12 | Programming Language :: Python :: 3 13 | License :: OSI Approved :: ISC License (ISCL) 14 | 15 | [options] 16 | packages = 17 | pan_chainguard 18 | scripts = 19 | bin/fling.py 20 | bin/sprocket.py 21 | bin/chain.py 22 | bin/chainring.py 23 | bin/link.py 24 | bin/guard.py 25 | bin/cert-fingerprints.sh 26 | install_requires = 27 | aiohttp>=3.9.0 28 | pan-python>=0.25.0 29 | treelib>=1.7.0 30 | python_requires = >=3.9 31 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | pan-chainguard is distributed with an ISC license 2 | (https://cvsweb.openbsd.org/src/share/misc/license.template?rev=HEAD): 3 | 4 | Copyright (c) 2024 Palo Alto Networks, Inc. 5 | 6 | Permission to use, copy, modify, and distribute this software for any 7 | purpose with or without fee is hereby granted, provided that the above 8 | copyright notice and this permission notice appear in all copies. 9 | 10 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | 18 | CCADB data is licensed under the CCADB Data Usage Terms: 19 | https://www.ccadb.org/rootstores/usage#ccadb-data-usage-terms. 20 | -------------------------------------------------------------------------------- /tests/test_ccadb_date.py: -------------------------------------------------------------------------------- 1 | from datetime import timedelta 2 | import unittest 3 | 4 | from pan_chainguard.ccadb import * 5 | from pan_chainguard.ccadb import _now 6 | 7 | FMT = '%Y.%m.%d' 8 | VALID_FROM = 'Valid From (GMT)' 9 | VALID_TO = 'Valid To (GMT)' 10 | 11 | now = _now() 12 | today = now.strftime(FMT) 13 | x = now - timedelta(days=1) 14 | yesterday = x.strftime(FMT) 15 | x = now + timedelta(days=1) 16 | tomorrow = x.strftime(FMT) 17 | 18 | 19 | class CcadbTest(unittest.TestCase): 20 | def test_01(self): 21 | x = today 22 | t = {VALID_FROM: x} 23 | r, err = valid_from(t) 24 | self.assertTrue(r, "%s: %s" % (x, err)) 25 | self.assertIsNone(err) 26 | 27 | def test_02(self): 28 | x = yesterday 29 | t = {VALID_FROM: x} 30 | r, err = valid_from(t) 31 | self.assertTrue(r, "%s: %s" % (x, err)) 32 | self.assertIsNone(err) 33 | 34 | def test_03(self): 35 | x = tomorrow 36 | t = {VALID_FROM: x} 37 | r, err = valid_from(t) 38 | self.assertFalse(r, "%s: %s" % (x, err)) 39 | self.assertIsNotNone(err) 40 | self.assertIn('Not yet valid', err) 41 | 42 | def test_04(self): 43 | x = tomorrow 44 | t = {VALID_TO: x} 45 | r, err = valid_to(t) 46 | self.assertTrue(r, "%s: %s" % (x, err)) 47 | self.assertIsNone(err) 48 | 49 | def test_05(self): 50 | x = yesterday 51 | t = {VALID_TO: x} 52 | r, err = valid_to(t) 53 | self.assertFalse(r, "%s: %s" % (x, err)) 54 | self.assertIsNotNone(err) 55 | self.assertIn('Expired', err) 56 | 57 | def test_06(self): 58 | x = today 59 | t = {VALID_TO: x} 60 | r, err = valid_to(t) 61 | self.assertFalse(r, "%s: %s" % (x, err)) 62 | self.assertIsNotNone(err) 63 | self.assertIn('Expired', err) 64 | 65 | def test_07(self): 66 | t = { 67 | VALID_FROM: yesterday, 68 | VALID_TO: tomorrow, 69 | } 70 | r, err = valid_from_to(t) 71 | self.assertTrue(r, "%s: %s" % (t, err)) 72 | self.assertIsNone(err) 73 | 74 | def test_08(self): 75 | t = { 76 | VALID_FROM: yesterday, 77 | VALID_TO: today, 78 | } 79 | r, err = valid_from_to(t) 80 | self.assertFalse(r, "%s: %s" % (t, err)) 81 | self.assertIsNotNone(err) 82 | self.assertIn('Expired', err) 83 | -------------------------------------------------------------------------------- /doc/links.md: -------------------------------------------------------------------------------- 1 | # pan-chainguard Data and Process Flow 2 | 3 | ```mermaid 4 | flowchart TD 5 | panos{{"PAN-OS NGFW, Panorama
Export Default Trusted CAs"}} 6 | panos2{{"PAN-OS NGFW, Panorama
Update Device Certificates"}} 7 | truststore[(trust-store.tgz)] 8 | truststoredir[(trust-store/)] 9 | trustpolicy[("policy.json
[mozilla,apple,microsoft,chrome]")] 10 | fling("fling.py
export PAN-OS trusted CAs") 11 | sprocket("sprocket.py
create custom root store") 12 | chain("chain.py
determine intermediate CAs") 13 | link("link.py
get CA certificates") 14 | guard("guard.py
update PAN-OS trusted CAs") 15 | chainring("chainring.py
certificate tree analysis and reporting") 16 | curl(curl) 17 | untar(untar) 18 | fingerprints(cert-fingerprints.sh) 19 | rootfingerprintscsv[(root-fingerprints.csv)] 20 | intfingerprintscsv[(intermediate-fingerprints.csv)] 21 | certificatetree[(certificate-tree.json)] 22 | treedocs[("certificate documents (txt, html, rst, ...)")] 23 | ccadb[("AllCertificateRecordsReport.csv
CCADB All Certificate Information")] 24 | roottrust[("AllIncludedRootCertsCSV.csv
CCADB All Included Root Certificate Trust Bits")] 25 | mozilla[("MozillaIntermediateCerts.csv
PublicAllIntermediateCertsWithPEMReport.csv
Intermediate CA certificates from Mozilla")] 26 | onecrl[("IntermediateCertsInOneCRL.csv
Intermediate certificates in Mozilla OneCRL")] 27 | crtsh>crt.sh:443] 28 | oldcertificates[(certificates-old.tgz)] 29 | newcertificates[(certificates-new.tgz)] 30 | 31 | panos<-->|XML API|fling 32 | fling-->truststore 33 | truststore-->untar 34 | untar-->truststoredir 35 | truststoredir-->fingerprints 36 | fingerprints-->|deprecated|rootfingerprintscsv 37 | curl-->ccadb 38 | ccadb-->chain 39 | curl-->roottrust 40 | curl-->onecrl 41 | onecrl-->chain 42 | trustpolicy-->sprocket 43 | ccadb-->sprocket 44 | roottrust-->sprocket 45 | sprocket-->rootfingerprintscsv 46 | rootfingerprintscsv-->chain 47 | chain-->intfingerprintscsv 48 | chain-->certificatetree 49 | certificatetree-->chainring 50 | chainring-->treedocs 51 | rootfingerprintscsv-->link 52 | intfingerprintscsv-->link 53 | curl-->mozilla 54 | mozilla-->link 55 | oldcertificates-->link 56 | crtsh-->|API|link 57 | link-->newcertificates 58 | newcertificates-->guard 59 | guard<-->|XML API|panos2 60 | ``` 61 | -------------------------------------------------------------------------------- /pan_chainguard/crtsh.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) 2024 Palo Alto Networks, Inc. 3 | # 4 | # Permission to use, copy, modify, and distribute this software for any 5 | # purpose with or without fee is hereby granted, provided that the above 6 | # copyright notice and this permission notice appear in all copies. 7 | # 8 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 9 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 10 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 11 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 12 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 13 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 14 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 15 | # 16 | 17 | # crt.sh API interface 18 | 19 | import aiohttp 20 | 21 | URL = 'https://crt.sh' 22 | 23 | 24 | class ArgsError(Exception): 25 | pass 26 | 27 | 28 | class CrtShApi: 29 | def __init__(self, *, 30 | timeout=None, 31 | headers=None): 32 | self.url = URL 33 | 34 | if isinstance(timeout, tuple): 35 | if len(timeout) != 2: 36 | raise ArgsError('timeout tuple length must be 2') 37 | x = aiohttp.ClientTimeout(sock_connect=timeout[0], 38 | sock_read=timeout[1]) 39 | else: 40 | x = aiohttp.ClientTimeout(total=timeout) 41 | 42 | self.session = aiohttp.ClientSession(timeout=x, 43 | headers=headers) 44 | 45 | async def __aenter__(self): 46 | return self 47 | 48 | async def __aexit__(self, *args): 49 | await self.session.close() 50 | 51 | async def content(self, *, resp=None): 52 | if resp is None: 53 | raise ArgsError('missing resp') 54 | 55 | if resp.status != 200: 56 | return None, None 57 | 58 | filename = None 59 | if (resp.content_disposition is not None and 60 | resp.content_disposition.filename is not None): 61 | filename = resp.content_disposition.filename 62 | 63 | x = await resp.text() 64 | 65 | return filename, x 66 | 67 | async def download(self, *, 68 | id=None, 69 | query_string=None): 70 | path = '/' 71 | url = self.url + path 72 | 73 | data = {} 74 | if id is not None: 75 | data['d'] = id 76 | if query_string is not None: 77 | data.update(query_string) 78 | 79 | kwargs = { 80 | 'url': url, 81 | 'data': data, 82 | } 83 | 84 | resp = await self.session.post(**kwargs) 85 | 86 | return resp 87 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | pan-chainguard - Manage Root Store and Intermediate Certificate Chains on PAN-OS 2 | ================================================================================ 3 | 4 | Overview 5 | -------- 6 | 7 | ``pan-chainguard`` is a Python application which uses 8 | `CCADB data 9 | `_ 10 | and allows PAN-OS SSL decryption administrators to: 11 | 12 | #. Create a custom, up-to-date trusted root store for PAN-OS. 13 | #. Determine intermediate certificate chains for trusted Certificate 14 | Authorities in PAN-OS so they can be `preloaded 15 | `_ 16 | as device certificates. 17 | 18 | Issue 1: Out-of-date Root Store 19 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 20 | 21 | The PAN-OS root store (*Default Trusted Certificate Authorities*) is 22 | updated only in PAN-OS major software releases; it is not currently 23 | managed by content updates. The root store for PAN-OS 10.x releases 24 | is now over 5 years old. 25 | 26 | The impact for PAN-OS SSL decryption administrators is when the root 27 | CA for the server certificate is not trusted, the firewall will 28 | provide the forward untrust certificate to the client. End users will 29 | then see errors such as *NET::ERR_CERT_AUTHORITY_INVALID* (Chrome) or 30 | *SEC_ERROR_UNKNOWN_ISSUER* (Firefox) until the missing trusted CAs are 31 | identified, the certificates are obtained, and the certificates are 32 | imported into PAN-OS. 33 | 34 | Issue 2: Misconfigured Servers 35 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 36 | 37 | Many TLS enabled origin servers suffer from a misconfiguration in 38 | which they: 39 | 40 | #. Do not return intermediate CA certificates. 41 | #. Return certificates out of order. 42 | #. Return intermediate certificates which are not related to the root 43 | CA for the server certificate. 44 | 45 | The impact for PAN-OS SSL decryption administrators is end users will 46 | see errors such as *unable to get local issuer certificate* until the 47 | sites that are misconfigured are 48 | `identified 49 | `_, 50 | the required intermediate certificates are obtained, and the 51 | certificates are imported into PAN-OS. 52 | 53 | Solution 1: Create Custom Root Store 54 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 55 | 56 | ``pan-chainguard`` can create a custom root store, using one or more 57 | of the major vendor root stores, which are managed by their CA 58 | certificate program: 59 | 60 | + `Mozilla `_ 61 | + `Apple `_ 62 | + `Chrome `_ 63 | + `Microsoft `_ 64 | 65 | The custom root store can then be added to PAN-OS as trusted CA device 66 | certificates. 67 | 68 | Solution 2: Intermediate CA Preloading 69 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 70 | 71 | ``pan-chainguard`` uses a root store and the 72 | *All Certificate Information (root and intermediate) in CCADB (CSV)* 73 | data file as input, and determines the intermediate certificate 74 | chains, if available, for each root CA certificate. These can then be 75 | added to PAN-OS as trusted CA device certificates. 76 | 77 | By preloading known intermediates for the trusted CAs, the number of 78 | TLS connection errors that users encounter for misconfigured servers 79 | can be reduced, without reactive actions by an administrator. 80 | 81 | Documentation 82 | ------------- 83 | 84 | - Administrator's Guide: 85 | 86 | https://github.com/PaloAltoNetworks/pan-chainguard/blob/main/doc/admin-guide.rst 87 | 88 | Install ``pan-chainguard`` 89 | -------------------------- 90 | 91 | ``pan-chainguard`` is available as a 92 | `release 93 | `_ 94 | on GitHub and as a 95 | `package 96 | `_ 97 | on PyPi. 98 | 99 | ``pan-chainguard-content`` - Certificate Content for pan-chainguard 100 | ------------------------------------------------------------------- 101 | 102 | `pan-chainguard-content 103 | `_ 104 | provides pre-generated, up-to-date content 105 | which can be used to simplify the deployment of ``pan-chainguard``. 106 | -------------------------------------------------------------------------------- /pan_chainguard/mozilla.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) 2024 Palo Alto Networks, Inc. 3 | # 4 | # Permission to use, copy, modify, and distribute this software for any 5 | # purpose with or without fee is hereby granted, provided that the above 6 | # copyright notice and this permission notice appear in all copies. 7 | # 8 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 9 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 10 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 11 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 12 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 13 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 14 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 15 | # 16 | 17 | import csv 18 | import sys 19 | from typing import Optional 20 | 21 | 22 | class MozillaError(Exception): 23 | pass 24 | 25 | 26 | # https://wiki.mozilla.org/CA/Intermediate_Certificates 27 | # 28 | # Intermediate CA Certificates: 29 | # https://ccadb.my.salesforce-sites.com/mozilla/PublicAllIntermediateCertsWithPEMCSV 30 | # 31 | 32 | # Non-revoked, non-expired Intermediate CA Certificates chaining up 33 | # to roots in Mozilla's program with the Websites trust bit set: 34 | # https://ccadb.my.salesforce-sites.com/mozilla/MozillaIntermediateCertsCSVReport 35 | 36 | class MozillaCaCerts: 37 | def __init__(self, *, 38 | path: str): 39 | def insert(row): 40 | self.certs[row[SHA256]] = row[PEM] 41 | 42 | self.certs = {} 43 | self.name = None 44 | 45 | try: 46 | with open(path, 'r', newline='') as csvfile: 47 | reader = csv.DictReader(csvfile, 48 | dialect='unix') 49 | row = next(reader) 50 | if 'SHA256' in row and 'PEM' in row: 51 | SHA256 = 'SHA256' 52 | PEM = 'PEM' 53 | self.name = 'MozillaIntermediateCerts' 54 | elif ('SHA-256 Fingerprint' in row and 55 | 'PEM Info' in row): 56 | SHA256 = 'SHA-256 Fingerprint' 57 | PEM = 'PEM Info' 58 | self.name = 'PublicAllIntermediateCerts' 59 | else: 60 | raise MozillaError('Invalid CSV') 61 | insert(row) 62 | 63 | for row in reader: 64 | insert(row) 65 | 66 | except OSError as e: 67 | raise MozillaError(str(e)) 68 | 69 | def get_cert_pem(self, *, sha256: str) -> Optional[str]: 70 | if sha256 not in self.certs: 71 | return 72 | 73 | pem = self.certs[sha256] 74 | 75 | # XXX PublicAllIntermediateCertsWithPEMReport.csv has 76 | # single quotes around the PEM. 77 | c = "'" 78 | if pem.startswith(c): 79 | pem = pem[1:] 80 | if pem.endswith(c): 81 | pem = pem[:-1] 82 | 83 | return pem 84 | 85 | 86 | # https://wiki.mozilla.org/CA/Intermediate_Certificates 87 | # 88 | # The following reports list the intermediate certificates that have 89 | # been added to OneCRL, and their revocation status as indicated by 90 | # the CA in the CCADB: 91 | # https://ccadb.my.salesforce-sites.com/mozilla/IntermediateCertsInOneCRLReportCSV 92 | 93 | class MozillaOneCrl: 94 | def __init__(self, *, 95 | path: str, 96 | debug: bool = False): 97 | self.certs = {} 98 | EXPECTED = { # subset 99 | 'OneCRL Bug Number', 100 | 'Comments', 101 | 'SHA-256 Fingerprint', 102 | } 103 | 104 | try: 105 | with open(path, 'r', newline='') as csvfile: 106 | reader = csv.DictReader(csvfile, 107 | dialect='unix') 108 | fieldnames = set(reader.fieldnames or []) 109 | missing = EXPECTED - fieldnames 110 | if missing: 111 | raise MozillaError('Invalid CSV') 112 | 113 | for row in reader: 114 | sha256 = row['SHA-256 Fingerprint'] 115 | if sha256 in self.certs and debug: 116 | print('Duplicate in OneCRL %s' % ( 117 | sha256), file=sys.stderr) 118 | self.certs[sha256] = row 119 | 120 | except OSError as e: 121 | raise MozillaError(str(e)) 122 | 123 | def get(self, *, sha256: str) -> Optional[dict[str, str]]: 124 | if sha256 not in self.certs: 125 | return 126 | 127 | return self.certs[sha256] 128 | -------------------------------------------------------------------------------- /tests/test_ccadb_misc.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from pan_chainguard.ccadb import * 4 | from pan_chainguard.ccadb import TrustBitsMap 5 | 6 | REVOCATION_STATUS = 'Revocation Status' 7 | DERIVED_TRUST_BITS = 'Derived Trust Bits' # Intermediate Certificate 8 | ROOT_TRUST_BITS = 'Trust Bits for Root Cert' # Root Certificate 9 | TRUST_BITS = { 10 | DERIVED_TRUST_BITS: { 11 | '': derived_trust_bits, 12 | 'list': derived_trust_bits_list, 13 | 'flag': trust_bits_flag, 14 | 'type': 'Intermediate Certificate', 15 | }, 16 | ROOT_TRUST_BITS: { 17 | '': root_trust_bits, 18 | 'list': root_trust_bits_list, 19 | 'flag': trust_bits_flag, 20 | 'type': 'Root Certificate', 21 | }, 22 | } 23 | 24 | 25 | class CcadbTest(unittest.TestCase): 26 | def test_01(self): 27 | x = '' 28 | t = {REVOCATION_STATUS: x} 29 | r, err = revoked(t) 30 | self.assertFalse(r, "%s: %s" % (x, err)) 31 | self.assertIsNone(err) 32 | 33 | def test_02(self): 34 | x = 'Not Revoked' 35 | t = {REVOCATION_STATUS: x} 36 | r, err = revoked(t) 37 | self.assertFalse(r, "%s: %s" % (x, err)) 38 | self.assertIsNone(err) 39 | 40 | def test_03(self): 41 | x = 'Revoked' 42 | t = {REVOCATION_STATUS: x} 43 | r, err = revoked(t) 44 | self.assertTrue(r, "%s: %s" % (x, err)) 45 | self.assertIsNotNone(err) 46 | self.assertEqual('Revoked', err) 47 | 48 | def test_04(self): 49 | x = 'Unknown' 50 | t = {REVOCATION_STATUS: x} 51 | r, err = revoked(t) 52 | self.assertTrue(r, "%s: %s" % (x, err)) 53 | self.assertIsNotNone(err) 54 | self.assertEqual('Unknown', err) 55 | 56 | def test_05(self): 57 | for k in TRUST_BITS: 58 | x = '' 59 | t = { 60 | 'Certificate Record Type': TRUST_BITS[k]['type'], 61 | k: x, 62 | } 63 | 64 | r = TRUST_BITS[k]['list'](t) 65 | self.assertListEqual([], r) 66 | 67 | bits = TRUST_BITS[k]['flag'](r) 68 | self.assertEqual(TrustBits.NONE, bits) 69 | 70 | bits = TRUST_BITS[k][''](t) 71 | self.assertEqual(TrustBits.NONE, bits) 72 | 73 | def test_06(self): 74 | for k in TRUST_BITS: 75 | x = 'abcd' 76 | t = { 77 | 'Certificate Record Type': TRUST_BITS[k]['type'], 78 | k: x, 79 | } 80 | 81 | r = TRUST_BITS[k]['list'](t) 82 | self.assertListEqual([x], r) 83 | 84 | bits = TRUST_BITS[k]['flag'](r) 85 | self.assertEqual(TrustBits.OTHER, bits) 86 | 87 | bits = TRUST_BITS[k][''](t) 88 | self.assertEqual(TrustBits.OTHER, bits) 89 | 90 | def test_07(self): 91 | for k in TRUST_BITS: 92 | x = 'Server Authentication' 93 | t = { 94 | 'Certificate Record Type': TRUST_BITS[k]['type'], 95 | k: x, 96 | } 97 | 98 | r = TRUST_BITS[k]['list'](t) 99 | self.assertListEqual([x], r) 100 | 101 | bits = TRUST_BITS[k]['flag'](r) 102 | self.assertEqual(TrustBitsMap[x], bits) 103 | 104 | bits = TRUST_BITS[k][''](t) 105 | self.assertEqual(TrustBitsMap[x], bits) 106 | 107 | def test_08(self): 108 | for k in TRUST_BITS: 109 | x = ';'.join(TrustBitsMap.keys()) 110 | t = { 111 | 'Certificate Record Type': TRUST_BITS[k]['type'], 112 | k: x, 113 | } 114 | 115 | r = TRUST_BITS[k]['list'](t) 116 | self.assertListEqual(list(TrustBitsMap.keys()), r) 117 | 118 | bits = TRUST_BITS[k]['flag'](r) 119 | for x in TrustBitsMap.values(): 120 | self.assertIn(x, bits) 121 | 122 | bits = TRUST_BITS[k][''](t) 123 | for x in TrustBitsMap.values(): 124 | self.assertIn(x, bits) 125 | 126 | def test_09(self): 127 | row = { 128 | "Certificate Record Type": "Intermediate Certificate", 129 | "Mozilla Status": "Not Included", 130 | "Apple Status": "Not Included", 131 | "Chrome Status": "Not Included", 132 | "Microsoft Status": "Not Included", 133 | } 134 | r = RootStatusBits.NONE 135 | 136 | with self.assertRaises(ValueError) as e: 137 | bits = root_status_bits_flag(row) 138 | x = ('certificate type "Intermediate Certificate" ' 139 | 'not "Root Certificate"') 140 | self.assertEqual(str(e.exception), x) 141 | 142 | def test_10(self): 143 | row = { 144 | "Certificate Record Type": "Root Certificate", 145 | "Mozilla Status": "Not Included", 146 | "Apple Status": "Not Included", 147 | "Chrome Status": "Not Included", 148 | "Microsoft Status": "Not Included", 149 | } 150 | r = RootStatusBits.NONE 151 | 152 | bits = root_status_bits_flag(row) 153 | self.assertEqual(bits, r) 154 | r = root_status_bits(bits) 155 | self.assertEqual(r, []) 156 | 157 | def test_11(self): 158 | row = { 159 | "Certificate Record Type": "Root Certificate", 160 | "Mozilla Status": "Included", 161 | "Apple Status": "Included", 162 | "Chrome Status": "Included", 163 | "Microsoft Status": "Included", 164 | } 165 | r = (RootStatusBits.MOZILLA | 166 | RootStatusBits.APPLE | 167 | RootStatusBits.CHROME | 168 | RootStatusBits.MICROSOFT) 169 | 170 | bits = root_status_bits_flag(row) 171 | self.assertEqual(bits, r) 172 | r = root_status_bits(bits) 173 | self.assertEqual(r, ['mozilla', 'apple', 'chrome', 'microsoft']) 174 | r = root_status_bits(bits, compact=True) 175 | self.assertEqual(r, 'MzApChMs') 176 | 177 | def test_12(self): 178 | row = { 179 | "Certificate Record Type": "Root Certificate", 180 | "Mozilla Status": "Included", 181 | "Apple Status": "Included", 182 | "Chrome Status": "Not Included", 183 | "Microsoft Status": "Not Included", 184 | } 185 | r = (RootStatusBits.MOZILLA | 186 | RootStatusBits.APPLE) 187 | 188 | bits = root_status_bits_flag(row) 189 | self.assertEqual(bits, r) 190 | r = root_status_bits(bits) 191 | self.assertEqual(r, ['mozilla', 'apple']) 192 | r = root_status_bits(bits, compact=True) 193 | self.assertEqual(r, 'MzAp') 194 | -------------------------------------------------------------------------------- /bin/fling.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2024 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import argparse 20 | import asyncio 21 | import io 22 | import logging 23 | import os 24 | import sys 25 | import tarfile 26 | import time 27 | 28 | try: 29 | import pan.xapi 30 | except ImportError: 31 | print('Install pan-python: https://pypi.org/project/pan-python/', 32 | file=sys.stderr) 33 | sys.exit(1) 34 | 35 | libpath = os.path.dirname(os.path.abspath(__file__)) 36 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 37 | 38 | from pan_chainguard import title, __version__ 39 | from pan_chainguard.util import s1_in_s2 40 | 41 | args = None 42 | 43 | 44 | def main(): 45 | global args 46 | args = parse_args() 47 | 48 | logger = logging.getLogger(pan.xapi.__name__) 49 | if args.xdebug == 3: 50 | logger.setLevel(pan.xapi.DEBUG3) 51 | elif args.xdebug == 2: 52 | logger.setLevel(pan.xapi.DEBUG2) 53 | elif args.xdebug == 1: 54 | logger.setLevel(pan.xapi.DEBUG1) 55 | 56 | log_format = '%(name)s: %(message)s' 57 | handler = logging.StreamHandler() 58 | formatter = logging.Formatter(log_format) 59 | handler.setFormatter(formatter) 60 | logger.addHandler(handler) 61 | 62 | asyncio.run(main_loop()) 63 | 64 | sys.exit(0) 65 | 66 | 67 | async def main_loop(): 68 | try: 69 | xapi = pan.xapi.PanXapi(tag=args.tag, 70 | debug=args.xdebug) 71 | except pan.xapi.PanXapiError as e: 72 | print('pan.xapi.PanXapi:', e, file=sys.stderr) 73 | sys.exit(1) 74 | 75 | create_archive(test=True) 76 | 77 | store = {} 78 | for cert_name, filename in get_trusted_certs(xapi): 79 | if args.verbose: 80 | print('Exporting %s => %s' % (cert_name, filename)) 81 | _, content = export_cert(xapi, cert_name) 82 | if content is not None: 83 | store[filename] = content 84 | 85 | total = create_archive(store) 86 | print('Exported %d PAN-OS trusted CAs to %s' % (total, args.certs)) 87 | 88 | 89 | def get_trusted_certs(xapi): 90 | kwargs = {'xpath': '/config/predefined/trusted-root-ca/entry'} 91 | api_request(xapi, xapi.get, kwargs, 'success', '19') 92 | entries = xapi.element_root.findall('./result/entry') 93 | 94 | for entry in entries: 95 | name = entry.attrib['name'] 96 | x = entry.find('./filename') 97 | filename = x.text 98 | yield name, filename 99 | 100 | 101 | def export_cert(xapi, name): 102 | kwargs = { 103 | 'category': 'certificate', 104 | 'extra_qs': { 105 | 'certificate-name': name, 106 | 'format': 'pem', 107 | 'include-key': 'no', 108 | }, 109 | } 110 | api_request(xapi, xapi.export, kwargs, 'success') 111 | if xapi.export_result is None: 112 | print("Can't export %s" % name, file=sys.stderr) 113 | return None, None 114 | 115 | return xapi.export_result['file'], xapi.export_result['content'] 116 | 117 | 118 | def api_request(xapi, func, kwargs, status=None, status_code=None): 119 | try: 120 | func(**kwargs) 121 | except pan.xapi.PanXapiError as e: 122 | print('%s: %s: %s' % (func.__name__, kwargs, e), 123 | file=sys.stderr) 124 | sys.exit(1) 125 | 126 | if status is not None and not s1_in_s2(xapi.status, status): 127 | print('%s: %s: status %s != %s' % 128 | (func.__name__, kwargs, 129 | xapi.status, status), 130 | file=sys.stderr) 131 | sys.exit(1) 132 | 133 | if (status_code is not None and 134 | not s1_in_s2(xapi.status_code, status_code)): 135 | print('%s: %s: status_code %s != %s' % 136 | (func.__name__, kwargs, 137 | xapi.status_code, status_code), 138 | file=sys.stderr) 139 | sys.exit(1) 140 | 141 | 142 | def create_archive(store=None, test=False): 143 | try: 144 | with tarfile.open(name=args.certs, mode='w:gz') as tar: 145 | if test: 146 | return 147 | total = 0 148 | for filename in sorted(store.keys()): 149 | member = tarfile.TarInfo(name=filename) 150 | member.size = len(store[filename]) 151 | member.mtime = time.time() 152 | f = io.BytesIO(store[filename]) 153 | tar.addfile(member, fileobj=f) 154 | total += 1 155 | except (tarfile.TarError, OSError) as e: 156 | print("tarfile %s: %s" % (args.certs, e), file=sys.stderr) 157 | sys.exit(1) 158 | 159 | return total 160 | 161 | 162 | def parse_args(): 163 | parser = argparse.ArgumentParser( 164 | usage='%(prog)s [options]', 165 | description='export PAN-OS trusted CAs') 166 | parser.add_argument('--tag', '-t', 167 | required=True, 168 | help='.panrc tagname') 169 | x = 'root-store.tgz' 170 | parser.add_argument('--certs', 171 | default=x, 172 | metavar='PATH', 173 | help='PAN-OS trusted CAs archive path' 174 | ' (default: %s)' % x) 175 | parser.add_argument('--xdebug', 176 | type=int, 177 | choices=[0, 1, 2, 3], 178 | default=0, 179 | help='pan.xapi debug') 180 | parser.add_argument('--verbose', 181 | action='store_true', 182 | help='enable verbosity') 183 | parser.add_argument('--debug', 184 | type=int, 185 | choices=[0, 1, 2, 3], 186 | default=0, 187 | help='enable debug') 188 | x = '%s %s' % (title, __version__) 189 | parser.add_argument('--version', 190 | action='version', 191 | help='display version', 192 | version=x) 193 | args = parser.parse_args() 194 | 195 | if args.debug: 196 | print(args, file=sys.stderr) 197 | 198 | return args 199 | 200 | 201 | if __name__ == '__main__': 202 | main() 203 | -------------------------------------------------------------------------------- /pan_chainguard/util.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) 2024 Palo Alto Networks, Inc. 3 | # 4 | # Permission to use, copy, modify, and distribute this software for any 5 | # purpose with or without fee is hereby granted, provided that the above 6 | # copyright notice and this permission notice appear in all copies. 7 | # 8 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 9 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 10 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 11 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 12 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 13 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 14 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 15 | # 16 | 17 | from collections import defaultdict 18 | import csv 19 | import io 20 | import os 21 | import re 22 | import tarfile 23 | import time 24 | import treelib 25 | from typing import Union 26 | 27 | NAME_PREFIX = 'LINK-' 28 | NAME_RE = r'^%s%s$' % (NAME_PREFIX, '[A-F0-9]{26,26}') 29 | NAME_RE_COMPAT = r'^(\d{4,4}|LINK)-[0-9A-F]{26,26}$' 30 | FIELDNAMES_FINGERPRINTS = [ 31 | 'type', 32 | 'sha256', 33 | ] 34 | 35 | 36 | class UtilError(Exception): 37 | pass 38 | 39 | 40 | def s1_in_s2(s1: str, s2: Union[str, list[str]]) -> bool: 41 | if isinstance(s2, str): 42 | return s1 == s2 43 | elif isinstance(s2, list): 44 | return s1 in s2 45 | else: 46 | raise ValueError('Invalid type for s2. ' 47 | 'Must be a string or a list of strings.') 48 | 49 | 50 | def is_writable(path: str) -> bool: 51 | # Check if the file exists 52 | if os.path.exists(path): 53 | # If the file exists, check if it is writable 54 | return os.access(path, os.W_OK) 55 | else: 56 | # If the file doesn't exist, check if the directory is writable 57 | parent_dir = os.path.dirname(path) or '.' 58 | return os.access(parent_dir, os.W_OK) 59 | 60 | 61 | def read_cert_archive(*, path: str) -> dict[str, tuple[str, str]]: 62 | def parse_name(name): 63 | pat = (r'^(intermediate|root)/' 64 | r'[0-9A-F]{64,64}\.pem$') 65 | if not re.search(pat, name): 66 | e = 'malformed path in archive: %s' % name 67 | raise UtilError(e) 68 | type_, pem = os.path.split(name) 69 | sha256 = pem[:64] 70 | 71 | return type_, sha256 72 | 73 | data = {} 74 | try: 75 | with tarfile.open(name=path, mode='r') as tar: 76 | for member in tar: 77 | if member.name in ['root', 'intermediate']: 78 | continue 79 | cert_type, sha256 = parse_name(member.name) 80 | f = tar.extractfile(member) 81 | content = f.read() 82 | data[sha256] = (cert_type, content) 83 | 84 | except (tarfile.TarError, OSError) as e: 85 | raise UtilError(str(e)) 86 | 87 | return data 88 | 89 | 90 | def write_cert_archive(*, path: str, data: dict[str, tuple[str, str]]): 91 | try: 92 | with tarfile.open(name=path, mode='w:gz') as tar: 93 | for k, v in data.items(): 94 | name = os.path.join(v[0], k + '.pem') 95 | member = tarfile.TarInfo(name=name) 96 | member.size = len(v[1]) 97 | member.mtime = time.time() 98 | content = (v[1].encode() if isinstance(v[1], str) 99 | else v[1]) 100 | f = io.BytesIO(content) 101 | tar.addfile(member, fileobj=f) 102 | except (tarfile.TarError, OSError) as e: 103 | raise UtilError(str(e)) 104 | 105 | 106 | def hash_to_name(*, sha256: str) -> str: 107 | # PAN-OS certificate-name max len 63 108 | # Panorama certificate-name max len 31 109 | # PAN-99186 won't do 110 | x = NAME_PREFIX + sha256 111 | return x[0:31] 112 | 113 | 114 | def read_fingerprints(*, path: str) -> list[dict[str, str]]: 115 | try: 116 | with open(path, 'r', newline='') as csvfile: 117 | reader = csv.DictReader(csvfile, 118 | dialect='unix') 119 | fieldnames = set(reader.fieldnames or []) 120 | if fieldnames != set(FIELDNAMES_FINGERPRINTS): 121 | raise UtilError('Invalid CSV') 122 | 123 | x = [] 124 | for row in reader: 125 | x.append(row) 126 | except OSError as e: 127 | raise UtilError(str(e)) 128 | 129 | return x 130 | 131 | 132 | def write_fingerprints(*, path: str, data: list[dict[str, str]]): 133 | try: 134 | with open(path, 'w', newline='') as csvfile: 135 | writer = csv.DictWriter(csvfile, 136 | dialect='unix', 137 | fieldnames=FIELDNAMES_FINGERPRINTS) 138 | writer.writeheader() 139 | for x in data: 140 | row = { 141 | 'type': x['type'], 142 | 'sha256': x['sha256'], 143 | } 144 | writer.writerow(row) 145 | 146 | except OSError as e: 147 | raise UtilError(str(e)) 148 | 149 | 150 | def tree_to_dict(*, tree: treelib.Tree) -> dict: 151 | nodes = [] 152 | 153 | for node in tree.all_nodes(): 154 | parent = tree.parent(node.identifier) 155 | parent = parent if parent is None else parent.identifier 156 | x = { 157 | 'identifier': node.identifier, 158 | 'tag': node.tag, 159 | 'data': node.data, 160 | 'parent': parent, 161 | } 162 | nodes.append(x) 163 | 164 | return {'nodes': nodes} 165 | 166 | 167 | def dict_to_tree(*, data: dict) -> treelib.Tree: 168 | root = { 169 | 'identifier': 0, 170 | 'tag': 'Root', 171 | 'parent': None, 172 | 'data': None, 173 | } 174 | 175 | if ('nodes' not in data or 176 | data['nodes'][0] != root): 177 | raise UtilError('Malformed tree dict') 178 | 179 | tree = treelib.Tree() 180 | 181 | for x in data['nodes']: 182 | tree.create_node( 183 | identifier=x['identifier'], 184 | tag=x['tag'], 185 | parent=x['parent'], 186 | data=x['data']) 187 | 188 | return tree 189 | 190 | 191 | def stats_from_tree(*, tree: treelib.Tree) -> dict[str, Union[float, int]]: 192 | all_nodes = tree.all_nodes() 193 | root_nodes = [node for node in all_nodes 194 | if tree.parent(node.identifier) is not None and 195 | tree.parent(node.identifier).identifier == 0] 196 | roots_with_no_children = [node for node in root_nodes 197 | if not len(tree.children(node.identifier))] 198 | children_counts = [len(tree.children(node.identifier)) 199 | for node in all_nodes] 200 | node_depths = [tree.depth(node.identifier) for node in all_nodes] 201 | level_counts = defaultdict(int) 202 | for x in node_depths: 203 | level_counts[x] += 1 204 | 205 | stats = { 206 | 'total_nodes': len(all_nodes), # includes root node 207 | 'total_roots': len(root_nodes), 208 | 'roots_with_no_children': len(roots_with_no_children), 209 | 'total_intermediates': len(all_nodes) - 1 - len(root_nodes), 210 | 'maximum_depth': max(node_depths, default=0), 211 | 'average_depth': 212 | sum(node_depths) / len(node_depths) if node_depths else 0, 213 | 'maximum_breadth': max(level_counts.values(), default=0), 214 | 'maximum_children': max(children_counts, default=0), 215 | 'average_children': 216 | sum(children_counts) / len(children_counts) if children_counts else 0, 217 | 'nodes_with_10+_children': sum(1 for c in children_counts if c >= 10), 218 | 'nodes_with_50+_children': sum(1 for c in children_counts if c >= 50), 219 | 'nodes_with_100+_children': 220 | sum(1 for c in children_counts if c >= 100), 221 | 'leaf_nodes': sum(1 for c in children_counts if c == 0), 222 | } 223 | 224 | return stats 225 | -------------------------------------------------------------------------------- /bin/chainring.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2024 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import argparse 20 | import asyncio 21 | from collections import defaultdict 22 | from datetime import datetime, timezone 23 | from html import escape 24 | import json 25 | import os 26 | import pprint 27 | import sys 28 | 29 | libpath = os.path.dirname(os.path.abspath(__file__)) 30 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 31 | 32 | from pan_chainguard import title, __version__ 33 | from pan_chainguard.ccadb import root_status_bits_flag, root_status_bits 34 | import pan_chainguard.util 35 | 36 | args = None 37 | 38 | 39 | def root_status(node): 40 | if not args.verbose or node.tag == 'Root': 41 | return '' 42 | 43 | data = node.data 44 | if data['Certificate Record Type'] == 'Root Certificate': 45 | bits = root_status_bits_flag(data) 46 | status = root_status_bits(bits, compact=True) 47 | return status 48 | else: 49 | return '' 50 | 51 | 52 | def tree_sort(node): 53 | return str(node.data['Certificate Name']) 54 | 55 | 56 | def format_text(tree): 57 | txt = tree.show(stdout=None, key=tree_sort) 58 | print(txt, end='') 59 | 60 | 61 | def format_stats(tree): 62 | stats = pan_chainguard.util.stats_from_tree(tree=tree) 63 | 64 | for k, v in stats.items(): 65 | name = k.replace('_', ' ').title() 66 | value = '%.4f' % v if isinstance(v, float) else '%d' % v 67 | print('%s: %s' % (name, value)) 68 | 69 | 70 | def format_rst(tree): 71 | def tree_to_rst(tree, node_id=None, level=-1): 72 | if node_id is None: 73 | node_id = tree.root 74 | 75 | node = tree[node_id] 76 | lines = [] 77 | 78 | sha256 = str(node_id) 79 | # skips root node 80 | if len(sha256) == 64: 81 | # XXX uncertain if we can monospace the anchor 82 | lines.append( 83 | f'{" " * level}* ' 84 | f'`{sha256} `_ ' 85 | f'{node.tag[64:]}' 86 | ) 87 | 88 | for i, child in enumerate(sorted(tree.children(node_id), 89 | key=tree_sort)): 90 | if i == 0: 91 | lines.append('') 92 | lines.extend(tree_to_rst(tree, child.identifier, level + 1)) 93 | 94 | return lines 95 | 96 | lines = tree_to_rst(tree) 97 | rst = '' 98 | if args.title: 99 | rst = f'{args.title}\n{"=" * len(args.title)}\n' 100 | rst += '\n'.join(lines) 101 | print(rst) 102 | 103 | 104 | def format_html(tree): 105 | def tree_to_html(tree, node_id=None): 106 | if node_id is None: 107 | node_id = tree.root 108 | 109 | node = tree[node_id] 110 | children = tree.children(node_id) 111 | root_vendors = root_status(node) 112 | if root_vendors: 113 | root_vendors = f' vendors:{root_vendors} ' 114 | html = '' 115 | 116 | sha256 = str(node_id) 117 | # skips root node 118 | if len(sha256) == 64: 119 | if node.data['Certificate Record Type'] == 'Root Certificate': 120 | try: 121 | tree_to_html.roots += 1 122 | except AttributeError: 123 | tree_to_html.roots = 1 124 | else: 125 | try: 126 | tree_to_html.intermediates += 1 127 | except AttributeError: 128 | tree_to_html.intermediates = 1 129 | 130 | html += (f'
  • ' 131 | f'{sha256}' 132 | f'{root_vendors}' 133 | f'{escape(node.tag[64:])}
    • \n') 134 | 135 | if children: 136 | html += '
      \n' 137 | for child in sorted(children, key=tree_sort): 138 | html += tree_to_html(tree, child.identifier) 139 | html += '
    \n' 140 | 141 | return html 142 | 143 | tree_html = tree_to_html(tree) 144 | 145 | html = '' 146 | if args.title: 147 | html += f'

    {escape(args.title)}

    \n' 148 | 149 | if args.verbose: 150 | html += f'''

    Certificate Totals

    151 |
    152 | Root Certificates: {tree_to_html.roots}
    153 | Intermediate Certificates: {tree_to_html.intermediates}
    154 |
    155 | ''' 156 | if args.verbose: 157 | html += '

    Certificate Tree

    \n' 158 | html += tree_html 159 | if args.verbose: 160 | stats = pan_chainguard.util.stats_from_tree(tree=tree) 161 | stats_ = '' 162 | for k, v in stats.items(): 163 | name = k.replace('_', ' ').title() 164 | value = '%.4f' % v if isinstance(v, float) else '%d' % v 165 | stats_ += '%s: %s
    \n' % (name, value) 166 | html += f'''

    Certificate Tree Statistics

    167 |
    168 | {stats_}
    169 | ''' 170 | 171 | now_utc = datetime.now(timezone.utc) 172 | date_str = now_utc.strftime('%Y-%m-%d %H:%M:%S UTC') 173 | html += f'''
    174 |

    Generated: {date_str}

    175 |
    176 | ''' 177 | 178 | print(html, end='') 179 | 180 | 181 | def format_json(tree): 182 | data = pan_chainguard.util.tree_to_dict(tree=tree) 183 | json_data = json.dumps(data, indent=4) 184 | print(json_data) 185 | 186 | 187 | formats = { 188 | 'txt': format_text, 189 | 'rst': format_rst, 190 | 'html': format_html, 191 | 'json': format_json, 192 | 'stats': format_stats, 193 | } 194 | 195 | 196 | def main(): 197 | global args 198 | args = parse_args() 199 | 200 | ret = asyncio.run(main_loop()) 201 | 202 | sys.exit(ret) 203 | 204 | 205 | async def main_loop(): 206 | tree = read_tree() 207 | 208 | if args.test_collisions: 209 | if not test_collisions(tree): 210 | return 1 211 | 212 | if args.format: 213 | for format in args.format: 214 | formats[format](tree) 215 | 216 | if args.fingerprint: 217 | for x in args.fingerprint: 218 | lookup(tree, x) 219 | 220 | return 0 221 | 222 | 223 | def read_tree(): 224 | try: 225 | with open(args.tree, 'r') as f: 226 | json_data = f.read() 227 | data = json.loads(json_data) 228 | except (OSError, ValueError) as e: 229 | print('%s: %s' % (args.tree, e), file=sys.stderr) 230 | sys.exit(1) 231 | 232 | try: 233 | tree = pan_chainguard.util.dict_to_tree(data=data) 234 | except pan_chainguard.util.UtilError as e: 235 | print('%s: %s' % (args.tree, e), file=sys.stderr) 236 | sys.exit(1) 237 | 238 | return tree 239 | 240 | 241 | # The first 26 characters of the SHA-256 fingerprint (length 64) are 242 | # used for the PAN-OS certificate name; test name for collisions. 243 | def test_collisions(tree): 244 | data = pan_chainguard.util.tree_to_dict(tree=tree) 245 | 246 | names = defaultdict(list) 247 | for node in data['nodes']: 248 | ident = node.get('identifier') 249 | if ident: 250 | name = pan_chainguard.util.hash_to_name(sha256=ident) 251 | names[name].append(ident) 252 | 253 | # keep only names that map to >1 identifiers 254 | collisions = {name: ids for name, ids in names.items() if len(ids) > 1} 255 | 256 | if collisions: 257 | print(f'{len(collisions)} certificate name collisions', 258 | file=sys.stderr) 259 | for name in sorted(collisions): 260 | for ident in collisions[name]: 261 | print(f' {name} {ident}', file=sys.stderr) 262 | return False 263 | else: 264 | print('no certificate name collisions', file=sys.stderr) 265 | return True 266 | 267 | 268 | def lookup(tree, sha256): 269 | nodes = [] 270 | 271 | if len(sha256) == 64: 272 | node = tree.get_node(sha256) 273 | if node is not None: 274 | nodes.append(node) 275 | else: 276 | s = sha256 277 | if s.startswith(pan_chainguard.util.NAME_PREFIX): 278 | s = s[len(pan_chainguard.util.NAME_PREFIX):] 279 | s = s.upper() 280 | matches = tree.filter_nodes( 281 | lambda n: (isinstance(n.identifier, str) and 282 | s in n.identifier) 283 | ) 284 | nodes.extend(matches) 285 | 286 | if not nodes: 287 | print('Not found: %s' % s, file=sys.stderr) 288 | return 289 | 290 | for node in nodes: 291 | data = node.data 292 | filtered_data = {k: v for k, v in data.items() if v != ''} 293 | print(pprint.pformat(filtered_data)) 294 | 295 | 296 | def parse_args(): 297 | parser = argparse.ArgumentParser( 298 | usage='%(prog)s [options]', 299 | description='certificate tree analysis and reporting', 300 | formatter_class=argparse.RawTextHelpFormatter) 301 | parser.add_argument('--tree', 302 | required=True, 303 | metavar='PATH', 304 | help='JSON certificate tree path') 305 | parser.add_argument('-f', '--format', 306 | action='append', 307 | choices=formats.keys(), 308 | help='output format') 309 | parser.add_argument('-t', '--title', 310 | help='report title') 311 | parser.add_argument('--test-collisions', 312 | action='store_true', 313 | help='test for certificate name collisions') 314 | parser.add_argument('-F', '--fingerprint', 315 | action='append', 316 | metavar='SHA-256', 317 | help='''\ 318 | lookup CCADB data by certificate SHA-256 fingerprint 319 | (partial fingerprint allowed)''') 320 | parser.add_argument('--verbose', 321 | action='store_true', 322 | help='enable verbosity') 323 | parser.add_argument('--debug', 324 | type=int, 325 | choices=[0, 1, 2, 3], 326 | default=0, 327 | help='enable debug') 328 | x = '%s %s' % (title, __version__) 329 | parser.add_argument('--version', 330 | action='version', 331 | help='display version', 332 | version=x) 333 | args = parser.parse_args() 334 | 335 | if args.debug: 336 | print(args, file=sys.stderr) 337 | 338 | return args 339 | 340 | 341 | if __name__ == '__main__': 342 | main() 343 | -------------------------------------------------------------------------------- /bin/link.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2024 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import aiohttp 20 | import argparse 21 | import asyncio 22 | import os 23 | import sys 24 | import time 25 | 26 | libpath = os.path.dirname(os.path.abspath(__file__)) 27 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 28 | 29 | from pan_chainguard import title, __version__ 30 | from pan_chainguard.ccadb import * 31 | from pan_chainguard.crtsh import ArgsError, CrtShApi 32 | import pan_chainguard.mozilla 33 | import pan_chainguard.util 34 | 35 | MAX_TASKS = 3 # concurrent crt.sh API requests 36 | CRT_SH_TIMEOUT = 60 37 | 38 | args = None 39 | 40 | 41 | def main(): 42 | global args 43 | args = parse_args() 44 | 45 | ret = asyncio.run(main_loop()) 46 | 47 | sys.exit(ret) 48 | 49 | 50 | async def main_loop(): 51 | if not pan_chainguard.util.is_writable(args.certs_new): 52 | print('%s: not writable' % args.certs_new, file=sys.stderr) 53 | sys.exit(1) 54 | 55 | fingerprints = [] 56 | for x in args.fingerprints: 57 | try: 58 | fingerprints.extend( 59 | pan_chainguard.util.read_fingerprints(path=x)) 60 | except pan_chainguard.util.UtilError as e: 61 | print(str(e), file=sys.stderr) 62 | sys.exit(1) 63 | 64 | mozilla = [] 65 | if args.certs_mozilla: 66 | for x in args.certs_mozilla: 67 | try: 68 | mozilla.append( 69 | pan_chainguard.mozilla.MozillaCaCerts(path=x)) 70 | except pan_chainguard.mozilla.MozillaError as e: 71 | print(str(e), file=sys.stderr) 72 | sys.exit(1) 73 | 74 | certs_old = {} 75 | if args.certs_old: 76 | try: 77 | certs_old = pan_chainguard.util.read_cert_archive( 78 | path=args.certs_old) 79 | except pan_chainguard.util.UtilError as e: 80 | print(str(e), file=sys.stderr) 81 | sys.exit(1) 82 | 83 | errors, certs = await get_certs(fingerprints, mozilla, certs_old) 84 | 85 | try: 86 | pan_chainguard.util.write_cert_archive( 87 | path=args.certs_new, data=certs) 88 | except pan_chainguard.util.UtilError as e: 89 | print(str(e), file=sys.stderr) 90 | sys.exit(1) 91 | 92 | if args.verbose: 93 | print('Total certs-new: %d' % len(certs)) 94 | 95 | return 2 if errors else 0 96 | 97 | 98 | async def get_certs(fingerprints, mozilla, certs_old): 99 | certs = {} 100 | crtsh_download = {} 101 | total_certs_old = 0 102 | total_mozilla = { 103 | 'MozillaIntermediateCerts': 0, 104 | 'PublicAllIntermediateCerts': 0, 105 | } 106 | total_crtsh = 0 107 | 108 | for row in fingerprints: 109 | cert_type = row['type'] 110 | sha256 = row['sha256'] 111 | 112 | # may be duplicates in multiple fingerprint files 113 | if sha256 in certs: 114 | continue 115 | 116 | if sha256 in certs_old: 117 | total_certs_old += 1 118 | certs[sha256] = certs_old[sha256] 119 | continue 120 | 121 | if cert_type == 'intermediate': 122 | for mozilla_certs in mozilla: 123 | pem = mozilla_certs.get_cert_pem(sha256=sha256) 124 | if pem is not None: 125 | total_mozilla[mozilla_certs.name] += 1 126 | certs[sha256] = (cert_type, pem) 127 | break 128 | if sha256 in certs: 129 | continue 130 | 131 | crtsh_download[sha256] = cert_type 132 | 133 | errors, certificates = await get_cert_files(crtsh_download.keys()) 134 | for sha256 in certificates: 135 | total_crtsh += 1 136 | certs[sha256] = (crtsh_download[sha256], certificates[sha256]) 137 | 138 | if args.verbose: 139 | print('certs-old: %d' % total_certs_old) 140 | for x in total_mozilla: 141 | print('%s: %d' % (x, total_mozilla[x])) 142 | print('crt.sh: %d' % total_crtsh) 143 | 144 | return errors, certs 145 | 146 | 147 | async def download(api, sha256): 148 | tries = 0 149 | MAX_TRIES = 5 150 | RETRY_SLEEP = 5.0 151 | 152 | RETRY_STATUS = [ 153 | 429, # Too Many Requests 154 | 502, # Bad Gateway 155 | 503, # Service Unavailable 156 | 504, # Gateway Time-out 157 | ] 158 | 159 | while True: 160 | tries += 1 161 | try: 162 | if args.verbose and tries == 1: 163 | print('Download using crt.sh API %s' % sha256) 164 | 165 | resp = await api.download(id=sha256) 166 | 167 | if resp.status in RETRY_STATUS: 168 | x = '%d %s %s' % ( 169 | resp.status, resp.reason, sha256) 170 | if tries == MAX_TRIES: 171 | print('no retry after try %d %s' % (tries, x), 172 | file=sys.stderr) 173 | x = 'download failed ' + x 174 | break 175 | print('retry after try %d %s' % (tries, x), 176 | file=sys.stderr) 177 | await asyncio.sleep(RETRY_SLEEP) 178 | elif resp.status != 200: 179 | x = 'download failed %d %s %s' % ( 180 | resp.status, resp.reason, sha256) 181 | break 182 | else: 183 | filename, content = await api.content(resp=resp) 184 | 185 | start = '-----BEGIN CERTIFICATE-----' 186 | end = '-----END CERTIFICATE-----\n' 187 | if (content.startswith(start) and 188 | content.endswith(end)): 189 | return content, sha256 190 | else: 191 | # XXX ephemeral? 192 | x = 'content malformed %s %s' % ( 193 | sha256, content) 194 | if tries == MAX_TRIES: 195 | print('no retry after try %d %s' % (tries, x), 196 | file=sys.stderr) 197 | x = 'download failed ' + x 198 | break 199 | print('retry after try %d %s' % (tries, x), 200 | file=sys.stderr) 201 | await asyncio.sleep(RETRY_SLEEP) 202 | 203 | except (asyncio.TimeoutError, 204 | asyncio.CancelledError, # XXX 205 | aiohttp.ClientError) as e: 206 | msg = e if str(e) else type(e).__name__ 207 | x = 'CrtShApi: %s %s' % (msg, sha256) 208 | if tries == MAX_TRIES: 209 | print('no retry after try %d %s' % (tries, x), 210 | file=sys.stderr) 211 | x = 'download failed ' + x 212 | break 213 | print('retry after try %d %s' % (tries, x), 214 | file=sys.stderr) 215 | # no sleep 216 | 217 | except ArgsError as e: 218 | x = 'CrtShApi: %s' % e 219 | break 220 | 221 | return None, x 222 | 223 | 224 | async def get_cert_files(sha256): 225 | user_agent = '%s/%s' % (title, __version__) 226 | headers = {'user-agent': user_agent} 227 | 228 | try: 229 | api = CrtShApi(timeout=CRT_SH_TIMEOUT, headers=headers) 230 | except ArgsError as e: 231 | print('CrtShApi: %s' % e, file=sys.stderr) 232 | sys.exit(1) 233 | else: 234 | errors, certificates = await download_certs(api, sha256) 235 | finally: 236 | await api.session.close() 237 | 238 | return errors, certificates 239 | 240 | 241 | async def download_certs(api, sha256): 242 | certificates = {} 243 | tasks = [] 244 | start = time.time() 245 | total = 0 246 | errors = 0 247 | 248 | for x in sha256: 249 | total += 1 250 | tasks.append(download(api, x)) 251 | 252 | if len(tasks) == MAX_TASKS: 253 | errors += await run_tasks(tasks, certificates) 254 | tasks = [] 255 | 256 | if len(tasks): 257 | errors += await run_tasks(tasks, certificates) 258 | 259 | if args.debug: 260 | end = time.time() 261 | elapsed = end - start 262 | mins = elapsed // 60 263 | secs = elapsed % 60 264 | avg = elapsed / total if total > 0 else 0 265 | print('%d tasks, elapsed %.2f seconds, %dm%ds, avg %.2fs' % 266 | (total, elapsed, mins, secs, avg), 267 | file=sys.stderr) 268 | 269 | return errors, certificates 270 | 271 | 272 | async def run_tasks(tasks, certificates): 273 | INTERVAL_WAIT = 3.3 # random throttle sweet spot to avoid TimeoutError 274 | errors = 0 275 | 276 | if args.debug: 277 | print('running %d tasks' % len(tasks), 278 | file=sys.stderr) 279 | start = time.time() 280 | 281 | for coro in asyncio.as_completed(tasks): 282 | content, sha256 = await coro 283 | if content is None: 284 | # error 285 | print(sha256, file=sys.stderr) 286 | errors += 1 287 | else: 288 | certificates[sha256] = content 289 | 290 | if args.debug: 291 | end = time.time() 292 | elapsed = end - start 293 | rate = len(tasks) / elapsed 294 | rate2 = elapsed / len(tasks) 295 | print('%d tasks complete, elapsed %.2f seconds, ' 296 | '%.2f tasks/sec, %.2f secs/task' % 297 | (len(tasks), elapsed, rate, rate2), file=sys.stderr) 298 | print('sleeping %.2fs' % INTERVAL_WAIT, file=sys.stderr) 299 | await asyncio.sleep(INTERVAL_WAIT) 300 | 301 | return errors 302 | 303 | 304 | def parse_args(): 305 | parser = argparse.ArgumentParser( 306 | usage='%(prog)s [options]', 307 | description='get CA certificates') 308 | parser.add_argument('-f', '--fingerprints', 309 | required=True, 310 | action='append', 311 | metavar='PATH', 312 | help='CA fingerprints CSV path') 313 | parser.add_argument('-m', '--certs-mozilla', 314 | action='append', 315 | metavar='PATH', 316 | help='Mozilla certs with PEM CSV path') 317 | parser.add_argument('--certs-old', 318 | metavar='PATH', 319 | help='old certificate archive path') 320 | parser.add_argument('--certs-new', 321 | required=True, 322 | metavar='PATH', 323 | help='new certificate archive path') 324 | parser.add_argument('--verbose', 325 | action='store_true', 326 | help='enable verbosity') 327 | parser.add_argument('--debug', 328 | type=int, 329 | choices=[0, 1, 2, 3], 330 | default=0, 331 | help='enable debug') 332 | x = '%s %s' % (title, __version__) 333 | parser.add_argument('--version', 334 | action='version', 335 | help='display version', 336 | version=x) 337 | args = parser.parse_args() 338 | 339 | if args.debug: 340 | print(args, file=sys.stderr) 341 | 342 | return args 343 | 344 | 345 | if __name__ == '__main__': 346 | main() 347 | -------------------------------------------------------------------------------- /HISTORY.rst: -------------------------------------------------------------------------------- 1 | Release History 2 | =============== 3 | 4 | 0.12.0 (2025-11-12) 5 | ------------------- 6 | 7 | - bin/guard.py: Skip unsupported certificates for FIPS-CC mode. #8 8 | 9 | - Use V4 All Certificate Information CSV 10 | (AllCertificateRecordsCSVFormatv4). 11 | 12 | - bin/chainring.py: Allow partial fingerprint for SHA-256 lookup. 13 | 14 | - README.rst: Add pan-chainguard-content. 15 | 16 | 0.11.0 (2025-10-07) 17 | ------------------- 18 | 19 | - etc/derailer.py: Use type=pathlib.Path for argparse. 20 | 21 | - bin/guard.py: Fix f-string quoting. Don't use PEP 701 syntax. 22 | 23 | - bin/chainring.py: Use sha256 for crt.sh query string parameter. No 24 | functional change. 25 | 26 | - bin/chainring.py: Sort tree output by subject. 27 | 28 | - bin/guard.py: Print pan_chainguard.util.stats_from_tree result when 29 | verbose for show-tree. 30 | 31 | - bin/guard.py: Detect same subject as ancestor condition in PAN-OS 32 | derived certificate tree. 33 | 34 | - bin/chain.py: Implement workaround for #7. 35 | 36 | - bin/chainring.py: Optimise and more detail for test collisions. 37 | 38 | - etc/derailer.py: Add detail on duplicates in CCADB. 39 | 40 | - bin/chainring.py: Use
    for certificate totals and 41 | statistics. 42 | 43 | - bin/chainring.py: Fix to use ValueError for json.loads(). 44 | 45 | - bin/sprocket.py: Shorten assignment for readability. 46 | 47 | - bin/guard.py: Catch ValueError for 'template set for firewall' 48 | error. 49 | 50 | - etc/derailer.py: Rework error message construction. 51 | 52 | - etc/derailer.py: Add return annotations to vendor fetchers. 53 | 54 | - etc/derailer.py: Add download timeout. 55 | 56 | - etc/derailer.py: Handle None return from vendor fetchers. 57 | 58 | - etc/derailer.py: Fix to use path argument. 59 | 60 | - etc/derailer.py: Catch cryptography warnings so we can display the 61 | problem certificates. 62 | 63 | - bin/guard.py: Recognise M-Series appliance models as Panorama. #6 64 | 65 | - etc/derailer.py: Print summary line for validity issues. 66 | 67 | - Documentation fixes and improvements. 68 | 69 | 0.10.0 (2025-08-29) 70 | ------------------- 71 | 72 | - etc/derailer.py: New utility program to perform vendor root CA 73 | program analysis. 74 | 75 | Currently determines vendor root store certificate counts from 76 | multiple sources, and can be used to verify sprocket.py stats 77 | counts. 78 | 79 | - Support CCADB All Included Root Certificate Trust Bits CSV file for 80 | root certificate inclusion. When utilised the equivalent of the 81 | SERVER_AUTHENTICATION trust bit must be set for the vendor. The 82 | bits are per-vendor, and not the same as the All Certificates 83 | Information "Trust Bits for Root Cert" field which is a union of all 84 | vendors. 85 | 86 | To use download: 87 | https://ccadb.my.salesforce-sites.com/ccadb/AllIncludedRootCertsCSV 88 | and specify the path using the sprocket.py --trust-settings option. 89 | 90 | - Use V3 All Certificate Information CSV 91 | (AllCertificateRecordsCSVFormatv3). 92 | 93 | - chain.py: For duplicate certificate fingerprints in CCADB, retain a 94 | root certificate, or intermediate when no root. 95 | 96 | - guard.py: Display count of disabled default trusted CAs for --show. 97 | 98 | - chain.py: Don't allow intermediate certificates with null 'Derived 99 | Trust Bits'. 100 | 101 | This occurs when the intersection of an intermediate's Extended Key 102 | Usage (EKU) values and its root certificate trust bits are disjoint 103 | (empty). 104 | 105 | - chainring.py: Add generated time to html format when verbose. 106 | 107 | - Documentation fixes and improvements. 108 | 109 | 0.9.0 (2025-07-10) 110 | ------------------ 111 | 112 | - chainring.py: Enhance verbose html format. 113 | 114 | Add section headings, move certificate totals to the top, and add 115 | certificate tree statistics to the bottom. 116 | 117 | - chainring.py: Move certificate tree statistics generation to 118 | pan_chainguard.util.stats_from_tree(). 119 | 120 | - chain.py: Don't add node to waiting nodes when parent is invalid. 121 | 122 | No functional change, reduces size of "Warning: nodes with no parent" 123 | debug log. 124 | 125 | - Support Mozilla OneCRL for intermediate certificate exclusion. 126 | 127 | To use download: 128 | https://ccadb.my.salesforce-sites.com/mozilla/IntermediateCertsInOneCRLReportCSV 129 | and specify the path using the chain.py --onecrl option. 130 | 131 | - sprocket.py: Fix bug in trust_bits usage for root certificates. 132 | 133 | This was using the AllCertificateRecordsCSVFormatv2 "Derived Trust 134 | Bits" field which only applies to intermediate certificates. The 135 | "Trust Bits for Root Cert" field was recently added which provides 136 | trust bits for root certificates, and that is now utilised. 137 | 138 | - ccadb.py: Add support for AllCertificateRecordsCSVFormatv2 "Trust 139 | Bits for Root Cert". 140 | 141 | - chainring.py: Add format stats with preliminary data from the 142 | certificate tree. 143 | 144 | - chainring.py: For html display totals when verbose. 145 | 146 | - chainring.py: Only use bold tag when there are vendors. 147 | 148 | - chainring.py: Add option to lookup CCADB data by certificate SHA-256 149 | fingerprint. 150 | 151 | - util.py: Allow root and intermediate directory members in the 152 | certificates tar file. The Python tarfile module doesn't add these 153 | but command-line tar command does and we may want to post-process 154 | the Python tar file. 155 | 156 | 0.8.0 (2025-04-13) 157 | ------------------ 158 | 159 | - guard.py: Add --show-tree, which outputs a tree created using the 160 | certificate subject-hash and issuer-hash configuration values. 161 | 162 | - fling.py is deprecated. It is recommended to use 163 | pan-chainguard-content or to create an up-to-date custom root store 164 | using sprocket.py. 165 | 166 | - Add use cases to admin guide. 167 | 168 | - Deprecate "google" for "chrome" in sprocket.py policy source vendor. 169 | 170 | - chainring.py: For html format when verbose, add vendor sources for 171 | root certificates. 172 | 173 | - chainring.py: Don't print Root node for format html, rst. 174 | 175 | - guard.py: Exclude cert with Signature Algorithm: rsassaPss. 176 | 177 | - Introduce the pan-chainguard-content repository which provides 178 | pre-generated, up-to-date content which can be used to simplify the 179 | deployment of pan-chainguard. 180 | 181 | - chainring.py: For RST document, create reference for certificate 182 | fingerprint to crt.sh. 183 | 184 | - Documentation fixes and improvements. 185 | 186 | 0.7.0 (2025-01-02) 187 | ------------------ 188 | 189 | - Major updates to guard.py: 190 | 191 | - Allow incremental certificate updates using --update (replaces 192 | --add) 193 | - Allow update of only root certificates 194 | - Add --dry-run to show what actions would be performed without 195 | updating PAN-OS 196 | - Add --show to show managed config 197 | - Add --update-trusted to fix out of sync trusted root CA certificate 198 | settings 199 | 200 | - chainring.py: For HTML document, create hyperlink for certificate 201 | fingerprint to crt.sh. 202 | 203 | - chain.py: 204 | 205 | For a root certificate the "Parent Certificate Name" is set to the 206 | "CA Owner" field; change node tag to use CA-Owner vs Issuer. Also 207 | quote Subject, Issuer, CA-Owner. 208 | 209 | - chainring.py: Add --test-collisions to test for PAN-OS certificate 210 | name collisions using the JSON certificate tree as input. 211 | 212 | 0.6.0 (2024-12-15) 213 | ------------------ 214 | 215 | - Split chain.py into separate programs for: 216 | 217 | - Intermediate certificate determination (chain.py) 218 | - Certificate download (link.py) 219 | 220 | - Re-implement chain.py to use a tree (using treelib package). 221 | 222 | - Add chainring.py to generate documents from JSON certificate tree. 223 | 224 | - Get CA certificates program (link.py) can use alternate certificate 225 | sources before downloading from crt.sh. 226 | 227 | - Allow update of root store only, without adding intermediate 228 | certificates. 229 | 230 | - Certificate name on PAN-OS has been changed to 231 | 'LINK-[0-9A-F]{26,26}' (sequence number replaced by 'LINK'). 232 | 233 | - Add tests for CCADB module. 234 | 235 | - guard.py: 236 | 237 | PAN-257401 is an issue where a specific certificate, when imported 238 | as a device certificate and set as a trusted root CA, results in a 239 | commit error due to a bad signature. Implement temporary workaround 240 | to not import this certificate on --add. 241 | 242 | - sprocket.py: Fix GitHub Issue #3. 243 | 244 | 0.5.0 (2024-10-07) 245 | ------------------ 246 | 247 | - chain.py: Fix bug where only a single child certificate chain for a 248 | root was used. 249 | 250 | - Add To Do List. 251 | 252 | - guard.py: When API import results in expired certificate error, skip 253 | that certificate. Allows use of an older certificate archive. 254 | 255 | - chain.py: Remove unneeded else. 256 | 257 | - chain.py: Raise debug level to 3 for revoked and expired logging. 258 | 259 | - chain.py: Fix incorrect indent for saving 'Intermediate with no 260 | parent' certificate. 261 | 262 | - Add features to allow a custom root store to replace the PAN-OS root 263 | store. 264 | 265 | 0.4.0 (2024-07-12) 266 | ------------------ 267 | 268 | - ccadb.py: Add functions for 'Derived Trust Bits' to ccadb module. 269 | 270 | - chain.py: Set user-agent header to pan-chainguard/version for crt.sh 271 | API. 272 | 273 | - chain.py, guard.py: Generalise some message strings previously 274 | specifying PAN-OS to prepare for using other root stores as input. 275 | 276 | - chain.py, ccadb.py: Add pan_chainguard.ccadb module. 277 | 278 | - Documentation improvements and fixes. 279 | 280 | 0.3.0 (2024-06-12) 281 | ------------------ 282 | 283 | - guard.py: Cache certificate names so we can use a single API request 284 | to enable them as trusted root CAs. 285 | 286 | - guard.py: When device is panorama and template specified, perform 287 | partial commit with template scope. 288 | 289 | - chain.py: Also retry download on 503 Service Unavailable. 290 | 291 | - guard.py: Fix partial commit using specific admin. In the XML cmd 292 | document, needs to be within container. 293 | 294 | - guard.py: Simplify Xpath() class. 295 | 296 | - admin-guide.rst: 297 | 298 | chainguard-api admin profile does require type=op because we use 299 | synchronous commit in pan.xapi which uses 'show jobs id id-num' to 300 | check job status. 301 | 302 | - guard.py: Fix use of panorama from removal of global. 303 | 304 | 0.2.0 (2024-05-30) 305 | ------------------ 306 | 307 | - guard.py: Add support for import to Panorama Template shared device 308 | certificates. 309 | 310 | - chain.py: 311 | 312 | Change 'Server Authentication' not in 'Derived Trust Bits' check to 313 | a warning. Safer to leave these valid until we can research this 314 | more. 315 | 316 | - Documentation improvements and fixes: 317 | 318 | + type=op not needed in admin role profile. 319 | 320 | + Add admin role profile for Panorama. 321 | 322 | + Document intermediate certificate name pattern. 323 | 324 | + There is a single *All Certificate Information (root and 325 | intermediate) in CCADB (CSV)* data file now. 326 | 327 | 0.1.0 (2024-04-09) 328 | ------------------ 329 | 330 | - fling.py, chain.py, guard.py, admin-guide.rst: 331 | 332 | Add --debug argument and use args.debug for all debugging related 333 | output, and be consistent in use of args.verbose for verbose output 334 | (e.g., progress messages). 335 | 336 | - chain.py: 337 | 338 | + Log when a CA certificate is not in any of Apple, Google Chrome, 339 | Microsoft, Mozilla root stores. 340 | + Log when 0 intermediates found for a CA certificate. 341 | 342 | - chain.py: 343 | 344 | Add message when all certificate chains were downloaded 345 | successfully. 346 | 347 | - chain.py: 348 | 349 | + Display PAN-OS certificates not in CCADB and consider them 350 | invalid, because we will not find intermediate certificate chains 351 | for these. 352 | + Output invalid PAN-OS certificate messages to stderr. 353 | + Display total invalid PAN-OS certificates found. 354 | 355 | - chain.py: Fix invalid path in error. 356 | 357 | - chain.py: Print download error to stderr. 358 | 359 | - chain.py: Also retry on status code 502, 504. 360 | 361 | - chain.py: Improve some messages. 362 | 363 | - chain.py: 364 | 365 | Since we don't use xapi.export_result 'file', check 'content' 366 | instead. There is currently an issue in pan.xapi export() where 367 | filename can be None. Fixes a bug where certificate names with 368 | parentheses were not saved to the archive. 369 | 370 | - chain.py: exit with status 2 when there are download failures. 371 | 372 | - chain.py: 373 | 374 | + Fix missing value for format string. 375 | + Change message to Error. 376 | 377 | - Documentation improvements and fixes. 378 | 379 | 0.0.0 (2024-03-15) 380 | ------------------ 381 | 382 | - Initial release. 383 | -------------------------------------------------------------------------------- /pan_chainguard/ccadb.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) 2024 Palo Alto Networks, Inc. 3 | # 4 | # Permission to use, copy, modify, and distribute this software for any 5 | # purpose with or without fee is hereby granted, provided that the above 6 | # copyright notice and this permission notice appear in all copies. 7 | # 8 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 9 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 10 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 11 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 12 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 13 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 14 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 15 | # 16 | 17 | # https://www.ccadb.org/resources 18 | # 19 | # All Certificate Information (root and intermediate) in CCADB (CSV) 20 | # utility functions. 21 | # https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv4 22 | # 23 | # All Included Root Certificate Trust Bit Settings: 24 | # https://ccadb.my.salesforce-sites.com/ccadb/AllIncludedRootCertsCSV 25 | 26 | import csv 27 | from datetime import datetime, timezone 28 | from enum import Flag, auto 29 | import sys 30 | from typing import Tuple, Union, Optional 31 | 32 | 33 | class CcadbError(Exception): 34 | pass 35 | 36 | 37 | _FMT = '%Y.%m.%d %z' 38 | 39 | __all__ = [ 40 | 'valid_from', 'valid_to', 'valid_from_to', 'revoked', 41 | 'TrustBits', 'trust_bits_flag', 42 | 'derived_trust_bits_list', 'derived_trust_bits', 43 | 'root_trust_bits_list', 'root_trust_bits', 44 | 'RootStatusBits', 'root_status_bits_flag', 'root_status_bits', 45 | ] 46 | 47 | 48 | class TrustBits(Flag): 49 | NONE = 0 50 | OTHER = auto() 51 | CLIENT_AUTHENTICATION = auto() 52 | CODE_SIGNING = auto() 53 | DOCUMENT_SIGNING = auto() 54 | OCSP_SIGNING = auto() 55 | SECURE_EMAIL = auto() 56 | SERVER_AUTHENTICATION = auto() 57 | TIME_STAMPING = auto() 58 | 59 | 60 | TrustBitsMap = { 61 | 'Client Authentication': TrustBits.CLIENT_AUTHENTICATION, 62 | 'Code Signing': TrustBits.CODE_SIGNING, 63 | 'Document Signing': TrustBits.DOCUMENT_SIGNING, 64 | 'OCSP Signing': TrustBits.OCSP_SIGNING, 65 | 'Secure Email': TrustBits.SECURE_EMAIL, 66 | 'Server Authentication': TrustBits.SERVER_AUTHENTICATION, 67 | 'Time Stamping': TrustBits.TIME_STAMPING, 68 | } 69 | 70 | TrustBitsMap2 = {v.name: v for v in TrustBitsMap.values()} 71 | 72 | 73 | class RootStatusBits(Flag): 74 | NONE = 0 75 | MOZILLA = auto() 76 | CHROME = auto() 77 | APPLE = auto() 78 | MICROSOFT = auto() 79 | 80 | 81 | RootStatusBitsMap = { 82 | 'Mozilla Status': RootStatusBits.MOZILLA, 83 | 'Apple Status': RootStatusBits.APPLE, 84 | 'Chrome Status': RootStatusBits.CHROME, 85 | 'Microsoft Status': RootStatusBits.MICROSOFT, 86 | } 87 | 88 | 89 | def _now(): 90 | now = datetime.now(timezone.utc) 91 | 92 | return now 93 | 94 | 95 | # CCADB Valid From/To fields are the date only in YYYY.MM.DD format. 96 | # When converted to a date with time, the time used is midnight; for 97 | # example: 98 | # 2024.10.27 -> 2024-10-27 00:00:00+00:00 99 | 100 | def valid_from(row: dict[str, str]) -> Tuple[bool, Union[str, None]]: 101 | k = 'Valid From (GMT)' 102 | valid_from = datetime.strptime(row[k] + ' +0000', _FMT) 103 | if _now() < valid_from: 104 | x = 'Not yet valid (valid from %s)' % row[k] 105 | return False, x 106 | 107 | return True, None 108 | 109 | 110 | def valid_to(row: dict[str, str]) -> Tuple[bool, Union[str, None]]: 111 | k = 'Valid To (GMT)' 112 | valid_to = datetime.strptime(row[k] + ' +0000', _FMT) 113 | if valid_to < _now(): 114 | x = 'Expired (valid to %s)' % row[k] 115 | return False, x 116 | 117 | return True, None 118 | 119 | 120 | def valid_from_to(row: dict[str, str]) -> Tuple[bool, Union[str, None]]: 121 | return valid_from(row) and valid_to(row) 122 | 123 | 124 | def revoked(row: dict[str, str]) -> Tuple[bool, Union[str, None]]: 125 | k = 'Revocation Status' 126 | if row[k] not in ['', 'Not Revoked']: 127 | return True, row[k] 128 | 129 | return False, None 130 | 131 | 132 | def _trust_bits_list(key: str, row: dict[str, str]) -> list[str]: 133 | x = row[key] 134 | if not x: 135 | return [] 136 | 137 | return x.split(';') 138 | 139 | 140 | def derived_trust_bits_list(row: dict[str, str]) -> list[str]: 141 | type_ = 'Intermediate Certificate' 142 | x = row['Certificate Record Type'] 143 | if x != type_: 144 | raise ValueError('certificate type "%s" not "%s"' % (x, type_)) 145 | 146 | key = 'Derived Trust Bits' 147 | return _trust_bits_list(key, row) 148 | 149 | 150 | def root_trust_bits_list(row: dict[str, str]) -> list[str]: 151 | type_ = 'Root Certificate' 152 | x = row['Certificate Record Type'] 153 | if x != type_: 154 | raise ValueError('certificate type "%s" not "%s"' % (x, type_)) 155 | 156 | key = 'Trust Bits for Root Cert' 157 | return _trust_bits_list(key, row) 158 | 159 | 160 | def trust_bits_flag(values: list[str]) -> TrustBits: 161 | bits = TrustBits.NONE 162 | for x in values: 163 | if x in TrustBitsMap: 164 | bits = bits | TrustBitsMap[x] 165 | else: 166 | bits = bits | TrustBits.OTHER 167 | 168 | return bits 169 | 170 | 171 | def derived_trust_bits(row: dict[str, str]) -> TrustBits: 172 | x = derived_trust_bits_list(row) 173 | return trust_bits_flag(x) 174 | 175 | 176 | def root_trust_bits(row: dict[str, str]) -> TrustBits: 177 | x = root_trust_bits_list(row) 178 | return trust_bits_flag(x) 179 | 180 | 181 | def root_status_bits_flag(row: dict[str, str]) -> RootStatusBits: 182 | type_ = 'Root Certificate' 183 | x = row['Certificate Record Type'] 184 | if x != type_: 185 | raise ValueError('certificate type "%s" not "%s"' % (x, type_)) 186 | 187 | bits = RootStatusBits.NONE 188 | for x in RootStatusBitsMap: 189 | if x in row and row[x] == 'Included': 190 | bits = bits | RootStatusBitsMap[x] 191 | 192 | return bits 193 | 194 | 195 | map_ = { 196 | ('mozilla', 'Mz'): RootStatusBits.MOZILLA, 197 | ('apple', 'Ap'): RootStatusBits.APPLE, 198 | ('chrome', 'Ch'): RootStatusBits.CHROME, 199 | ('microsoft', 'Ms'): RootStatusBits.MICROSOFT, 200 | } 201 | 202 | 203 | def root_status_bits(bits: RootStatusBits, 204 | compact: bool = False) -> Union[str, list]: 205 | r = [] 206 | 207 | for k, v in map_.items(): 208 | if v in bits: 209 | r.append(k[1] if compact else k[0]) 210 | 211 | return ''.join(r) if compact else r 212 | 213 | 214 | class CcadbRootTrustSettings: 215 | def __init__(self, *, 216 | path: str, 217 | debug: bool = False): 218 | self._certs = {} 219 | self._debug = debug 220 | EXPECTED = { # subset 221 | 'Google Chrome Status', 222 | 'Microsoft EKUs', 223 | 'SHA-256 Fingerprint', 224 | } 225 | 226 | try: 227 | with open(path, 'r', newline='') as csvfile: 228 | reader = csv.DictReader(csvfile, 229 | dialect='unix') 230 | fieldnames = set(reader.fieldnames or []) 231 | missing = EXPECTED - fieldnames 232 | if missing: 233 | raise CcadbError('Invalid CSV') 234 | 235 | for row in reader: 236 | sha256 = row['SHA-256 Fingerprint'] 237 | if self._debug and sha256 in self.certs: 238 | print('Duplicate in CCADB Root Trust Settings %s' % ( 239 | sha256), file=sys.stderr) 240 | self._certs[sha256] = row 241 | 242 | except OSError as e: 243 | raise CcadbError(str(e)) 244 | 245 | @property 246 | def certs(self): 247 | return self._certs 248 | 249 | def get(self, *, sha256: str) -> Optional[dict[str, str]]: 250 | if sha256 not in self._certs: 251 | return 252 | 253 | return self._certs[sha256] 254 | 255 | def root_status_bits_flag(self, *, sha256: str) -> Optional[ 256 | RootStatusBits]: 257 | if sha256 not in self._certs: 258 | return 259 | row = self._certs[sha256] 260 | 261 | bits_map = RootStatusBitsMap.copy() 262 | del bits_map['Chrome Status'] 263 | bits_map['Google Chrome Status'] = RootStatusBits.CHROME 264 | 265 | bits = RootStatusBits.NONE 266 | for x in bits_map: 267 | if x in row and row[x] == 'Included': 268 | bits = bits | bits_map[x] 269 | 270 | return bits 271 | 272 | # There is no 'Google Chrome Trust Bits'. 273 | # The Chrome Root Store only trusts Server Authentication. 274 | def chrome_trust_bits_list(self, *, sha256: str) -> Optional[list]: 275 | if sha256 not in self._certs: 276 | return 277 | 278 | bits = self.root_status_bits_flag(sha256=sha256) 279 | if RootStatusBits.CHROME not in bits: 280 | return [] 281 | 282 | return ['Server Authentication'] 283 | 284 | def chrome_trust_bits(self, *, sha256: str) -> Optional[TrustBits]: 285 | values = self.chrome_trust_bits_list(sha256=sha256) 286 | if values is None: 287 | return 288 | 289 | if not values: 290 | return TrustBits.NONE 291 | 292 | return TrustBits.SERVER_AUTHENTICATION 293 | 294 | def mozilla_trust_bits_list(self, *, sha256: str) -> Optional[list]: 295 | if sha256 not in self._certs: 296 | return 297 | 298 | return _trust_bits_list('Mozilla Trust Bits', self._certs[sha256]) 299 | 300 | def mozilla_trust_bits(self, *, sha256: str) -> Optional[TrustBits]: 301 | values = self.mozilla_trust_bits_list(sha256=sha256) 302 | if values is None: 303 | return 304 | 305 | if self._debug and not values: 306 | print('Mozilla Trust Bits null %s' % sha256, file=sys.stderr) 307 | 308 | if values == ['All Trust Bits Turned Off']: 309 | return TrustBits.NONE 310 | 311 | MAP = { 312 | 'Websites': TrustBits.SERVER_AUTHENTICATION, 313 | 'Email': TrustBits.SECURE_EMAIL, 314 | } 315 | 316 | bits = TrustBits.NONE 317 | for x in values: 318 | if x in MAP: 319 | bits = bits | MAP[x] 320 | else: 321 | bits = bits | TrustBits.OTHER 322 | 323 | if self._debug and TrustBits.OTHER in bits: 324 | print('Mozilla Trust Bits other %s %s' % ( 325 | values, sha256), file=sys.stderr) 326 | 327 | return bits 328 | 329 | def apple_trust_bits_list(self, *, sha256: str) -> Optional[list]: 330 | if sha256 not in self._certs: 331 | return 332 | 333 | return _trust_bits_list('Apple Trust Bits', self._certs[sha256]) 334 | 335 | def apple_trust_bits(self, *, sha256: str) -> Optional[TrustBits]: 336 | values = self.apple_trust_bits_list(sha256=sha256) 337 | if values is None: 338 | return 339 | 340 | if self._debug and not values: 341 | print('Apple Trust Bits null %s' % sha256, file=sys.stderr) 342 | 343 | MAP = { 344 | 'serverAuth': TrustBits.SERVER_AUTHENTICATION, 345 | 'clientAuth': TrustBits.CLIENT_AUTHENTICATION, 346 | 'emailProtection': TrustBits.SECURE_EMAIL, 347 | 'timeStamping': TrustBits.TIME_STAMPING, 348 | 'codeSigning': TrustBits.CODE_SIGNING, 349 | # 'BrandIndicatorforMessageIdentification': TrustBits.OTHER, 350 | } 351 | 352 | bits = TrustBits.NONE 353 | for x in values: 354 | if x in MAP: 355 | bits = bits | MAP[x] 356 | else: 357 | bits = bits | TrustBits.OTHER 358 | 359 | if self._debug and TrustBits.OTHER in bits: 360 | print('Apple Trust Bits other %s %s' % ( 361 | values, sha256), file=sys.stderr) 362 | 363 | return bits 364 | 365 | def microsoft_trust_bits_list(self, *, sha256: str) -> Optional[list]: 366 | if sha256 not in self._certs: 367 | return 368 | 369 | # DTBs - Derived Trust Bits 370 | return _trust_bits_list('Microsoft EKUs', self._certs[sha256]) 371 | 372 | def microsoft_trust_bits(self, *, sha256: str) -> Optional[TrustBits]: 373 | values = self.microsoft_trust_bits_list(sha256=sha256) 374 | if values is None: 375 | return 376 | 377 | if self._debug and not values: 378 | print('Microsoft Trust Bits null %s' % sha256, file=sys.stderr) 379 | 380 | bits = trust_bits_flag(values) 381 | 382 | return bits 383 | -------------------------------------------------------------------------------- /bin/sprocket.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2024 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import argparse 20 | import asyncio 21 | import csv 22 | import json 23 | import os 24 | import pprint 25 | import sys 26 | 27 | libpath = os.path.dirname(os.path.abspath(__file__)) 28 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 29 | 30 | from pan_chainguard import title, __version__ 31 | from pan_chainguard.ccadb import (revoked, valid_from_to, 32 | root_status_bits_flag, RootStatusBits, 33 | root_trust_bits, TrustBits, TrustBitsMap2, 34 | CcadbRootTrustSettings, CcadbError) 35 | import pan_chainguard.util 36 | 37 | 38 | DEFAULT_POLICY = { 39 | 'sources': ['mozilla'], 40 | 'operation': 'union', 41 | 'trust_bits': [], 42 | } 43 | 44 | SOURCES_MAP = { 45 | 'mozilla': RootStatusBits.MOZILLA, 46 | 'chrome': RootStatusBits.CHROME, 47 | 'google': RootStatusBits.CHROME, # XXX deprecated 48 | 'apple': RootStatusBits.APPLE, 49 | 'microsoft': RootStatusBits.MICROSOFT, 50 | } 51 | 52 | args = None 53 | 54 | 55 | def main(): 56 | global args 57 | args = parse_args() 58 | 59 | asyncio.run(main_loop()) 60 | 61 | sys.exit(0) 62 | 63 | 64 | async def main_loop(): 65 | certs = read_certs() 66 | root_trust_settings = get_trust_settings() 67 | policy = get_policy() 68 | 69 | if args.debug > 1: 70 | print('total', len(certs), file=sys.stderr) 71 | if args.debug > 2: 72 | print(pprint.pformat(certs, width=160), file=sys.stderr) 73 | if args.verbose: 74 | print('policy:', policy) 75 | 76 | if args.stats: 77 | stats(certs, root_trust_settings, policy) 78 | return 79 | 80 | policy_certs = get_root_certs(certs, root_trust_settings, policy) 81 | 82 | if args.verbose: 83 | print("%s: %d total certificates" % ( 84 | ', '.join(policy['sources']), len(policy_certs))) 85 | 86 | if args.fingerprints is not None: 87 | write_fingerprints(policy_certs) 88 | 89 | 90 | def read_certs(): 91 | certs = {} 92 | EXPECTED = { # subset 93 | 'Certificate Record Type', 94 | 'SHA-256 Fingerprint', 95 | } 96 | 97 | try: 98 | with open(args.ccadb, 'r', newline='') as csvfile: 99 | reader = csv.DictReader(csvfile, 100 | dialect='unix') 101 | fieldnames = set(reader.fieldnames or []) 102 | missing = EXPECTED - fieldnames 103 | if missing: 104 | print('%s: Invalid CSV' % args.ccadb, file=sys.stderr) 105 | sys.exit(1) 106 | 107 | for row in reader: 108 | if row['Certificate Record Type'] != 'Root Certificate': 109 | continue 110 | 111 | sha256 = row['SHA-256 Fingerprint'] 112 | name = row['Certificate Name'] 113 | 114 | ret, err = revoked(row) 115 | if ret: 116 | x = '%s %s %s' % (err, sha256, name) 117 | if args.debug > 2: 118 | print(x, file=sys.stderr) 119 | continue 120 | 121 | ret, err = valid_from_to(row) 122 | if not ret: 123 | x = '%s %s %s' % (err, sha256, name) 124 | if args.debug > 2: 125 | print(x, file=sys.stderr) 126 | continue 127 | 128 | certs[sha256] = row 129 | 130 | except OSError as e: 131 | print('%s: %s' % (args.ccadb, e), file=sys.stderr) 132 | sys.exit(1) 133 | 134 | return certs 135 | 136 | 137 | def get_trust_settings(): 138 | if not args.trust_settings: 139 | return 140 | 141 | try: 142 | settings = CcadbRootTrustSettings(path=args.trust_settings, 143 | debug=bool(args.debug)) 144 | except CcadbError as e: 145 | print('CcadbRootTrustSettings: %s' % e, file=sys.stderr) 146 | sys.exit(1) 147 | 148 | return settings 149 | 150 | 151 | def get_policy(): 152 | def isvalid(policy): 153 | if not isinstance(policy['sources'], list): 154 | x = '"sources" not list' 155 | elif not len(policy['sources']): 156 | x = '"sources" empty' 157 | elif not all(item in SOURCES_MAP for item in policy['sources']): 158 | x = 'Invalid "sources"' 159 | elif policy['operation'] not in ['union', 'intersection']: 160 | x = 'Invalid "operation"' 161 | elif not all(item in TrustBitsMap2 162 | for item in policy['trust_bits']): 163 | x = 'Invalid "trust_bits"' 164 | else: 165 | return True, None 166 | 167 | return False, x 168 | 169 | if args.policy is None: 170 | return DEFAULT_POLICY 171 | 172 | if os.path.isfile(args.policy): 173 | try: 174 | with open(args.policy, 'r') as f: 175 | x = json.load(f) 176 | except (OSError, ValueError) as e: 177 | print('%s: %s' % (args.policy, e), file=sys.stderr) 178 | sys.exit(1) 179 | 180 | else: 181 | try: 182 | x = json.loads(args.policy) 183 | except ValueError as e: 184 | print('%s: %s' % (e, args.policy), file=sys.stderr) 185 | sys.exit(1) 186 | 187 | if 'sources' not in x: 188 | x['sources'] = DEFAULT_POLICY['sources'] 189 | if 'operation' not in x: 190 | x['operation'] = DEFAULT_POLICY['operation'] 191 | if 'trust_bits' not in x: 192 | x['trust_bits'] = DEFAULT_POLICY['trust_bits'] 193 | 194 | ret, e = isvalid(x) 195 | 196 | if not ret: 197 | msg = '%s: ' % args.policy if os.path.isfile(args.policy) else '' 198 | msg += '%s: %s' % (e, x) 199 | print(msg, file=sys.stderr) 200 | sys.exit(1) 201 | 202 | return x 203 | 204 | 205 | def get_certs(certs, root_trust_settings, policy): 206 | certs_ = [] 207 | 208 | for row in certs.values(): 209 | sha256 = row['SHA-256 Fingerprint'] 210 | 211 | if policy_match(policy, root_trust_settings, row): 212 | certs_.append(sha256) 213 | 214 | return certs_ 215 | 216 | 217 | def get_all_sources(certs, root_trust_settings, policy): 218 | sources = {} 219 | for x in SOURCES_MAP.keys(): 220 | if x == 'google': 221 | # XXX deprecated 222 | continue 223 | policy_ = { 224 | 'sources': [x], 225 | 'trust_bits': policy['trust_bits'], 226 | 'operation': policy['operation'], 227 | } 228 | source_certs = get_certs(certs, root_trust_settings, policy_) 229 | sources[x] = source_certs 230 | 231 | return sources 232 | 233 | 234 | def stats(certs, root_trust_settings, policy): 235 | sources = get_all_sources(certs, root_trust_settings, policy) 236 | 237 | sets = [] 238 | for x in sources.keys(): 239 | print("%s: %d total certificates" % (x, len(sources[x]))) 240 | sets.append(set(sources[x])) 241 | 242 | x = ', '.join(sources.keys()) 243 | 244 | intersection = set.intersection(*sets) 245 | print('%s: %d total certificates in all (intersection)' % ( 246 | x, len(intersection))) 247 | 248 | union = set.union(*sets) 249 | print('%s: %d total certificates in any (union)' % ( 250 | x, len(union))) 251 | 252 | 253 | def get_root_certs(certs, root_trust_settings, policy): 254 | sources = get_all_sources(certs, root_trust_settings, policy) 255 | 256 | sets = [] 257 | for x in policy['sources']: 258 | k = 'chrome' if x == 'google' else x # XXX deprecated 259 | sets.append(set(sources[k])) 260 | 261 | if policy['operation'] == 'union': 262 | result = set.union(*sets) 263 | elif policy['operation'] == 'intersection': 264 | result = set.intersection(*sets) 265 | else: 266 | assert False, 'Invalid operation: %s' % policy['operation'] 267 | 268 | return result 269 | 270 | 271 | def policy_match(policy, root_trust_settings, row): 272 | def sources_match(status_bits, source): 273 | if SOURCES_MAP[source] in status_bits: 274 | return True 275 | return False 276 | 277 | def trust_bits_match(bits_cert, bits_pol): 278 | if bits_cert == TrustBits.NONE: 279 | return True 280 | 281 | for x in bits_pol: 282 | if TrustBitsMap2[x] not in bits_cert: 283 | return False 284 | return True 285 | 286 | certificate_name = row['Certificate Name'] 287 | sha256 = row['SHA-256 Fingerprint'] 288 | 289 | status_bits = root_status_bits_flag(row) 290 | trust_bits = root_trust_bits(row) 291 | 292 | if args.debug > 1: 293 | print(certificate_name, 294 | 'status_bits', status_bits, 295 | 'trust_bits', trust_bits, file=sys.stderr) 296 | 297 | assert len(policy["sources"]) == 1, ( 298 | f"Must be single source: {policy['sources']}" 299 | ) 300 | 301 | if (sources_match(status_bits, policy['sources'][0]) and 302 | trust_bits_match(trust_bits, policy['trust_bits'])): 303 | if root_trust_settings: 304 | MAP = { 305 | 'mozilla': root_trust_settings.mozilla_trust_bits, 306 | 'chrome': root_trust_settings.chrome_trust_bits, 307 | 'apple': root_trust_settings.apple_trust_bits, 308 | 'microsoft': root_trust_settings.microsoft_trust_bits, 309 | } 310 | 311 | if args.debug: 312 | status_bits2 = root_trust_settings.root_status_bits_flag( 313 | sha256=sha256) 314 | if status_bits2 != status_bits: 315 | print(f'AllCertificateRecords {status_bits} != ' 316 | f'AllIncludedRootCerts {status_bits2} {sha256}', 317 | file=sys.stderr) 318 | 319 | bits = MAP[policy['sources'][0]](sha256=sha256) 320 | if TrustBits.SERVER_AUTHENTICATION in bits: 321 | return True 322 | else: 323 | return True 324 | 325 | if args.debug > 1: 326 | print('no match', certificate_name, sha256, file=sys.stderr) 327 | 328 | return False 329 | 330 | 331 | def write_fingerprints(policy_certs): 332 | data = [] 333 | for x in policy_certs: 334 | row = {} 335 | row['type'] = 'root' 336 | row['sha256'] = x 337 | data.append(row) 338 | 339 | try: 340 | pan_chainguard.util.write_fingerprints(path=args.fingerprints, 341 | data=data) 342 | except pan_chainguard.util.UtilError as e: 343 | print('%s: %s' % (args.fingerprints, e), file=sys.stderr) 344 | sys.exit(1) 345 | 346 | 347 | def parse_args(): 348 | parser = argparse.ArgumentParser( 349 | usage='%(prog)s [options]', 350 | description='create custom root store') 351 | # https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv4 352 | parser.add_argument('-c', '--ccadb', 353 | required=True, 354 | metavar='PATH', 355 | help='CCADB all certificate information CSV path') 356 | parser.add_argument('-f', '--fingerprints', 357 | metavar='PATH', 358 | help='root CA fingerprints CSV path') 359 | # https://ccadb.my.salesforce-sites.com/ccadb/AllIncludedRootCertsCSV 360 | parser.add_argument('-T', '--trust-settings', 361 | metavar='PATH', 362 | help='CCADB root certificate trust bit settings' 363 | ' CSV path') 364 | parser.add_argument('--policy', 365 | metavar='JSON', 366 | help='JSON policy object path or string') 367 | parser.add_argument('--stats', 368 | action='store_true', 369 | help='print source stats') 370 | parser.add_argument('--verbose', 371 | action='store_true', 372 | help='enable verbosity') 373 | parser.add_argument('--debug', 374 | type=int, 375 | choices=[0, 1, 2, 3], 376 | default=0, 377 | help='enable debug') 378 | x = '%s %s' % (title, __version__) 379 | parser.add_argument('--version', 380 | action='version', 381 | help='display version', 382 | version=x) 383 | args = parser.parse_args() 384 | 385 | if args.debug: 386 | print(args, file=sys.stderr) 387 | 388 | return args 389 | 390 | 391 | if __name__ == '__main__': 392 | main() 393 | -------------------------------------------------------------------------------- /bin/chain.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2024 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import argparse 20 | import asyncio 21 | from collections import defaultdict 22 | import csv 23 | import json 24 | import os 25 | import sys 26 | from treelib import Node, Tree 27 | 28 | libpath = os.path.dirname(os.path.abspath(__file__)) 29 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 30 | 31 | from pan_chainguard import title, __version__ 32 | from pan_chainguard.ccadb import * 33 | from pan_chainguard.mozilla import MozillaOneCrl, MozillaError 34 | import pan_chainguard.util 35 | 36 | args = None 37 | 38 | 39 | def main(): 40 | global args 41 | args = parse_args() 42 | 43 | ret = asyncio.run(main_loop()) 44 | 45 | sys.exit(ret) 46 | 47 | 48 | async def main_loop(): 49 | onecrl = None 50 | if args.onecrl: 51 | try: 52 | onecrl = MozillaOneCrl(path=args.onecrl) 53 | except MozillaError as e: 54 | print('MozillaOneCrl: %s' % e, file=sys.stderr) 55 | sys.exit(1) 56 | 57 | certs, invalid, warning = get_certs(onecrl) 58 | 59 | tree = get_tree(certs, invalid) 60 | if args.debug > 2: 61 | x = tree.show(stdout=False) 62 | print(x, file=sys.stderr, end='') 63 | 64 | total_invalid, newtree, intermediates = get_intermediates( 65 | tree, invalid, warning) 66 | 67 | if args.debug: 68 | x = newtree.show(stdout=False) 69 | print(x, file=sys.stderr, end='') 70 | 71 | write_fingerprints(intermediates) 72 | write_tree(newtree) 73 | 74 | return 0 75 | 76 | 77 | def get_certs(onecrl): 78 | invalid = {} 79 | warning = {} 80 | certs = {} 81 | duplicates = defaultdict(list) 82 | EXPECTED = { # subset 83 | 'Certificate Record Type', 84 | 'SHA-256 Fingerprint', 85 | } 86 | 87 | try: 88 | with open(args.ccadb, 'r', newline='') as csvfile: 89 | reader = csv.DictReader(csvfile, 90 | dialect='unix') 91 | fieldnames = set(reader.fieldnames or []) 92 | missing = EXPECTED - fieldnames 93 | if missing: 94 | print('%s: Invalid CSV' % args.ccadb, file=sys.stderr) 95 | sys.exit(1) 96 | 97 | for row in reader: 98 | sha256 = row['SHA-256 Fingerprint'] 99 | name = row['Certificate Name'] 100 | cert_type = row['Certificate Record Type'] 101 | parent_sha256 = row['Parent SHA-256 Fingerprint'] 102 | 103 | ret, err = revoked(row) 104 | if ret: 105 | x = '%s %s %s' % (err, sha256, name) 106 | if args.debug > 2: 107 | print(x, file=sys.stderr) 108 | invalid[sha256] = x 109 | continue 110 | 111 | ret, err = valid_from_to(row) 112 | if not ret: 113 | x = '%s %s %s' % (err, sha256, name) 114 | if args.debug > 2: 115 | print(x, file=sys.stderr) 116 | invalid[sha256] = x 117 | continue 118 | 119 | # For duplicate certificate fingerprints in CCADB, 120 | # retain a root certificate, or intermediate when no 121 | # root. 122 | if sha256 in certs: 123 | if sha256 not in duplicates: 124 | duplicates[sha256].append(certs[sha256]) 125 | duplicates[sha256].append(row) 126 | 127 | if (certs[sha256]['Certificate Record Type'] == 128 | 'Root Certificate'): 129 | if args.debug > 1: 130 | print('Retain duplicate %s %s' % ( 131 | certs[sha256]['Certificate Record Type'], 132 | sha256), file=sys.stderr) 133 | continue 134 | elif cert_type == 'Root Certificate': 135 | if args.debug > 1: 136 | print('Replace duplicate %s' 137 | ' with %s %s' % ( 138 | certs[sha256]['Certificate Record Type'], 139 | cert_type, sha256), file=sys.stderr) 140 | else: 141 | if args.debug > 1: 142 | print('Skip duplicate %s, retain %s %s' % ( 143 | cert_type, 144 | certs[sha256]['Certificate Record Type'], 145 | sha256), file=sys.stderr) 146 | continue 147 | 148 | if cert_type == 'Root Certificate': 149 | if parent_sha256: 150 | x = 'Root with parent: %s' % sha256 151 | if args.debug > 1: 152 | print(x, file=sys.stderr) 153 | invalid[sha256] = x 154 | continue 155 | 156 | if cert_type == 'Intermediate Certificate': 157 | if onecrl: 158 | r = onecrl.get(sha256=sha256) 159 | if r is not None: 160 | x = 'In OneCRL %s "%s" "%s"' % ( 161 | sha256, name, r['Revocation Status']) 162 | if r['Comments']: 163 | x += ' "%s"' % ( 164 | r['Comments'].replace('\r\n', ' ')) 165 | if args.debug > 1: 166 | print(x, file=sys.stderr) 167 | invalid[sha256] = x 168 | continue 169 | 170 | if not parent_sha256: 171 | x = 'Intermediate with no parent: %s' % sha256 172 | if args.debug > 1: 173 | print(x, file=sys.stderr) 174 | invalid[sha256] = x 175 | continue 176 | 177 | trust_bits = derived_trust_bits(row) 178 | if TrustBits.SERVER_AUTHENTICATION not in trust_bits: 179 | x = 'Missing %s in %s %s' % ( 180 | TrustBits.SERVER_AUTHENTICATION.name, 181 | trust_bits, sha256) 182 | if args.debug > 1: 183 | print(x, file=sys.stderr) 184 | invalid[sha256] = x 185 | continue 186 | 187 | certs[sha256] = row 188 | 189 | except OSError as e: 190 | print('%s: %s' % (args.ccadb, e), file=sys.stderr) 191 | sys.exit(1) 192 | 193 | if args.debug > 1: 194 | for sha256 in duplicates: 195 | print('Duplicate certificates %s' % sha256, file=sys.stderr) 196 | for x in duplicates[sha256]: 197 | print(' %s %s %s' % (x['Certificate Name'], 198 | x['Certificate Record Type'], 199 | x['Salesforce Record ID']), 200 | file=sys.stderr) 201 | 202 | return certs, invalid, warning 203 | 204 | 205 | def get_tree(certs, invalid): 206 | nodes = {} 207 | 208 | for row in certs.values(): 209 | sha256 = row['SHA-256 Fingerprint'] 210 | name = row['Certificate Name'] 211 | cert_type = row['Certificate Record Type'] 212 | parent_name = row['Parent Certificate Name'] 213 | 214 | tag = f'{sha256} Subject: "{name}"' 215 | if cert_type == 'Root Certificate': 216 | tag += f' CA-Owner: "{parent_name}"' 217 | else: 218 | tag += f' Issuer: "{parent_name}"' 219 | nodes[sha256] = Node(tag=tag, identifier=sha256, data=row) 220 | 221 | waiting_nodes = defaultdict(list) 222 | 223 | tree = Tree() 224 | tree.add_node(Node(tag='Root', identifier=0)) 225 | 226 | for node in nodes.values(): 227 | row = node.data 228 | sha256 = row['SHA-256 Fingerprint'] 229 | parent_sha256 = row['Parent SHA-256 Fingerprint'] 230 | 231 | if not parent_sha256: 232 | tree.add_node(nodes[sha256], parent=0) 233 | else: 234 | if tree.contains(parent_sha256): 235 | tree.add_node(nodes[sha256], parent=nodes[parent_sha256]) 236 | # is node a parent for any waiting nodes 237 | if sha256 in waiting_nodes: 238 | for x in waiting_nodes[sha256]: 239 | tree.add_node(nodes[x], parent=nodes[sha256]) 240 | del waiting_nodes[sha256] 241 | elif parent_sha256 not in invalid: 242 | # node's parent not in tree, add node to waiting list 243 | waiting_nodes[parent_sha256].append(sha256) 244 | 245 | if args.debug > 1 and waiting_nodes: 246 | print('Warning: nodes with no parent', file=sys.stderr) 247 | for x in waiting_nodes: 248 | print('parent', x, 'nodes', waiting_nodes[x], file=sys.stderr) 249 | 250 | return tree 251 | 252 | 253 | def get_intermediates(tree, invalid, warning): 254 | intermediates = [] 255 | total_invalid = 0 256 | 257 | EXCLUDE_INTERMEDIATES = [ 258 | 'C0A6F4DC63A24BFDCF54EF2A6A082A0A72DE35803E2FF5FF527AE5D87206DFD5', 259 | ] 260 | 261 | try: 262 | data = pan_chainguard.util.read_fingerprints( 263 | path=args.root_fingerprints) 264 | except pan_chainguard.util.UtilError as e: 265 | print('%s: %s' % (args.root_fingerprints, e), file=sys.stderr) 266 | sys.exit(1) 267 | 268 | newtree = Tree() 269 | newtree.add_node(Node(tag='Root', identifier=0)) 270 | 271 | for row in data: 272 | sha256 = row['sha256'] 273 | 274 | if sha256 in warning: 275 | print('Certificate warning: %s' % ( 276 | warning[sha256]), file=sys.stderr) 277 | 278 | if sha256 in invalid: 279 | print('Invalid certificate: %s' % ( 280 | invalid[sha256]), file=sys.stderr) 281 | total_invalid += 1 282 | continue 283 | 284 | if not tree.contains(sha256): 285 | print('Not found in CCADB: %s' % ( 286 | sha256), file=sys.stderr) 287 | total_invalid += 1 288 | continue 289 | 290 | node = tree.get_node(sha256) 291 | status_root = node.data['Status of Root Cert'] 292 | statuses = status_root.split(';') 293 | included = [': Included' in x for x in statuses] 294 | if not any(included): 295 | print('Certificate not in common root store: ' 296 | '%s: %s' % (sha256, status_root), 297 | file=sys.stderr) 298 | 299 | if sha256 in EXCLUDE_INTERMEDIATES: 300 | print('Skip intermediates for root: %s' % ( 301 | sha256), file=sys.stderr) 302 | newtree.add_node(node=node, parent=0) 303 | continue 304 | 305 | subtree = tree.subtree(sha256) 306 | nodes = subtree.all_nodes() 307 | for node in nodes[1:]: 308 | intermediates.append(node.identifier) 309 | for node in nodes: 310 | parent = subtree.parent(node.identifier) 311 | newtree.add_node( 312 | node=node, 313 | parent=0 if parent is None else parent.identifier) 314 | 315 | if args.debug > 1: 316 | print(subtree, end='', file=sys.stderr) 317 | nodes_ = [x.identifier for x in nodes] 318 | print(len(nodes_), nodes_, file=sys.stderr) 319 | 320 | return total_invalid, newtree, intermediates 321 | 322 | 323 | def write_fingerprints(intermediates): 324 | if not args.int_fingerprints: 325 | return 326 | 327 | data = [] 328 | for x in intermediates: 329 | row = { 330 | 'type': 'intermediate', 331 | 'sha256': x, 332 | } 333 | data.append(row) 334 | 335 | try: 336 | pan_chainguard.util.write_fingerprints( 337 | path=args.int_fingerprints, 338 | data=data) 339 | except pan_chainguard.util.UtilError as e: 340 | print('%s: %s' % (args.int_fingerprints, e), file=sys.stderr) 341 | sys.exit(1) 342 | 343 | if args.verbose: 344 | print('%d total intermediate certificates' % len(data)) 345 | 346 | 347 | def write_tree(tree): 348 | if not args.tree: 349 | return 350 | 351 | data = pan_chainguard.util.tree_to_dict(tree=tree) 352 | 353 | try: 354 | with open(args.tree, 'w') as f: 355 | json.dump(data, f, separators=(',', ':')) 356 | except (OSError, TypeError) as e: 357 | print('%s: %s' % (args.tree, e), file=sys.stderr) 358 | sys.exit(1) 359 | 360 | 361 | def parse_args(): 362 | parser = argparse.ArgumentParser( 363 | usage='%(prog)s [options]', 364 | description='determine intermediate CAs') 365 | # https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv4 366 | parser.add_argument('-c', '--ccadb', 367 | required=True, 368 | metavar='PATH', 369 | help='CCADB all certificate information CSV path') 370 | parser.add_argument('-r', '--root-fingerprints', 371 | required=True, 372 | metavar='PATH', 373 | help='root CA fingerprints CSV path') 374 | # https://ccadb.my.salesforce-sites.com/mozilla/IntermediateCertsInOneCRLReportCSV 375 | parser.add_argument('-o', '--onecrl', 376 | metavar='PATH', 377 | help='Mozilla OneCRL CSV path') 378 | parser.add_argument('-i', '--int-fingerprints', 379 | metavar='PATH', 380 | help='intermediate CA fingerprints CSV path') 381 | parser.add_argument('--tree', 382 | metavar='PATH', 383 | help='save certificate tree as JSON to path') 384 | parser.add_argument('--verbose', 385 | action='store_true', 386 | help='enable verbosity') 387 | parser.add_argument('--debug', 388 | type=int, 389 | choices=[0, 1, 2, 3], 390 | default=0, 391 | help='enable debug') 392 | x = '%s %s' % (title, __version__) 393 | parser.add_argument('--version', 394 | action='version', 395 | help='display version', 396 | version=x) 397 | args = parser.parse_args() 398 | 399 | if args.debug: 400 | print(args, file=sys.stderr) 401 | 402 | return args 403 | 404 | 405 | if __name__ == '__main__': 406 | main() 407 | -------------------------------------------------------------------------------- /etc/derailer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2025 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import aiohttp 20 | import argparse 21 | import asyncio 22 | from collections import defaultdict 23 | import csv 24 | import hashlib 25 | import os 26 | from pathlib import Path 27 | import re 28 | import sys 29 | from typing import Optional, Tuple 30 | import warnings 31 | 32 | try: 33 | from cryptography import x509 34 | from cryptography.hazmat.primitives import hashes 35 | except ImportError: 36 | print('Install cryptography: https://pypi.org/project/cryptography/', 37 | file=sys.stderr) 38 | sys.exit(1) 39 | 40 | libpath = os.path.dirname(os.path.abspath(__file__)) 41 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 42 | 43 | from pan_chainguard import title, __version__ 44 | from pan_chainguard.ccadb import revoked, valid_from, valid_to 45 | 46 | DOWNLOAD_TIMEOUT = 5 47 | 48 | args = None 49 | downloads = {} 50 | 51 | 52 | def main(): 53 | global args 54 | args = parse_args() 55 | 56 | asyncio.run(main_loop()) 57 | 58 | sys.exit(0) 59 | 60 | 61 | async def main_loop(): 62 | if args.debug: 63 | for k, v in vendors.items(): 64 | if k == 'all': 65 | continue 66 | for x in v: 67 | print(k, x.__name__, x.url, file=sys.stderr) 68 | 69 | vendors_ = set(args.vendor) 70 | timeout = aiohttp.ClientTimeout(total=DOWNLOAD_TIMEOUT) 71 | 72 | urls = set([x.url for vendor in vendors_ 73 | for x in vendors[vendor]]) 74 | urls = list(urls) 75 | async with aiohttp.ClientSession(timeout=timeout) as session: 76 | tasks = [download(session, url) for url in urls] 77 | results = await asyncio.gather(*tasks) 78 | for idx, url in enumerate(urls): 79 | downloads[url] = results[idx] 80 | ok, r = downloads[url] 81 | if ok and args.verbose: 82 | print(f'Downloaded {len(r)} bytes from {url}') 83 | 84 | if args.debug > 1: 85 | for x in downloads: 86 | r, v = downloads[x] 87 | if not r: 88 | print(x, v, file=sys.stderr) 89 | else: 90 | print(x, len(v), file=sys.stderr) 91 | 92 | ccadb = None 93 | if args.ccadb: 94 | ccadb = load_ccadb(args.ccadb) 95 | 96 | for k, v in vendors.items(): 97 | if k in vendors_: 98 | for x in v: 99 | fingerprints = x() 100 | if fingerprints is None: 101 | continue 102 | messages = check_validity(ccadb, fingerprints) 103 | m = [] 104 | for type_ in sorted(messages.keys()): 105 | m.append(f'{len(messages[type_])} {type_}') 106 | summary = '' 107 | if m: 108 | summary = ', '.join(m) 109 | summary = ' (%s)' % summary 110 | print(f'{x.__name__}: ' 111 | f'{len(fingerprints)} root CAs{summary}') 112 | if args.verbose: 113 | for type_ in sorted(messages.keys()): 114 | for msg in messages[type_]: 115 | print(f'{x.__name__}: {msg}') 116 | if args.save: 117 | path = args.save / f'{x.__name__}.txt' 118 | digest = save(path, sorted(fingerprints)) 119 | if digest and args.verbose: 120 | print(f'Saved fingerprints ' 121 | f'(SHA256 {digest}) to {path}') 122 | 123 | 124 | async def download( 125 | session: aiohttp.ClientSession, 126 | url: str 127 | ) -> Tuple[bool, str]: 128 | try: 129 | async with session.get(url) as response: 130 | if response.status != 200: 131 | return False, f'Error: status code {response.status}' 132 | 133 | content = await response.text() 134 | return True, content 135 | 136 | except (aiohttp.ClientError, asyncio.TimeoutError) as e: 137 | msg = str(e) or type(e).__name__ 138 | return False, f'Request failed: {msg}' 139 | except Exception as e: 140 | msg = str(e) 141 | detail = f': {msg}' if msg else '' 142 | return False, f'Unexpected {type(e).__name__}{detail}' 143 | 144 | 145 | def pem_cert_fingerprint(data: bytes) -> Optional[str]: 146 | try: 147 | with warnings.catch_warnings(record=True) as w: 148 | warnings.simplefilter('always') 149 | cert = x509.load_pem_x509_certificate(data) 150 | sha256 = cert.fingerprint(hashes.SHA256()).hex().upper() 151 | 152 | if args.debug: 153 | for warn in w: 154 | print(f'{warn.category.__name__}: {warn.message}', 155 | file=sys.stderr) 156 | print('SHA256', sha256, file=sys.stderr) 157 | 158 | return sha256 159 | 160 | except Exception as e: 161 | # Future-proof: cryptography may raise instead of warn 162 | pem_sha256 = hashlib.sha256(data).hexdigest().upper() 163 | print(f'{type(e).__name__}: {e}', file=sys.stderr) 164 | print('Failed to load cert. PEM blob SHA256', pem_sha256, 165 | file=sys.stderr) 166 | return 167 | 168 | 169 | def load_ccadb(path): 170 | ccadb = defaultdict(list) 171 | EXPECTED = { # subset 172 | 'Certificate Record Type', 173 | 'SHA-256 Fingerprint', 174 | } 175 | 176 | try: 177 | with open(path, 'r', newline='') as csvfile: 178 | reader = csv.DictReader(csvfile, 179 | dialect='unix') 180 | fieldnames = set(reader.fieldnames or []) 181 | missing = EXPECTED - fieldnames 182 | if missing: 183 | print('%s: Invalid CSV' % path, file=sys.stderr) 184 | sys.exit(1) 185 | 186 | for row in reader: 187 | sha256 = row['SHA-256 Fingerprint'] 188 | ccadb[sha256].append(row) 189 | 190 | except OSError as e: 191 | print('%s: %s' % (path, e), file=sys.stderr) 192 | sys.exit(1) 193 | 194 | return ccadb 195 | 196 | 197 | def check_validity(ccadb, fingerprints): 198 | if not ccadb: 199 | return {} 200 | 201 | messages = defaultdict(list) 202 | 203 | for sha256 in fingerprints: 204 | if sha256 not in ccadb: 205 | msg = f'Not in CCADB: {sha256}' 206 | messages['not in ccadb'].append(msg) 207 | continue 208 | 209 | row = ccadb[sha256][0] 210 | name = row['Certificate Name'] 211 | cert_type = row['Certificate Record Type'] 212 | 213 | x = 'Root Certificate' 214 | if len(ccadb[sha256]) == 1 and cert_type != x: 215 | msg = f'Not a {x}: {sha256} "{name}"' 216 | messages['not root'].append(msg) 217 | 218 | if len(ccadb[sha256]) > 1: 219 | total = defaultdict(int) 220 | for x in ccadb[sha256]: 221 | if x['Certificate Record Type'] == 'Root Certificate': 222 | total['R'] += 1 223 | elif (x['Certificate Record Type'] == 224 | 'Intermediate Certificate'): 225 | total['I'] += 1 226 | else: 227 | raise RuntimeError(x['Certificate Record Type']) 228 | totals = ' '.join(f'{k}={v}' for k, v in total.items()) 229 | msg = (f'{len(ccadb[sha256])} duplicate certificates: ' 230 | f'{sha256} {totals} "{name}"') 231 | messages['duplicates'].append(msg) 232 | 233 | ret, err = revoked(row) 234 | if ret: 235 | msg = f'{err}: {sha256} "{name}"' 236 | messages['revoked'].append(msg) 237 | 238 | ret, err = valid_from(row) 239 | if not ret: 240 | msg = f'{err}: {sha256} "{name}"' 241 | messages['not yet valid'].append(msg) 242 | 243 | ret, err = valid_to(row) 244 | if not ret: 245 | msg = f'{err}: {sha256} "{name}"' 246 | messages['expired'].append(msg) 247 | 248 | return messages 249 | 250 | 251 | def save(path, fingerprints): 252 | data = '\n'.join(fingerprints) + '\n' 253 | try: 254 | with open(path, 'w') as f: 255 | f.write(data) 256 | except OSError as e: 257 | print(f'{path}: {e}', file=sys.stderr) 258 | return 259 | 260 | return hashlib.sha256(data.encode()).hexdigest() 261 | 262 | 263 | def websites_trusted(vendor, row): 264 | def mozilla(row): 265 | ret = (row['Mozilla Status'] == 'Included' and 266 | 'Websites' in row['Mozilla Trust Bits'].split(';')) 267 | return ret 268 | 269 | def microsoft(row): 270 | ret = (row['Microsoft Status'] == 'Included' and 271 | 'Server Authentication' in row['Microsoft EKUs'].split(';')) 272 | return ret 273 | 274 | def chrome(row): 275 | ret = row['Google Chrome Status'] == 'Included' 276 | return ret 277 | 278 | def apple(row): 279 | ret = (row['Apple Status'] == 'Included' and 280 | 'serverAuth' in row['Apple Trust Bits'].split(';')) 281 | return ret 282 | 283 | return locals()[vendor](row) 284 | 285 | 286 | def set_url(url): 287 | def decorator(func): 288 | func.url = url 289 | return func 290 | return decorator 291 | 292 | 293 | @set_url('https://ccadb.my.salesforce-sites.com/ccadb/' 294 | 'AllIncludedRootCertsCSV') 295 | def mozilla_0() -> Optional[list]: 296 | url = mozilla_0.url 297 | ok, r = downloads[url] 298 | if not ok: 299 | print(f'{mozilla_0.__name__}: {r}', file=sys.stderr) 300 | return 301 | 302 | reader = csv.DictReader(r.splitlines()) 303 | 304 | fingerprints = [] 305 | for row in reader: 306 | if websites_trusted('mozilla', row): 307 | fingerprints.append(row['SHA-256 Fingerprint']) 308 | 309 | return fingerprints 310 | 311 | 312 | # https://wiki.mozilla.org/CA/Included_Certificates 313 | # PEM of Root Certificates in Mozilla's Root Store with the 314 | # Websites (TLS/SSL) Trust Bit Enabled (CSV) 315 | @set_url('https://ccadb.my.salesforce-sites.com/mozilla/' 316 | 'IncludedRootsDistrustTLSSSLPEMCSV?TrustBitsInclude=Websites') 317 | def mozilla_1() -> Optional[list]: 318 | url = mozilla_1.url 319 | ok, r = downloads[url] 320 | if not ok: 321 | print(f'{mozilla_1.__name__}: {r}', file=sys.stderr) 322 | return 323 | 324 | reader = csv.DictReader(r.splitlines()) 325 | 326 | fingerprints = [] 327 | for row in reader: 328 | pem = row.get('PEM') 329 | if pem: 330 | fingerprint = pem_cert_fingerprint(pem.encode()) 331 | if fingerprint is not None: 332 | fingerprints.append(fingerprint) 333 | 334 | return fingerprints 335 | 336 | 337 | # https://wiki.mozilla.org/CA/Included_Certificates 338 | # Included Root CA Certificates (CSV) 339 | @set_url('https://ccadb.my.salesforce-sites.com/mozilla/' 340 | 'IncludedRootCertificateReportCSVFormat') 341 | def mozilla_2() -> Optional[list]: 342 | url = mozilla_2.url 343 | ok, r = downloads[url] 344 | if not ok: 345 | print(f'{mozilla_2.__name__}: {r}', file=sys.stderr) 346 | return 347 | 348 | reader = csv.DictReader(r.splitlines()) 349 | 350 | fingerprints = [] 351 | for row in reader: 352 | trust_bits = row['Trust Bits'].split(';') 353 | if 'Websites' in trust_bits: 354 | fingerprints.append(row['SHA-256 Fingerprint']) 355 | 356 | return fingerprints 357 | 358 | 359 | @set_url('https://ccadb.my.salesforce-sites.com/ccadb/' 360 | 'AllIncludedRootCertsCSV') 361 | def microsoft_0() -> Optional[list]: 362 | url = microsoft_0.url 363 | ok, r = downloads[url] 364 | if not ok: 365 | print(f'{microsoft_0.__name__}: {r}', file=sys.stderr) 366 | return 367 | 368 | reader = csv.DictReader(r.splitlines()) 369 | 370 | fingerprints = [] 371 | for row in reader: 372 | if websites_trusted('microsoft', row): 373 | fingerprints.append(row['SHA-256 Fingerprint']) 374 | 375 | return fingerprints 376 | 377 | 378 | @set_url('https://ccadb.my.salesforce-sites.com/microsoft/' 379 | 'IncludedRootsPEMCSVForMSFT?MicrosoftEKUs=Server Authentication') 380 | def microsoft_1() -> Optional[list]: 381 | url = microsoft_1.url 382 | ok, r = downloads[url] 383 | if not ok: 384 | print(f'{microsoft_1.__name__}: {r}', file=sys.stderr) 385 | return 386 | 387 | reader = csv.DictReader(r.splitlines()) 388 | 389 | fingerprints = [] 390 | for row in reader: 391 | pem = row.get('PEM') 392 | if pem: 393 | fingerprint = pem_cert_fingerprint(pem.encode()) 394 | if fingerprint is not None: 395 | fingerprints.append(fingerprint) 396 | 397 | return fingerprints 398 | 399 | 400 | @set_url('https://ccadb.my.salesforce-sites.com/microsoft/' 401 | 'IncludedCACertificateReportForMSFTCSV') 402 | def microsoft_2() -> Optional[list]: 403 | url = microsoft_2.url 404 | ok, r = downloads[url] 405 | if not ok: 406 | print(f'{microsoft_2.__name__}: {r}', file=sys.stderr) 407 | return 408 | 409 | reader = csv.DictReader(r.splitlines()) 410 | 411 | fingerprints = [] 412 | for row in reader: 413 | status = row['Microsoft Status'] 414 | if status != 'Included': 415 | continue 416 | sha256 = row['SHA-256 Fingerprint'] 417 | trust_bits = row['Microsoft EKUs'].split(';') 418 | if 'Server Authentication' not in trust_bits: 419 | continue 420 | fingerprints.append(sha256) 421 | 422 | return fingerprints 423 | 424 | 425 | @set_url('https://ccadb.my.salesforce-sites.com/ccadb/' 426 | 'AllIncludedRootCertsCSV') 427 | def chrome_0() -> Optional[list]: 428 | url = chrome_0.url 429 | ok, r = downloads[url] 430 | if not ok: 431 | print(f'{chrome_0.__name__}: {r}', file=sys.stderr) 432 | return 433 | 434 | reader = csv.DictReader(r.splitlines()) 435 | 436 | fingerprints = [] 437 | for row in reader: 438 | if websites_trusted('chrome', row): 439 | fingerprints.append(row['SHA-256 Fingerprint']) 440 | 441 | return fingerprints 442 | 443 | 444 | @set_url('https://raw.githubusercontent.com/chromium/chromium/' 445 | 'main/net/data/ssl/chrome_root_store/root_store.certs') 446 | def chrome_1() -> Optional[list]: 447 | url = chrome_1.url 448 | ok, r = downloads[url] 449 | if not ok: 450 | print(f'{chrome_1.__name__}: {r}', file=sys.stderr) 451 | return 452 | 453 | PEM_PATTERN = re.compile( 454 | r'-----BEGIN CERTIFICATE-----(.*?)-----END CERTIFICATE-----', 455 | re.DOTALL 456 | ) 457 | matches = PEM_PATTERN.findall(r) 458 | certs = [ 459 | f'-----BEGIN CERTIFICATE-----{m}-----END CERTIFICATE-----' 460 | for m in matches 461 | ] 462 | 463 | fingerprints = [] 464 | for pem in certs: 465 | fingerprint = pem_cert_fingerprint(pem.encode()) 466 | if fingerprint is not None: 467 | fingerprints.append(fingerprint) 468 | 469 | return fingerprints 470 | 471 | 472 | @set_url('https://ccadb.my.salesforce-sites.com/ccadb/' 473 | 'AllIncludedRootCertsCSV') 474 | def apple_0() -> Optional[list]: 475 | url = apple_0.url 476 | ok, r = downloads[url] 477 | if not ok: 478 | print(f'{apple_0.__name__}: {r}', file=sys.stderr) 479 | return 480 | 481 | reader = csv.DictReader(r.splitlines()) 482 | 483 | fingerprints = [] 484 | for row in reader: 485 | if websites_trusted('apple', row): 486 | fingerprints.append(row['SHA-256 Fingerprint']) 487 | 488 | return fingerprints 489 | 490 | 491 | vendors = { 492 | 'mozilla': [mozilla_0, mozilla_1, mozilla_2], 493 | 'microsoft': [microsoft_0, microsoft_1, microsoft_2], 494 | 'chrome': [chrome_0, chrome_1], 495 | 'apple': [apple_0], 496 | } 497 | vendors['all'] = [x for vendor in vendors 498 | for x in vendors[vendor]] 499 | 500 | 501 | def parse_args(): 502 | parser = argparse.ArgumentParser( 503 | usage='%(prog)s [options]', 504 | description='vendor root CA program analysis') 505 | parser.add_argument('-V', '--vendor', 506 | action='append', 507 | required=True, 508 | choices=vendors.keys(), 509 | help='vendor') 510 | parser.add_argument('-s', '--save', 511 | metavar='DIR', 512 | type=Path, 513 | help='save fingerprints to directory') 514 | # https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv4 515 | parser.add_argument('-c', '--ccadb', 516 | metavar='PATH', 517 | type=Path, 518 | help='CCADB all certificate information CSV path') 519 | parser.add_argument('--verbose', 520 | action='store_true', 521 | help='enable verbosity') 522 | parser.add_argument('--debug', 523 | type=int, 524 | choices=[0, 1, 2, 3], 525 | default=0, 526 | help='enable debug') 527 | x = '%s %s' % (title, __version__) 528 | parser.add_argument('--version', 529 | action='version', 530 | help='display version', 531 | version=x) 532 | args = parser.parse_args() 533 | 534 | if args.debug: 535 | print(args, file=sys.stderr) 536 | 537 | return args 538 | 539 | 540 | if __name__ == '__main__': 541 | main() 542 | -------------------------------------------------------------------------------- /bin/guard.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # 4 | # Copyright (c) 2024 Palo Alto Networks, Inc. 5 | # 6 | # Permission to use, copy, modify, and distribute this software for any 7 | # purpose with or without fee is hereby granted, provided that the above 8 | # copyright notice and this permission notice appear in all copies. 9 | # 10 | # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11 | # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12 | # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13 | # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14 | # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15 | # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16 | # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17 | # 18 | 19 | import argparse 20 | import asyncio 21 | from collections import defaultdict 22 | import logging 23 | import os 24 | import pprint 25 | import re 26 | import sys 27 | import time 28 | from treelib import Node, Tree 29 | import xml.etree.ElementTree as etree 30 | 31 | try: 32 | import pan.xapi 33 | except ImportError: 34 | print('Install pan-python: https://pypi.org/project/pan-python/', 35 | file=sys.stderr) 36 | sys.exit(1) 37 | 38 | libpath = os.path.dirname(os.path.abspath(__file__)) 39 | sys.path[:0] = [os.path.join(libpath, os.pardir)] 40 | 41 | from pan_chainguard import title, __version__ 42 | import pan_chainguard.util 43 | 44 | # XXX keep compatible with sequence based naming for now 45 | NAME_RE = pan_chainguard.util.NAME_RE_COMPAT 46 | 47 | args = None 48 | 49 | 50 | class Xpath(): 51 | def __init__(self, *, 52 | panorama=False, 53 | template=None, 54 | vsys=None): 55 | self.type = 'panorama' if panorama else 'firewall' 56 | if self.type == 'firewall' and template is not None: 57 | raise ValueError('template set for firewall') 58 | self.template = template 59 | self.vsys = vsys 60 | 61 | def __str__(self): 62 | x = self.__dict__ 63 | return ', '.join((': '.join((k, str(x[k])))) 64 | for k in sorted(x)) 65 | 66 | @property 67 | def panorama(self): 68 | return self.type == 'panorama' 69 | 70 | def _variant(self, xpath): 71 | if self.type == 'firewall': 72 | if self.vsys is None: 73 | x = xpath[self.type]['shared'] 74 | else: 75 | x = xpath[self.type]['vsys'] % self.vsys 76 | elif self.type == 'panorama': 77 | if self.template is None: 78 | x = xpath[self.type]['panorama'] 79 | elif self.vsys is None: 80 | x = xpath[self.type]['template']['shared'] % self.template 81 | else: 82 | x = xpath[self.type]['template']['vsys'] % ( 83 | self.template, self.vsys) 84 | else: 85 | assert False, 'Malformed type: %s' % self.type 86 | 87 | return x 88 | 89 | def trusted_root_ca(self): 90 | xpath = { 91 | 'firewall': { 92 | 'shared': '/config/shared/ssl-decrypt/trusted-root-CA', 93 | 'vsys': ("/config/devices/entry" 94 | "[@name='localhost.localdomain']" 95 | "/vsys/entry[@name='%s']/ssl-decrypt" 96 | "/trusted-root-CA") 97 | }, 98 | 99 | 'panorama': { 100 | 'panorama': '/config/panorama/ssl-decrypt/trusted-root-CA', 101 | 'template': { 102 | 'shared': ("/config/devices/entry" 103 | "[@name='localhost.localdomain']" 104 | "/template/entry[@name='%s']/config/shared" 105 | "/ssl-decrypt/trusted-root-CA"), 106 | 'vsys': ("/config/devices/entry" 107 | "[@name='localhost.localdomain']" 108 | "/template/entry[@name='%s']/config/devices" 109 | "/entry[@name='localhost.localdomain']/vsys" 110 | "/entry[@name='%s']/ssl-decrypt/trusted-root-CA") 111 | }, 112 | }, 113 | } 114 | 115 | return self._variant(xpath) 116 | 117 | def certificates(self): 118 | xpath = { 119 | 'firewall': { 120 | 'shared': '/config/shared/certificate', 121 | 'vsys': ("/config/devices/entry[@name='localhost.localdomain']" 122 | "/vsys/entry[@name='%s']/certificate"), 123 | }, 124 | 125 | 'panorama': { 126 | 'panorama': '/config/panorama/certificate', 127 | 'template': { 128 | 'shared': ("/config/devices/entry" 129 | "[@name='localhost.localdomain']" 130 | "/template/entry[@name='%s']" 131 | "/config/shared/certificate"), 132 | 'vsys': ("/config/devices/entry" 133 | "[@name='localhost.localdomain']" 134 | "/template/entry[@name='%s']/config/devices" 135 | "/entry[@name='localhost.localdomain']" 136 | "/vsys/entry[@name='%s']/certificate") 137 | }, 138 | }, 139 | } 140 | 141 | return self._variant(xpath) 142 | 143 | def root_ca_exclude_list(self): 144 | xpath = { 145 | 'firewall': { 146 | 'shared': '/config/shared/ssl-decrypt/root-ca-exclude-list', 147 | 'vsys': ("/config/devices/entry[@name='localhost.localdomain']" 148 | "/vsys/entry[@name='%s']/ssl-decrypt/" 149 | "root-ca-exclude-list"), 150 | }, 151 | 152 | 'panorama': { 153 | 'panorama': ('/config/panorama/ssl-decrypt/' 154 | 'root-ca-exclude-list'), 155 | 'template': { 156 | 'shared': ("/config/devices/entry" 157 | "[@name='localhost.localdomain']" 158 | "/template/entry[@name='%s']" 159 | "/config/shared/ssl-decrypt/" 160 | "root-ca-exclude-list"), 161 | 'vsys': ("/config/devices/entry" 162 | "[@name='localhost.localdomain']" 163 | "/template/entry[@name='%s']/config/devices" 164 | "/entry[@name='localhost.localdomain']" 165 | "/vsys/entry[@name='%s']/ssl-decrypt/" 166 | "root-ca-exclude-list") 167 | }, 168 | }, 169 | } 170 | 171 | return self._variant(xpath) 172 | 173 | 174 | def main(): 175 | global args 176 | args = parse_args() 177 | 178 | logger = logging.getLogger(pan.xapi.__name__) 179 | if args.xdebug == 3: 180 | logger.setLevel(pan.xapi.DEBUG3) 181 | elif args.xdebug == 2: 182 | logger.setLevel(pan.xapi.DEBUG2) 183 | elif args.xdebug == 1: 184 | logger.setLevel(pan.xapi.DEBUG1) 185 | 186 | log_format = '%(name)s: %(message)s' 187 | handler = logging.StreamHandler() 188 | formatter = logging.Formatter(log_format) 189 | handler.setFormatter(formatter) 190 | logger.addHandler(handler) 191 | 192 | asyncio.run(main_loop()) 193 | 194 | sys.exit(0) 195 | 196 | 197 | async def main_loop(): 198 | xapi = None 199 | panorama = False 200 | 201 | try: 202 | xapi = pan.xapi.PanXapi(tag=args.tag, 203 | debug=args.xdebug) 204 | xapi.ad_hoc(modify_qs=True, 205 | qs={'type': 'version'}) 206 | elem = xapi.element_root.find('./result/model') 207 | if elem is not None: 208 | if elem.text == 'Panorama' or elem.text.startswith('M-'): 209 | panorama = True 210 | else: 211 | print("Can't get model", file=sys.stderr) 212 | except pan.xapi.PanXapiError as e: 213 | print('pan.xapi.PanXapi:', e, file=sys.stderr) 214 | sys.exit(1) 215 | 216 | try: 217 | xpath = Xpath(panorama=panorama, 218 | vsys=args.vsys, 219 | template=args.template) 220 | except ValueError as e: 221 | print(e, file=sys.stderr) 222 | sys.exit(1) 223 | 224 | if args.debug > 1: 225 | print('Xpath():', str(xpath), file=sys.stderr) 226 | print(xpath.certificates(), file=sys.stderr) 227 | print(xpath.trusted_root_ca(), file=sys.stderr) 228 | print(xpath.root_ca_exclude_list(), file=sys.stderr) 229 | 230 | if args.show: 231 | show(xapi, xpath) 232 | if args.show_tree: 233 | show_tree(xapi, xpath) 234 | 235 | # experimental 236 | if args.enable_trusted: 237 | enable_trusted(xapi, xpath) 238 | 239 | # experimental 240 | if args.disable_trusted: 241 | disable_trusted(xapi, xpath) 242 | 243 | if args.update_trusted: 244 | update_trusted_root_cas(xapi, xpath, quiet=False) 245 | 246 | if args.delete: 247 | delete_certs(xapi, xpath) 248 | 249 | if args.update: 250 | if args.certs is None: 251 | print('--certs argument required', file=sys.stderr) 252 | sys.exit(1) 253 | if args.type is None: 254 | print('--type argument required', file=sys.stderr) 255 | sys.exit(1) 256 | 257 | update_certs(xapi, xpath) 258 | 259 | if args.commit: 260 | commit(xapi, panorama) 261 | 262 | 263 | def exclude_cert(sha256): 264 | EXCLUDE = [ 265 | # This root is in microsoft root store and is PAN-OS predefined 266 | # 0216_CCA_India_2015_SPL. 267 | # Expires Jan 29 11:36:43 2025 GMT. 268 | # 'openssl verify -check_ss_sig' fails with bad signature 269 | # PAN-257401 270 | 'C34C5DF53080078FFE45B21A7F600469917204F4F0293F1D7209393E5265C04F', 271 | # XXX Signature Algorithm: rsassaPss 272 | '233525D6E906A9B99176204E3C2B4FBF5CEE03F2D126B2E64428BDF97CBC6138', 273 | ] 274 | 275 | if sha256 in EXCLUDE: 276 | if args.debug: 277 | print('Skip problem certificate %s' % sha256, file=sys.stderr) 278 | return True 279 | 280 | return False 281 | 282 | 283 | def show_disabled_trusted(xapi, xpath): 284 | kwargs = {'xpath': xpath.root_ca_exclude_list()} 285 | api_request(xapi, xapi.get, kwargs, 'success', ['7', '19']) 286 | disabled = xapi.element_root.findall( 287 | './result/root-ca-exclude-list/member') 288 | 289 | if len(disabled): 290 | print('Warning: %d Default Trusted CAs are disabled; ' 291 | 'to enable run guard.py --enable-trusted' % len(disabled)) 292 | 293 | 294 | def enable_trusted(xapi, xpath): 295 | kwargs = {'xpath': xpath.root_ca_exclude_list()} 296 | api_request(xapi, xapi.get, kwargs, 'success', ['7', '19']) 297 | disabled = xapi.element_root.findall( 298 | './result/root-ca-exclude-list/member') 299 | 300 | if args.dry_run: 301 | kwargs2 = {'xpath': '/config/predefined/trusted-root-ca'} 302 | api_request(xapi, xapi.get, kwargs2, 'success', '19') 303 | total = xapi.element_root.findall( 304 | './result/trusted-root-ca/entry') 305 | 306 | print('enable-trusted dry-run: %d default trusted root CAs,' 307 | ' %d are disabled, %d to enable' % ( 308 | len(total), len(disabled), len(disabled))) 309 | return 310 | 311 | if len(disabled): 312 | api_request(xapi, xapi.delete, kwargs, 'success', '20') 313 | 314 | print('%d default trusted root CAs enabled' % len(disabled)) 315 | 316 | 317 | def disable_trusted(xapi, xpath): 318 | kwargs = {'xpath': '/config/predefined/trusted-root-ca'} 319 | api_request(xapi, xapi.get, kwargs, 'success', '19') 320 | entries = xapi.element_root.findall('./result/trusted-root-ca/entry') 321 | 322 | kwargs = {'xpath': xpath.root_ca_exclude_list()} 323 | api_request(xapi, xapi.get, kwargs, 'success', ['7', '19']) 324 | disabled = xapi.element_root.findall( 325 | './result/root-ca-exclude-list/member') 326 | 327 | if args.dry_run: 328 | enabled = len(entries) - len(disabled) 329 | print('disable-trusted dry-run: %d default trusted root CAs,' 330 | ' %d are enabled, %d to disable' % ( 331 | len(entries), enabled, enabled)) 332 | return 333 | 334 | disabled_ = [] 335 | for entry in disabled: 336 | name = entry.text 337 | disabled_.append(name) 338 | 339 | members = [] 340 | for entry in entries: 341 | name = entry.attrib['name'] 342 | if name in disabled_: 343 | continue 344 | member = '%s' % name 345 | members.append(member) 346 | 347 | if args.verbose: 348 | print('disabling', name, file=sys.stderr) 349 | 350 | if members: 351 | root_ca_exclude = xpath.root_ca_exclude_list() 352 | element = ''.join(members) 353 | 354 | kwargs = { 355 | 'xpath': root_ca_exclude, 356 | 'element': element, 357 | } 358 | api_request(xapi, xapi.set, kwargs, 'success', '20') 359 | 360 | print('%d default trusted root CAs disabled' % len(members)) 361 | 362 | 363 | def get_certs(xapi, xpath): 364 | certificates = xpath.certificates() 365 | kwargs = {'xpath': certificates} 366 | api_request(xapi, xapi.get, kwargs, 'success', ['7', '19']) 367 | if xapi.status_code == '7': 368 | return [] 369 | 370 | entries = xapi.element_root.findall('./result/certificate/entry') 371 | 372 | data = {} 373 | prog = re.compile(NAME_RE) 374 | progcn = re.compile(r'/CN=([^/]+)') 375 | 376 | for entry in entries: 377 | name = entry.attrib['name'] 378 | if prog.search(name): 379 | subject = entry.find('./subject').text 380 | subject_cn = None 381 | match = progcn.search(subject) 382 | if match: 383 | subject_cn = match.group(1) 384 | issuer = entry.find('./issuer').text 385 | issuer_cn = None 386 | match = progcn.search(issuer) 387 | if match: 388 | issuer_cn = match.group(1) 389 | expiry = entry.find('./expiry-epoch').text 390 | try: 391 | if time.time() > int(expiry): 392 | expired = True 393 | else: 394 | expired = False 395 | except ValueError as e: 396 | print('%s expiry-epoch %s: %s' % ( 397 | name, expiry, e), file=sys.stderr) 398 | 399 | v = { 400 | 'cert-name': name, 401 | 'subject': subject, 402 | 'subject-cn': subject_cn, 403 | 'subject-hash': entry.find('./subject-hash').text, 404 | 'issuer': issuer, 405 | 'issuer-cn': issuer_cn, 406 | 'issuer-hash': entry.find('./issuer-hash').text, 407 | 'expired': expired, 408 | } 409 | data[name] = v 410 | 411 | return data 412 | 413 | 414 | def delete_certs(xapi, xpath): 415 | data = get_certs(xapi, xpath) 416 | 417 | if args.dry_run: 418 | print('delete dry-run: %d to delete' % len(data)) 419 | return 420 | 421 | for name in data: 422 | delete_cert(xapi, xpath, name) 423 | 424 | print('%d certificates deleted' % len(data)) 425 | 426 | 427 | def delete_cert(xapi, xpath, name): 428 | rootca = xpath.trusted_root_ca() 429 | member = "/member[text()='%s']" % name 430 | kwargs = {'xpath': rootca + member} 431 | api_request(xapi, xapi.delete, kwargs, 'success', ['7', '20']) 432 | 433 | certificates = xpath.certificates() 434 | entry = "/entry[@name='%s']" % name 435 | kwargs = {'xpath': certificates + entry} 436 | # XXX can return status_code 7 intermittently; workaround is 437 | # to re-run 438 | api_request(xapi, xapi.delete, kwargs, 'success', '20') 439 | if args.verbose: 440 | print('deleted', name, file=sys.stderr) 441 | 442 | 443 | def get_trusted_root_cas(xapi, xpath): 444 | trusted_root_ca = xpath.trusted_root_ca() 445 | kwargs = {'xpath': trusted_root_ca} 446 | api_request(xapi, xapi.get, kwargs, 'success', ['7', '19']) 447 | 448 | prog = re.compile(NAME_RE) 449 | data = [] 450 | if xapi.status_code == '19': 451 | entries = xapi.element_root.findall( 452 | './result/trusted-root-CA/member') 453 | for entry in entries: 454 | name = entry.text 455 | if prog.search(name): 456 | data.append(name) 457 | 458 | return data 459 | 460 | 461 | def update_trusted_root_cas(xapi, xpath, quiet=True): 462 | cert_names = get_certs(xapi, xpath) 463 | add = [] 464 | 465 | if cert_names: 466 | data = get_trusted_root_cas(xapi, xpath) 467 | if data: 468 | for name in cert_names: 469 | if name not in data: 470 | add.append(name) 471 | else: 472 | add.extend(cert_names) 473 | 474 | if args.dry_run: 475 | print('update-trusted dry-run: %d certificates' 476 | ' to enable as trusted root CAs' % len(add)) 477 | return 478 | 479 | if add: 480 | add_trusted_root_cas(xapi, xpath, add) 481 | 482 | if not quiet: 483 | print('%d certificates enabled as trusted root CA' % len(add)) 484 | 485 | 486 | def add_trusted_root_cas(xapi, xpath, cert_names): 487 | members = [] 488 | 489 | for x in cert_names: 490 | members.append('%s' % x) 491 | 492 | trusted_root_ca = xpath.trusted_root_ca() 493 | element = ''.join(members) 494 | 495 | kwargs = { 496 | 'xpath': trusted_root_ca, 497 | 'element': element, 498 | } 499 | 500 | api_request(xapi, xapi.set, kwargs, 'success', '20') 501 | 502 | 503 | def update_certs(xapi, xpath): 504 | old = get_certs(xapi, xpath) 505 | 506 | new = {} 507 | try: 508 | data = pan_chainguard.util.read_cert_archive( 509 | path=args.certs) 510 | except pan_chainguard.util.UtilError as e: 511 | print(str(e), file=sys.stderr) 512 | sys.exit(1) 513 | 514 | for sha256 in data: 515 | if exclude_cert(sha256): 516 | continue 517 | cert_type, content = data[sha256] 518 | if cert_type not in args.type: 519 | continue 520 | cert_name = pan_chainguard.util.hash_to_name(sha256=sha256) 521 | new[cert_name] = content 522 | 523 | old_set = set(old) 524 | new_set = set(new.keys()) 525 | 526 | if args.debug > 2: 527 | print('old', pprint.pformat(old_set), file=sys.stderr) 528 | print('new', pprint.pformat(new_set), file=sys.stderr) 529 | 530 | delete = list(old_set - new_set) 531 | add = list(new_set - old_set) 532 | 533 | if args.debug > 1: 534 | print('update delete', pprint.pformat(delete), file=sys.stderr) 535 | print('update add', pprint.pformat(add), file=sys.stderr) 536 | 537 | if args.dry_run: 538 | print('update dry-run: %d to delete, %d to add' % ( 539 | len(delete), len(add))) 540 | return 541 | 542 | for name in delete: 543 | delete_cert(xapi, xpath, name) 544 | print('%d certificates deleted' % len(delete)) 545 | 546 | total = 0 547 | for name in add: 548 | if add_cert(xapi, xpath, name, new[name]): 549 | total += 1 550 | 551 | if total: 552 | add_trusted_root_cas(xapi, xpath, add_cert.cert_names) 553 | 554 | print('%d certificates added' % total) 555 | 556 | 557 | def add_cert(xapi, xpath, cert_name, content): 558 | kwargs = { 559 | 'category': 'certificate', 560 | 'file': content, 561 | 'extra_qs': { 562 | 'certificate-name': cert_name, 563 | 'format': 'pem', 564 | }, 565 | } 566 | 567 | if xpath.panorama: 568 | if args.template is not None: 569 | kwargs['extra_qs']['target-tpl'] = args.template 570 | if args.vsys is not None: 571 | # XXX does not work; PAN-257229 572 | kwargs['extra_qs']['target-tpl-vsys'] = args.vsys 573 | elif args.vsys is not None: 574 | kwargs['vsys'] = args.vsys 575 | 576 | SKIP_ERRORS = [ 577 | 'Certificate is expired', 578 | 'Unsupported digest or keys used in FIPS-CC mode', 579 | ] 580 | 581 | try: 582 | xapi.import_file(**kwargs) 583 | except pan.xapi.PanXapiError as e: 584 | for error in SKIP_ERRORS: 585 | if error in str(e): 586 | if args.verbose: 587 | print('%s skipped: %s' % (cert_name, e), 588 | file=sys.stderr) 589 | return False 590 | 591 | print('%s: %s: %s' % (xapi.import_file.__name__, kwargs, e), 592 | file=sys.stderr) 593 | sys.exit(1) 594 | 595 | # Use function attribute to cache certificate names so we can use 596 | # a single API request to enable them as trusted root CAs. 597 | try: 598 | add_cert.cert_names.append(cert_name) 599 | except AttributeError: 600 | add_cert.cert_names = [cert_name] 601 | 602 | if args.verbose: 603 | print('added %s' % cert_name, file=sys.stderr) 604 | 605 | return True 606 | 607 | 608 | def show(xapi, xpath): 609 | data = get_certs(xapi, xpath) 610 | out = [] 611 | num_expired = 0 612 | 613 | for x in data: 614 | expired = '' 615 | if data[x]['expired']: 616 | num_expired += 1 617 | expired = ' (expired)' 618 | if data[x]['subject'] == data[x]['issuer']: 619 | # Root 620 | # XXX subject == issuer can be an intermediate 621 | type_ = 'R' 622 | else: 623 | # Intermediate 624 | type_ = 'I' 625 | m = '%s %s%s Subject: "%s" Issuer: "%s"' % ( 626 | x, type_, expired, 627 | data[x]['subject'], data[x]['issuer']) 628 | out.append(m) 629 | 630 | expired = '' 631 | if num_expired: 632 | expired = ' (%d expired)' % num_expired 633 | print('%d Device Certificates%s' % (len(out), expired)) 634 | if out and args.verbose: 635 | print('\n'.join(out)) 636 | 637 | if out: 638 | data = get_trusted_root_cas(xapi, xpath) 639 | num = len(data) 640 | print('%d Trusted Root CA Certificates' % num) 641 | if num < len(out): 642 | print('Warning: %d certificates not trusted; ' 643 | 'run guard.py --update-trusted' % (len(out) - num)) 644 | 645 | show_disabled_trusted(xapi, xpath) 646 | 647 | 648 | def duplicates_in_path(tree: Tree) -> list[dict]: 649 | def dfs(nid: str, seen: set[str], out: list[dict]) -> None: 650 | new_seen = set(seen) 651 | node = tree.get_node(nid) 652 | if node.tag != 'Root': 653 | key = node.data['subject-hash'] 654 | 655 | if key in seen: 656 | out.append({ 657 | 'node-id': nid, 658 | 'tag': str(node.tag), 659 | 'key': key, 660 | 'node-data': node.data, 661 | }) 662 | return # stop this root-to-leaf path here 663 | 664 | else: 665 | new_seen.add(key) 666 | 667 | children = tree.children(nid) 668 | if not children: 669 | return # leaf reached, no dup on this path 670 | 671 | for child in children: 672 | dfs(child.identifier, new_seen, out) 673 | 674 | duplicates = [] 675 | dfs(tree.root, set(), duplicates) 676 | 677 | return duplicates 678 | 679 | 680 | def show_tree(xapi, xpath): 681 | data = get_certs(xapi, xpath) 682 | issuers = defaultdict(list) 683 | subjects = defaultdict(list) 684 | roots = [] 685 | 686 | for x in data: 687 | issuers[data[x]['issuer-hash']].append(data[x]) 688 | subjects[data[x]['subject-hash']].append(data[x]) 689 | if data[x]['subject-hash'] == data[x]['issuer-hash']: 690 | roots.append(data[x]) 691 | 692 | orphans = [] 693 | 694 | for x in data: 695 | if data[x]['issuer-hash'] not in subjects: 696 | orphans.append(data[x]) 697 | 698 | if args.debug: 699 | print('issuers', pprint.pformat(issuers), file=sys.stderr) 700 | print('subjects', pprint.pformat(subjects), file=sys.stderr) 701 | print('roots', pprint.pformat(roots), file=sys.stderr) 702 | print('orphans', pprint.pformat(orphans), file=sys.stderr) 703 | 704 | tree = Tree() 705 | tree.add_node(Node(tag='Root', identifier=0)) 706 | 707 | def build_node(x): 708 | subject = x['subject-cn'] if x['subject-cn'] else x['subject'] 709 | issuer = x['issuer-cn'] if x['issuer-cn'] else x['issuer'] 710 | tag = (f"{x['cert-name']} " 711 | f'Subject: "{subject}" ' 712 | f'Issuer: "{issuer}"') 713 | 714 | node = Node(tag=tag, identifier=x['cert-name'], data=x) 715 | 716 | return node 717 | 718 | def add_children(parent, issuer): 719 | if issuer in issuers: 720 | for x in issuers[issuer]: 721 | if (tree.contains(x['cert-name']) or 722 | x['subject-hash'] == x['issuer-hash']): 723 | continue 724 | 725 | tree.add_node(build_node(x), parent=parent) 726 | add_children(x['cert-name'], x['subject-hash']) 727 | 728 | def format_stats(tree): 729 | stats = pan_chainguard.util.stats_from_tree(tree=tree) 730 | 731 | for k, v in stats.items(): 732 | name = k.replace('_', ' ').title() 733 | value = '%.4f' % v if isinstance(v, float) else '%d' % v 734 | print('%s: %s' % (name, value)) 735 | 736 | for x in roots + orphans: 737 | tree.add_node(build_node(x), parent=0) 738 | add_children(x['cert-name'], x['subject-hash']) 739 | 740 | print(tree.show(stdout=False), end='') 741 | 742 | if args.verbose: 743 | format_stats(tree) 744 | 745 | duplicates = duplicates_in_path(tree) 746 | if args.verbose and duplicates: 747 | print(f'Info: {len(duplicates)} duplicate subject in tree path') 748 | for x in duplicates: 749 | print(f"{x['key']} {x['tag']}") 750 | 751 | treelen = len(tree) - 1 # don't count root node 752 | sublen = 0 753 | for x in subjects: 754 | sublen += len(subjects[x]) 755 | if treelen != sublen: 756 | print('Error: tree length != number of certificates:' 757 | ' %d != %d' % (treelen, sublen)) 758 | 759 | duplicates = [] 760 | for x in subjects: 761 | if len(subjects[x]) > 1: 762 | duplicates.append(subjects[x][0]['subject']) 763 | 764 | if args.verbose and duplicates: 765 | print('Info: %d duplicate certificate subjects' % 766 | len(duplicates)) 767 | print('\n'.join(duplicates)) 768 | 769 | 770 | def commit(xapi, panorama): 771 | root = etree.Element('commit') 772 | partial = etree.SubElement(root, 'partial') 773 | desc = etree.SubElement(partial, 'description') 774 | desc.text = '%s %s' % (title, __version__) 775 | if args.admin: 776 | admin = etree.SubElement(partial, 'admin') 777 | admin_member = etree.SubElement(admin, 'member') 778 | admin_member.text = args.admin 779 | 780 | policy_and_objects = etree.Element('policy-and-objects') 781 | policy_and_objects.text = 'excluded' 782 | device_and_network = etree.Element('device-and-network') 783 | device_and_network.text = 'excluded' 784 | shared_object = etree.Element('shared-object') 785 | shared_object.text = 'excluded' 786 | 787 | if panorama: 788 | if args.template: 789 | # commit scope: template 790 | # no template vsys scope 791 | template = etree.SubElement(partial, 'template') 792 | template_member = etree.SubElement(template, 'member') 793 | template_member.text = args.template 794 | else: 795 | # commit scope: device-and-network 796 | partial.append(shared_object) 797 | else: 798 | # firewall 799 | if args.vsys: 800 | # commit scope: vsys 801 | vsys = etree.SubElement(partial, 'vsys') 802 | vsys_member = etree.SubElement(vsys, 'member') 803 | vsys_member.text = args.vsys 804 | partial.append(device_and_network) 805 | partial.append(shared_object) 806 | else: 807 | # commit scope: shared-object 808 | partial.append(device_and_network) 809 | partial.append(policy_and_objects) 810 | 811 | cmd = etree.tostring(root).decode() 812 | if args.debug: 813 | print(cmd, file=sys.stderr) 814 | 815 | kwargs = { 816 | 'cmd': cmd, 817 | 'sync': True, 818 | } 819 | 820 | if args.dry_run: 821 | return 822 | 823 | if args.verbose: 824 | print('commit config for admin %s' % args.admin) 825 | 826 | api_request(xapi, xapi.commit, kwargs, 'success') 827 | if args.debug: 828 | print(xapi.xml_root(), file=sys.stderr) 829 | 830 | if xapi.status_code is not None: 831 | code = ' [code=\"%s\"]' % xapi.status_code 832 | else: 833 | code = '' 834 | if xapi.status is not None: 835 | print('commit: %s%s' % (xapi.status, code), end='') 836 | if args.verbose and xapi.status_detail is not None: 837 | print(': "%s"' % xapi.status_detail.rstrip(), end='') 838 | print() 839 | 840 | 841 | def api_request(xapi, func, kwargs, status=None, status_code=None): 842 | try: 843 | func(**kwargs) 844 | except pan.xapi.PanXapiError as e: 845 | print('%s: %s: %s' % (func.__name__, kwargs, e), 846 | file=sys.stderr) 847 | sys.exit(1) 848 | 849 | status_detail = (' "%s"' % xapi.status_detail 850 | if xapi.status_detail is not None 851 | else '') 852 | if (status is not None and 853 | not pan_chainguard.util.s1_in_s2(xapi.status, status)): 854 | print('%s: %s:%s status %s != %s' % 855 | (func.__name__, kwargs, xapi.status_detail, 856 | xapi.status, status), 857 | file=sys.stderr) 858 | sys.exit(1) 859 | 860 | if (status_code is not None and 861 | not pan_chainguard.util.s1_in_s2(xapi.status_code, status_code)): 862 | print('%s: %s:%s status_code %s != %s' % 863 | (func.__name__, kwargs, status_detail, 864 | xapi.status_code, status_code), 865 | file=sys.stderr) 866 | sys.exit(1) 867 | 868 | 869 | def parse_args(): 870 | def check_vsys(x): 871 | if x.isdigit(): 872 | x = 'vsys' + x 873 | return x 874 | 875 | parser = argparse.ArgumentParser( 876 | usage='%(prog)s [options]', 877 | description='update PAN-OS trusted CAs') 878 | parser.add_argument('--tag', '-t', 879 | required=True, 880 | help='.panrc tagname') 881 | parser.add_argument('--vsys', 882 | type=check_vsys, 883 | help='vsys name or number') 884 | parser.add_argument('--template', 885 | help='Panorama template') 886 | parser.add_argument('--certs', 887 | metavar='PATH', 888 | help='certificate archive path') 889 | parser.add_argument('--update', 890 | action='store_true', 891 | help='update certificates') 892 | parser.add_argument('--delete', 893 | action='store_true', 894 | help='delete all previously added certificates') 895 | parser.add_argument('-T', '--type', 896 | action='append', 897 | choices=['root', 'intermediate'], 898 | help='certificate type(s) for update') 899 | # XXX experimental 900 | # We don't want to have a condition where default trusted CAs 901 | # are disabled but there are no replacement root CAs. Better 902 | # to just keep default trusted CAs enabled. 903 | parser.add_argument('--disable-trusted', 904 | action='store_true', 905 | help=argparse.SUPPRESS) 906 | # help='disable all default trusted root CAs') 907 | parser.add_argument('--enable-trusted', 908 | action='store_true', 909 | help=argparse.SUPPRESS) 910 | # help='enable all default trusted root CAs') 911 | parser.add_argument('--update-trusted', 912 | action='store_true', 913 | help='update trusted root CA for all certificates') 914 | parser.add_argument('--commit', 915 | action='store_true', 916 | help='commit configuration') 917 | parser.add_argument('--dry-run', 918 | action='store_true', 919 | help="don't update PAN-OS") 920 | parser.add_argument('--show', 921 | action='store_true', 922 | help='show %s managed config' % title) 923 | parser.add_argument('--show-tree', 924 | action='store_true', 925 | help='show %s managed certificates in tree format' % 926 | title) 927 | parser.add_argument('--admin', 928 | help='commit admin') 929 | parser.add_argument('--xdebug', 930 | type=int, 931 | choices=[0, 1, 2, 3], 932 | default=0, 933 | help='pan.xapi debug') 934 | parser.add_argument('--verbose', 935 | action='store_true', 936 | help='enable verbosity') 937 | parser.add_argument('--debug', 938 | type=int, 939 | choices=[0, 1, 2, 3], 940 | default=0, 941 | help='enable debug') 942 | x = '%s %s' % (title, __version__) 943 | parser.add_argument('--version', 944 | action='version', 945 | help='display version', 946 | version=x) 947 | args = parser.parse_args() 948 | 949 | if args.debug: 950 | print(args, file=sys.stderr) 951 | 952 | return args 953 | 954 | 955 | if __name__ == '__main__': 956 | main() 957 | -------------------------------------------------------------------------------- /doc/admin-guide.rst: -------------------------------------------------------------------------------- 1 | .. 2 | Copyright (c) 2024 Palo Alto Networks, Inc. 3 | 4 | Permission to use, copy, modify, and distribute this software for any 5 | purpose with or without fee is hereby granted, provided that the above 6 | copyright notice and this permission notice appear in all copies. 7 | 8 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 9 | WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 10 | MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 11 | ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 12 | WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 13 | ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 14 | OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 15 | 16 | pan-chainguard Administrator's Guide 17 | ==================================== 18 | 19 | .. contents:: 20 | 21 | Overview 22 | -------- 23 | 24 | ``pan-chainguard`` is a Python application which uses 25 | `CCADB data 26 | `_ 27 | and allows PAN-OS SSL decryption administrators to: 28 | 29 | #. Create a custom, up-to-date trusted root store for PAN-OS. 30 | #. Determine intermediate certificate chains for trusted Certificate 31 | Authorities in PAN-OS so they can be `preloaded 32 | `_ 33 | as device certificates. 34 | 35 | Issue 1: Out-of-date Root Store 36 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 37 | 38 | The PAN-OS root store (*Default Trusted Certificate Authorities*) is 39 | updated only in PAN-OS major software releases; it is not currently 40 | managed by content updates. The root store for PAN-OS 10.x releases 41 | is now over 5 years old. 42 | 43 | The impact for PAN-OS SSL decryption administrators is when the root 44 | CA for the server certificate is not trusted, the firewall will 45 | provide the forward untrust certificate to the client. End users will 46 | then see errors such as *NET::ERR_CERT_AUTHORITY_INVALID* (Chrome) or 47 | *SEC_ERROR_UNKNOWN_ISSUER* (Firefox) until the missing trusted CAs are 48 | identified, the certificates are obtained, and the certificates are 49 | imported into PAN-OS. 50 | 51 | Issue 2: Misconfigured Servers 52 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 53 | 54 | Many TLS enabled origin servers suffer from a misconfiguration in 55 | which they: 56 | 57 | #. Do not return intermediate CA certificates. 58 | #. Return certificates out of order. 59 | #. Return intermediate certificates which are not related to the root 60 | CA for the server certificate. 61 | 62 | The impact for PAN-OS SSL decryption administrators is end users will 63 | see errors such as *unable to get local issuer certificate* until the 64 | sites that are misconfigured are 65 | `identified 66 | `_, 67 | the required intermediate certificates are obtained, and the 68 | certificates are imported into PAN-OS. 69 | 70 | Solution 1: Create Custom Root Store 71 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 72 | 73 | ``pan-chainguard`` can create a custom root store, using one or more 74 | of the major vendor root stores, which are managed by their CA 75 | certificate program: 76 | 77 | + `Mozilla `_ 78 | + `Apple `_ 79 | + `Chrome `_ 80 | + `Microsoft `_ 81 | 82 | The custom root store can then be added to PAN-OS as trusted CA device 83 | certificates. 84 | 85 | Solution 2: Intermediate CA Preloading 86 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 87 | 88 | ``pan-chainguard`` uses a root store and the 89 | *All Certificate Information (root and intermediate) in CCADB (CSV)* 90 | data file as input, and determines the intermediate certificate 91 | chains, if available, for each root CA certificate. These can then be 92 | added to PAN-OS as trusted CA device certificates. 93 | 94 | By preloading known intermediates for the trusted CAs, the number of 95 | TLS connection errors that users encounter for misconfigured servers 96 | can be reduced, without reactive actions by an administrator. 97 | 98 | AIA Fetching 99 | ............ 100 | 101 | Another approach used is AIA fetching, or AIA chasing, which uses the 102 | *CA Issuers* field in the *Authority Information Access* X509v3 103 | extension of the server certificate to obtain missing issuer 104 | certificates. This discloses a source IP address to the CA that 105 | issued the server certificate, which may be considered a privacy 106 | concern. There will also be connection delays for the certificate 107 | download. Intermediate CA preloading does not have these issues. AIA 108 | fetching is reactive, based upon what server certificates are seen; 109 | intermediate preloading as performed by ``pan-chainguard`` is 110 | proactive and uses a known trusted CA store as its starting point. 111 | 112 | pan-chainguard 113 | -------------- 114 | 115 | Install pan-chainguard 116 | ~~~~~~~~~~~~~~~~~~~~~~ 117 | 118 | The ``pan-chainguard`` source repository is hosted on GitHub at: 119 | `https://github.com/PaloAltoNetworks/pan-chainguard 120 | `_. 121 | 122 | It requires the following Python packages: 123 | 124 | + `aiohttp `_ 125 | + `pan-python `_ 126 | + `treelib `_ 127 | 128 | ``pan-chainguard`` should run on any Unix system with Python 3.9 or 129 | greater, and OpenSSL or LibreSSL; it has been tested on OpenBSD 7.6, 130 | Ubuntu 22.04 and 24.04, and macOS 14. 131 | 132 | Get pan-chainguard using ``git clone`` 133 | ...................................... 134 | 135 | :: 136 | 137 | $ python3 -m pip install aiohttp 138 | 139 | $ python3 -m pip install pan-python 140 | 141 | $ python3 -m pip install treelib 142 | 143 | $ git clone https://github.com/PaloAltoNetworks/pan-chainguard.git 144 | 145 | $ cd pan-chainguard 146 | 147 | $ bin/chain.py --version 148 | pan-chainguard 0.6.0 149 | 150 | $ bin/guard.py --version 151 | pan-chainguard 0.6.0 152 | 153 | Install pan-chainguard using ``pip`` 154 | .................................... 155 | 156 | :: 157 | 158 | $ python3 -m pip install pan-chainguard 159 | 160 | $ chain.py --version 161 | pan-chainguard 0.6.0 162 | 163 | $ guard.py --version 164 | pan-chainguard 0.6.0 165 | 166 | pan-chainguard Command Line Programs 167 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 168 | 169 | ``pan-chainguard`` provides 6 Python command line programs and a shell 170 | script: 171 | 172 | - ``fling.py`` - Deprecated 173 | 174 | Command line program which exports the PEM encoded X.509 175 | certificates from the PAN-OS Default Trusted CA store. 176 | 177 | - ``cert-fingerprints.sh`` - Deprecated 178 | 179 | A shell script which takes as input the X.509 certificates 180 | exported by ``fling.py`` and creates a CSV file containing 181 | the SHA-256 fingerprint for each certificate. 182 | 183 | - ``sprocket.py`` 184 | 185 | Command line program which creates a custom root store according a 186 | user-defined policy. 187 | 188 | - ``chain.py`` 189 | 190 | Command line program which takes as input: 191 | 192 | + The root CA fingerprint CSV file created by 193 | ``cert-fingerprints.sh`` or ``sprocket.py`` 194 | 195 | + The *All Certificate Information (root and 196 | intermediate) in CCADB* CSV file (`AllCertificateRecordsCSVFormatv4 197 | `_) 198 | 199 | + Optional: The *Intermediate CA Certificates in OneCRL* CSV 200 | file (`IntermediateCertsInOneCRL 201 | `_) 202 | 203 | and creates: 204 | 205 | + A CSV file containing the fingerprints of the intermediate 206 | certificate chains found for the CAs in the root store 207 | 208 | + A JSON file containing the tree representation of the root 209 | and intermediate certificates 210 | 211 | - ``chainring.py`` 212 | 213 | Command line program which takes as input the JSON file created by 214 | ``chain.py`` and creates multiple representations of the certificate 215 | tree, including HTML and text. 216 | 217 | - ``link.py`` 218 | 219 | Command line program which obtains PEM encoded X.509 certificates 220 | from different sources including: 221 | 222 | + Mozilla certificates with PEM CSV files 223 | + Old (previous) certificate archive 224 | + crt.sh API 225 | 226 | - ``guard.py`` 227 | 228 | Command line program which takes as input the certificate archive 229 | created by ``link.py`` and imports the certificates (root and 230 | intermediate) as trusted CA device certificates on PAN-OS. 231 | 232 | Command options can be displayed using ``--help`` (e.g., 233 | ``chain.py --help``). 234 | 235 | Data and Process Flow 236 | ..................... 237 | 238 | A `data and process flow diagram 239 | `_ 240 | illustrates the programs, execution sequence, and data inputs and 241 | outputs. 242 | 243 | .. _panrc: 244 | 245 | pan-chainguard PAN-OS XML API Usage 246 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 247 | 248 | ``fling.py`` and ``guard.py`` use the `pan.xapi module 249 | `_ 250 | to make configuration updates. 251 | 252 | A `.panrc file 253 | `_ 254 | is used to specify the hostname and API key for the PAN-OS XML API. 255 | A `short tutorial 256 | `_ is available 257 | to assist with the creation of an API key and .panrc file. 258 | 259 | Role Based Admin 260 | ................ 261 | 262 | As a best practice it is recommended to use an application specific 263 | role based admin for the XML API operations. The following PAN-OS 264 | firewall configuration creates a ``chainguard-api`` admin role profile 265 | and ``chainguard`` admin:: 266 | 267 | set shared admin-role chainguard-api role device xmlapi config enable 268 | set shared admin-role chainguard-api role device xmlapi op enable 269 | set shared admin-role chainguard-api role device xmlapi commit enable 270 | set shared admin-role chainguard-api role device xmlapi export enable 271 | set shared admin-role chainguard-api role device xmlapi import enable 272 | set shared admin-role chainguard-api role device webui 273 | set shared admin-role chainguard-api role device restapi 274 | 275 | set mgt-config users chainguard permissions role-based custom profile chainguard-api 276 | set mgt-config users chainguard password 277 | 278 | .. note:: Also ensure access to all *Web UI* (webui) and *REST API* 279 | (restapi) features are disabled. 280 | 281 | .. note:: Operational requests are needed because a synchronous commit 282 | is used which requires ``show jobs id id-num`` to poll for 283 | job completion. 284 | 285 | The admin role profile for Panorama:: 286 | 287 | set shared admin-role chainguard-api role panorama xmlapi config enable 288 | set shared admin-role chainguard-api role panorama xmlapi op enable 289 | set shared admin-role chainguard-api role panorama xmlapi commit enable 290 | set shared admin-role chainguard-api role panorama xmlapi export enable 291 | set shared admin-role chainguard-api role panorama xmlapi import enable 292 | set shared admin-role chainguard-api role panorama webui 293 | set shared admin-role chainguard-api role panorama restapi 294 | 295 | When using ``guard.py`` to commit the configuration, the ``--admin`` 296 | option should be used to specify the ``pan-chainguard`` specific admin 297 | to guarantee only changes made by the admin are committed. 298 | 299 | Running pan-chainguard 300 | ---------------------- 301 | 302 | Identify Source Root Store 303 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 304 | 305 | ``pan-chainguard`` can use a root store from PAN-OS or a custom 306 | root store as input. 307 | 308 | PAN-OS Root Store Updates 309 | ......................... 310 | 311 | The PAN-OS root store (*Default Trusted Certificate Authorities*) is 312 | updated as part of a PAN-OS major software releases; it is not 313 | currently managed by content updates. 314 | 315 | The root store was updated for PAN-OS 10.0, which was released in 316 | July 2020. All 10.x.x releases contain the same root store (10.0.x, 317 | 10.1.x and 10.2.x). 318 | 319 | The root store was updated for PAN-OS 11.0, which was released in 320 | November 2022. All 11.x.x releases contain the same root store 321 | (11.0.x, 11.1.x and 11.2.x). 322 | 323 | The root store was not updated for PAN-OS 12.1.2, which was released 324 | in August 2025. 12.1.2 contains the 11.0 root store from 325 | November 2022. 326 | 327 | To use a PAN-OS root store, run the ``fling.py`` program as described 328 | below. 329 | 330 | Custom Root Store 331 | ................. 332 | 333 | You can create a custom root store, using one or more of the 334 | major vendor root stores, which are managed by their CA certificate 335 | program: 336 | 337 | + `Mozilla `_ 338 | + `Apple `_ 339 | + `Chrome `_ 340 | + `Microsoft `_ 341 | 342 | To use a custom root store, run the ``sprocket.py`` program as 343 | described below. 344 | 345 | sprocket.py 346 | ~~~~~~~~~~~ 347 | 348 | ``sprocket.py`` is used to create a custom root store using the 349 | following policy attributes: 350 | 351 | #. Source vendor root store (one or more) 352 | 353 | + mozilla (default) 354 | + apple 355 | + chrome 356 | + microsoft 357 | 358 | #. Set operation to use when combining multiple source sets 359 | 360 | + union - set of elements which are in any (default) 361 | + intersection - set of elements which are in all 362 | 363 | #. "Trust Bits for Root Cert" field from CCADB 364 | 365 | + CLIENT_AUTHENTICATION 366 | + CODE_SIGNING 367 | + DOCUMENT_SIGNING 368 | + OCSP_SIGNING 369 | + SECURE_EMAIL 370 | + SERVER_AUTHENTICATION 371 | + TIME_STAMPING 372 | 373 | The root store policy is specified as a JSON object; the default is: 374 | 375 | :: 376 | 377 | { 378 | "sources": ["mozilla"], 379 | "operation": "union", 380 | "trust_bits": [] 381 | } 382 | 383 | The following example can be used to specify a root store with 384 | **mozilla** and **chrome** sources and trust bits of 385 | **SERVER_AUTHENTICATION**: 386 | 387 | :: 388 | 389 | { 390 | "sources": ["mozilla", "chrome"], 391 | "operation": "union", 392 | "trust_bits": ["SERVER_AUTHENTICATION"] 393 | } 394 | 395 | sprocket.py Usage 396 | ................. 397 | 398 | :: 399 | 400 | $ bin/sprocket.py --help 401 | usage: sprocket.py [options] 402 | 403 | create custom root store 404 | 405 | options: 406 | -h, --help show this help message and exit 407 | -c PATH, --ccadb PATH 408 | CCADB all certificate information CSV path 409 | -f PATH, --fingerprints PATH 410 | root CA fingerprints CSV path 411 | -T PATH, --trust-settings PATH 412 | CCADB root certificate trust bit settings CSV path 413 | --policy JSON JSON policy object path or string 414 | --stats print source stats 415 | --verbose enable verbosity 416 | --debug {0,1,2,3} enable debug 417 | --version display version 418 | 419 | sprocket.py Example 420 | ................... 421 | 422 | The CCADB ``AllCertificateRecordsCSVFormatv4`` CSV file needs to be 423 | downloaded before running ``sprocket.py``. 424 | 425 | :: 426 | 427 | $ pwd 428 | /home/ksteves/git/pan-chainguard 429 | 430 | $ cd tmp 431 | 432 | $ curl -sOJ https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv4 433 | 434 | $ ls -lh AllCertificateRecordsReport.csv 435 | -rw-r--r-- 1 ksteves ksteves 8.5M Aug 26 09:46 AllCertificateRecordsReport.csv 436 | 437 | The CCADB ``AllIncludedRootCertsCSV.csv`` CSV file *should* be 438 | downloaded before running ``sprocket.py``. This is currently 439 | optional, however it may become required in the future. 440 | 441 | :: 442 | 443 | $ curl -sOJ https://ccadb.my.salesforce-sites.com/ccadb/AllIncludedRootCertsCSV 444 | 445 | $ ls -lh AllIncludedRootCertsCSV.csv 446 | -rw-r--r-- 1 ksteves ksteves 98.9K Aug 26 09:46 AllIncludedRootCertsCSV.csv 447 | 448 | $ cd .. 449 | 450 | $ bin/sprocket.py --verbose --ccadb tmp/AllCertificateRecordsReport.csv \ 451 | > --trust-settings tmp/AllIncludedRootCertsCSV.csv \ 452 | > --fingerprints tmp/root-fingerprints.csv 453 | policy: {'sources': ['mozilla'], 'operation': 'union', 'trust_bits': []} 454 | mozilla: 145 total certificates 455 | 456 | fling.py - Deprecated 457 | ~~~~~~~~~~~~~~~~~~~~~ 458 | 459 | ``fling.py`` is used to export the PEM encoded X.509 certificates from 460 | the PAN-OS Default Trusted CA store. It is only used when you have 461 | chosen to use the PAN-OS native root store; it is recommended 462 | to use 463 | `pan-chainguard-content `_ 464 | or to create an up-to-date custom root store using ``sprocket.py``. 465 | 466 | fling.py Usage 467 | .............. 468 | 469 | :: 470 | 471 | $ bin/fling.py --help 472 | usage: fling.py [options] 473 | 474 | export PAN-OS trusted CAs 475 | 476 | options: 477 | -h, --help show this help message and exit 478 | --tag TAG, -t TAG .panrc tagname 479 | --certs PATH PAN-OS trusted CAs archive path (default: root-store.tgz) 480 | --xdebug {0,1,2,3} pan.xapi debug 481 | --verbose enable verbosity 482 | --debug {0,1,2,3} enable debug 483 | --version display version 484 | 485 | fling.py Example 486 | ................ 487 | 488 | :: 489 | 490 | $ pwd 491 | /home/ksteves/git/pan-chainguard 492 | 493 | $ mkdir -p tmp/root-store 494 | 495 | $ bin/fling.py --tag pa-460-chainguard --certs tmp/root-store/root-store.tgz 496 | Exported 293 PAN-OS trusted CAs to tmp/root-store/root-store.tgz 497 | 498 | $ cd tmp/root-store/ 499 | $ tar xzf root-store.tgz 500 | $ ls -1 | head 501 | 0001_Hellenic_Academic_and_Research_Institutions_RootCA_2011.cer 502 | 0003_USERTrust_ECC_Certification_Authority.cer 503 | 0004_CHAMBERS_OF_COMMERCE_ROOT_-_2016.cer 504 | 0008_VRK_Gov._Root_CA.cer 505 | 0012_Hellenic_Academic_and_Research_Institutions_RootCA_2015.cer 506 | 0013_SZAFIR_ROOT_CA.cer 507 | 0014_EE_Certification_Centre_Root_CA.cer 508 | 0016_ePKI_Root_Certification_Authority.cer 509 | 0017_thawte_Primary_Root_CA_-_G2.cer 510 | 0019_GeoTrust_Universal_CA_2.cer 511 | 512 | cert-fingerprints.sh - Deprecated 513 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 514 | 515 | Run ``cert-fingerprints.sh`` if you use ``fling.py`` to export the root 516 | store from PAN-OS. 517 | 518 | cert-fingerprints.sh Usage 519 | .......................... 520 | 521 | :: 522 | 523 | $ bin/cert-fingerprints.sh --help 524 | usage: cert-fingerprints.sh cert-directory 525 | 526 | cert-fingerprints.sh Example 527 | ............................ 528 | 529 | :: 530 | 531 | $ pwd 532 | /home/ksteves/git/pan-chainguard 533 | 534 | $ bin/cert-fingerprints.sh tmp/root-store > tmp/root-fingerprints.csv 535 | 536 | $ head tmp/root-fingerprints.csv 537 | "type","sha256" 538 | "root","BC104F15A48BE709DCA542A7E1D4B9DF6F054527E802EAA92D595444258AFE71" 539 | "root","4FF460D54B9C86DABFBCFC5712E0400D2BED3FBC4D4FBDAA86E06ADCD2A9AD7A" 540 | "root","04F1BEC36951BC1454A904CE32890C5DA3CDE1356B7900F6E62DFA2041EBAD51" 541 | "root","F008733EC500DC498763CC9264C6FCEA40EC22000E927D053CE9C90BFA046CB2" 542 | "root","A040929A02CE53B4ACF4F2FFC6981CE4496F755E6D45FE0B2A692BCD52523F36" 543 | "root","FABCF5197CDD7F458AC33832D3284021DB2425FD6BEA7A2E69B7486E8F51F9CC" 544 | "root","3E84BA4342908516E77573C0992F0979CA084E4685681FF195CCBA8A229B8A76" 545 | "root","C0A6F4DC63A24BFDCF54EF2A6A082A0A72DE35803E2FF5FF527AE5D87206DFD5" 546 | "root","A4310D50AF18A6447190372A86AFAF8B951FFB431D837F1E5688B45971ED1557" 547 | 548 | chain.py 549 | ~~~~~~~~ 550 | 551 | ``chain.py`` is used to determine intermediate certificate chains for 552 | the CAs in the root store. It can also save the certificate metadata 553 | as a JSON tree structure for use in generating documents which describe 554 | the certificate hierarchy. 555 | 556 | chain.py Usage 557 | .............. 558 | 559 | :: 560 | 561 | $ bin/chain.py --help 562 | usage: chain.py [options] 563 | 564 | determine intermediate CAs 565 | 566 | options: 567 | -h, --help show this help message and exit 568 | -c PATH, --ccadb PATH 569 | CCADB all certificate information CSV path 570 | -r PATH, --root-fingerprints PATH 571 | root CA fingerprints CSV path 572 | -o PATH, --onecrl PATH 573 | Mozilla OneCRL CSV path 574 | -i PATH, --int-fingerprints PATH 575 | intermediate CA fingerprints CSV path 576 | --tree PATH save certificate tree as JSON to path 577 | --verbose enable verbosity 578 | --debug {0,1,2,3} enable debug 579 | --version display version 580 | 581 | chain.py Example 582 | ................ 583 | 584 | The CCADB ``AllCertificateRecordsCSVFormatv4`` CSV file needs to be 585 | downloaded before running ``chain.py``. If you downloaded it previously 586 | to run ``sprocket.py`` you do not need to download it again. 587 | 588 | The Mozilla ``IntermediateCertsInOneCRL`` CSV file *should* (it is 589 | optional) be downloaded to allow ``chain.py`` to check if an 590 | intermediate certificate is in OneCRL and exclude it. 591 | 592 | :: 593 | 594 | $ pwd 595 | /home/ksteves/git/pan-chainguard 596 | 597 | $ cd tmp 598 | 599 | $ curl -sOJ https://ccadb.my.salesforce-sites.com/mozilla/IntermediateCertsInOneCRLReportCSV 600 | 601 | $ ls -lh IntermediateCertsInOneCRL.csv 602 | rw-r--r-- 1 ksteves ksteves 543K Aug 26 09:46 IntermediateCertsInOneCRL.csv 603 | 604 | $ cd .. 605 | 606 | $ bin/chain.py --verbose -c tmp/AllCertificateRecordsReport.csv -r tmp/root-fingerprints.csv \ 607 | > -o tmp/IntermediateCertsInOneCRL.csv 608 | > -i tmp/intermediate-fingerprints.csv --tree tmp/certificate-tree.json 609 | 1737 total intermediate certificates 610 | 611 | 612 | chainring.py 613 | ~~~~~~~~~~~~ 614 | 615 | ``chainring.py`` is used to: 616 | 617 | + Create documents which describe the certificate hierarchy in various 618 | formats including: 619 | 620 | + txt - Text 621 | + rst - reStructuredText 622 | + html - Hypertext Markup Language 623 | + json - pretty printed JSON 624 | + stats - statistics about the certificate tree 625 | 626 | + Test for collisions in PAN-OS certificate names, which are derived 627 | using the first 26 characters of the certificate SHA-256 628 | fingerprint, which is 64 characters 629 | 630 | + Lookup CCADB data by full or partial certificate SHA-256 fingerprint, 631 | including ``pan-chainguard`` managed **LINK-** certificate names 632 | 633 | chainring.py Usage 634 | .................. 635 | 636 | :: 637 | 638 | $ bin/chainring.py --help 639 | usage: chainring.py [options] 640 | 641 | certificate tree analysis and reporting 642 | 643 | options: 644 | -h, --help show this help message and exit 645 | --tree PATH JSON certificate tree path 646 | -f {txt,rst,html,json,stats}, --format {txt,rst,html,json,stats} 647 | output format 648 | -t TITLE, --title TITLE 649 | report title 650 | --test-collisions test for certificate name collisions 651 | -F SHA-256, --fingerprint SHA-256 652 | lookup CCADB data by certificate SHA-256 fingerprint 653 | (partial fingerprint allowed) 654 | --verbose enable verbosity 655 | --debug {0,1,2,3} enable debug 656 | --version display version 657 | 658 | chainring.py Example 659 | .................... 660 | 661 | :: 662 | 663 | $ pwd 664 | /home/ksteves/git/pan-chainguard 665 | 666 | $ bin/chainring.py --tree tmp/certificate-tree.json --format txt > tmp/certificate-tree.txt 667 | 668 | $ head tmp/certificate-tree.txt 669 | Root 670 | ├── 8AC552AD577E37AD2C6808D72AA331D6A96B4B3FEBFF34CE9BC0578E08055EC3 Subject: "A-Trust-Root-07" CA-Owner: "A-Trust" 671 | ├── 7A38F708A35A31E42E1CF3220F9A2D273E7666354618B2464657D43D8E77ADC2 Subject: "A-Trust-Root-09" CA-Owner: "A-Trust" 672 | ├── D7A7A0FB5D7E2731D771E9484EBCDEF71D5F0C3E0A2948782BC83EE0EA699EF4 Subject: "AAA Certificate Services" CA-Owner: "Sectigo" 673 | │ ├── 70DB9DED944DD35D474EA15FF2AA4E25F393A893ECDA54359D305BC319649817 Subject: "Apple Public Server ECC CA 12 - G1" Issuer: "AAA Certificate Services" 674 | │ ├── 0B405CFE9A6BEB098FFB969121C5F6710F3F7FA9EA101A6418F7AF201D3D3938 Subject: "Apple Public Server RSA CA 12 - G1" Issuer: "AAA Certificate Services" 675 | │ ├── 53612513970B9F264CA4BCC3BFD84DBC5FE774E3C6295B3EBB99EB9D74069E2A Subject: "COMODO ECC Certification Authority" Issuer: "AAA Certificate Services" 676 | │ ├── 38392F17CE7B682C198D29C6E71D2740964A2074C8D2558E6CFF64C27823F129 Subject: "COMODO RSA Certification Authority" Issuer: "AAA Certificate Services" 677 | │ ├── 1286173E6F0102F7BDD32C2F830910953489BF22C16295D84DD90A3DA137164A Subject: "COMODO SHA-2 Pro Series Secure Server CA" Issuer: "AAA Certificate Services" 678 | │ ├── E11E06861C4D308FD944BF17BE5E9072A034C4F93034CB59C02D512D30F7FC45 Subject: "COMODO SHA-2 Pro Series Secure Server CA" Issuer: "AAA Certificate Services" 679 | 680 | link.py 681 | ~~~~~~~ 682 | 683 | ``link.py`` obtains PEM encoded X.509 certificates from different 684 | sources including: 685 | 686 | + `Mozilla certificates with PEM CSV files 687 | `_ 688 | 689 | * `Intermediate CA Certificates 690 | `_ 691 | 692 | * `Non-revoked, non-expired Intermediate CA Certificates chaining up to 693 | roots in Mozilla's program with the Websites trust bit set 694 | `_ 695 | 696 | + Old (previous) certificate archive 697 | 698 | + crt.sh API 699 | 700 | The `crt.sh API `_ can be slow. ``link.py`` 701 | implements concurrent API requests using asyncio, however the server 702 | throttles response times in addition to returning "429 Too many 703 | requests" response status when too many concurrent requests are 704 | performed. Timeout, connection and response content errors have also 705 | been observed, and when seen will be retried up to 4 times (total 5 706 | tries). 707 | 708 | Updating (or refreshing) the certificate archive only needs to be 709 | performed periodically when the root store is updated by 710 | ``sprocket.py`` and/or ``chain.py`` is used to determine intermediate 711 | certificates for updates in CCADB. 712 | 713 | link.py Usage 714 | ............. 715 | 716 | :: 717 | 718 | $ bin/link.py --help 719 | usage: link.py [options] 720 | 721 | get CA certificates 722 | 723 | options: 724 | -h, --help show this help message and exit 725 | -f PATH, --fingerprints PATH 726 | CA fingerprints CSV path 727 | -m PATH, --certs-mozilla PATH 728 | Mozilla certs with PEM CSV path 729 | --certs-old PATH old certificate archive path 730 | --certs-new PATH new certificate archive path 731 | --verbose enable verbosity 732 | --debug {0,1,2,3} enable debug 733 | --version display version 734 | 735 | link.py Example 736 | ................ 737 | 738 | This example performs an initial download without an old certificate 739 | archive. 740 | 741 | :: 742 | 743 | $ pwd 744 | /home/ksteves/git/pan-chainguard 745 | 746 | $ cd tmp 747 | 748 | $ rm -f MozillaIntermediateCerts.csv 749 | $ curl -sOJ https://ccadb.my.salesforce-sites.com/mozilla/MozillaIntermediateCertsCSVReport 750 | 751 | $ rm -f PublicAllIntermediateCertsWithPEMReport.csv 752 | $ curl -sOJ https://ccadb.my.salesforce-sites.com/mozilla/PublicAllIntermediateCertsWithPEMCSV 753 | 754 | $ cd .. 755 | 756 | $ bin/link.py --verbose -f tmp/root-fingerprints.csv -f tmp/intermediate-fingerprints.csv \ 757 | > -m tmp/MozillaIntermediateCerts.csv -m tmp/PublicAllIntermediateCertsWithPEMReport.csv \ 758 | > --certs-old tmp/certificates-old.tgz --certs-new tmp/certificates-new.tgz >tmp/stdout.txt 2>tmp/stderr.txt 759 | 760 | $ echo $? 761 | 0 762 | 763 | $ tail tmp/stdout.txt 764 | Download using crt.sh API 55903859C8C0C3EBB8759ECE4E2557225FF5758BBD38EBD48276601E1BD58097 765 | Download using crt.sh API ADA5A71AF2121B569104BE385E746FA975617E81DBFAF6F722E62352471BD838 766 | Download using crt.sh API E7FA0F67C9B6D886C868408996DBDFC3680E8B9EC47628EEFB4824C23A287693 767 | Download using crt.sh API D793D934DD1B9FF9F6A76D438C760ED44B72BCDE660B49A77DBCF81EC7CEB3A9 768 | Download using crt.sh API F7B09EEA79096A4498F6A2B8D6F1183228A3769EA988050D1B32A380EABC4F9E 769 | certs-old: 0 770 | MozillaIntermediateCerts: 1718 771 | PublicAllIntermediateCerts: 15 772 | crt.sh: 178 773 | Total certs-new: 1911 774 | 775 | ``link.py`` exits with the following status codes: 776 | 777 | =========== ========= 778 | Status Code Condition 779 | =========== ========= 780 | 0 success, all certificates were obtained 781 | 1 fatal error 782 | 2 error, some certificates were not obtained 783 | =========== ========= 784 | 785 | Review ``tmp/stderr.txt`` for warnings and errors. 786 | 787 | The tar archive uses the following directory structure: 788 | 789 | :: 790 | 791 | root/ 792 | certificate-SHA-256.pem 793 | intermediate/ 794 | certificate-SHA-256.pem 795 | 796 | For example: 797 | 798 | :: 799 | 800 | $ tar tzf tmp/certificates-new.tgz | head 801 | root/55926084EC963A64B96E2ABE01CE0BA86A64FBFEBCC7AAB5AFC155B37FD76066.pem 802 | root/2E44102AB58CB85419451C8E19D9ACF3662CAFBC614B6A53960A30F7D0E2EB41.pem 803 | root/8ECDE6884F3D87B1125BA31AC3FCB13D7016DE7F57CC904FE1CB97C6AE98196E.pem 804 | root/1BA5B2AA8C65401A82960118F80BEC4F62304D83CEC4713A19C39C011EA46DB4.pem 805 | root/18CE6CFE7BF14E60B2E347B8DFE868CB31D02EBB3ADA271569F50343B46DB3A4.pem 806 | root/E35D28419ED02025CFA69038CD623962458DA5C695FBDEA3C22B0BFB25897092.pem 807 | root/568D6905A2C88708A4B3025190EDCFEDB1974A606A13C6E5290FCB2AE63EDAB5.pem 808 | root/D8E0FEBC1DB2E38D00940F37D27D41344D993E734B99D5656D9778D4D8143624.pem 809 | root/6B328085625318AA50D173C98D8BDA09D57E27413D114CF787A0F5D06C030CF6.pem 810 | root/5C58468D55F58E497E743982D2B50010B6D165374ACF83A7D4A32DB768C4408E.pem 811 | 812 | This example performs a subsequent download using an old certificate 813 | archive. 814 | 815 | :: 816 | 817 | $ pwd 818 | /home/ksteves/git/pan-chainguard 819 | 820 | $ cd tmp 821 | 822 | $ mv certificates-new.tgz certificates-old.tgz 823 | 824 | $ cd .. 825 | 826 | $ bin/link.py --verbose -f tmp/root-fingerprints.csv -f tmp/intermediate-fingerprints.csv \ 827 | > --certs-old tmp/certificates-old.tgz --certs-new tmp/certificates-new.tgz 828 | certs-old: 1911 829 | MozillaIntermediateCerts: 0 830 | PublicAllIntermediateCerts: 0 831 | crt.sh: 0 832 | Total certs-new: 1911 833 | 834 | guard.py 835 | ~~~~~~~~ 836 | 837 | guard.py Usage 838 | .............. 839 | 840 | :: 841 | 842 | $ bin/guard.py --help 843 | usage: guard.py [options] 844 | 845 | update PAN-OS trusted CAs 846 | 847 | options: 848 | -h, --help show this help message and exit 849 | --tag TAG, -t TAG .panrc tagname 850 | --vsys VSYS vsys name or number 851 | --template TEMPLATE Panorama template 852 | --certs PATH certificate archive path 853 | --update update certificates 854 | --delete delete all previously added certificates 855 | -T {root,intermediate}, --type {root,intermediate} 856 | certificate type(s) for update 857 | --update-trusted update trusted root CA for all certificates 858 | --commit commit configuration 859 | --dry-run don't update PAN-OS 860 | --show show pan-chainguard managed config 861 | --show-tree show pan-chainguard managed certificates in tree format 862 | --admin ADMIN commit admin 863 | --xdebug {0,1,2,3} pan.xapi debug 864 | --verbose enable verbosity 865 | --debug {0,1,2,3} enable debug 866 | --version display version 867 | 868 | guard.py Example 869 | ................ 870 | 871 | ``guard.py`` uses the certificate archive created by ``link.py`` to 872 | import the certificates as trusted CA device certificates on PAN-OS: 873 | 874 | + ``--tag`` specifies the .panrc tagname which can be a Panorama or 875 | firewall. 876 | 877 | + ``--template`` is used to specify the Panorama template to update. 878 | 879 | + ``--vsys`` is used to specify the vsys for multi VSYS firewalls and 880 | multi VSYS Panorama templates. 881 | 882 | + ``--delete`` is used to delete all previously added certificates. 883 | 884 | + ``--update`` is used to perform an initial update or incremental 885 | update of certificates. 886 | 887 | + ``--certs`` specifies the certificate archive for the update. 888 | 889 | + ``--type`` specifies the certificate type(s) for the update: 890 | 891 | * root - update only root certificates; this is used to update the 892 | default PAN-OS root store with a custom root store. 893 | 894 | * root and intermediate - update root and intermediate certificates; 895 | this is used to update the default PAN-OS root store with a custom 896 | root store and their intermediate certificates. 897 | 898 | * intermediate - update only intermediate certificates. 899 | 900 | + ``--dry-run`` is used to show what actions ``guard.py`` would 901 | perform without updating PAN-OS. 902 | 903 | + ``--show`` is used the show the pan-chainguard managed 904 | configuration. 905 | 906 | The device certificate names can have a maximum length of 31 907 | characters on Panorama and 63 on PAN-OS. They are constructed in a 908 | way to avoid conflict with other user and machine defined certificate 909 | names, and also to have a well-defined pattern so ``guard.py`` can 910 | manage certificates it owns. The PAN-OS certificate name pattern 911 | (format) used is: 912 | 913 | + The length is 31 characters (the maximum length on Panorama) 914 | 915 | + Starts with 'LINK' 916 | 917 | + Followed by a single dash '-' 918 | 919 | + Followed by the first 26 characters of the uppercase hexadecimal 920 | certificate fingerprint 921 | 922 | .. note:: ``chainring.py --test-collisions`` can be used to test for 923 | collisions in PAN-OS certificate names. 924 | 925 | .. note:: Panorama support: 926 | 927 | + Import to Panorama device certificates 928 | + Import to Template single VSYS device certificates 929 | + Import to Template multi VSYS device certificates 930 | 931 | * Issue ID PAN-257229 932 | * Fixed in PAN-OS 12.1.0 933 | * Does not work (not fixed) in PAN-OS 10.x, 11.x 934 | 935 | + Commit to Panorama 936 | 937 | :: 938 | 939 | $ pwd 940 | /home/ksteves/git/pan-chainguard 941 | 942 | $ bin/guard.py -t pa-460-chainguard --show 943 | 0 Device Certificates 944 | 945 | $ bin/guard.py -t pa-460-chainguard --update -T root -T intermediate \ 946 | > --certs tmp/certificates-new.tgz --dry-run 947 | update dry-run: 0 to delete, 1911 to add 948 | 949 | $ bin/guard.py -t pa-460-chainguard --update -T root -T intermediate \ 950 | > --certs tmp/certificates-new.tgz --commit 951 | 0 certificates deleted 952 | 1911 certificates added 953 | commit: success 954 | 955 | $ bin/guard.py -t pa-460-chainguard --update -T root -T intermediate \ 956 | > --certs tmp/certificates-new.tgz --dry-run 957 | update dry-run: 0 to delete, 0 to add 958 | 959 | $ bin/guard.py -t pa-460-chainguard --show 960 | 1911 Device Certificates 961 | 1911 Trusted Root CA Certificates 962 | 963 | pan-chainguard-content - Certificate Content for pan-chainguard 964 | --------------------------------------------------------------- 965 | 966 | `pan-chainguard-content 967 | `_ 968 | provides pre-generated, up-to-date content which can be used to 969 | simplify the deployment of pan-chainguard. 970 | 971 | Use Cases 972 | --------- 973 | 974 | The use case prerequisites include: 975 | 976 | + Install `pan-chainguard `_ 977 | 978 | + Set up a `.panrc `_ file 979 | 980 | + Configure `role based admin `_ 981 | 982 | Use Case 1: Update Out-of-date PAN-OS Root Store - All Vendor Sources 983 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 984 | 985 | A PAN-OS SSL decryption administrator wants to refresh the system root 986 | store (Default Trusted Certificate Authorities) on a firewall. They 987 | want their updated trusted root store to contain the root certificates 988 | from the 4 common vendor stores: 989 | 990 | + Mozilla 991 | + Apple 992 | + Chrome 993 | + Microsoft 994 | 995 | This is the simplest deployment because it can directly use the 996 | certificate archive from ``pan-chainguard-content``, which is updated 997 | daily. 998 | 999 | The steps to implement this use case include: 1000 | 1001 | #. Download ``pan-chainguard-content`` certificate archive 1002 | #. Run ``guard.py`` to update PAN-OS trusted CAs 1003 | 1004 | Download ``pan-chainguard-content`` certificate archive 1005 | ....................................................... 1006 | 1007 | ``pan-chainguard-content`` creates an updated certificate archive 1008 | daily using a policy of the union of all 4 common vendor root 1009 | certificate stores, and includes intermediate certificates for the 1010 | root certificates, which are not used for this use case. 1011 | 1012 | :: 1013 | 1014 | $ pwd 1015 | /home/ksteves/git/pan-chainguard/tmp 1016 | 1017 | $ curl -sLO https://raw.githubusercontent.com/PaloAltoNetworks/pan-chainguard-content/main/latest-certs/certificates-new.tgz 1018 | 1019 | $ ls -lh certificates-new.tgz 1020 | -rw-r--r-- 1 ksteves ksteves 2.0M Mar 24 11:19 certificates-new.tgz 1021 | 1022 | Run ``guard.py`` to update PAN-OS trusted CAs 1023 | ............................................. 1024 | 1025 | :: 1026 | 1027 | $ pwd 1028 | /home/ksteves/git/pan-chainguard 1029 | 1030 | $ bin/guard.py -t pa-460-chainguard --show 1031 | 0 Device Certificates 1032 | 1033 | $ bin/guard.py -t pa-460-chainguard --admin chainguard --certs tmp/certificates-new.tgz --update --type root --dry-run 1034 | update dry-run: 0 to delete, 298 to add 1035 | 1036 | $ bin/guard.py -t pa-460-chainguard --admin chainguard --certs tmp/certificates-new.tgz --update --type root 1037 | 0 certificates deleted 1038 | 298 certificates added 1039 | 1040 | $ bin/guard.py -t pa-460-chainguard --show 1041 | 298 Device Certificates 1042 | 298 Trusted Root CA Certificates 1043 | 1044 | $ bin/guard.py -t pa-460-chainguard --admin chainguard --commit 1045 | commit: success 1046 | 1047 | Then repeat the certificate archive download and update periodically 1048 | to ensure the root store remains up-to-date. These subsequent updates 1049 | are performed incrementally, resulting in fast update times. 1050 | 1051 | Use Case 2: Update Out-of-date PAN-OS Root Store - Custom Vendor Sources 1052 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1053 | 1054 | A PAN-OS SSL decryption administrator wants to refresh the system root 1055 | store (Default Trusted Certificate Authorities) on a firewall. They 1056 | want their updated trusted root store to contain the root certificates 1057 | from: 1058 | 1059 | + Mozilla 1060 | + Chrome 1061 | 1062 | The steps to implement this use case include: 1063 | 1064 | #. Download *CCADB All Certificate Information* CSV file 1065 | #. Download *CCADB All Included Root Certificate Trust Bits* CSV file 1066 | #. Create ``sprocket.py policy.json`` file 1067 | #. Run ``sprocket.py`` to create *root CA fingerprints* CSV file 1068 | #. Download ``pan-chainguard-content`` certificate archive 1069 | #. Run ``link.py`` to create new certificate archive 1070 | #. Run ``guard.py`` to update PAN-OS trusted CAs 1071 | 1072 | Download *CCADB All Certificate Information* CSV file 1073 | ..................................................... 1074 | 1075 | :: 1076 | 1077 | $ pwd 1078 | /home/ksteves/git/pan-chainguard/tmp 1079 | 1080 | $ curl --clobber -sOJ https://ccadb.my.salesforce-sites.com/ccadb/AllCertificateRecordsCSVFormatv4 1081 | 1082 | $ ls -lh AllCertificateRecordsReport.csv 1083 | -rw-r--r-- 1 ksteves ksteves 8.5M Aug 28 16:34 AllCertificateRecordsReport.csv 1084 | 1085 | Download *CCADB All Included Root Certificate Trust Bits* CSV file 1086 | .................................................................. 1087 | 1088 | :: 1089 | 1090 | $ pwd 1091 | /home/ksteves/git/pan-chainguard/tmp 1092 | 1093 | $ curl --clobber -sOJ https://ccadb.my.salesforce-sites.com/ccadb/AllIncludedRootCertsCSV 1094 | 1095 | $ ls -lh AllIncludedRootCertsCSV.csv 1096 | -rw-r--r-- 1 ksteves ksteves 98.9K Aug 28 16:35 AllIncludedRootCertsCSV.csv 1097 | 1098 | Create ``sprocket.py policy.json`` file 1099 | ....................................... 1100 | 1101 | :: 1102 | 1103 | $ pwd 1104 | /home/ksteves/git/pan-chainguard/tmp 1105 | 1106 | $ echo '{"sources":["mozilla","chrome"]}' > policy.json 1107 | 1108 | Run ``sprocket.py`` to create *root CA fingerprints* CSV file 1109 | ............................................................. 1110 | 1111 | :: 1112 | 1113 | $ pwd 1114 | /home/ksteves/git/pan-chainguard 1115 | 1116 | $ bin/sprocket.py --verbose -c tmp/AllCertificateRecordsReport.csv \ 1117 | > --trust-settings tmp/AllIncludedRootCertsCSV.csv \ 1118 | > --policy tmp/policy.json -f tmp/root-fingerprints.csv 1119 | policy: {'sources': ['mozilla', 'chrome'], 'operation': 'union', 'trust_bits': []} 1120 | mozilla, chrome: 148 total certificates 1121 | 1122 | Download ``pan-chainguard-content`` certificate archive 1123 | ....................................................... 1124 | 1125 | ``pan-chainguard-content`` creates an updated certificate archive 1126 | daily using a policy of the union of all 4 common vendor root 1127 | certificate stores, and includes intermediate certificates for the 1128 | root certificates, which are not used for this use case. 1129 | 1130 | :: 1131 | 1132 | $ pwd 1133 | /home/ksteves/git/pan-chainguard/tmp 1134 | 1135 | $ curl -so certificates-old.tgz https://raw.githubusercontent.com/PaloAltoNetworks/pan-chainguard-content/main/latest-certs/certificates-new.tgz 1136 | 1137 | $ ls -lh certificates-old.tgz 1138 | -rw-r--r-- 1 ksteves ksteves 2.1M Aug 28 16:41 certificates-old.tgz 1139 | 1140 | Run ``link.py`` to create new certificate archive 1141 | ................................................. 1142 | 1143 | :: 1144 | 1145 | $ pwd 1146 | /home/ksteves/git/pan-chainguard 1147 | 1148 | $ bin/link.py --verbose -f tmp/root-fingerprints.csv --certs-old tmp/certificates-old.tgz --certs-new tmp/certificates-new.tgz 1149 | certs-old: 148 1150 | MozillaIntermediateCerts: 0 1151 | PublicAllIntermediateCerts: 0 1152 | crt.sh: 0 1153 | Total certs-new: 148 1154 | 1155 | Run ``guard.py`` to update PAN-OS trusted CAs 1156 | ............................................. 1157 | 1158 | :: 1159 | 1160 | $ pwd 1161 | /home/ksteves/git/pan-chainguard 1162 | 1163 | $ bin/guard.py -t pa-460-chainguard --show 1164 | 0 Device Certificates 1165 | 1166 | $ bin/guard.py -t pa-460-chainguard --certs tmp/certificates-new.tgz --update --type root --dry-run 1167 | 1168 | $ bin/guard.py -t pa-460-chainguard --certs tmp/certificates-new.tgz --update --type root 1169 | 1170 | $ bin/guard.py -t pa-460-chainguard --show 1171 | 1172 | $ bin/guard.py -t pa-460-chainguard --admin chainguard --commit 1173 | commit: success 1174 | 1175 | Then repeat the certificate archive download and update periodically 1176 | to ensure the root store remains up-to-date. These subsequent updates 1177 | are performed incrementally, resulting in fast update times. 1178 | 1179 | About the Name 1180 | -------------- 1181 | 1182 | ``pan-chainguard`` is named after a bicycle chain guard. This chain 1183 | guard serves to guard and protect against an out-of-date root store 1184 | and missing intermediate certificate chains. ``fling.py`` is named 1185 | after anti-fling grease used on chains. 1186 | 1187 | References 1188 | ---------- 1189 | 1190 | - `PAN-OS Repair Incomplete Certificate Chains 1191 | `_ 1192 | 1193 | - `Common CA Database - Useful Resources 1194 | `_ 1195 | 1196 | - `Firefox Intermediate CA Preloading 1197 | `_ 1198 | 1199 | - `Mozilla CA/Included Certificates 1200 | `_ 1201 | 1202 | - `Mozilla CA/Intermediate Certificates 1203 | `_ 1204 | 1205 | - `Mozilla OneCRL 1206 | `_ 1207 | 1208 | - `Mozilla Root Store Policy 1209 | `_ 1210 | 1211 | - `pan-python 1212 | `_ 1213 | 1214 | - `pan-chainguard GitHub Repository 1215 | `_ 1216 | 1217 | - `pan-chainguard-content GitHub Repository 1218 | `_ 1219 | 1220 | - `crt.sh API Usage 1221 | `_ 1222 | --------------------------------------------------------------------------------